primitive JSON

Added by Philip about 7 years ago

We are subscribing to NNet Test Server using one connection for 10+ symbols, now everything is working fine until 1000+ returned messages. we notice we are having this data...

volume":2044651,"close":116.00,"high":118.40,"last":115.90,"last_volume":125696,"lot_size":1,"low":113.80,"open":116.50,"vwap":116.04,"turnover":112975483441.40,"turnover_volume":973622237}}
{"cmd":"price","args":{"i":"402","m":11,"t":"price","trade_timestamp":"2012-06-11 18:13:31","tick_timestamp":"2012-06-11 18:13:32","bid":153.00,"bid_volume":1515700,"ask":153.20,"ask_volume":537386,"close":153.00,"high":155.80,"last":153.00,"last_volume":202248,"lot_size":1,"low":150.10,"open":153.00,"vwap":152.95,"turnover":150691916664.80,"turnover_volume":985235186}}
{"cmd":"price","args":{"i":"45","m":11,"t":"price","trade_timestamp":"2012-06-11 18:13:32","tick_timestamp":"2012-06-11 18:13:32","bid":115.90,"bid_volume":1009860,"ask":116.30,"ask_volume":2044651,"close":116.00,"high":118.40,"last":115.90,"last_volume":811905,"lot_size":1,"low":113.80,"open":116.50,"vwap":116.04,"turnover":113080483394.10,"turnover_volume":974528190}}
{"cmd":"price","args":{"i":"18634","m":11,"t":"price","trade_timestamp":"2012-06-11 18:13:32","tick_timestamp":"2012-06-11 18:13:32","bid":120.80,"bid_volume":2170041,"ask":121.10,"ask_volume":1227561,"close":121.20,"high":122.70,"last":120.80,"last_volume":417238,"lot_size":1,"low":118.10,"open":121.00,"vwap":120.87,"turnover":113386683530.10,"turnover_volume":938050979}}

Before that we only get data in this format

{"cmd":"price","args":{"i":"45","m":11,"t":"price","trade_timestamp":"2012-06-11 18:13:32","tick_timestamp":"2012-06-11 18:13:32","bid":115.90,"bid_volume":1009860,"ask":116.30,"ask_volume":2044651,"close":116.00,"high":118.40,"last":115.90,"last_volume":811905,"lot_size":1,"low":113.80,"open":116.50,"vwap":116.04,"turnover":113080483394.10,"turnover_volume":974528190}}

We understand that we need to handle undocumented JSON stream but would like to ask the following questions :
Do we need to have separate connection for each symbol during subscription to feeds?
Is there some best practice on subscribing to feeds on NNet?

Thanks


Replies (16)

RE: primitive JSON - Added by Nordnet Simon about 7 years ago

Hi, you are using the feed in the correct way. One connection and multiple subscriptions.

We send each message in one tcp send chunk but as you probably know using TCP/IP there is no guarantee that the chunk is intact in the other end. That is why we end all JSON messages with newline.

So even if we send them one by one they can end up like this in your receives

Receive 1:


{"cmd":"price","args":{"i":"402","m":11,"t":"price","trade_timestamp":"2012-06-11 18:13:31","tick_timestamp":"2012-06-11 18:13:32","bid":153.00,"bid_volume":1515700,"ask":153.20,"ask_volume":537386,"close":153.00,"high":155.80,"last":153.00,"last_volume":202248,"lot_size":1,"low":150.10,"open":153.00,"vwap":152.95,"turnover":150691916664.80,"turnover_volume":985235186}}
{"cmd":"price","args":{"i":"45","m":11,"t":"price","trade_timestamp":"2012-06-11 18:13:32",

Receive 2:


"tick_timestamp":"2012-06-11 18:13:32","bid":115.90,"bid_volume":1009860,"ask":116.30,"ask_volume":2044651,"close":116.00,"high":118.40,"last":115.90,"last_volume":811905,"lot_size":1,"low":113.80,"open":116.50,"vwap":116.04,"turnover":113080483394.10,"turnover_volume":974528190}}
{"cmd":"price","args":{"i":"18634","m":11,"t":"price","trade_timestamp":"2012-06-11 18:13:32","tick_timestamp":"2012-06-11 18:13:32","bid":120.80,"bid_volume":2170041,"ask":121.10,"ask_volume":1227561,"close":121.20,"high":122.70,"last":120.80,"last_volume":417238,"lot_size":1,"low":118.10,"open":121.00,"vwap":120.87,"turnover":113386683530.10,"turnover_volume":938050979}}

So you need to buffer all incoming messages and split them by the newline because each receive you have in your code can get either:
  • A complete JSON message.
  • A part of a JSON message.
  • More than one JSON message.

You need to handle all cases. This has nothing to do with nExt it is always like this receiving data over TCP.

Philip wrote:

We understand that we need to handle undocumented JSON stream but would like to ask the following questions :

Please let me know what part of the JSON stream documentation that you find is lacking documentation. We will try to fix it.

Do we need to have separate connection for each symbol during subscription to feeds?
Is there some best practice on subscribing to feeds on NNet?

You are using it correct. One feed and multiple subscriptions

RE: primitive JSON - Added by Philip about 7 years ago

We thought we already fix this issue with buffering the stream.. however we getting this error again.

We have this sample data.

{"cmd":"price","args":{"i":"161","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:32","tick_timestamp":"2012-07-23 18:55:32","bid":130.30,"bid_volume":67575,"ask":130.40,"ask_volume":1452352,"close":130.40,"high":134.20,"last":130.40,"last_volume":457771,"lot_size":1,"low":127.90,"open":129.30,"vwap":130.15,"turnover":311008379429.80,"turnover_volume":2389523255}}
{"cmd":"price","args":{"i":"120","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:29","tick_timestamp":"2012-07-23 18:55:32","bid":94.60,"bid_volume":1482813,"ask":94.95,"ask_volume":1458795,"close":95.15,"high":96.05,"last":94.95,"last_volume":162972,"lot_size":1,"low":92.40,"open":95.85,"vwap":94.93,"turnover":237187180792.70,"turnover_volume":2498515930}}
{"cmd":"price","args":{"i":"161","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:32","tick_timestamp":"2012-07-23 18:55:32","bid":130.30,"bid_volume":67575,"ask":130.40,"ask_volume":538651,"close":130.40,"high":134.20,"last":130.40,"last_volume":457771,"lot_size":1,"low":127.90,"open":129.30,"vwap":130.15,"turnover":311008379429.80,"turnover_volume":2389523255}}
{"cmd":"price","args":{"i":"3966","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:30","tick_timestamp":"2012-07-23 18:55:32","bid":140.70,"bid_volume":1553793,"ask":140.90,"ask_volume":737769,"close":140.60,"high":142.70,"last":141.10,"last_volume":217240,"lot_size":1,"low":112.00,"open":115.00,"vwap":140.80,"turnover":339012645351.90,"turnover_volume":2407803983}}
{"cmd":"price","args":{"i":"18634","m":11,"t":"pric

We try to split the data using

Dim delimiter As Char() = New Char() {vbCr, vbCrLf, Environment.NewLine}
Dim strStream As String = sbStream.ToString()
Dim arrStream As String() = strStream.Split(delimiter)

For some reason we get primitive JSON error, which means that the split does not work. so what/how should we split the stream what is the proper newline character we should use to split?

Any idea on what we are missing?

Thanks in advance

RE: primitive JSON - Added by Philip about 7 years ago

What character should we use to to split the stream in .NET since vbCr or Environment.NewLine does not work?

RE: primitive JSON - Added by Nordnet Simon about 7 years ago

I used "\n" in a test client when but I did not use the Split method I just did


        private void ProcessString(string data )
        {
            try
            {
                int index = data.IndexOf("\n");
                // Not a complete message add it to the unprocessed data
                if (index < 0)
                {
                    unprocessed += data;
                }
                else
                {
                    // Process first message
                    string newMessage = unprocessed + data.Substring(0, index);
                    unprocessed = "";
                    ParseMessage(newMessage);
                    if (data.Length > index + 1)
                    {
                        // Find more messages in the data
                        ProcessString(data.Substring(index + 1));
                    }
                }
            }
            catch (Exception e)
            {
              // Handle error
            }   
        }


If index > 0 then I parsed the part between 0 and index. I index < 0 I stored the messages and waited for the rest.

Since I don't have all the code I have a bit of a problem understanding your problem.

You get:


{"cmd":"price","args":{"i":"161","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:32","tick_timestamp":"2012-07-23 18:55:32","bid":130.30,"bid_volume":67575,"ask":130.40,"ask_volume":1452352,"close":130.40,"high":134.20,"last":130.40,"last_volume":457771,"lot_size":1,"low":127.90,"open":129.30,"vwap":130.15,"turnover":311008379429.80,"turnover_volume":2389523255}}
{"cmd":"price","args":{"i":"120","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:29","tick_timestamp":"2012-07-23 18:55:32","bid":94.60,"bid_volume":1482813,"ask":94.95,"ask_volume":1458795,"close":95.15,"high":96.05,"last":94.95,"last_volume":162972,"lot_size":1,"low":92.40,"open":95.85,"vwap":94.93,"turnover":237187180792.70,"turnover_volume":2498515930}}
{"cmd":"price","args":{"i":"161","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:32","tick_timestamp":"2012-07-23 18:55:32","bid":130.30,"bid_volume":67575,"ask":130.40,"ask_volume":538651,"close":130.40,"high":134.20,"last":130.40,"last_volume":457771,"lot_size":1,"low":127.90,"open":129.30,"vwap":130.15,"turnover":311008379429.80,"turnover_volume":2389523255}}
{"cmd":"price","args":{"i":"3966","m":11,"t":"price","trade_timestamp":"2012-07-23 18:55:30","tick_timestamp":"2012-07-23 18:55:32","bid":140.70,"bid_volume":1553793,"ask":140.90,"ask_volume":737769,"close":140.60,"high":142.70,"last":141.10,"last_volume":217240,"lot_size":1,"low":112.00,"open":115.00,"vwap":140.80,"turnover":339012645351.90,"turnover_volume":2407803983}}
{"cmd":"price","args":{"i":"18634","m":11,"t":"pric


in one read from the socket?

Then you tries to split it? What does the result look like before you parse the Json, can you make a debug print of arrStream?

Are you sure it is the first item that fails? It could be that the first 4 are ok and


{"cmd":"price","args":{"i":"18634","m":11,"t":"pric


fails because you are not waiting for the rest of the message before parsing it.

RE: primitive JSON - Added by Philip about 7 years ago

Hi Simon, thanks for the reply.

Heres how we process the data using VB.NET

    Private Sub ReadMessage()

        ' Read the  message sent by the server.
        Dim buffer(2047) As Byte
        Dim bytes As Integer = -1
        Do

            Try

                bytes = oSslStream.Read(buffer, 0, buffer.Length)

                ' Use Decoder class to convert from bytes to UTF8
                ' in case a character spans two buffers.
                Dim decoder As Decoder = Encoding.UTF8.GetDecoder()
                Dim chars(decoder.GetCharCount(buffer, 0, bytes) - 1) As Char
                decoder.GetChars(buffer, 0, bytes, chars, 0)

                sbStream.Append(chars)

                ProcessStream()

            Catch ex As Exception

                oSslStream.Close()
                oSslStream = Nothing

            End Try

        Loop While bytes <> 0

    End Sub

    Private Sub ProcessStream()

        'Dim delimiter As Char() = New Char() {vbCr, vbCrLf, Environment.NewLine, "\n"}
        Dim strStream As String = sbStream.ToString()
        Dim arrStream As String() = strStream.Split("\n")

        For i As Integer = 0 To arrStream.Length - 1
            If ProcessFeed(arrStream(i)) Then
                If i = (arrStream.Length - 1) Then
                    sbStream = New StringBuilder("")
                Else
                    sbStream.Remove(0, arrStream(i).Length + 1)
                End If
            End If
        Next

    End Sub

ProcessFeed function returns true if the data is in valid JSON format otherwise false.

Hope to hear from you soon.

BTW Will also try your approach tommorrow when data stream is available on test server and post our findings here.

Thanks

RE: primitive JSON - Added by Philip about 7 years ago

Hi Simon,

We tried your suggestion last night but still did not get any luck.

We tried the following codes, just to look for newline character with no luck on all of them.

data.IndexOf(Environment.NewLine)
data.IndexOf("\r\n")
data.IndexOf("\n")
data.IndexOf(vbCr)
data.IndexOf(vbCrLf)

Is there something wrong with how we read the message? Our routine for reading the message is posted in my previous post.

Thanks

RE: primitive JSON - Added by Philip about 7 years ago

Hi all. any idea on this issue?

Thanks

RE: primitive JSON - Added by Nordnet Simon about 7 years ago

Hi, it is hard to say without the complete code.

As I asked previously can you make a debug print of arrStream?

Do a mark in the output with - or some other charachter so we can se exactly how each element in arrStream looks before it is sent to ProcessFeed.

Then it is simpler to say what is wrong.

RE: primitive JSON - Added by Philip about 7 years ago

Right now we can't provide you with the actual data when the issue occur since we don't have stream from test server. However attached are the screenshot of the contents of arrStream.

BTW Is there a way for us to send you a private message?

arrStream1.png (225 KB)

arrStream2.png (226 KB)

arrStream3.png (259 KB)

RE: primitive JSON - Added by Nordnet Simon about 7 years ago

Hi, it is hard to see what the problem is from the screenshot.

The best way to nail this problem is to logg strStream and each then each element in arrStream.

When it fails it will be easy to inspect exactly what you got and also exactly what was splitted.

RE: primitive JSON - Added by Nordnet Simon about 7 years ago

Philip wrote:

BTW Is there a way for us to send you a private message?

Hi, the best way is to use this forum. That way other users of the API can help. We at Nordnet knows the API but we are far from experts in all languages and development environments.

RE: primitive JSON - Added by Philip about 7 years ago

I understand simon. If only there is a way for us to test our code with data similar to production/live data anytime we should have fix this issue a long time ago. It sucks because we can only test on fridays and mondays or if is their any other way?

Thanks

RE: primitive JSON - Added by Hans about 7 years ago

Hei,

jeg ble kvitt dette problemet ved å bufre opp som følger:
har 2 like objekter: 1 public feed og 1 private feed
objektet initialiseres med

Private previous_message_data As String = ""

se nedenfor for read_feed logikken

med vennlig hilsen

Hans

Public Function read_feed(ByVal sslStream As SslStream) As String

Dim buffer(_buffer_size) As Byte
Dim messageData As New StringBuilder()
Dim bytes As Integer = -1
If previous_message_data.Length > 0 Then
For i As Integer = 0 To previous_message_data.Length - 1
messageData.Append(previous_message_data(i))
Next
End If
previous_message_data = ""
try
While True
If True Then
bytes = sslStream.Read(buffer, 0, buffer.Length) ' 2047
Dim decoder As Decoder = Encoding.UTF8.GetDecoder() ' Use Decoder class to convert from bytes to UTF8' in case a character spans two buffers.
Dim chars(decoder.GetCharCount(buffer, 0, bytes) - 1) As Char
decoder.GetChars(buffer, 0, bytes, chars, 0)
messageData.Append(chars)
ElseIf first Then
first = False
For i As Integer = 0 To p2.Length - 1
messageData.Append(p2(i))
Next
End If
Dim tx As String = messageData.ToString()
Dim I_lf As Integer = tx.IndexOf(vbLf)
If I_lf <> 1 Then 'vi har et telegram, minimum, f.eks 3.13 telegrammer
If tx.Length - 1 > I_lf Then
previous_message_data = tx.Substring(I_lf + 1)
Else
previous_message_data = ""
End If
Return tx.Substring(0, I_lf + 1) '<----------------------------

Exit While
End If
End While
Catch ex As Exception
If feed = "private" Then
_restart_private_handle.Set() ' re Login feed required
Else
_restart_public_handle.Set()
End If
_logger.log(_feed & " <---ABORT - abort thread:" & ex.ToString, "SSL_feed", "read_feed", logger_levels.error
)
Return "ERROR" 
End Try
End Function

RE: primitive JSON - Added by Hans about 7 years ago

Formatteringen ble snodig så jeg prøver igjen:

Private previous_message_data As String = ""

Public Function read_feed(ByVal sslStream As SslStream) As String

Dim buffer(_buffer_size) As Byte
Dim messageData As New StringBuilder()
Dim bytes As Integer = -1
If previous_message_data.Length > 0 Then
For i As Integer = 0 To previous_message_data.Length - 1
messageData.Append(previous_message_data(i))
Next
End If
previous_message_data = ""
try
While True
If True Then
bytes = sslStream.Read(buffer, 0, buffer.Length) ' 2047
' Use Decoder class to convert from bytes to UTF8' in case a character spans two buffers.
Dim decoder As Decoder = Encoding.UTF8.GetDecoder()              
Dim chars(decoder.GetCharCount(buffer, 0, bytes) - 1) As Char
decoder.GetChars(buffer, 0, bytes, chars, 0)
messageData.Append(chars)
ElseIf first Then
first = False
For i As Integer = 0 To p2.Length - 1
messageData.Append(p2(i))
Next
End If
Dim tx As String = messageData.ToString()
Dim I_lf As Integer = tx.IndexOf(vbLf) ' EOF
If I_lf < > -1 Then 'vi har et minimum ett telegram, f.eks 3.13 telegrammer
If tx.Length - 1 > I_lf Then
previous_message_data = tx.Substring(I_lf + 1)
Else
previous_message_data = ""
End If
Return tx.Substring(0, I_lf + 1)  '#################
Exit While
End If
End While
Catch ex As Exception
If feed = "private" Then
_restart_private_handle.Set() ' re Login feed required
Else
_restart_public_handle.Set()
End If
_logger.log(_feed & " <---ABORT - abort thread:" & ex.ToString, "SSL_feed", "read_feed", logger_levels.error
)
Return "ERROR" 
End Try
End Function

RE: primitive JSON - Added by Philip about 7 years ago

Hi Hans,

I think you have the answer.

to make the story short the newline equivalent to use in .NET is vbLf

Thanks

RE: primitive JSON - Added by Nordnet Simon almost 7 years ago

Hi, glad that it works now.

(1-16/16)