De-Serializing msg_container objects returned from the telegram.org server
<p>I'm having trouble de-serializing a <code>msg_container</code> object. The issue seems to be my understanding of where the <code>flags</code> field is located in the <code>message</code> as it doesn't make sense to me. My assumptions below come from reading this page: <a href="https://core.telegram.org/mtproto/TL" rel="nofollow">https://core.telegram.org/mtproto/TL</a></p>

<p>Here is what I am seeing, I'm hoping someone can clarify.</p>

<p>A response from the telegram server looks like this:</p>

<pre><code>    ('server_answer: ', '\xdc\xf8\xf1s\x02\x00\x00\x00\x018\xcb\x8cCu\xcdW\x01\x00\x00\x00\x1c\x00\x00\x00\x08\t\xc2\x9e\x00t\xbe?Cu\xcdW\x82\x0e:\x11\xe0\xda\xed\x05\xd7\xbezI4\x05Tf\x01\xdc\xcb\x8cCu\xcdW\x02\x00\x00\x00\x14\x00\x00\x00Y\xb4\xd6b\x15\xc4\xb5\x1c\x01\x00\x00\x00\x00t\xbe?Cu\xcdW')

 0 | DC F8 F1 73 02 00 00 00
 8 | 01 38 CB 8C 43 75 CD 57
16 | 01 00 00 00 1C 00 00 00
24 | 08 09 C2 9E 00 74 BE 3F
32 | 43 75 CD 57 82 0E 3A 11
40 | E0 DA ED 05 D7 BE 7A 49
48 | 34 05 54 66 01 DC CB 8C
56 | 43 75 CD 57 02 00 00 00
64 | 14 00 00 00 59 B4 D6 62
72 | 15 C4 B5 1C 01 00 00 00
80 | 00 74 BE 3F 43 75 CD 57
</code></pre>

<p>Which indicates a <code>MessageContainer</code> object, which looks like this:</p>

<pre><code>msg_container#73f1f8dc messages:vector&lt;message&gt; = MessageContainer;
</code></pre>

<p>(Note: 73f1f8dc is the first four bytes returned, in little-endian order)</p>

<p>Since a <code>msg_container</code> is a <code>vector</code> of <code>messages</code> the next integer is the <code>message</code> count, which is <code>00 00 00 02</code>, so there are two messages in this vector.</p>

<p>A <code>message</code> looks like this:</p>

<pre><code>message#c09be45f flags:# out:flags.1?true mentioned:flags.4?true media_unread:flags.5?true silent:flags.13?true post:flags.14?true id:int from_id:flags.8?int to_id:Peer fwd_from:flags.2?MessageFwdHeader via_bot_id:flags.11?int reply_to_msg_id:flags.3?int date:int message:string media:flags.9?MessageMedia reply_markup:flags.6?ReplyMarkup entities:flags.7?Vector&lt;MessageEntity&gt; views:flags.10?int edit_date:flags.15?int = Message;
messageService#9e19a1f6 flags:# out:flags.1?true mentioned:flags.4?true media_unread:flags.5?true silent:flags.13?true post:flags.14?true id:int from_id:flags.8?int to_id:Peer reply_to_msg_id:flags.3?int date:int action:MessageAction = Message;
</code></pre>

<p>So I am assuming the next four bytes will be the first parameter in the first message, which is the <code>flags</code> field. But the value of that four bytes is <code>8c cb 38 01</code> which is 2362128385L as an unsigned integer, which is suspiciously large for a simple bit mask. Plus, if I use this value to de-serialize the optional parameters from the rest of the byte stream they don't make sense and eventually fail with a type mismatch. I've tried the <code>flags</code> field as a signed integer as well with similar results.</p>

<p>What is incorrect about my understanding of how the telegram server responds with msg_container objects?</p>

<p>Here is a simple approach to use in handling messages packed in a container:</p>

<pre><code>Container_type_header
Content_count
[messages]
</code></pre>

<p>Each message is packed like so:</p>

<pre><code>msgs_id
seq_no
mgs_len
msg_body
</code></pre>

<p>given your sample data:</p>

<pre><code>DCF8F173 --&gt; container_type_header
02000000 --&gt; message_count_in_container

0138CB8C4375CD57 --&gt; msg_id of first message
01000000 --&gt; seq_no of first message
1C000000 --&gt; length of first message; 1C == 28 bytes
0809C29E0074BE3F4375CD57820E3A11E0DAED05D7BE7A4934055466 --&gt; 28 byte msg_body

01DCCB8C4375CD57 --&gt; msg_id of second message
02000000 --&gt; seq_number of second message
14000000 --&gt; lenght of second message; 14 == 20 bytes
59B4D66215C4B51C010000000074BE3F4375CD57 --&gt; 20 byte msg_body
</code></pre>

<p>when decoded first message gives:</p>

<pre><code>0809C29E0074BE3F4375CD57820E3A11E0DAED05D7BE7A4934055466

New_Session_Created{first_msg_id: 6326841983218119680,  server_salt: 7373524212041563863, unique_id: 427238195566612098}
</code></pre>

<p>when decoded second message gives:</p>

<pre><code>0x59B4D66215C4B51C010000000074BE3F4375CD57 

Msgs_Ack{msg_ids: [6326841983218119680]}
</code></pre>

python api telegram