How do I tell python what my data structure (that is in binary) looks like so I can plot it?
<p>I have a data set that looks like this.</p>

<pre><code>b'\xa3\x95\x80\x80YFMT\x00BBnNZ\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00Type,Length,Name,Format,Columns\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa3\x95\x80\x81\x17PARMNf\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00Name,Value\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa3\x95\x80\x82-GPS\x00BIHBcLLeeEefI\x00\x00\x00Status,TimeMS,Week,NSats,HDop,Lat,Lng,RelAlt,Alt,Spd,GCrs,VZ,T\x00\x00\xa3\x95\x80\x83\x1fIMU\x00Iffffff\x00\x00\x00\x00\x00\x00\x00\x00\x00TimeMS,GyrX,GyrY,G
</code></pre>

<p>I have been reading around to try and find how do I implement a code into python that will allow me to parse this data so that I can plot some of the column against each other (Mostly time).</p>

<p>Some things I found that may help in doing this:</p>

<p>There is a code that will allow me to convert this data into a CSV file. I know how to use the code and convert it to a CSV file and plot from there, but for a learning experience I want to be able to do this without converting it to a CSV file. Now I tried reading that code but I am clueless since I am very new to python. Here is the link to the code:</p>

<pre><code>https://github.com/PX4/Firmware/blob/master/Tools/sdlog2/sdlog2_dump.py
</code></pre>

<p>Also, Someone posted this saying this might be the log format, but again I couldn't understand or run any code on that page.</p>

<pre><code>http://dev.px4.io/advanced-ulog-file-format.html
</code></pre>

<p>A good starting point for parsing binary data is the struct module <a href="https://docs.python.org/3/library/struct.html" rel="nofollow">https://docs.python.org/3/library/struct.html</a> and it's <code>unpack</code> function. That's what the CSV dump routine you linked to is doing as well. If you walk through the <code>process</code> method, it's doing the following:</p>

<ul>
<li>Read a chunk of binary data</li>
<li>Figure out if it has a valid header</li>
<li>Check the message type - if it's a FORMAT message parse that. If it's
a description message, parse that. </li>
<li>Dump out a CSV row</li>
</ul>

<p>You could modify this code to essentially replace the <code>__printCSVRow</code> method with something that captures the data into a pandas dataframe (or other handy data structure) so that when the main routine is all done you can grab all the data from the dataframe and plot it.</p>

python