Can I efficiently create a pandas data frame from semi-structured binary data?
<p>I need to convert large binary files to n x 3 arrays. The data is a series of image frames defined by (x, y, time) coordinates. Each frame uses two 32-bit integers to define the n x 3 dimensions, and n triplets of 16-bit integers to define the (x, y, time) values. The result is a binary structure that looks like:</p>

<p><code>int32, int32, uint16, uint16, uint16, ..., int32, int32, uint16, uint16, uint16</code>, and so on. </p>

<p>My first attempt involved converting the binary data to a 1D array and then adding the sections I wanted to a data frame. The current data is already sorted in such a way that frame separation can be reconstructed without the two <code>int32</code> values, so they can be dropped if necessary. If this weren't the case, the same effect could be achieved by sorting each frame individually before adding it to the final data frame.</p>

<pre><code>import numpy as np
import pandas as pd

def frame_extract(index):
    n = data[index]
    subarray=data[index+4:index+(3*n+4)]
    subarray=np.reshape(subarray, (len(subarray)/3,3))
    frame = pd.DataFrame(data=subarray, columns=['x','y','t'])
    return frame

def indexer(index):
    n = data[index]
    new_index = index+(3*n+4)
    return new_index

data = np.fromfile('file.bin', dtype='&lt;u2')
framedata = pd.DataFrame()

index = 0
while index &lt;= len(data)-1:
    framedata = framedata.append(frame_extract(index), ignore_index=True)
    index = indexer(index)
print(framedata)
</code></pre>

<p>The above works, but the while loop is very slow, especially when compared to the following structured method, which would work fine (and orders of magnitude faster) if the <code>int32</code> values were not in the way:</p>

<pre><code>dt = np.dtype([('x', '&lt;u2'), ('y', '&lt;u2'), ('time', '&lt;u2')])
data = np.fromfile("file.bin", dtype=dt)
df = pd.DataFrame(data.tolist(), columns=data.dtype.names)
</code></pre>

<p>Is there a more efficient way of approaching this? If so, would it be easier to do while unpacking the binary data, or after it has been converted to integers?</p>

<p>I'm currently considering using a generator to read the binary file as a series of chunks (i.e. to use the two 32-bit integers to decide how large the 16-bit integer chunk I need is), but I'm not yet familiar enough with these to know if that's the right approach. </p>

<p>The <code>count</code> parameter simplified this by allowing <code>np.fromfile</code> to take advantage of the structure defined by the <code>int32</code> values. The following <code>for</code> loop creates each image frame individually:</p>

<pre><code>f = open('file.bin', 'rb')
for i in np.arange(1,15001,1): 
    m, n = np.fromfile(f, dtype='&lt;i', count=2)
    frame = np.reshape(np.fromfile(f, dtype='&lt;u2', count=m*n), (m, n))
</code></pre>

<p>Each frame can be added to a list and converted to a data frame using: </p>

<pre><code>f = open('file.bin', 'rb')
xyt_data = list()
for i in np.arange(1,15001,1):
    m, n = np.fromfile(f, dtype='&lt;i', count=2)
    frame = np.reshape(np.fromfile(f, dtype='&lt;u2', count=m*n), (m, n))
    xyt_data.append(frame)
df = pd.DataFrame(np.vstack(xyt_data), columns=['x','y','t'])
</code></pre>

<p>The result is about three orders of magnitude faster than the version described in the original question.</p>

python pandas numpy binary