Python readinto: How to convert from an array.array to a custom ctype structure
<p>I have created an array of integers and I would like them to be interpreted by the structure definition which I have created</p>

<pre><code>from ctypes import *
from array import array

class MyStruct(Structure):
    _fields_ = [("init", c_uint),
                ("state", c_char),
                ("constant", c_int),
                ("address", c_uint),
                ("size", c_uint),
                ("sizeMax", c_uint),
                ("start", c_uint),
                ("end", c_uint),
                ("timestamp", c_uint),
                ("location", c_uint),
                ("nStrings", c_uint),
                ("nStringsMax", c_uint),
                ("maxWords", c_uint),
                ("sizeFree", c_uint),
                ("stringSizeMax", c_uint),
                ("stringSizeFree", c_uint),
                ("recordCount", c_uint),
                ("categories", c_uint),
                ("events", c_uint),
                ("wraps", c_uint),
                ("consumed", c_uint),
                ("resolution", c_uint),
                ("previousStamp", c_uint),
                ("maxTimeStamp", c_uint),
                ("threshold", c_uint),
                ("notification", c_uint),
                ("version", c_ubyte)]

# arr = array.array('I', [1])
# How can I do this?
# mystr = MyStruct(arr) magic
# (mystr.helloworld == 1) == True
</code></pre>

<p>I can do the following:</p>

<pre><code>mystr = MyStruct()
rest = array.array('I')
with open('myfile.bin', 'rb') as binaryFile:
    binaryFile.readinto(mystr)
    rest.fromstring(binaryFile.read())

# Now create another struct with rest
rest.readinto(mystr) # Does not work
</code></pre>

<p>How can I avoid using a file to convert an array of Ints to a struct if the data is contained in an array.array('I')? I am not sure what the Structure constructor accepts or how the readinto works.</p>

<h2>Solution #1: Star unpacking for one-line initialization</h2>

<p>Star-unpacking will work, but only if all the fields in your structure are integer types. In Python 2.x, <code>c_char</code> cannot be initialized from an <code>int</code> (it works fine in 3.5). If you change the type of <code>state</code> to <code>c_byte</code>, then you can just do:</p>

<pre><code>mystr = MyStruct(*myarr)
</code></pre>

<p>This doesn't actually benefit from any <code>array</code> specific magic (the values are briefly converted to Python <code>int</code>s in the unpacking step, so you're not reducing peak memory usage), so you'd only bother with an <code>array</code> if initializing said <code>array</code> was easier than directly reading into the structure for whatever reason.</p>

<p>If you go the star unpacking route, reading <code>.state</code> will now get you <code>int</code> values instead of len 1 <code>str</code> values. If you want to initialize with <code>int</code>, but read as one character <code>str</code>, you can use a protected name wrapped in a <code>property</code>:</p>

<pre><code>class MyStruct(Structure):
    _fields_ = [...
                ("_state", c_byte),  # "Protected" name int-like; constructor expects int
                ...]

    @property
    def state(self):
        return chr(self._state)

    @state.setter
    def state(self, x):
        if isinstance(x, basestring):
            x = ord(x)
        self._state = x
</code></pre>

<p>A similar technique could be used without <code>property</code>s by defining your own <code>__init__</code> that converted the <code>state</code> argument passed:</p>

<pre><code>class MyStruct(Structure):
    _fields_ = [("init", c_uint),
                ("state", c_char),
                ...]

    def __init__(self, init=0, state=b'\0', *args, **kwargs):
        if not isinstance(state, basestring):
            state = chr(state)
        super(MyStruct, self).__init__(init, state, *args, **kwargs)
</code></pre>

<h2>Solution #2: Direct <code>memcpy</code>-like solutions to reduce temporaries</h2>

<p>You can use some <code>array</code> specific magic to avoid the temporary Python level <code>int</code>s though (and avoid the need to change <code>state</code> to <code>c_byte</code>) without real file objects using a fake (in-memory) file-like object:</p>

<pre><code>import io

mystr = MyStruct()  # Default initialize

# Use BytesIO to gain the ability to write the raw bytes to the struct
# because BytesIO's readinto isn't finicky about exact buffer formats
io.BytesIO(myarr.tostring()).readinto(mystr)

# In Python 3, where array implements the buffer protocol, you can simplify to:
io.BytesIO(myarr).readinto(mystr)
# This still performs two memcpys (one occurs internally in BytesIO), but
# it's faster by avoiding a Python level method call
</code></pre>

<p>This only works because your non-<code>c_int</code> width attributes are followed by <code>c_int</code> width attributes (so they're padded out to four bytes anyway); if you had two <code>c_ubyte</code>/<code>c_char</code>/etc. types back to back, then you'd have problems (because one value of the <code>array</code> would initialize two fields in the struct, which does not appear to be what you want).</p>

<p>If you were using Python 3, you could benefit from <code>array</code> specific magic to avoid the cost of both unpacking and the two step <code>memcpy</code> of the <code>BytesIO</code> technique (from <code>array</code> -> <code>bytes</code> -> struct). It works in Py3 because Py3's <code>array</code> type supports the buffer protocol (it didn't in Py2), and because Py3's <code>memoryview</code> features a <code>cast</code> method that lets you change the format of the <code>memoryview</code> to make it directly compatible with <code>array</code>:</p>

<pre><code>mystr = MyStruct()  # Default initialize

# Make a view on mystr's underlying memory that behaves like a C array of
# unsigned ints in native format (matching array's type code)
# then perform a "memcpy" like operation using empty slice assignment
# to avoid creating any Python level values.
memoryview(mystr).cast('B').cast('I')[:] = myarr
</code></pre>

<p>Like the <code>BytesIO</code> solution, this only works because your fields all happen to pad to four bytes in size</p>

<h2>Performance</h2>

<p>Performance-wise, star unpacking wins for small numbers of fields, but for large numbers of fields (your case has a couple dozen), direct <code>memcpy</code> based approaches win out; in tests for a 23 field class, the <code>BytesIO</code> solution won over star unpacking on my Python 2.7 install by a factor of 2.5x (star unpacking was 2.5 microseconds, <code>BytesIO</code> was 1 microsecond).</p>

<p>The <code>memoryview</code> solution scales similarly to the <code>BytesIO</code> solution, though as of 3.5, it's slightly slower than the <code>BytesIO</code> approach (likely a result of the need to construct several temporary <code>memoryview</code>s to perform the necessary casting operations and/or the <code>memoryview</code> slice assignment code being general purpose for many possible formats, so it's not simple <code>memcpy</code> in implementation). <code>memoryview</code> might scale better for much larger copies (if the losses are due to the fixed <code>cast</code> overhead), but it's rare that you'd have a struct large enough to matter; it would only be in more general purpose copying scenarios (to and from <code>ctypes</code> arrays or the like) that <code>memoryview</code> would potentially win.</p>

python arrays ctypes python-2.x