Memory-friendly way to add a field to a structured ndarray â without duplicating data?
<p>To add a field to a structured numpy array, it is quite simply to create a new array with a new dtype, copy over the old fields, and add the new field.  However, I need to do this for an array that takes a lot of memory, and I would rather not duplicate all of it.  Both my own implementation and the (slow) implementation in <code>numpy.lib.recfunctions.append_fields</code> duplicate memory.</p>

<p>Is there a way to add a field to a structured <code>ndarray</code>, without duplicating memory?  That means, either a way that avoids creating a new <code>ndarray</code>, <em>or</em> a way to create a new <code>ndarray</code> that points to the same data as the old?</p>

<p>Solutions that do duplicate RAM:</p>

<ul>
<li><p><a href="http://stackoverflow.com/questions/1201817/adding-a-field-to-a-structured-numpy-array">Adding a field to a structured numpy array</a></p></li>
<li><p><a href="http://stackoverflow.com/questions/5288736/adding-a-field-to-a-structured-numpy-array-2">Adding a field to a structured numpy array (2)</a></p></li>
<li><p><a href="http://stackoverflow.com/questions/21413947/adding-a-field-to-a-structured-numpy-array-3">Adding a field to a structured numpy array (3)</a></p></li>
</ul>

<p>There is a <a href="http://stackoverflow.com/questions/37079175/how-to-remove-a-column-from-a-structured-numpy-array-without-copying-it">similar question</a> where the challenge is to <em>remove</em>, not add, fields.  The solution uses a view, which should work for a subset of the original data, but I'm not sure if it can be amended when I rather want to <em>add</em> fields.</p>

<p>A structured array is stored, like a regular one, as a contiguous buffer of bytes, one record following the previous.  The records are, thus, a bit like the last dimension of a multidimensional array.  You can't add a column to a 2d array without making a new array via concatenation.</p>

<p>Adding a field, say <code>I4</code> dtype to dtype that is, say, 20 bytes long, means changing the record (element) length to 24, i.e. adding 4 bytes to the buffer every 20th byte.  <code>numpy</code> can't do that without making a new data buffer and copying values from the old (and the new).</p>

<p>Actually even if we were talking about adding a new record to the array, i.e. concatenating on a new array, it would still require creating a new data buffer.  Arrays are fixed sized.</p>

<p>Fields in a structured array are not like objects in a list or a dictionary.  You can't add a field by just adding a pointer to an object elsewhere in memory.</p>

<p>Maybe you should be using a dictionary, with <code>item</code> being an array.  Then you can freely add a key/item without copying the existing ones.  But then access by 'rows' will be slow.</p>

python arrays numpy memory structured-array