Realizing different column formats with numpy.savetxt
<p>I want to create a <code>.csv</code> file using <code>numpy.savetxt</code>. Each <code>row</code> of the file indicates a certain event. Every row has multiple <code>columns</code> indicating different elements of the event. The information stored in each <code>column</code> is different. Certain <code>columns</code> will contain single <code>float</code> values while others should contain two <code>floats</code> that are connected to each other. If I would call that column when loading the <code>.csv</code> I should obtain the two <code>float</code> values. </p>

<p>I have the following code:</p>

<pre><code>import numpy

rows = 5
columns = 2 
save_values = numpy.zeros((rows, columns))

for idx in xrange(rows):
  column_0 = float(idx)
  column_1 = [idx + 5., idx + 15.]

  save_values[idx, :] = column_0, column_1

numpy.savetxt("outfile.csv", save_values, delimiter = ",")
</code></pre>

<p>This however results in the following error message: </p>

<pre><code>  save_values[idx, :] = column_0, column_1
ValueError: setting an array element with a sequence.
</code></pre>

<p>which is understandable. However, despite knowing why it is going wrong I am having a hard time realizing my goal.</p>

<p>How can I achieve my goal?</p>

<p>You aren't even getting to the <code>savetxt</code> step.</p>

<pre><code>save_values[idx, :] = column_0, column_1
</code></pre>

<p>the target is 2 values (2 columns).  The source is <code>idx</code> and a list.</p>

<p>That's why it's giving you the 'setting with a sequence' error.  It can't put the list in <code>save_values[idx,1]</code>.</p>

<p>You could define a <code>save_values</code> array that has 2 fields, and one of the fields having 2 elements.  But how would you save it?</p>

<p>How should the text file appear - 3 columns separated by <code>,</code>?  or two columns with special structure inside the 2nd?, e.g.</p>

<pre><code> 1.2, 3.5, 4.2   # or
 1.2, [2.5, 4.2]
</code></pre>

<p>That in turn raises the issue of what can be loaded.  <code>genfromtxt</code> can handle the 3 columns; it can't readily hand the nested columns.  As a default <code>genfromtxt</code> would read the 3 column case as 3 columns, but it is possible to give it the 2 fields <code>dtype</code>.</p>

<p>Anyways, for saving I think generating 3 columns is simplest.  Reloading could be done with columns or fields.</p>

<p>I can generate a compound dtype array with:</p>

<pre><code>In [329]: dt = np.dtype('i,(2,)f')
In [330]: dt
Out[330]: dtype([('f0', '&lt;i4'), ('f1', '&lt;f4', (2,))])
In [331]: save_values = np.zeros((5,),dtype=dt)
In [332]: for i in range(5):
     ...:     save_values[i]=(i,(i+5., i+15.))
     ...:     
In [333]: save_values
Out[333]: 
array([(0, [5.0, 15.0]), (1, [6.0, 16.0]), (2, [7.0, 17.0]),
       (3, [8.0, 18.0]), (4, [9.0, 19.0])], 
      dtype=[('f0', '&lt;i4'), ('f1', '&lt;f4', (2,))])
</code></pre>

<p>But if I try to save it I get an error</p>

<pre><code>In [334]: np.savetxt('test.txt',save_values,delimiter=',')
...
TypeError: Mismatch between array dtype ('[('f0', '&lt;i4'), ('f1', '&lt;f4', (2,))]') and format specifier ('%.18e,%.18e')
</code></pre>

<p>I can save it by spelling out the write format, <code>fmt%tuple(save_values[0])</code>, but that puts <code>[]</code> in the output:</p>

<pre><code>In [335]: np.savetxt('test.txt',save_values,fmt='%10d, %s')
In [336]: cat test.txt
         0, [  5.  15.]
         1, [  6.  16.]
         2, [  7.  17.]
         3, [  8.  18.]
         4, [  9.  19.]
</code></pre>

<p>I can flatten the array dtype with a view (here I'm keeping the 1st field integer just to keep things interesting):</p>

<pre><code>In [337]: dt1=np.dtype('i,f,f')
In [338]: save_values.view(dt1)
Out[338]: 
array([(0, 5.0, 15.0), (1, 6.0, 16.0), (2, 7.0, 17.0), (3, 8.0, 18.0),
       (4, 9.0, 19.0)], 
      dtype=[('f0', '&lt;i4'), ('f1', '&lt;f4'), ('f2', '&lt;f4')])
</code></pre>

<p>Now I can save it as 3 columns:</p>

<pre><code>In [340]: np.savetxt('test.txt',save_values.view(dt1),fmt='%10d, %10f, %10f')
In [341]: cat test.txt
         0,   5.000000,  15.000000
         1,   6.000000,  16.000000
         2,   7.000000,  17.000000
         3,   8.000000,  18.000000
         4,   9.000000,  19.000000
</code></pre>

<p>and I can reload it with either dtype:</p>

<pre><code>In [342]: np.genfromtxt('test.txt',delimiter=',',dtype=dt)
Out[342]: 
array([(0, [5.0, 15.0]), (1, [6.0, 16.0]), (2, [7.0, 17.0]),
       (3, [8.0, 18.0]), (4, [9.0, 19.0])], 
      dtype=[('f0', '&lt;i4'), ('f1', '&lt;f4', (2,))])

In [343]: np.genfromtxt('test.txt',delimiter=',',dtype=dt1)
Out[343]: 
array([(0, 5.0, 15.0), (1, 6.0, 16.0), (2, 7.0, 17.0), (3, 8.0, 18.0),
       (4, 9.0, 19.0)], 
      dtype=[('f0', '&lt;i4'), ('f1', '&lt;f4'), ('f2', '&lt;f4')])
</code></pre>

<p>I could have also created the <code>text.txt</code> with a 5x3 array of floats.</p>

<p>The key point with <code>savetxt</code> is that it iterates over the rows your array, formats them, and writes that line to the file.  So your array has to work with:</p>

<pre><code>for row in myarray:
   print(fmt % tuple(row))
</code></pre>

<p><code>fmt</code> may be be spelled out, or may build from a single field format, eg.</p>

<pre><code>fmt = ','.join(['%10f']*3)   # or
fmt = '%10d, %10f, %10f'
</code></pre>

<p>So it comes down to standard Python string formatting.</p>

python csv numpy save