Write multiple Numpy arrays of different dtype as columns of CSV file
<p>What would be the best way to write multiple numpy arrays of different dtype as different columns of a single CSV file?</p>

<p>For instance, given the following arrays:</p>

<pre><code>array([[1, 2],
       [3, 4],
       [5, 6]])

array([[ 10.,  20.],
       [ 30.,  40.],
       [ 50.,  60.]])
</code></pre>

<p>I would like to obtain a file (delimiter irrelevant):</p>

<pre><code>1 2 10.0 20.0
3 4 30.0 40.0
5 6 50.0 60.0
</code></pre>

<p>Optimally, I would like to be able to write a list of arrays this way, where a format/dtype can be different for every array.</p>

<p>I tried looking at <code>savetxt</code>, but it's not clear to me how to use it if the arrays have a different type.</p>

<pre><code>In [38]: a=np.arange(1,7).reshape(3,2)
In [39]: b=np.arange(10,70.,10).reshape(3,2)
In [40]: c=np.concatenate((a,b),axis=1)
In [41]: c
Out[41]: 
array([[  1.,   2.,  10.,  20.],
       [  3.,   4.,  30.,  40.],
       [  5.,   6.,  50.,  60.]])
</code></pre>

<p>All values are float; default <code>savetxt</code> is a general float:</p>

<pre><code>In [43]: np.savetxt('test.csv',c)
In [44]: cat test.csv
1.000000000000000000e+00 2.000000000000000000e+00 1.000000000000000000e+01 2.000000000000000000e+01
3.000000000000000000e+00 4.000000000000000000e+00 3.000000000000000000e+01 4.000000000000000000e+01
5.000000000000000000e+00 6.000000000000000000e+00 5.000000000000000000e+01 6.000000000000000000e+01
</code></pre>

<p>With a custom <code>fmt</code> I can get:</p>

<pre><code>In [46]: np.savetxt('test.csv',c,fmt='%2d %2d %5.1f %5.1f')
In [47]: cat test.csv
 1  2  10.0  20.0
 3  4  30.0  40.0
 5  6  50.0  60.0
</code></pre>

<p>More generally we can make a <code>c</code> with a compound dtype.  It isn't needed here with just floats and ints, but with strings it would matter.  But we still need a long <code>fmt</code> to display the columns correctly.</p>

<p><code>np.rec.fromarrays</code> is an easy way to generate a structured arrays.  Unfortunately it only works with flattened arrays.  So for your (3,2) arrays I need to list the columns separately.</p>

<pre><code>In [52]: c = np.rec.fromarrays((a[:,0],a[:,1],b[:,0],b[:,1]))
In [53]: c
Out[53]: 
rec.array([(1, 2, 10.0, 20.0), (3, 4, 30.0, 40.0), (5, 6, 50.0, 60.0)], 
          dtype=[('f0', '&lt;i4'), ('f1', '&lt;i4'), ('f2', '&lt;f8'), ('f3', '&lt;f8')])
In [54]: np.savetxt('test.csv',c,fmt='%2d %2d %5.1f %5.1f')
In [55]: cat test.csv
 1  2  10.0  20.0
 3  4  30.0  40.0
 5  6  50.0  60.0
</code></pre>

<p>I'm using the same <code>savetxt</code>.</p>

<p>I could also make a structured array with 2 fields, each being 2 columns.  I'm not sure if <code>savetxt</code> would work with that or not.</p>

<p><code>savetxt</code> essentially iterates over the 1st dimension of your array, and does a formatted write on each row, roughly:</p>

<pre><code>for row in arr:
    f.write(fmt%tuple(row))
</code></pre>

<p>where <code>fmt</code> is derived from your parameter.</p>

<p>It wouldn't be hard to write your own version that iterates on 2 arrays, and does a separate formatted write for each pair of rows.</p>

<pre><code>for r1,r2 in zip(a,b):
    print('%2d %2d'%tuple(r1), '%5.1f %5.1f'%tuple(r2))
</code></pre>

<p>===================</p>

<p>Trying a compound dtype</p>

<pre><code>In [60]: np.dtype('2i,2f')
Out[60]: dtype([('f0', '&lt;i4', (2,)), ('f1', '&lt;f4', (2,))])
In [61]: c=np.zeros(a.shape[0], np.dtype('2i,2f'))
In [62]: c['f0']=a
In [63]: c['f1']=b
In [64]: c
Out[64]: 
array([([1, 2], [10.0, 20.0]), ([3, 4], [30.0, 40.0]),
       ([5, 6], [50.0, 60.0])], 
      dtype=[('f0', '&lt;i4', (2,)), ('f1', '&lt;f4', (2,))])
In [65]: np.savetxt('test.csv',c,fmt='%2d %2d %5.1f %5.1f')
---
ValueError: fmt has wrong number of % formats:  %2d %2d %5.1f %5.1f
</code></pre>

<p>So writing a compound dtype like this does not work.  Considering that a row of <code>c</code> looks like:</p>

<pre><code>In [69]: tuple(c[0]) 
Out[69]: (array([1, 2], dtype=int32), array([ 10.,  20.], dtype=float32))
</code></pre>

<p>I shouldn't be surprised.</p>

<p>I can save the two blocks with <code>%s</code> format, but that leaves me with brackets.</p>

<pre><code>In [66]: np.savetxt('test.csv',c,fmt='%s %s')
In [67]: cat test.csv
[1 2] [ 10.  20.]
[3 4] [ 30.  40.]
[5 6] [ 50.  60.]
</code></pre>

<p>I think there is a <code>np.rec</code> function that flattens the dtype.  But I can also do that with a <code>view</code>:</p>

<pre><code>In [72]: np.savetxt('test.csv',c.view('i,i,f,f'),fmt='%2d %2d %5.1f %5.1f')
In [73]: cat test.csv
 1  2  10.0  20.0
 3  4  30.0  40.0
 5  6  50.0  60.0
</code></pre>

<p>So as long as you are dealing with numeric values, the simple concatenate is just as good as the more complex structured approaches.</p>

<p>============</p>

python csv numpy