numpy.genfromtxt- ValueError- Line # (got n columns instead of m)
<p>So, I've been writing up code to read in a dataset from a file and separate it out for analysis.  </p>

<p>The data in question is read from a .dat file, and looks like this:</p>

<pre><code>14        HO2       O3        OH        O2        O2
15        HO2       HO2       H2O2      O2
16        H2O2      OH        HO2       H2O
17        O         O         O2
18        O         O2        O3
19        O         O3        O2        O2
</code></pre>

<p>The code I've written looks like this:</p>

<pre><code>edge_data=np.genfromtxt('Early_earth_reaction.dat', dtype = str, 
missing_values=True, filling_values=bool)
</code></pre>

<p>The plan was that I'd then run the values from the dataset and build a paired list from them.</p>

<pre><code>edge_list=[]
for i in range(360):
    edge_list.append((edge_data[i,0],edge_data[i,2]))
    edge_list.append((edge_data[i,1],edge_data[i,2]))
    print edge_data[i,0]
    if edge_data[i,3] != None:
        edge_list.append((edge_data[i,0],edge_data[i,3]))
        edge_list.append((edge_data[i,1],edge_data[i,3]))
    if edge_data[i,4]!= None:
        edge_list.append((edge_data[i,0],edge_data[i,4]))
        edge_list.append((edge_data[i,1,edge_data[i,4]))
</code></pre>

<p>However, upon running it, I get the error message </p>

<pre><code>File "read_early_earth.py", line 52, in main
edge_data=np.genfromtxt('Early_earth_reaction.dat', dtype = str,  
usecols=(1,2,3,4,5), missing_values=True, filling_values=bool)
File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 1667, 
in genfromtxt
raise ValueError(errmsg)
ValueError: Some errors were detected !
Line #6 (got 4 columns instead of 5)
Line #14 (got 6 columns instead of 5)
Line #17 (got 4 columns instead of 5)
</code></pre>

<p>And so on and so forth. As far as I can tell, this is happening because there are rows where not all the columns have values in them, which apparently throws numpy for a loop.  </p>

<p>Is there a work-around for this in numpy?  Alternatively, is there another way to accomplish this task?  I know, worse comes to worse, I can torture some regular expressions into doing the job, but I'd prefer a method that's a bit more efficient if at all possible.</p>

<p>Thanks!</p>

<p>It looks like your data is nicely aligned in fields of exactly 10 characters.  If that is always the case, you can tell <code>genfromtxt</code> the field widths to use by specifying the sequence of field widths in the <code>delimiter</code> argument.</p>

<p>Here's an example.</p>

<p>First, your data file:</p>

<pre><code>In [20]: !cat reaction.dat
14        HO2       O3        OH        O2        O2
15        HO2       HO2       H2O2      O2
16        H2O2      OH        HO2       H2O
17        O         O         O2
18        O         O2        O3
19        O         O3        O2        O2
</code></pre>

<p>For convenience, I'll define the number of fields and the field width here. (In general, it is not necessary that all the fields have the same width.)</p>

<pre><code>In [21]: numfields = 6

In [22]: fieldwidth = 10
</code></pre>

<p>Tell <code>genfromtxt</code> that the data is in fixed width columns by passing in the argument <code>delimiter=(10, 10, 10, 10, 10, 10)</code>:</p>

<pre><code>In [23]: data = genfromtxt('reaction.dat', dtype='S%d' % fieldwidth, delimiter=(fieldwidth,)*numfields)
</code></pre>

<p>Here's the result.  Note that "missing" fields are empty strings.  Also note that non-empty fields include the white space, and the last non-empty field in each row includes the newline character:</p>

<pre><code>In [24]: data
Out[24]: 
array([[b'14        ', b'HO2       ', b'O3        ', b'OH        ',
        b'O2        ', b'O2\n'],
       [b'15        ', b'HO2       ', b'HO2       ', b'H2O2      ',
        b'O2\n', b''],
       [b'16        ', b'H2O2      ', b'OH        ', b'HO2       ',
        b'H2O\n', b''],
       [b'17        ', b'O         ', b'O         ', b'O2\n', b'', b''],
       [b'18        ', b'O         ', b'O2        ', b'O3\n', b'', b''],
       [b'19        ', b'O         ', b'O3        ', b'O2        ',
        b'O2\n', b'']], 
      dtype='|S10')

In [25]: data[1]
Out[25]: 
array([b'15        ', b'HO2       ', b'HO2       ', b'H2O2      ', b'O2\n',
       b''], 
      dtype='|S10')
</code></pre>

<p>We could clean up the strings in a second step, or we can have <code>genfromtxt</code> do it by providing a converter for each field that simply strips the white space from the field:</p>

<pre><code>In [26]: data = genfromtxt('reaction.dat', dtype='S%d' % fieldwidth, delimiter=(fieldwidth,)*numfields, converters={k: lambda s: s.
    ...: strip() for k in range(numfields)})

In [27]: data
Out[27]: 
array([[b'14', b'HO2', b'O3', b'OH', b'O2', b'O2'],
       [b'15', b'HO2', b'HO2', b'H2O2', b'O2', b''],
       [b'16', b'H2O2', b'OH', b'HO2', b'H2O', b''],
       [b'17', b'O', b'O', b'O2', b'', b''],
       [b'18', b'O', b'O2', b'O3', b'', b''],
       [b'19', b'O', b'O3', b'O2', b'O2', b'']], 
      dtype='|S10')

In [28]: data[:,0]
Out[28]: 
array([b'14', b'15', b'16', b'17', b'18', b'19'], 
      dtype='|S10')

In [29]: data[:,5]
Out[29]: 
array([b'O2', b'', b'', b'', b'', b''], 
      dtype='|S10')
</code></pre>

python numpy genfromtxt