numpy.genfromtxt- ValueError- Line # (got n columns instead of m)
<p>So, I've been writing up code to read in a dataset from a file and separate it out for analysis.  </p>

<p>The data in question is read from a .dat file, and looks like this:</p>

<pre><code>14        HO2       O3        OH        O2        O2
15        HO2       HO2       H2O2      O2
16        H2O2      OH        HO2       H2O
17        O         O         O2
18        O         O2        O3
19        O         O3        O2        O2
</code></pre>

<p>The code I've written looks like this:</p>

<pre><code>edge_data=np.genfromtxt('Early_earth_reaction.dat', dtype = str, 
missing_values=True, filling_values=bool)
</code></pre>

<p>The plan was that I'd then run the values from the dataset and build a paired list from them.</p>

<pre><code>edge_list=[]
for i in range(360):
    edge_list.append((edge_data[i,0],edge_data[i,2]))
    edge_list.append((edge_data[i,1],edge_data[i,2]))
    print edge_data[i,0]
    if edge_data[i,3] != None:
        edge_list.append((edge_data[i,0],edge_data[i,3]))
        edge_list.append((edge_data[i,1],edge_data[i,3]))
    if edge_data[i,4]!= None:
        edge_list.append((edge_data[i,0],edge_data[i,4]))
        edge_list.append((edge_data[i,1,edge_data[i,4]))
</code></pre>

<p>However, upon running it, I get the error message </p>

<pre><code>File "read_early_earth.py", line 52, in main
edge_data=np.genfromtxt('Early_earth_reaction.dat', dtype = str,  
usecols=(1,2,3,4,5), missing_values=True, filling_values=bool)
File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 1667, 
in genfromtxt
raise ValueError(errmsg)
ValueError: Some errors were detected !
Line #6 (got 4 columns instead of 5)
Line #14 (got 6 columns instead of 5)
Line #17 (got 4 columns instead of 5)
</code></pre>

<p>And so on and so forth. As far as I can tell, this is happening because there are rows where not all the columns have values in them, which apparently throws numpy for a loop.  </p>

<p>Is there a work-around for this in numpy?  Alternatively, is there another way to accomplish this task?  I know, worse comes to worse, I can torture some regular expressions into doing the job, but I'd prefer a method that's a bit more efficient if at all possible.</p>

<p>Thanks!</p>

<p>Looks like you've already read <code>genfromtxt</code> about missing values.  Does it say anything about the use of delimiters?</p>

<p>I think it can handle missing values with lines like</p>

<pre><code>'one, 1, 234.4, , ,'
'two, 3, , 4, 5'
</code></pre>

<p>but when the delimiter is the default 'white-space' it can't.  One of the first steps after reading a line is</p>

<pre><code> strings = line.split(delimiter)
</code></pre>

<p>And objects if <code>len(strings)</code> doesn't match with the initial target.  Apparently it does not try to guess that you want to pad the line with <code>n-len(strings)</code> missing values.</p>

<p>Options that come to mind:</p>

<ul>
<li><p>try Pandas; it may make more effort to guess your intentions</p></li>
<li><p>write your own reader.  Pandas is compiled; <code>genfromtxt</code> is plain numpy Python.  It reads the file line by line, splits and converts fields, and appends the list to a master list.  It converts that list of lists into array at the end.  Your own reader should be just as efficient.</p></li>
<li><p>preprocess your file to add the missing values or change the delimiter.  <code>genfromtxt</code> accepts anything that feeds it lines.  So it works with a list of strings, a file reader that yields modified lines, etc.  This may be simplest.</p>

<p>def foo(astr):
    strs=astr.split()
    if len(strs)&lt;6:
        strs.extend([b' ']*(6-len(strs)))
    return b','.join(strs)</p></li>
</ul>

<p>Simulating with a list of strings (in Py3):</p>

<pre><code>In [139]: txt=b"""14        HO2       O3        OH        O2        O2
     ...: 15        HO2       HO2       H2O2      O2
     ...: 16        H2O2      OH        HO2       H2O
     ...: 17        O         O         O2
     ...: 18        O         O2        O3
     ...: 19        O         O3        O2        O2""".splitlines()

In [140]: [foo(l) for l in txt]
Out[140]: 
[b'14,HO2,O3,OH,O2,O2',
 b'15,HO2,HO2,H2O2,O2, ',
 b'16,H2O2,OH,HO2,H2O, ',
 b'17,O,O,O2, , ',
 b'18,O,O2,O3, , ',
 b'19,O,O3,O2,O2, ']

In [141]: np.genfromtxt([foo(l) for l in txt], dtype=None, delimiter=',')
Out[141]: 
array([(14, b'HO2', b'O3', b'OH', b'O2', b'O2'),
       (15, b'HO2', b'HO2', b'H2O2', b'O2', b''),
       (16, b'H2O2', b'OH', b'HO2', b'H2O', b''),
       (17, b'O', b'O', b'O2', b' ', b''),
       (18, b'O', b'O2', b'O3', b' ', b''),
       (19, b'O', b'O3', b'O2', b'O2', b'')], 
      dtype=[('f0', '&lt;i4'), ('f1', 'S4'), ('f2', 'S3'), ('f3', 'S4'), ('f4', 'S3'), ('f5', 'S2')])
</code></pre>

python numpy genfromtxt