NumPy genfromxt TypeError: data type not understood error
<p>I would like to read in this file (test.txt)</p>

<pre><code>01.06.2015;00:00:00;0.000;0;-9.999;0;8;0.00;18951;(SPECTRUM)ZERO(/SPECTRUM)
01.06.2015;00:01:00;0.000;0;-9.999;0;8;0.00;18954;(SPECTRUM)ZERO(/SPECTRUM)
01.06.2015;00:02:00;0.000;0;-9.999;0;8;0.00;18960;(SPECTRUM)ZERO(/SPECTRUM)
01.06.2015;09:23:00;0.327;61;25.831;39;29;0.18;19006;01.06.2015;09:23:00;0.327;61;25.831;39;29;0.18;19006;(SPECTRUM);;;;;;;;;;;;;;1;1;;;1;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;1;;;;;;;;;;;;(/SPECTRUM)
01.06.2015;09:24:00;0.000;0;-9.999;0;29;0.00;19010;(SPECTRUM)ZERO(/SPECTRUM)
</code></pre>

<p>...I tried it with the numpy function genfromtxt() (see below in the code excerpt).</p>

<pre><code>import numpy as np
col_names = ["date", "time", "rain_intensity", "weather_code_1", "radar_ref", "weather_code_2", "val6", "rain_accum", "val8", "val9"]
types = ["object", "object", "float", "uint8", "float", "uint8", "uint8", "float", "uint8","|S10"]
# Read in the file with np.genfromtxt
mydata = np.genfromtxt("test.txt", delimiter=";", names=col_names, dtype=types)
</code></pre>

<p>Now when I execute the code I get the following error --></p>

<pre><code>raise ValueError(errmsg)ValueError: Some errors were detected !
    Line #4 (got 79 columns instead of 10)
</code></pre>

<p>Now I think that the difficulties come from the last column (val9) with the many <code>;;;;;;;</code><br>
It is obvious that the delimeters and the signs in the last column<code>;</code> are the same!</p>

<p>How can I read in the file without an error, maybe there is a possibility to skip the last column, or to replace the <code>;</code> only in the last column?</p>

<p>From the <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html" rel="nofollow">numpy documentation</a></p>

<blockquote>
  <p><strong>invalid_raise</strong> : bool, optional<br>
  If True, an exception is raised if an inconsistency is detected in the
  number of columns. If False, a warning is emitted and the offending
  lines are skipped.</p>
</blockquote>

<pre><code>mydata = np.genfromtxt("test.txt", delimiter=";", names=col_names, dtype=types, invalid_raise = False)
</code></pre>

<p>Note that there were errors in your code which I have corrected (delimiter spelled incorrectly, and <code>types</code> list referred to as <code>dtypes</code> in function call)</p>

<p><strong>Edit</strong>: From your comment, I see I slightly misunderstood. You meant that you want to skip the last <em>column</em> not the last <em>row</em>.</p>

<p>Take a look at the following code. I have defined a generator that only returns the first ten elements of each row. This will allow <code>genfromtxt()</code> to complete without error and you now get column #3 from all rows.</p>

<p>Note though, that you are still going to lose some data, as if you look carefully you will see that the problem line is actually two lines concatenated together with garbage where the other lines have <code>ZERO</code>. So you are still going to lose this second line. You could maybe modify the generator to parse each line and deal with this differently, but I'll leave that as a fun exercise :)</p>

<pre><code>import numpy as np

def filegen(filename):
    with open(filename, 'r') as infile:
        for line in infile:
            yield ';'.join(line.split(';')[:10])

col_names = ["date", "time", "rain_intensity", "weather_code_1", "radar_ref", "weather_code_2", "val6", "rain_accum", "val8", "val9"]
dtypes = ["object", "object", "float", "uint8", "float", "uint8", "uint8", "float", "uint8","|S10"]
# Read in the file with np.genfromtxt
mydata = np.genfromtxt(filegen('temp.txt'), delimiter=";", names=col_names, dtype = dtypes)
</code></pre>

<p><strong>Output</strong></p>

<pre><code>[('01.06.2015', '00:00:00', 0.0, 0, -9.999, 0, 8, 0.0, 7, '(SPECTRUM)')
 ('01.06.2015', '00:01:00', 0.0, 0, -9.999, 0, 8, 0.0, 10, '(SPECTRUM)')
 ('01.06.2015', '00:02:00', 0.0, 0, -9.999, 0, 8, 0.0, 16, '(SPECTRUM)')
 ('01.06.2015', '09:23:00', 0.327, 61, 25.831, 39, 29, 0.18, 62, '01.06.2015')
 ('01.06.2015', '09:24:00', 0.0, 0, -9.999, 0, 29, 0.0, 66, '(SPECTRUM)')]
</code></pre>

python numpy genfromtxt