Numpy dtype invalid index
<pre><code>I'm trying to load csv data file:
ACCEPT,organizer@t.net,t,p1@t.net,0,UK,3600000,3,1475917200000,1475920800000,MON,9,0,0,0
</code></pre>

<p>in following way:</p>

<pre><code>dataset = genfromtxt('./training_set.csv', delimiter=',', dtype='a20, a20, a20, a8, i8, a20, i8, i8, i8, i8, a3, i8, i8, i8, i8')
print(dataset)
target = [x[0] for x in dataset]
train = [x[1:] for x in dataset]
</code></pre>

<p>in last line above I've got an error:</p>

<pre><code>---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
&lt;ipython-input-66-5d58edf06039&gt; in &lt;module&gt;()
      4 print(dataset)
      5 target = [x[0] for x in dataset]
----&gt; 6 train = [x[1:] for x in dataset]
      7 
      8 #rf = RandomForestClassifier(n_estimators=100)

&lt;ipython-input-66-5d58edf06039&gt; in &lt;listcomp&gt;(.0)
      4 print(dataset)
      5 target = [x[0] for x in dataset]
----&gt; 6 train = [x[1:] for x in dataset]
      7 
      8 #rf = RandomForestClassifier(n_estimators=100)

IndexError: invalid index
</code></pre>

<p>How to handle this?</p>

<p>With that <code>dtype</code> you have created a structured array - it is 1d with a compound dtype.</p>

<p>I have a sample structured array from another problem:</p>

<pre><code>In [26]: data
Out[26]: 
array([(b'1Q11', 252.0, 0.0166), (b'2Q11', 212.4, 0.0122),
       (b'3Q11', 425.9, 0.0286), (b'4Q11', 522.3, 0.0322),
       (b'1Q12', 263.2, 0.0185), (b'2Q12', 238.6, 0.0131),
       ...
       (b'1Q14', 264.5, 0.0179), (b'2Q14', 211.2, 0.0116)], 
      dtype=[('Qtrs', 'S4'), ('Y', '&lt;f8'), ('X', '&lt;f8')])
</code></pre>

<p>One record is: </p>

<pre><code>In [27]: data[0]
Out[27]: (b'1Q11', 252.0, 0.0166)
</code></pre>

<p>While I can access elements within that a number, it does not accept a slice:</p>

<pre><code>In [36]: data[0][1]
Out[36]: 252.0
In [37]: data[0][1:]
....
IndexError: invalid index
</code></pre>

<p>The preferred way of accessing elements with a structured record is with the field name:</p>

<pre><code>In [38]: data[0]['X']
Out[38]: 0.0166
</code></pre>

<p>Such a name allows me to access that field across all records:</p>

<pre><code>In [39]: data['X']
Out[39]: 
array([ 0.0166,  0.0122,  0.0286, ...  0.0116])
</code></pre>

<p>Fetching multiple fields requires a list of field names (and is more wordy than 2d slicing):</p>

<pre><code>In [42]: data.dtype.names[1:]
Out[42]: ('Y', 'X')

In [44]: data[list(data.dtype.names[1:])]
Out[44]: 
array([(252.0, 0.0166), (212.4, 0.0122),... (211.2, 0.0116)], 
      dtype=[('Y', '&lt;f8'), ('X', '&lt;f8')])
</code></pre>

<p>===============</p>

<p>With your sample line (replicated 3 times) I can load:</p>

<pre><code>In [53]: dataset=np.genfromtxt(txt,dtype=None,delimiter=',')
In [54]: dataset
Out[54]: 
array([ (b'ACCEPT', b'organizer@t.net', b't', b'p1@t.net', 0, b'UK', 3600000, 3, 1475917200000, 1475920800000, b'MON', 9, 0, 0, 0),
       (b'ACCEPT', b'organizer@t.net', b't', b'p1@t.net', 0, b'UK', 3600000, 3, 1475917200000, 1475920800000, b'MON', 9, 0, 0, 0),
       (b'ACCEPT', b'organizer@t.net', b't', b'p1@t.net', 0, b'UK', 3600000, 3, 1475917200000, 1475920800000, b'MON', 9, 0, 0, 0)], 
      dtype=[('f0', 'S6'), ('f1', 'S15'), ('f2', 'S1'), ('f3', 'S8'), ('f4', '&lt;i4'), ('f5', 'S2'), ('f6', '&lt;i4'), ('f7', '&lt;i4'), ('f8', '&lt;i8'), ('f9', '&lt;i8'), ('f10', 'S3'), ('f11', '&lt;i4'), ('f12', '&lt;i4'), ('f13', '&lt;i4'), ('f14', '&lt;i4')])
In [55]: 
</code></pre>

<p><code>dtype=None</code> produces something similar to your explicit <code>dtype</code>;</p>

<p>To get your desired output (as arrays, not lists):</p>

<pre><code>target = dataset['f0']
names=dataset.dtype.names[1:]
train = dataset[list(names)]
</code></pre>

<p>=====================</p>

<p>You could also refine the dtype to make the task simpler.  Define 2 fields, with the 2nd containing most of csv columns.  <code>genfromtxt</code> handles this sort of dtype nesting - just so long as the total field count is correct.</p>

<pre><code>In [106]: dt=[('target','a20'), 
       ('train','a20, a20, a8, i8, a20, i8, i8, i8, i8, a3, i8, i8, i8, i8')]
In [107]: dataset=np.genfromtxt(txt,dtype=dt,delimiter=',')
In [108]: dataset
Out[108]: 
array([ (b'ACCEPT', (b'organizer@t.net', b't', b'p1@t.net', 0, b'UK', 3600000, 3, 1475917200000, 1475920800000, b'MON', 9, 0, 0, 0)),
...], 
      dtype=[('target', 'S20'), ('train', [('f0', 'S20'), ('f1', 'S20'), ('f2', 'S8'), ('f3', '&lt;i8'), ('f4', 'S20'), ('f5', '&lt;i8'), ('f6', '&lt;i8'), ('f7', '&lt;i8'), ('f8', '&lt;i8'), ('f9', 'S3'), ('f10', '&lt;i8'), ('f11', '&lt;i8'), ('f12', '&lt;i8'), ('f13', '&lt;i8')])])
</code></pre>

<p>Now just select the 2 top level fields:</p>

<pre><code>In [109]: dataset['target']
Out[109]: 
array([b'ACCEPT', b'ACCEPT', b'ACCEPT'], 
      dtype='|S20')

In [110]: dataset['train']
Out[110]: 
array([ (b'organizer@t.net', b't', b'p1@t.net', 0, b'UK', 3600000, 3, 1475917200000, 1475920800000, b'MON', 9, 0, 0, 0),
...], 
      dtype=[('f0', 'S20'), ('f1', 'S20'), ...])
</code></pre>

<p>I could nest further, grouping the <code>i8</code> columns into groups of 4:</p>

<pre><code>dt=[('target','a20'), ('train','a20, a20, a8, i8, a20, (4,)i8, a3, (4,)i8')]
</code></pre>

python numpy