how to convert string array of mixed data types
<p>Let's say I have read and loaded a file into a 2D matrix of mixed data as strings(an example has been provided below)</p>

<pre><code># an example row of the matrix
['529997' '46623448' '2122110124' '2310' '2054' '2' '66' '' '2010/11/03-12:42:08' '26' 'CLEARING' '781' '30' '3' '0' '0' '1']
</code></pre>

<p>I want to convert this chunk of data into their data types to be able to do statistical analysis on it with numpy and scipy.</p>

<p>The datatype for all of the columns is integer <strong>except</strong> the 8th index this is DateTime and the 10th index is pure string.</p>

<h3>Question:</h3>

<p>What is the easiest way to this conversation?</p>

<hr>

<h1>EDIT</h1>

<p>Performance is very important than readability, I have to convert <strong>4.5m</strong> rows of data and then process them!</p>

<p>Usually when people ask about reading <code>csv</code> files we ask for a sample of the file.  I've attempted to reconstruct your line from the string list:</p>

<pre><code>In [590]: txt
Out[590]: b'529997, 46623448, 2122110124, 2310, 2054, 2, 66, , 2010/11/03-12:42:08, 26, CLEARING, 781, 30, 3, 0, 0, 1'
</code></pre>

<p>(<code>b</code> for bytestring in Py3, which is how genfromtxt expects its input)</p>

<p><code>genfromtxt</code> expects a filename, open file, or anything that feeds it lines.  So a list of lines works fine:</p>

<p>With <code>dtype=None</code> it deduces column types.</p>

<pre><code>In [591]: data=np.genfromtxt([txt], dtype=None, delimiter=',', autostrip=True)
In [592]: data
Out[592]: 
array((529997, 46623448, 2122110124, 2310, 2054, 2, 66, False, b'2010/11/03-12:42:08', 26, b'CLEARING', 781, 30, 3, 0, 0, 1), 
      dtype=[('f0', '&lt;i4'), ('f1', '&lt;i4'), ('f2', '&lt;i4'), ('f3', '&lt;i4'), ('f4', '&lt;i4'), ('f5', '&lt;i4'), ('f6', '&lt;i4'), ('f7', '?'), ('f8', 'S19'), ('f9', '&lt;i4'), ('f10', 'S8'), ('f11', '&lt;i4'), ('f12', '&lt;i4'), ('f13', '&lt;i4'), ('f14', '&lt;i4'), ('f15', '&lt;i4'), ('f16', '&lt;i4')])
</code></pre>

<p>The result is a bunch of <code>int</code> fields, 2 string fields.  The blank is interpreted as boolean.</p>

<p>If I spell out the columns types I get a slightly different array</p>

<pre><code>In [593]: dt=[int,int,int,int,int,int,int,float,'U20',int, 'U10',int,int,int,int,int,int]
In [594]: data=np.genfromtxt([txt], dtype=dt, delimiter=',', autostrip=True)
In [595]: data
Out[595]: 
array((529997, 46623448, 2122110124, 2310, 2054, 2, 66, nan, '2010/11/03-12:42:08', 26, 'CLEARING', 781, 30, 3, 0, 0, 1), 
      dtype=[('f0', '&lt;i4'), ('f1', '&lt;i4'), ('f2', '&lt;i4'), ('f3', '&lt;i4'), ('f4', '&lt;i4'), ('f5', '&lt;i4'), ('f6', '&lt;i4'), ('f7', '&lt;f8'), ('f8', '&lt;U20'), ('f9', '&lt;i4'), ('f10', '&lt;U10'), ('f11', '&lt;i4'), ('f12', '&lt;i4'), ('f13', '&lt;i4'), ('f14', '&lt;i4'), ('f15', '&lt;i4'), ('f16', '&lt;i4')])
</code></pre>

<p>I specified <code>float</code> for the blank column, which it then interprets as <code>nan</code>.  Handling of blacks can be refined.</p>

<p>I changed the string files to unicode (the default py3 string).</p>

<p>I should be able to specify a datetime conversion, for example to <code>np.datetime64</code>.</p>

<p>With just one line, <code>data</code> is a single element array, 0d, with a compound <code>dtype</code>.</p>

<p>Fields are accessed by name</p>

<pre><code>In [598]: data['f8']
Out[598]: 
array('2010/11/03-12:42:08', 
      dtype='&lt;U20')
In [599]: data['f2']
Out[599]: array(2122110124)
</code></pre>

<p>Speed wise this probably is the same as your custom reader.  <code>genfromtxt</code> reads the file line by line, and parses it.  It collects the parsed lines in a list, and creates an array once at the end (I don't recall if parsed lines are lists or dtype arrays - I suspect lists, but would have to study the code).</p>

<p>To handle the date, I have to use <code>'datetime64[s]'</code>, and some how change the date to read <code>"2010-11-03T12:42:08"</code>, probably in a <code>converter</code>.</p>

<p>===================</p>

<p>I can make a converter based on your <code>datetime</code> parsing:</p>

<pre><code>In [649]: from datetime import datetime
In [650]: dateconvert=lambda x: datetime.strptime(x.decode(),"%Y/%m/%d-%H:%M:%S")
In [651]: data=np.genfromtxt([txt], dtype=dt, delimiter=',',  autostrip=True, converters={8:dateconvert})
In [652]: data
Out[652]: 
array((529997, 46623448, 2122110124, 2310, 2054, 2, 66, nan, datetime.datetime(2010, 11, 3, 12, 42, 8), 26, 'CLEARING', 781, 30, 3, 0, 0, 1), 
      dtype=[('f0', '&lt;i4'), ('f1', '&lt;i4'), ('f2', '&lt;i4'), ('f3', '&lt;i4'), ('f4', '&lt;i4'), ('f5', '&lt;i4'), ('f6', '&lt;i4'), ('f7', '&lt;f8'), ('f8', '&lt;M8[s]'), ('f9', '&lt;i4'), ('f10', '&lt;U10'), ('f11', '&lt;i4'), ('f12', '&lt;i4'), ('f13', '&lt;i4'), ('f14', '&lt;i4'), ('f15', '&lt;i4'), ('f16', '&lt;i4')])
</code></pre>

python arrays numpy converter mixed