Auto convert strings and float columns using genfromtxt from numpy/python
<p>I have several different data files that I need to import using genfromtxt. Each data file has different content. For example, file 1 may have all floats, file 2 may have all strings, and file 3 may have a combination of floats and strings etc. Also the number of columns vary from file to file, and since there are hundreds of files, I don't know which columns are floats and strings in each file. However, all the entries in each column are the same data type. </p>

<p>Is there a way to set up a converter for genfromtxt that will detect the type of data in each column and convert it to the right data type?</p>

<p>Thanks!</p>

<p>If you're able to use the Pandas library, <code>pandas.read_csv</code> is <em>much</em> more generally useful than <code>np.genfromtxt</code>, and will automatically handle the kind of type inference mentioned in your question.  The result will be a dataframe, but you can get out a numpy array in one of several ways. e.g.</p>

<pre><code>import pandas as pd
data = pd.read_csv(filename)

# get a numpy array; this will be an object array if data has mixed/incompatible types
arr = data.values

# get a record array; this is how numpy handles mixed types in a single array
arr = data.to_records()
</code></pre>

<p><code>pd.read_csv</code> has dozens of options for various forms of text inputs; see more in the <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html" rel="nofollow">pandas.read_csv documentation</a>.</p>

python numpy data-type-conversion genfromtxt