Python Import Text Array with Numpy
<p>I have a text file that looks like this:</p>

<pre><code>...
5   [0, 1]  [512, 479]  991
10  [1, 0]  [706, 280]  986
15  [1, 0]  [807, 175]  982
20  [1, 0]  [895, 92]   987
...
</code></pre>

<p>Each column is tab separated, but there are arrays in some of the columns. Can I import these with <code>np.genfromtxt</code> in some way?</p>

<p>The resulting unpacked lists should be, for example:</p>

<pre><code>data1 = [..., 5, 10, 15, 20, ...]
data2 = [..., [512, 479], [706, 280], ... ] (i.e. a 2D list)
etc.
</code></pre>

<p>I tried </p>

<p><code>data1, data2, data3, data4 = np.genfromtxt('data.txt', dtype=None, delimiter='\t', unpack=True)</code></p>

<p>but <code>data2</code> and <code>data3</code> are lists containing 'nan'.</p>

<p>As an alternative to <code>genfromtxt</code> you might try <code>fromregex</code>.  You basically set up an analogy between regular expression groups and the fields of a numpy structured dtype.</p>

<p>In this example, I parse out all numbers without worrying about whether they are single numbers or arrays.  Then I switch to a dtype that specifies which columns have arrays.</p>

<pre><code>import numpy as np

# regular expression that will extract 6 numbers from each line
re = 6 * r"(\d+)\D*"

# dtype for a single number
dt_num = np.int

# structured dtype of 6 numbers
dt = 6 * [('', dt_num)]

# parse the file
a = np.fromregex("data.txt", re, dt)

# change to a more descriptive structured dtype
a.dtype = [
  ('data1', dt_num),
  ('data2', dt_num, (2,)),
  ('data3', dt_num, (2,)),
  ('data4', dt_num)
]

print(a['data1'])
print(a['data2'])
print(a['data3'])
print(a['data4'])
</code></pre>

<p>The nice thing about switching the dtype of a numpy array is that it does not have to process or make new copies of the data, it just reinterprets what you get when you access the data.</p>

<p>One of the down sides of this solution is that building complex regular expressions and building structured dtypes can get ugly.  And in this case, you have to keep the two in sync with each other.</p>

python arrays numpy genfromtxt