Speed up creation of numpy array from list
<p>I have a 33620x160 <code>pandas</code> <code>DataFrame</code> which has one column that contains lists of numbers. Each list entry in the <code>DataFrame</code> contains 30 elements.</p>

<pre><code>df['dlrs_col']

0        [0.048142470608688, 0.047021138711858, 0.04573...
1        [0.048142470608688, 0.047021138711858, 0.04573...
2        [0.048142470608688, 0.047021138711858, 0.04573...
3        [0.048142470608688, 0.047021138711858, 0.04573...
4        [0.048142470608688, 0.047021138711858, 0.04573...
5        [0.048142470608688, 0.047021138711858, 0.04573...
6        [0.048142470608688, 0.047021138711858, 0.04573...
7        [0.048142470608688, 0.047021138711858, 0.04573...
8        [0.048142470608688, 0.047021138711858, 0.04573...
9        [0.048142470608688, 0.047021138711858, 0.04573...
10       [0.048142470608688, 0.047021138711858, 0.04573...
</code></pre>

<p>I'm creating a 33620x30 array whose entries are the unlisted values from that single <code>DataFrame</code> column. I'm currently doing this as:</p>

<pre><code>np.array(df['dlrs_col'].tolist(), dtype = 'float64')
</code></pre>

<p>This works just fine, but it takes a significant amount of time, especially when considering I do a similar calculation for 6 additional columns of lists. Any ideas on how I can speed this up?</p>

<p>You can first convert to <code>numpy array</code> by <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.values.html" rel="nofollow"><code>values</code></a>:</p>

<pre><code>df = pd.DataFrame({'dlrs_col':[
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573]]})

print (df)
                                          dlrs_col
0  [0.048142470608688, 0.047021138711858, 0.04573]
1  [0.048142470608688, 0.047021138711858, 0.04573]
2  [0.048142470608688, 0.047021138711858, 0.04573]
3  [0.048142470608688, 0.047021138711858, 0.04573]
4  [0.048142470608688, 0.047021138711858, 0.04573]
5  [0.048142470608688, 0.047021138711858, 0.04573]
6  [0.048142470608688, 0.047021138711858, 0.04573]
7  [0.048142470608688, 0.047021138711858, 0.04573]

print (np.array(df['dlrs_col'].values.tolist(), dtype = 'float64'))
[[ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]]
</code></pre>

<p><strong>Timings</strong>:</p>

<pre><code>In [56]: %timeit (np.array(df['dlrs_col'].values.tolist(), dtype = 'float64'))
The slowest run took 9.76 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 14.1 Âµs per loop

In [57]: %timeit (np.array(df['dlrs_col'].tolist(), dtype = 'float64'))
The slowest run took 9.33 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 28.4 Âµs per loop
</code></pre>

python arrays performance pandas numpy