Fail to transfer the list in data frame to numpy array with python-pandas
<p>For a data frame <code>df</code>:</p>

<pre><code>name       list1                    list2
a          [1, 3, 10, 12, 20..]     [2, 6, 23, 29...]
b          [2, 10, 14, 3]           [4, 7, 8, 13...]
c          []                       [98, 101, 200]
...
</code></pre>

<p>I want to transfer the <code>list1</code> and <code>list2</code> to <code>np.array</code> and then <code>hstack</code> them. Here is what I did:</p>

<pre><code>df.pv = df.apply(lambda row: np.hstack((np.asarray(row.list1), np.asarray(row.list2))), axis=1)
</code></pre>

<p>And I got such an error:</p>

<pre><code>ValueError: Shape of passed values is (138493, 175), indices imply (138493, 4)
</code></pre>

<p>Where <code>138493==len(df)</code></p>

<p>Please note that some value in <code>list1</code> and <code>list2</code> is empty list, <code>[]</code>. And the length of list are different among rows. Do you know what is the reason how can I fix the problem? Thanks in advance!</p>

<p>EDIT:</p>

<p>When I just try to convert one list to array:</p>

<pre><code>df.apply(lambda row: np.asarray(row.list1), axis=1)
</code></pre>

<p>An error also occurs:</p>

<pre><code>ValueError: Empty data passed with indices specified.
</code></pre>

<p>Your apply function is <em>almost</em> correct. All you have to do - convert the output of the <code>np.hstack()</code> function back to a python list.</p>

<pre><code>df.apply(lambda row: list(np.hstack((np.asarray(row.list1), np.asarray(row.list2)))), axis=1)
</code></pre>

<p>The code is shown below (including the df creation):</p>

<pre><code>df = pd.DataFrame([('a',[1, 3, 10, 12, 20],[2, 6, 23, 29]),
                   ('b',[2, 10, 1.4, 3],[4, 7, 8, 13]),
                   ('c',[],[98, 101, 200])],
                   columns = ['name','list1','list2'])

df['list3'] = df.apply(lambda row: list(np.hstack((np.asarray(row.list1), np.asarray(row.list2)))), axis=1)

print(df)
</code></pre>

<p>Output:</p>

<pre><code>0              [1, 3, 10, 12, 20, 2, 6, 23, 29]
1    [2.0, 10.0, 1.4, 3.0, 4.0, 7.0, 8.0, 13.0]
2                          [98.0, 101.0, 200.0]
Name: list3, dtype: object
</code></pre>

<p>If you want a numpy array, the only way I could get it to work is:</p>

<pre><code>df['list3'] = df['list3'].apply(lambda x: np.array(x))

print(type(df['list3'].ix[0]))
Out[] : numpy.ndarray
</code></pre>

python pandas numpy