Process a list in a Dataframe column
<p>I created a DataFrame <code>neighbours</code> using <code>sim_measure_i</code> which is also a DataFrame.</p>

<pre><code>neighbours= sim_measure_i.apply(lambda s: s.nlargest(k).index.tolist(), axis =1)
</code></pre>

<p><code>neighbours</code> looks like this: </p>

<pre><code>1500                       [0, 1, 2, 3, 4]
1501                       [0, 1, 2, 3, 4]
1502                       [0, 1, 2, 3, 4]
1503     [7230, 12951, 13783, 8000, 18077]
1504                     [1, 3, 6, 27, 47]
</code></pre>

<p>The second column here has lists - I want to iterate over this DataFrame and work on the list such that I can read each element in the list - say 7230 and lookup a score for 7230 in another DataFrameI have which contains (id, score). </p>

<p>I would then like to add a column to this DataFrame such that it looks like </p>

<pre><code>test_case_id               nbr_list             scores             
1500                       [0, 1, 2, 3, 4]        [+1, -1, -1, +1, -1]
1501                       [0, 1, 2, 3, 4]        [+1, +1, +1, -1, -1]
1502                       [0, 1, 2, 3, 4]        [+1, +1, +1, -1, -1]
1503     [7230, 12951, 13783, 8000, 18077]        [+1, +1, +1, -1, -1]
1504                     [1, 3, 6, 27, 47]        [+1, +1, +1, -1, -1]
</code></pre>

<p>Edit: I've written a method <code>get_scores()</code></p>

<pre><code>def get_scores(list_of_neighbours):
    score_matrix = []
    for x, val in enumerate(list_of_neighbours):
        score_matrix.append(df.iloc[val].score)
    return score_matrix
</code></pre>

<p>When I try to use <code>lambda</code> on each of <code>nbr_list</code>, I get this error: </p>

<pre><code>TypeError: ("cannot do positional indexing on &lt;class 'pandas.indexes.numeric.Int64Index'&gt; with these indexers [0] of &lt;type 'str'&gt;", u'occurred at index 1500')
</code></pre>

<p>The code causing this issue: </p>

<pre><code>def nearest_neighbours(similarity_matrix, k):
    neighbours = pd.DataFrame(similarity_matrix.apply(lambda s: s.nlargest(k).index.tolist(), axis =1))
    neighbours = neighbours.rename(columns={0 : 'nbr_list'})

    nbr_scores = neighbours.apply(lambda l: get_scores(l.nbr_list), axis=1)

    print neighbours
</code></pre>

<p>You can try a nested loop:</p>

<pre><code>for i in range(neighbours.shape[0]): #iterate over each row
    for j in range(len(neighbours['neighbours_lists'].iloc[i])): #iterate over each element of the list
        a = neighbours['neighbours_lists'].iloc[i][j] #access the element of the list index j in cell location of row i 
</code></pre>

<p>where <code>i</code> is the outer loop variable which iterates over each row and <code>j</code> is the inner loop variable which iterates over the length of the list inside each cell.</p>

python pandas numpy dataframe