Perform operation on each cell of one column with each cell of other columns in pandas
<p>I have a data frame(df) consisting of more than 1000 columns. Each cell consists of a list.
e.g.</p>

<pre><code>           0            1             2  .... n
0    [1,2,3]      [3,7,9]       [1,2,1]  ....[x,y,z]
1    [2,5,6]      [2,3,1]       [3,3,3]  ....[x1,y1,z1]
2       None      [2,0,1]       [2,2,2]  ....[x2,y2,z2]
3       None      [9,5,9]          None  ....None
</code></pre>

<p>This list is actually the dimensions. I need to find the euclidean distance of each cell in column 0 with every other cell in column 1 and store the minimum.
Similarly from column 0 to column 2 and then to column 3 so on..</p>

<h2>Example</h2>

<pre><code>distance of df[0][0] from df[1][0], df[1][1], df[1][2]
then of df[0][1] from df[1][0], df[1][1], df[1][2] and so on...
</code></pre>

<p>Currently i am doing it with help of for loops but it is taking a lot of time for large data. 
Following is the implementation::</p>

<pre><code>for n in range(len(df.columns)):
    for m in range(n+1,len(df.columns)):
        for q in range(df.shape[0]):
            min1=9999
            r=0
            while(r&lt;df.shape[0]):
                if(df[n][q] is not None or df[m][r] is not None):
                    dist=distance.euclidean(df[n][q],df[m][r])
                    if(d&lt;min1):
                            min1=d
                    if(min1==0):  *#because distance can never be less than zero*
                        break
                    r=r+1
</code></pre>

<p>Is there any other way to do this?</p>

<p>You can use pandas apply. The example below will do the distance between 0 and 1 and create a new column.</p>

<pre><code>import pandas as pd
import numpy as np

df = pd.DataFrame({'0':[[1,2,3],[2,5,6]],'1':[[3,7,9],[2,3,1]]})

def eucledian(row):
    x = np.sqrt((row[0][0]-row[1][0])**2)
    y = np.sqrt((row[0][1]-row[1][1])**2)
    z = np.sqrt((row[0][2]-row[1][2])**2)
    return [x,y,z]

df['dist0-1'] = df.apply(eucledian,axis=1)
</code></pre>

<p>With this said I highly recommend to unroll your variables to separate columns, e.g. 0.x, 0.y, 0.z, etc and then use numpy to operate directly on columns. This will be much faster if you have a large amount of data.</p>

python pandas dataframe