More efficient way of computing distance matrix in Python
<p>Hi Everyone I am trying to write code (using python 2) that returns a matrix that contains the distance between all pairs of rows. Below is an implementation that I have written. It works as expected but can get very slow as the number of rows gets large. Hence I was wondering if anyone has any suggestions as to how the code can be made more efficient for large number of rows.</p>

<p>Thanks in advance</p>

<pre><code>def gendist(x,alpha=2):
    (n,p) = x.shape
    len = 0
    for ii in range(1,n):
        len = len + ii
    d = np.empty((len,p))
    ind = 0
    for ii in range(0,n):
        for jj in range(1,n):
            if ii &lt; jj:
                d[ind,] = (x[ii,]-x[jj,])**alpha
                ind = ind + 1
    return d
</code></pre>

<p>I see you use <code>X.shape</code>, for me, it is find to assume that you are using <code>NumPy</code></p>

<p>Code:</p>

<pre><code>#!/usr/bin/env python3
import numpy as np
import scipy.spatial.distance as dist

a = np.random.randint(0, 10, (5, 3))
b = dist.pdist(a)
print('Matrix:')
print(a)
print('Pdist')
for d in b:
    print(d)
</code></pre>

<p>Output:</p>

<pre><code>Matrix:
[[4 7 6]
 [8 2 8]
 [8 3 5]
 [2 4 7]
 [0 7 5]]
Pdist
6.7082039325
5.74456264654
3.74165738677
4.12310562562
3.16227766017
6.40312423743
9.89949493661
6.40312423743
8.94427191
4.12310562562
</code></pre>

<p>where the order of combinations is (0,1), (0,2), (0,3), (0,4), (1,2), (1,3), (1,4), (2,3), (2,4), ...</p>

<p>The default metric is the Euclidean distance.
See <a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html#scipy.spatial.distance.pdist" rel="nofollow"><code>pdist</code></a> to apply other metrics.</p>

python performance distance-matrix