compute the difference of all possible rows
<p>Based on a selection <code>ds</code> of a dataframe <code>d</code> with:</p>

<p><code>{ 'x': d.x, 'y': d.y, 'a':d.a, 'b':d.b, 'c':d.c 'row:d.n'})</code> </p>

<p>Having <code>n</code> rows, <code>x</code> ranges from <code>0</code> to <code>n-1</code>. The column <code>n</code> is needed since it's a selection and indices need to be kept for a later query.</p>

<p>How do you efficiently compute the difference between each row (e.g.<code>a_0, a_1, etc</code>) of each column (<code>a, b, c</code>) without losing the rows information (e.g. new column with the indices of the rows that were used) ?</p>

<p><strong>MWE</strong> </p>

<p>Sample selection <code>ds</code>:</p>

<pre><code>             x           y      a     b      c     n

    554.607085  400.971878   9789  4151   6837   146
    512.231450  405.469524   8796  3811   6596   225
    570.427284  694.369140   1608  2019   2097   291
</code></pre>

<p>Desired output:</p>

<p><code>dist</code> euclidean distance <code>math.hypot(x2 - x1, y2 - y1)</code></p>

<p><code>da, db, dc</code> for <code>da: np.abs(a1-a2)</code> </p>

<p><code>ns</code> a string with both <code>n</code>s of the employed rows</p>

<p>the result would look like:</p>

<pre><code>             dist          da        db       dc         ns
42.61365102824963         993       340      241    146-225
293.82347069813255       8181      2132     4740    146-291
                ..         ..        ..       ..    225-291
</code></pre>

<p>This approach makes good use of Pandas and the underlying numpy capabilities, but the matrix manipulations are a little hard to keep track of:</p>

<pre><code>import pandas as pd, numpy as np

ds = pd.DataFrame(
    [
        [554.607085, 400.971878,  9789, 4151,  6837,  146], 
        [512.231450, 405.469524,  8796, 3811,  6596,  225],
        [570.427284, 694.369140,  1608, 2019,  2097,  291]
    ],
    columns = ['x', 'y', 'a', 'b', 'c', 'n']
)

def concat_str(*arrays):
    result = arrays[0]
    for arr in arrays[1:]:
        result = np.core.defchararray.add(result, arr)
    return result

# Make a panel with one item for each column, with a square data frame for 
# each item, showing the differences between all row pairs.
# This creates perpendicular matrices of values based on the underlying numpy arrays;
# then numpy broadcasts them along the missing axis when calculating the differences
p = pd.Panel(
    (ds.values[np.newaxis,:,:] - ds.values[:,np.newaxis,:]).transpose(), 
    items=['d'+c for c in ds.columns], major_axis=ds.index, minor_axis=ds.index
)
# calculate euclidian distance
p['dist'] = np.hypot(p['dx'], p['dy'])
# create strings showing row relationships
p['ns'] = concat_str(ds['n'].values.astype(str)[:,np.newaxis], '-', ds['n'].values.astype(str)[np.newaxis,:])
# remove unneeded items
del p['dx'], p['dy'], p['dn']
# convert to frame
diffs = p.to_frame().reindex_axis(['dist', 'da', 'db', 'dc', 'ns'], axis=1)
diffs
</code></pre>

<p>This gives:</p>

<pre><code>                   dist    da    db    dc       ns
major minor                                       
0     0        0.000000     0     0     0  146-146
      1       42.613651   993   340   241  146-225
      2      293.823471  8181  2132  4740  146-291
1     0       42.613651  -993  -340  -241  225-146
      1        0.000000     0     0     0  225-225
      2      294.702805  7188  1792  4499  225-291
2     0      293.823471 -8181 -2132 -4740  291-146
      1      294.702805 -7188 -1792 -4499  291-225
      2        0.000000     0     0     0  291-291
</code></pre>

python pandas