digest of a pandas Index
<p>I would like to <strong>efficiently</strong> compute a digest of a Pandas DataFrame that uniquely and reproducibly identifies its content (for versioning purposes).  Assume for now that I don't worry about endianness, dtypes, types of the index nor columns.  Also assume that both index and columns are already sorted monotonic_increasing.</p>

<p>Things go reasonably well with the values (again, for simplification, assume <code>np.float64</code>).  But I am having trouble with the index (and columns) and don't get a consistent digest.  Of course I can do things such as converting the index into String and then utf-8 bytes, but that is slow.</p>

<p>Here is a simplified example:</p>

<pre><code>import hashlib
def pd_val_sha1(df):
    x = df.values
    if not x.flags.c_contiguous:
        x = x.copy(order='C')
    return hashlib.sha1(x).hexdigest()
</code></pre>

<p>Test:</p>

<pre><code>import pandas as pd
import io

str = """s,e,id,x,y,z
2012-01-01,2013-01-01,b,NaN,2,3
2015-10-27,2015-11-03,a,0.04,12.7,NaN
2015-11-15,2016-01-01,a,7.3,-1.2,8
"""
df = pd.read_csv(io.StringIO(str), parse_dates=[0,1], index_col=[0,1,2]).sort_index()
df
</code></pre>

<p>Out:</p>

<pre><code>                            x     y    z
s          e          id                 
2012-01-01 2013-01-01 b    NaN   2.0  3.0
2015-10-27 2015-11-03 a   0.04  12.7  NaN
2015-11-15 2016-01-01 a   7.30  -1.2  8.0
</code></pre>

<p>SHA-1 of the values:</p>

<pre><code>pd_val_sha1(df)
&gt;&gt;&gt; 'a7f0335988a967606bd030864e0e30ce03f32ec9'

pd_val_sha1(df.head())
&gt;&gt;&gt; 'a7f0335988a967606bd030864e0e30ce03f32ec9'

pd_val_sha1(pd.concat([df.ix[0:2], df.ix[2:3]]))
&gt;&gt;&gt; 'a7f0335988a967606bd030864e0e30ce03f32ec9'
</code></pre>

<p>So far, so good.  But when it comes to the index:</p>

<pre><code>pd_val_sha1(df.index)
&gt;&gt;&gt; inconsistent value (re-run the example from read_csv and we'll get
... a different result).
</code></pre>

<p>I tried various other things, e.g. using <code>index.data</code> or <code>index.to_native_types()</code> or <code>np.array(index.tolist())</code> instead of <code>index.values</code>, but I still get inconsistent results, as I suppose the underlying data can vary.</p>

<p>One thing that seems to be working so far is <code>hashlib.sha1(np.array(df.index.format())).hexdigest()</code>.  But it is slow, e.g. 2min 34s for a (5000000,12) dataframe, whereas the content itself is fingerprinted in 900ms. </p>

<p>Any suggestions?</p>

<p>Sometimes the solution is right under our nose...</p>

<pre><code>from sklearn.externals import joblib

%%time
joblib.hash(df, hash_name='sha1')
&gt;&gt;&gt; consistent value that depends on values and axes
Wall time: 1.66 s  (for the (5000000,12) DataFrame mentioned above) 
</code></pre>

python pandas digest