Python deduplicate records - dedupe
<p>I want to use <a href="https://github.com/datamade/dedupe" rel="nofollow">https://github.com/datamade/dedupe</a> to deduplicate some records in python. Looking at their examples </p>

<pre><code>data_d = {}
for row in data:
    clean_row = [(k, preProcess(v)) for (k, v) in row.items()]
    row_id = int(row['id'])
    data_d[row_id] = dict(clean_row)
</code></pre>

<p>the dictionary consumes quite a lot of memory compared to e.g. a dictionary created by pandas out of a pd.Datafrmae, or even a normal pd.Dataframe.</p>

<p>If this format is required, how can I convert a pd.Dataframe efficiently to such a dictionary?</p>

<h1>edit</h1>

<p>Example what pandas generates</p>

<pre><code>{'column1': {0: 1389225600000000000,
  1: 1388707200000000000,
  2: 1388707200000000000,
  3: 1389657600000000000,....
</code></pre>

<p>Example what dedupe expects</p>

<pre><code>{'1': {column1: 1389225600000000000, column2: "ddd"},
 '2': {column1: 1111, column2: "ddd} ...}
</code></pre>

<p>You can try something like this:</p>

<pre><code>df = pd.DataFrame({'A': [1,2,3,4,5], 'B': [6,7,8,9,10]})
A   B
0  1   6
1  2   7
2  3   8
3  4   9
4  5  10

print(df.T.to_dict())
{0: {'A': 1, 'B': 6}, 1: {'A': 2, 'B': 7}, 2: {'A': 3, 'B': 8}, 3: {'A': 4, 'B': 9}, 4: {'A': 5, 'B': 10}}
</code></pre>

<p><em>This is the same output as in @chthonicdaemon answer so his answer is probably better. I am using <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.T.html" rel="nofollow">pandas.DataFrame.T</a> to transpose  index and columns.</em></p>

python pandas dictionary duplicates dedupeplugin