Python deduplicate records - dedupe
<p>I want to use <a href="https://github.com/datamade/dedupe" rel="nofollow">https://github.com/datamade/dedupe</a> to deduplicate some records in python. Looking at their examples </p>

<pre><code>data_d = {}
for row in data:
    clean_row = [(k, preProcess(v)) for (k, v) in row.items()]
    row_id = int(row['id'])
    data_d[row_id] = dict(clean_row)
</code></pre>

<p>the dictionary consumes quite a lot of memory compared to e.g. a dictionary created by pandas out of a pd.Datafrmae, or even a normal pd.Dataframe.</p>

<p>If this format is required, how can I convert a pd.Dataframe efficiently to such a dictionary?</p>

<h1>edit</h1>

<p>Example what pandas generates</p>

<pre><code>{'column1': {0: 1389225600000000000,
  1: 1388707200000000000,
  2: 1388707200000000000,
  3: 1389657600000000000,....
</code></pre>

<p>Example what dedupe expects</p>

<pre><code>{'1': {column1: 1389225600000000000, column2: "ddd"},
 '2': {column1: 1111, column2: "ddd} ...}
</code></pre>

<p>It appears that <code>df.to_dict(orient='index')</code> will produce the representation you are looking for:</p>

<p>import pandas</p>

<pre><code>data = [[1, 2, 3], [4, 5, 6]]
columns = ['a', 'b', 'c']

df = pandas.DataFrame(data, columns=columns)

df.to_dict(orient='index')
</code></pre>

<p>results in </p>

<pre><code>{0: {'a': 1, 'b': 2, 'c': 3}, 1: {'a': 4, 'b': 5, 'c': 6}}
</code></pre>

python pandas dictionary duplicates dedupeplugin