Python deduplicate records - dedupe
<p>I want to use <a href="https://github.com/datamade/dedupe" rel="nofollow">https://github.com/datamade/dedupe</a> to deduplicate some records in python. Looking at their examples </p>

<pre><code>data_d = {}
for row in data:
    clean_row = [(k, preProcess(v)) for (k, v) in row.items()]
    row_id = int(row['id'])
    data_d[row_id] = dict(clean_row)
</code></pre>

<p>the dictionary consumes quite a lot of memory compared to e.g. a dictionary created by pandas out of a pd.Datafrmae, or even a normal pd.Dataframe.</p>

<p>If this format is required, how can I convert a pd.Dataframe efficiently to such a dictionary?</p>

<h1>edit</h1>

<p>Example what pandas generates</p>

<pre><code>{'column1': {0: 1389225600000000000,
  1: 1388707200000000000,
  2: 1388707200000000000,
  3: 1389657600000000000,....
</code></pre>

<p>Example what dedupe expects</p>

<pre><code>{'1': {column1: 1389225600000000000, column2: "ddd"},
 '2': {column1: 1111, column2: "ddd} ...}
</code></pre>

<p>A python dictionary is not required, you just need an object that allows indexing by column name. i.e. <code>row['col_name']</code></p>

<p>So, assuming <code>data</code> is a pandas dataframe should just be able to do something like:</p>

<pre><code>data_d = {}
for row_id, row in data.iterrows():
    data_d[row_id] = row
</code></pre>

<p>That said, the memory overhead of python dicts is not going to be where you have memory bottlenecks in dedupe.</p>

python pandas dictionary duplicates dedupeplugin