How to drop duplicated rows using pandas in a big data file?
<p>I have a csv file that too big to load to memory.I need to drop duplicated rows of the file.So I follow this way:</p>

<pre><code>chunker = pd.read_table(AUTHORS_PATH, names=['Author ID', 'Author name'],      encoding='utf-8', chunksize=10000000)

for chunk in chunker:
    chunk.drop_duplicates(['Author ID'])
</code></pre>

<p>But if duplicated rows distribute in different chunk seems like above script can't get the expected results.</p>

<p>Is there any better wayï¼</p>

<p>You could try something like this.</p>

<p>First, create your chunker.</p>

<pre><code>chunker = pd.read_table(AUTHORS_PATH, names=['Author ID', 'Author name'], encoding='utf-8', chunksize=10000000)
</code></pre>

<p>Now create a set of ids:</p>

<pre><code>ids = set()
</code></pre>

<p>Now iterate over the chunks:</p>

<pre><code>for chunk in chunker:
    chunk.drop_duplicates(['Author ID'])
</code></pre>

<p>However, now, within the body of the loop, drop also ids already in the set of known ids:</p>

<pre><code>    chunk = chunk[~chunk['Author ID'].isin(ids)]
</code></pre>

<p>Finally, still within the body of the loop, add the new ids</p>

<pre><code>    ids.update(chunk['Author ID'].values)
</code></pre>

<hr>

<p>If <code>ids</code> is too large to fit into main memory, you might need to use some disk-based database.</p>

python database pandas bigdata