Pandas fuzzy detect duplicates
<p>How can use fuzzy matching in pandas to detect duplicate rows (efficiently) </p>

<p><a href="http://i.stack.imgur.com/yZxK6.jpg" rel="nofollow"><img src="http://i.stack.imgur.com/yZxK6.jpg" alt="enter image description here"></a></p>

<p>How to find duplicates of one column vs. all the other ones without a gigantic for loop of converting row_i toString() and then comparing it to all the other ones?</p>

<p>Not pandas specific, but within the python ecosystem the <a href="https://github.com/datamade/dedupe" rel="nofollow">dedupe python library</a> would seem to do what you want. In particular, it allows you to compare each column of a row separately and then combine the information into a single probability score of a match.</p>

python pandas fuzzy-search locality-sensitive-hash