Grouping by almost similar strings
<p>I have a dataset with city names and counts of crimes. The data is dirty such that a name of a city for example 'new york', is written as 'newyork', 'new york us', 'new york city', 'manhattan new york' etc. How can I group all these cities together and sum their crimes?</p>

<p>I tried the 'difflib' package in python that matches strings and gives you a score. It doesn't work well. I also tried the geocode package in python. It has limits on number of times you can access the api, and doesnt work well either. Any suggestions? </p>

<p>Another approach is to go through each entry and strip the white space and see if they contain a base city name. For example 'newyork', 'new york us', 'new york city', 'manhattan new york' when stripped of white space would be 'newyork', 'newyorkus', 'newyorkcity', 'manhattannewyork', which all contain the word 'newyork'.</p>

<p>There are two approaches with this method, you can go through and replace the all the 'new york' strings with ones that have no white space and are just 'newyork' or you can just check them on the fly.</p>

<p>I wrote down an example below, but since I don't know how your data is formatted, I'm not sure how helpful it is.</p>

<pre><code>crime_count = 0
for (key, val) in dataset:
    if 'newyork' in key.replace(" ", ""):
        crime_count = crime_count + val
</code></pre>

python pandas