Sentence clustering
<p>I have a huge number of names from different sources.</p>

<ol>
<li>I need to extract all the groups (part of the names), which repeat from one to another.
In the example below program should locate: Post, Office, Post Office.</li>
<li>I need to get popularity count.</li>
</ol>

<p>So I want to extract a sorted by popularity list of phrases.</p>

<p>Here is an example of names:</p>

<pre><code>Post Office - High Littleton
Post Office Pilton Outreach Services
Town Street Post Office
post office St Thomas
</code></pre>

<p>Basically need to find out some algorithm or better library, to get such results:</p>

<pre><code>Post Office: 16999
Post: 17934
Office: 16999
Tesco: 7300
...
</code></pre>

<p>Here is the full <a href="https://gist.github.com/maZahaca/a54046a4cc7ab27f9d06751b89aa7446" rel="nofollow">example of names</a>.</p>

<p>I wrote a code which is fine for single words, but not for sentences:</p>

<pre><code>from textblob import TextBlob
import operator

title_file = open("names.txt", 'r')
blob = TextBlob(title_file.read())
list = sorted(blob.word_counts.items(), key=operator.itemgetter(1))
print list
</code></pre>

<p>You are not looking for clustering (and that is probably why "all of them suck" for @andrewmatte).</p>

<p>What you are looking for is <em>word counting</em> (or more precisely, n-gram-counting). Which is actually a much easier problem. Thst is why you won't be finding any library for that...</p>

<p>Well, actually you jave some libraries. In python, for example, the <code>collections</code> module has the class <code>Counter</code> that has much of the reusable code.</p>

<p>An untested, very basic code:</p>

<pre><code>from collections import Counter
counter = Counter()
for s in sentences:
  words = s.split(" ")
  for i in range(len(words)):
    counter.add(words[i])
    if i &gt; 0: counter.add((words[i-1], words[i]))
</code></pre>

<p>You csn get the most frequent from <code>counter</code>. If you want words and word pairs separate, feel free to use two counters. If you need longer phrases, add an inner loop. You may also want to clean sentences (e.g. lowercase) and use a regexp for splitting.</p>

python machine-learning cluster-analysis