Tokenizing a concatenated string
<p>I got a set of strings that contain concatenated words like the followings:</p>

<pre><code>longstring (two English words)
googlecloud (a name and an English word)
</code></pre>

<p>When I type these terms into Google, it recognizes the words with "did you mean?" ("long string", "google cloud"). I need similar functionality in my application.</p>

<p>I looked into the options provided by Python and ElasticSearch. All the tokenizing examples I found are based on whitespace, upper case, special characters etc. </p>

<p>What are my options provided the strings are in English (but they may contain names)? It doesn't have to be on a specific technology.</p>

<p>Can I get this done with Google BigQuery?</p>

<p>Can you also roll your own implementation? I am thinking of an algorithm like this:</p>

<ol>
<li>Get a dictionary with all words you want to distinguish</li>
<li>Build a data structure that allows quick lookup (I am thinking of a <a href="https://en.wikipedia.org/wiki/Trie" rel="nofollow"><code>trie</code></a>)</li>
<li>Try to find the first word (starting with one character and increasing it until a word is found); if found, use the remaining string and do the same until nothing is left. If it doesn't find anything, backtrack and extend the previous word. </li>
</ol>

<p>Should be ok-ish if the string can be split, but will try all possibilities if its gibberish. Of course, it depends on how big your dictionary is going to be. But this was just a quick thought, maybe it helps.</p>

python elasticsearch machine-learning google-bigquery