Tokenizing a concatenated string
<p>I got a set of strings that contain concatenated words like the followings:</p>

<pre><code>longstring (two English words)
googlecloud (a name and an English word)
</code></pre>

<p>When I type these terms into Google, it recognizes the words with "did you mean?" ("long string", "google cloud"). I need similar functionality in my application.</p>

<p>I looked into the options provided by Python and ElasticSearch. All the tokenizing examples I found are based on whitespace, upper case, special characters etc. </p>

<p>What are my options provided the strings are in English (but they may contain names)? It doesn't have to be on a specific technology.</p>

<p>Can I get this done with Google BigQuery?</p>

<p>If you do choose to solve this with BigQuery, then the following is a candidate solution:</p>

<ol>
<li><p>Load list of all possible English words into a table called <code>words</code>. For example, <a href="https://github.com/dwyl/english-words" rel="nofollow">https://github.com/dwyl/english-words</a> has list of ~350,000 words. There are other datasets (i.e. WordNet) freely available in Internet too.</p></li>
<li><p>Using Standard SQL, run the following query over list of candidates:</p></li>
</ol>

<p><code>SELECT first, second FROM (
    SELECT word AS first, SUBSTR(candidate, LENGTH(word) + 1) AS second
    FROM dataset.words
  CROSS JOIN (
    SELECT candidate
    FROM UNNEST(["longstring", "googlecloud", "helloxiuhiewuh"]) candidate)
  WHERE STARTS_WITH(candidate, word))
WHERE second IN (SELECT word FROM dataset.words)</code></p>

<p>For this example it produces:</p>

<pre><code>Row first   second   
1   long    string   
2   google  cloud
</code></pre>

<p>Even very big list of English words would be only couple of MBs, so the cost of this query is minimal. First 1 TB scan is free - which is good enough for about 500,000 scans on 2 MB table. After that each additional scan is 0.001 cents.</p>

python elasticsearch machine-learning google-bigquery