Best way to create a similarity matrix for given set of items with tags
<p>We have data in format:</p>

<pre><code>{
  '1': ['Lathi Charge', 'NIT', 'Nirmal Singh']
  '2': ['Kangana Ranaut', 'Hrithik Roshan']
  '3': ['Hrithik Roshan', 'mohenjo daro', 'release date'],
  '4': ['NIT', 'Placements']
  ...
}
</code></pre>

<p>Keys of dictionary are items and values are tags associated with them. Numbers of entries are around 10 million, We want to calculate similarity between each items?</p>

<p>One more question associated with same is if any new entry comes on run time what would be best way to calculate its similarity with existing entries.</p>

<p>There's only one similarity score that I can think of which would reliably handle the data you're looking at (it looks like your data is similar to that which would be gathered in a corpus study where documents are represented as bags of words, each potentially containing several words not present in the other documents in the corpus): that's cosine similarity. <a href="http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/" rel="nofollow">Here</a> is a pretty good explanation with some python code to go along.</p>

python nlp scikit-learn