How to create 2 gram shingles?
<p>I have this code which i got from some tutorial-:</p>

<pre><code>list1 = [['hello','there','you','too'],['hello','there','you','too','there'],['there','you','hello']]

def get_shingle(size,f):
    #shingles = set()
    for i in range (0,len(f)-2+1):
        yield f[i:i+2]

#shingles1 = set(get_shingle(list1[0],2))
#shingles2 = set(get_shingle(list1[1],2))
shingles1 = set(get_shingle(2,list1[0]))
shingles2 = set(get_shingle(2,list1[1]))

print shingles1
print shingles2
print "done"
</code></pre>

<p>When i try to run this code i am getting an error -: </p>

<pre><code>Traceback (most recent call last):
  File "E:\Research\Shingle Method\create_shingle.py", line 10, in &lt;module&gt;
    shingles1 = set(get_shingle(2,list1[0]))
TypeError: unhashable type: 'list'
</code></pre>

<p>If list1 is set then the error does not come. But i cannot convert list1 into set as <strong>it removes duplicate words</strong> and also i need it to be a list for my major code which processes a huge text file in the form of list. 
Why am i getting this 'unhashable list'? Can't we pass list as argument?</p>

<p>Yield command generates a generator and set(iterator) expects an iterator which is immutable </p>

<p>So something like this will work</p>

<pre><code>shingles1 = set(get_shingle(2,list1[0]))
set(tuple(x) for x in shingles1)
</code></pre>

python