calculate the totall number of uniqe words of the first column of a list
<p>I have a file that consists of many Persian sentences. each line contains a sentence, then a "tab", then a word, again a "tab" and then an English word. I have to know just the number of unique words of the sentences (the words after tabs should not be in calculation). For that I changed the file to a list, so I have a list that contains a lot of lines and each line contains three indices; the sentence, a Persian word, an English word. Now I can achieve the sentences. The problem is that, the code I wrote returns the number of unique words of each line separately. For example if the file has 100 lines it returns 100 numbers, each in a new line. But I want the summation of all the numbers and have just one number which shows the total number of unique words. How can I change the code?</p>

<pre><code>from hazm import*

def WordsProbs (file):
    with open (file, encoding = "utf-8") as f1:
        normalizer = Normalizer()
        for line in f1:
        tmp = line.strip().split("\t")
        tmp[0] = normalizer.normalize(tmp[0])
        corpus.append(tmp)
   for row in corpus:
       UniqueWords = len(set(row[0].split()))
       print (UniqueWords)
</code></pre>

<p>The sample data:</p>

<p>Ø¨Ø§Ø¯  Ø¨Ø§Ø±Ø´ Ø¨Ø±Ù ÙØ²Ø´ Ø¨Ø§Ø¯Ø Ú©ÙÙØ§Ú© ÛØ®Ø¨ÙØ¯Ø§Ù Ø³Ø·Ø­  wind</p>

<p>You an use <code>collections.Counter</code> in order to count the number of words after splitting:</p>

<pre><code>from collections import Counter
from itertools import chain

def WordsProbs (file_name):
    with open (file_name, encoding = "utf-8") as f1:
        all_words = chain.from_iterable(word_tokenizer(line.split(None, 1)[0]) for line in f1)
        return Counter(all_words)
</code></pre>

<p>The <code>chain.from_iterable</code> will chain the splitted words together as an integrated iterator, so that the <code>Counter</code> can create a counter object from all of the words.</p>

<p>If you just want the number of all unique words <code>len(WordsProbs(file_name))</code> will give you that, but another way is using a set on the iterator that we created with <code>cahin.from_iterable</code>:</p>

<pre><code>def WordsProbs (file_name):
    with open (file_name, encoding = "utf-8") as f1:
        all_words = chain.from_iterable(word_tokenizer(line.split(None, 1)[0]) for line in f1)
        return len(set(all_words))
</code></pre>

<p>And if you want the number of unique words for each line:</p>

<pre><code>def WordsProbs (file_name):
    with open (file_name, encoding = "utf-8") as f1:
        each_line_numbers = [len(set(word_tokenizer(line.split(None, 1)[0]))) for line in f1)
        return each_line_numbers
</code></pre>

python nlp