calculate the totall number of uniqe words of the first column of a list
<p>I have a file that consists of many Persian sentences. each line contains a sentence, then a "tab", then a word, again a "tab" and then an English word. I have to know just the number of unique words of the sentences (the words after tabs should not be in calculation). For that I changed the file to a list, so I have a list that contains a lot of lines and each line contains three indices; the sentence, a Persian word, an English word. Now I can achieve the sentences. The problem is that, the code I wrote returns the number of unique words of each line separately. For example if the file has 100 lines it returns 100 numbers, each in a new line. But I want the summation of all the numbers and have just one number which shows the total number of unique words. How can I change the code?</p>

<pre><code>from hazm import*

def WordsProbs (file):
    with open (file, encoding = "utf-8") as f1:
        normalizer = Normalizer()
        for line in f1:
        tmp = line.strip().split("\t")
        tmp[0] = normalizer.normalize(tmp[0])
        corpus.append(tmp)
   for row in corpus:
       UniqueWords = len(set(row[0].split()))
       print (UniqueWords)
</code></pre>

<p>The sample data:</p>

<p>Ø¨Ø§Ø¯  Ø¨Ø§Ø±Ø´ Ø¨Ø±Ù ÙØ²Ø´ Ø¨Ø§Ø¯Ø Ú©ÙÙØ§Ú© ÛØ®Ø¨ÙØ¯Ø§Ù Ø³Ø·Ø­  wind</p>

<p>There is a simple solution. As you said you have list of lines. So the following code should get you what you want</p>

<pre><code>sample_data = """This is One sentence   word1   word2
This is Second sentence word1   word2"""
lines = sample_data.split("\n")
word_list = []
for line in lines:
    line = line.split("\t")[0]
    word_list.extend(line.split(" "))
print len(set(word_list))
</code></pre>

python nlp