Using Python to create a (random) sample of n words from text files
<p>For my PhD project I am evaluating all existing Named Entity Recogition Taggers for Dutch. In order to check the precision and recall for those taggers I want to manually annotate all Named Entities in a random sample from my corpus. That manually annotated sample will function as the 'gold standard' to which I will compare the results of the different taggers. </p>

<p>My corpus consists of 170 Dutch novels. I am writing a Python script to generate a random sample of a specific amount of words for each novel (which I will use to annotate afterwards). All novels will be stored in the same directory. The following script is meant to generate for each novel in that directory a random sample of n-lines:</p>

<pre><code>import random
import os
import glob
import sys
import errno

path = '/Users/roelsmeets/Desktop/libris_corpus_clean/*.txt'
files = glob.glob(path)  

for text in files:
    try:
        with open(text, 'rt', encoding='utf-8') as f:
             # number of lines from txt file
             random_sample_input = random.sample(f.readlines(),100) 

    except IOError as exc:
    # Do not fail if a directory is found, just ignore it.
        if exc.errno != errno.EISDIR: 
            raise 


# This block of code writes the result of the previous to a new file
random_sample_output = open("randomsample", "w", encoding='utf-8') 
random_sample_input = map(lambda x: x+"\n", random_sample_input)
random_sample_output.writelines(random_sample_input)
random_sample_output.close()
</code></pre>

<p>There are two problems with this code:</p>

<ol>
<li><p>Currently, I have put two novels (.txt files) in the directory. But the code only outputs a random sample for one of each novels. </p></li>
<li><p>Currently, the code samples a random amount of LINES from each .txt file, but I prefer to generate a random amount of WORDS for each .txt file. Ideally, I would like to generate a sample of, say, the first or last 100 words of each of the 170 .txt-files. In that case, the sample won't be random at all; but thus far, I couldn't find a way to create a sample without using the random library.</p></li>
</ol>

<p>Could anyone give a suggestion how to solve both problems? I am still new to Python and programming in general (I am a literary scholar), so I would be pleased to learn different approaches. Many thanks in advance!</p>

<p>You just have to split your lines into words, store them somewhere, and then, after having read all of your files and stored their words, pick 100 with <code>random.sample</code>. It it what I did in the code below. However, I am not quite sure if it is able to deal with 170 novels, since it will likely result in a lot of memory usage.</p>

<pre><code>import random
import os
import glob
import sys
import errno

path = '/Users/roelsmeets/Desktop/libris_corpus_clean/*.txt'
files = glob.glob(path)
words = []

for text in files:
    try:
        with open(text, 'rt', encoding='utf-8') as f:
             # number of lines from txt file
             for line in f:
                 for word in line.split():
                     words.append(word)

    except IOError as exc:
    # Do not fail if a directory is found, just ignore it.
        if exc.errno != errno.EISDIR: 
            raise 

random_sample_input = random.sample(words, 100)

# This block of code writes the result of the previous to a new file
random_sample_output = open("randomsample", "w", encoding='utf-8') 
random_sample_input = map(lambda x: x+"\n", random_sample_input)
random_sample_output.writelines(random_sample_input)
random_sample_output.close()
</code></pre>

<p>In the above code, the more words a novel has, the more likely is to be represented in the output sample. That may or may not be the desired behaviour. If you want each novel to have the same ponderation, you can select, let's say, 100 words from it to add in the <code>words</code> variable, and then select 100 hundred words from there at the end. It will also have the side effect of using a lot less memory, since only one novel will be stored at a time.</p>

<pre><code>import random
import os
import glob
import sys
import errno

path = '/Users/roelsmeets/Desktop/libris_corpus_clean/*.txt'
files = glob.glob(path)
words = []

for text in files:
    try:
        novel = []
        with open(text, 'rt', encoding='utf-8') as f:
             # number of lines from txt file
             for line in f:
                 for word in line.split():
                     novel.append(word)
             words.append(random.sample(novel, 100))


    except IOError as exc:
    # Do not fail if a directory is found, just ignore it.
        if exc.errno != errno.EISDIR: 
            raise 


random_sample_input = random.sample(words, 100)

# This block of code writes the result of the previous to a new file
random_sample_output = open("randomsample", "w", encoding='utf-8') 
random_sample_input = map(lambda x: x+"\n", random_sample_input)
random_sample_output.writelines(random_sample_input)
random_sample_output.close()
</code></pre>

<p>Third version, this one will deal with sentences instead of words, and keep the punctuation. Also, each book has the same "weight" on the final sentences kept, regardless of its size. Keep in mind that the sentence detection is done by an algorithm that is quite clever, but not infallible.</p>

<pre><code>import random
import os
import glob
import sys
import errno
import nltk.data

path = '/home/clement/Documents/randomPythonScripts/data/*.txt'
files = glob.glob(path)

sentence_detector = nltk.data.load('tokenizers/punkt/dutch.pickle')
listOfSentences = []

for text in files:
    try:
        with open(text, 'rt', encoding='utf-8') as f:
            fullText = f.read()
        listOfSentences += [x.replace("\n", " ").replace("  "," ").strip() for x in random.sample(sentence_detector.tokenize(fullText), 30)]

    except IOError as exc:
    # Do not fail if a directory is found, just ignore it.
        if exc.errno != errno.EISDIR:
            raise

random_sample_input = random.sample(listOfSentences, 15)
print(random_sample_input)

# This block of code writes the result of the previous to a new file
random_sample_output = open("randomsample", "w", encoding='utf-8')
random_sample_input = map(lambda x: x+"\n", random_sample_input)
random_sample_output.writelines(random_sample_input)
random_sample_output.close()
</code></pre>

python text random nlp named-entity-recognition