Using Python to remove paratext (or 'noise') from txt files
<p>I am in the proces of preparing a corpus of textfiles, consisting of 170 Dutch novels. I am a literary scholar and relatively new to Python, and also to programming in general. What I am trying to do is writing a Python script for removing everything from each .txt file that does NOT belong to the actual content of the novel (i.e. the story). Things I want to remove are: added biographies of the author, blurbs, and other pieces of information that comes with converting an ePub to .txt. </p>

<p>My idea is to manually decide for every .txt file at which line the actual content of the novel begins and where it ends. I am using the following block of code for the purpose of removing every information in the .txt file that is not contained between those two line numbers:</p>

<pre><code>def removeparatext(inputFilename, outputFilename):
    inputfile = open(inputFilename,'rt', encoding='utf-8')
    outputfile = open(outputFilename, 'w', encoding='utf-8')

    for line_number, line in enumerate(inputfile, 1):
        if line_number &gt;= 80 and line_number &lt;= 2741: 
            outputfile.write(inputfile.readline())

    inputfile.close()
    outputfile.close()

removeparatext(inputFilename, outputFilename)
</code></pre>

<p>The numbers 80 and 2741 are the start and end numbers for the actual content of one specific novel. However, the outputfile only outputs a .txt file with the text removed BEFORE linenumber 80, it still contains everyhing AFTER line number 2741. I do not seem to understand why. Perhaps I am not using the enumerate() function in the right way. </p>

<p>Another thing is that I would like to get rid of all unnecessary spaces in the .txt-file. But the .strip() method does not seem to work when I implement it in this block of code. </p>

<p>Could anyone give me a suggestion as to how to solve this problem? Many thanks in advance!</p>

<p><code>enumerate</code> already provides the <em>line</em> alongside its index, so you don't need call <code>readline</code> on the file object again as that would lead to unpredictable behavior - more like reading the file object at a double pace:</p>

<pre><code>for line_number, line in enumerate(inputfile, 1):
    if line_number &gt;= 80 and line_number &lt;= 2741: 
        outputfile.write(line)
#                        ^^^^
</code></pre>

<hr>

<p>As an alternative to using <code>enumerate</code> and iterating through the entire file, you may consider <em>slicing</em> the file object using <a href="https://docs.python.org/2/library/itertools.html#itertools.islice" rel="nofollow"><code>itertools.islice</code></a> which takes the start and stop indices, and then writing the <em>sliced sequence</em> to the output file using <a href="http://python-reference.readthedocs.io/en/latest/docs/file/writelines.html" rel="nofollow"><code>writelines</code></a>:</p>

<pre><code>from itertools import islice

def removeparatext(inputFilename, outputFilename):
    inputfile = open(inputFilename,'rt', encoding='utf-8')
    outputfile = open(outputFilename, 'w', encoding='utf-8')

    # use writelines to write sliced sequence of lines 
    outputfile.writelines(islice(inputfile, 79, 2741)) # indices start from zero

    inputfile.close()
    outputfile.close()
</code></pre>

<hr>

<p>In addition, you can <em>open</em> files and leave the closing/cleanup to Python by using a context manager <em>with</em> the <code>with</code> statement. See <a href="http://stackoverflow.com/questions/9282967/how-to-open-a-file-using-the-open-with-statement">How to open a file using the open with statement</a>.</p>

<pre><code>from itertools import islice

def removeparatext(inputFilename, outputFilename):
    with open(inputFilename,'rt', encoding='utf-8') as inputfile,\
         open(outputFilename, 'w', encoding='utf-8') as outputfile:    
        # use writelines to write sliced sequence of lines 
        outputfile.writelines(islice(inputfile, 79, 2741))


removeparatext(inputFilename, outputFilename)
</code></pre>

python enumerate data-cleaning