Reading large text files line-by-line is still using all my memory
<p>I'm writing a simple program that is supposed to read a large file (263.5gb to be exact) with JSON on each line (<a href="https://www.reddit.com/r/datasets/comments/3mg812/full_reddit_submission_corpus_now_available_2006/" rel="nofollow">link here</a>). I've done some research and the best method I've found is to read each line by line. Mine looks like this (<a href="https://trinket.io/python/4d1aa528e4" rel="nofollow">full code here</a>):</p>

<pre><code>with open(dumpLocation, "r") as f:
for line in f:

    # Read line, convert to dictionary and assign it to 'c'
    c = json.loads(f.readline())

    for n in files:
        if n.lower() in c["title"].lower():

            try:
                # Collect data
                timestamp = str(c["retrieved_on"])
                sr_id = c["subreddit_id"]
                score = str(c["score"])
                ups = str(c["ups"])
                downs = str(c["downs"])
                title = ('"' + c["title"] + '"')

                # Append data to file
                files[n].write(timestamp + ","
                               + sr_id + ","
                               + score + ","
                               + ups + ","
                               + downs + ","
                               + title + ","
                               + "\n")
                found += 1
            except:
                numberOfErrors += 1
                errors[comments] = sys.exc_info()[0]

        comments += 1

        # Updates user
        print("Comments scanned: " + str(comments) + "\nFound: " + str(found) + "\n")
</code></pre>

<p>Now I can get this to run, and it ran for a good hour before it crashed (approximately 1.3 million lines). I noticed in the processes that the memory usage was slowly growing and got to about 2gb before crashing.</p>

<p>There is approximately 200 million lines I need to sort through, and I am also writing to files if specific words are found (searched for 5, found 337 before crash). Is there a better way of doing this? My computer usually only has about 2gb RAM to spare</p>

<p>I worked out where the memory leak was. On this line where I was printing to the console after every line:</p>

<pre><code> print("Comments scanned: " + str(comments) + "\nFound: " + str(found) + "\n")
</code></pre>

<p>Print 200 million times and your computer is bound to run out of memory trying to hold it all in the console at once. Removed it and it worked perfectly :)</p>

python file optimization text