Why is file.close() slowing my code
<p>I am writing a program which generates a specified number of sentences, which are each written to a file. I have been trying to optimize the code for cases with 10 million sentences or more.  </p>

<p>I recently specified the buffer parameter in my open calls to 512MB in order to improve write performance however, my code actually became 3 seconds slower. The culprit is something called <code>{method 'close' of '_io.TextIOWrapper' objects}</code>. I think this has something to do with the file close method but, this is my first time comparing profile output.</p>

<p>This is how slow my write speed used to be:</p>

<pre><code>10000000   49.057    0.000   49.057    0.000 {method 'write' of '_io.TextIOWrapper' objects}
</code></pre>

<p>This is how it is now:</p>

<pre><code>10000000    3.184    0.000    3.184    0.000 {method 'write' of '_io.TextIOWrapper' objects}
</code></pre>

<p>Quite a substantial improvement.</p>

<p>This is my old close method:</p>

<pre><code>3    4.003    1.334    4.003    1.334 {method 'close' of '_io.TextIOWrapper' objects}
</code></pre>

<p>This is my new one:</p>

<pre><code> 1   62.668   62.668   62.668   62.668 {method 'close' of '_io.TextIOWrapper' objects}
</code></pre>

<p>Here is my code:</p>

<pre><code>def sentence_maker(nouns, verbs, number_of_sentences, file_name):
    writer = open(file_name, "w", 536870912)
    for num in range(number_of_sentences):
        string = (choice(nouns) + " " + choice(verbs) + " " + choice(nouns))
        writer.write(string + "\n")
    writer.close()
</code></pre>

<p>Why is <code>close()</code> so slow?</p>

<p>Note: somewhere earlier in the program I used to have some <code>close()</code> statements hence the <code>ncalls = 3</code> in my old <code>close()</code> example. I have determined that these have no discernible impact on performance.</p>

<p>Writing to disk is slow, so many programs store up writes into large chunks which they write all-at-once. This is called buffering, and Python does it automatically when you open a file. When you write to the file, you're actually writing to a "buffer" in memory. When it fills up, Python will automatically write it to disk or upon calling close().</p>

python performance io