Why is file.close() slowing my code
<p>I am writing a program which generates a specified number of sentences, which are each written to a file. I have been trying to optimize the code for cases with 10 million sentences or more.  </p>

<p>I recently specified the buffer parameter in my open calls to 512MB in order to improve write performance however, my code actually became 3 seconds slower. The culprit is something called <code>{method 'close' of '_io.TextIOWrapper' objects}</code>. I think this has something to do with the file close method but, this is my first time comparing profile output.</p>

<p>This is how slow my write speed used to be:</p>

<pre><code>10000000   49.057    0.000   49.057    0.000 {method 'write' of '_io.TextIOWrapper' objects}
</code></pre>

<p>This is how it is now:</p>

<pre><code>10000000    3.184    0.000    3.184    0.000 {method 'write' of '_io.TextIOWrapper' objects}
</code></pre>

<p>Quite a substantial improvement.</p>

<p>This is my old close method:</p>

<pre><code>3    4.003    1.334    4.003    1.334 {method 'close' of '_io.TextIOWrapper' objects}
</code></pre>

<p>This is my new one:</p>

<pre><code> 1   62.668   62.668   62.668   62.668 {method 'close' of '_io.TextIOWrapper' objects}
</code></pre>

<p>Here is my code:</p>

<pre><code>def sentence_maker(nouns, verbs, number_of_sentences, file_name):
    writer = open(file_name, "w", 536870912)
    for num in range(number_of_sentences):
        string = (choice(nouns) + " " + choice(verbs) + " " + choice(nouns))
        writer.write(string + "\n")
    writer.close()
</code></pre>

<p>Why is <code>close()</code> so slow?</p>

<p>Note: somewhere earlier in the program I used to have some <code>close()</code> statements hence the <code>ncalls = 3</code> in my old <code>close()</code> example. I have determined that these have no discernible impact on performance.</p>

<p>You explicitly elected to use a huge buffer (that 536870912 is the number of bytes buffered before flushing the buffer, about half a GB of memory). <code>close</code> includes an implicit <code>flush</code> of whatever is left in the buffer, and assuming you're writing a lot, that's going to mean it involves writing it all out.</p>

<p>You have to pay for the actual I/O at some point; a large buffer makes <code>write</code> cheap (because it doesn't actually perform any I/O), but a buffer that large is just deferring the pain, not avoiding it. I doubt any buffer size beyond 1 MB would actually save meaningful work (and the limit may be lower); the cost of performing the system calls is high if you do it constantly, but the difference between one call per MB and one call per 512 MB is not meaningful when the work done per call (the actual physical I/O) outweighs both of them by an order of magnitude or more.</p>

<p>For comparison, the reason to buffer is that system calls have high overhead compared to regular function calls (<a href="http://stackoverflow.com/a/22366251/364696">a few hundred clock ticks, vs. a dozen or less for most function calls</a>). CPython has some extra system calls involved in I/O (releasing and recovering the GIL), so the incremental cost of system call to <code>write</code> vs. a <code>memcpy</code> is a difference of maybe 100-1000x. But even 2000 ticks is still in the microsecond range for overhead on a modern CPU. But I/O itself is much more expensive than that; writing 10 MB of data is likely to take a tenth of a second or so. <strong>Saving a few <em>milliseconds</em> on system calls by using larger buffer sizes doesn't matter much when the I/O itself has costs measured in <em>seconds</em>.</strong> And a buffer that large is going to start introducing cache misses (and possible page faults) that a smaller buffer would avoid.</p>

python performance io