How to speed up printing line by line to a file in Python?
<p>Say I'm printing numbers from two arrays into a file:</p>

<pre><code>from numpy import random
number_of_points = 10000
a = random.rand(number_of_points)
b = random.rand(number_of_points)
fh = open('file.txt', 'w')
for i in range(number_of_points):
    for j in range(number_of_points):
        print('%f %f' % (a[i], b[j]), file=fh)
</code></pre>

<p>I feel this is making lots of calls to the system to print, whereas sending one call containing this information would be faster. Is this correct? If so, how could I do this? Are there faster ways to implement this?</p>

<p><code>print</code> has a lot of bells and whistles you're not using, and you're using C-style looping with indexing instead of direct iteration, both of which add needless overhead. You might be able to speed it up a bit by limiting the Python level work, pushing it to the C layer.</p>

<p>For example, in this case, you could replace the whole doubly-nested loop structure with:</p>

<pre><code>import itertools

# You could use '%f %f\n'.__mod__ as the map function if you like, I just
# find the modern format strings a little nicer
fh.writelines(itertools.starmap('{} {}\n'.format, itertools.product(a, b)))
</code></pre>

<p>which uses <code>product</code> to produce the results of your nested loops and indexing directly, <code>starmap</code>+<code>str.format</code> to create the lines, and <code>fh.writelines</code> to exhaust the generator created by <code>starmap</code>, writing all of its outputs directly to the file with a single function call, instead of 100,000,000 calls to to <code>print</code>.</p>

<p>Aside from the fixed (unrelated to number of items iterated) setup cost to create the generators and pass the final generator to <code>fh.writelines</code>, the actual iteration, formatting and I/O work will take place entirely at the C layer on the CPython reference interpreter, so it should run quite fast.</p>

python python-3.x