Python performance on reading files with extremely long lines
<p>I've got a file with around 6MB of data. All of the data are written in a single line. Why is the following command taking more than 15 minutes to finish? Is it normal?</p>

<pre><code>infile = open('file.txt')
outfile = open('out.txt', 'w')
for line in infile.readlines():
    outfile.write(line);
</code></pre>

<p><strong>Details:</strong></p>

<p>I'm using Python 2.7.</p>

<p>Output from <strong>wc</strong>:  </p>

<ul>
<li>newline count: 2</li>
<li>word count: 3475246 </li>
<li>byte count: 6951140 </li>
</ul>

<p><strong>Evaluation 1:</strong></p>

<p><em>Reference</em> code using in file (suggestted by Ahsanul Haque and Daewon Lee):</p>

<pre><code>for line in infile:
    output.write(line):
</code></pre>

<p><em>Time: 959.487 secs.</em></p>

<p>Try the following code snippet. The <code>readlines()</code> loads all data on memory, which seems to cause a long time.</p>

<pre><code>infile = open('file.txt', 'r')
outfile = open('out.txt', 'w')
for line in infile:
    outfile.write(line)
</code></pre>

<p>With my python 3.5 (64bit) on Windows 10 OS, the following code snippet finished within a few seconds.</p>

<pre><code>import time

start = time.time()

with open("huge_text.txt", "w") as fout:
    for i in range(1737623):
        fout.write("ABCD ")
    fout.write('\n')   
    for i in range(1737623):
        fout.write("EFGH ")
    fout.write('\n')
# end of with

infile = open('huge_text.txt', 'r')
outfile = open('out.txt', 'w')
for line in infile:
    outfile.write(line)

outfile.close()
infile.close()

end = time.time()
print("Time elapsed: ", end - start)

"""
&lt;Output&gt;
Time elapsed:  1.557690143585205
"""
</code></pre>

python performance file