Fastest way to concatenate multiple files column wise - Python
<p><strong>What is the fastest method to concatenate multiple files column wise (within Python)?</strong></p>

<p>Assume that I have two files with 1,000,000,000 lines and ~200 UTF8 characters per line.</p>

<p><strong>Method 1:</strong> Cheating with <code>paste</code></p>

<p>I could concatenate the two files under a linux system by using <code>paste</code> in shell and I could cheat using <code>os.system</code>, i.e.:</p>

<pre><code>def concat_files_cheat(file_path, file1, file2, output_path, output):
    file1 = os.path.join(file_path, file1)
    file2 = os.path.join(file_path, file2)
    output = os.path.join(output_path, output)
    if not os.path.exists(output):
        os.system('paste ' + file1 + ' ' + file2 + ' &gt; ' + output)
</code></pre>

<p><strong>Method 2:</strong> Using nested context manager with <code>zip</code>:</p>

<pre><code>def concat_files_zip(file_path, file1, file2, output_path, output):
    with open(output, 'wb') as fout:
        with open(file1, 'rb') as fin1, open(file2, 'rb') as fin2:
            for line1, line2 in zip(fin1, fin2):
                fout.write(line1 + '\t' + line2)
</code></pre>

<p><strong>Method 3:</strong> Using <code>fileinput</code></p>

<p>Does <code>fileinput</code> iterate through the files in parallel? Or will they iterate through each file sequentially on after the other?</p>

<p>If it is the former, I would assume it would look like this:</p>

<pre><code>def concat_files_fileinput(file_path, file1, file2, output_path, output):
    with fileinput.input(files=(file1, file2)) as f:
        for line in f:
            line1, line2 = process(line)
            fout.write(line1 + '\t' + line2)
</code></pre>

<p><strong>Method 4</strong>: Treat them like <code>csv</code></p>

<pre><code>with open(output, 'wb') as fout:
    with open(file1, 'rb') as fin1, open(file2, 'rb') as fin2:
        writer = csv.writer(w)
        reader1, reader2 = csv.reader(fin1), csv.reader(fin2)
        for line1, line2 in zip(reader1, reader2):
          writer.writerow(line1 + '\t' + line2)
</code></pre>

<p>Given the data size, which would be the fastest? </p>

<p>Why would one choose one over the other? Would I lose or add information? </p>

<p>For each method how would I choose a different delimiter other than <code>,</code> or <code>\t</code>?</p>

<p>Are there other ways of achieving the same concatenation column wise? Are they as fast?</p>

<p>Method #1 is the fastest because it uses native (instead of Python) code to concatenate the files. However it is definitively cheating.</p>

<p>If you want to cheat, you may also consider also writing your own C extension for Python - it may be even faster, depending on your coding skills.</p>

<p>I am afraid Method #4 would not be working, since you are concatenating lists with string. I would go with <code>writer.writerow(line1 + line2)</code> instead. You may use <code>delimiter</code> parameter of both <code>csv.reader</code> and <code>csv.writer</code> to customise the delimiter (see <a href="https://docs.python.org/2/library/csv.html" rel="nofollow">https://docs.python.org/2/library/csv.html</a>).</p>

python shell text-files delimiter paste