Comparing the contents of very large files efficiently
<p>I need to compare two files of differing formats quickly and I'm not sure how to do it. I would very much appreciate it if someone could point me in the right direction.</p>

<p>I am working on CentOS 6 and I am most comfortable with Python (both Python 2 and Python 3 are available).</p>

<hr>

<h2>The problem</h2>

<p>I am looking to compare the contents two large files (quickly). The files, unfortunately, differ in content; I will need to modify the contents of one before I can compare them. They are not particularly well-organized, so I can't move linearly down each and compare line-by-line.</p>

<p>Here's an example of the files:</p>

<pre>
<i>File 1                    File 2</i>
Job,Time                  Job,Start,End
0123,3-00:00:00           0123,2016-01-01T00:00:00,2016-01-04T00:00:00
1111,05:30:00             1111,2016-01-01T00:00:00,2016-01-01T05:30:00
0000,00:00:05             9090.abc,2016-01-01T12:00:00,2016-01-01T22:00:00
9090.abc,10:00:00         0000,2015-06-01T00:00:00,2015-06-01T00:00:05
...                       ...
</pre>

<p>I would like to compare the contents of lines with the same "Job" field, like so:</p>

<pre>
<i>Job        File 1 Content   File 2 Content</i>
0123       3-00:00:00       2016-01-01T00:00:00,2016-01-04T00:00:00
1111       05:30:00         2016-01-01T00:00:00,2016-01-01T05:30:00
0000       00:00:05         2015-06-01T00:00:00,2015-06-01T00:00:05
9090.abc   10:00:00         2016-01-01T12:00:00,2016-01-01T22:00:00
...        ...              ...
</pre>

<p>I will be performing calculations on the <i>File 1 Content</i> and <i>File 2 Content</i> and comparing the two (for each line).</p>

<p>What is the most efficient way of doing this (matching lines)?</p>

<hr>

<p>The system currently in place loops through one file in its entirety for each line in the other (until a match is found). This process may take hours to complete, and the files are always growing. I am looking to make the process of comparing them as efficient as possible, but even marginal improvements in performance can have a drastic effect.</p>

<p>I appreciate any and all help.</p>

<p>Thank you!</p>

<p>This is simple utility to convert File 2 format to  File 1 like format (i hope i understand question right, python 2 used)
save code to file <code>util1.py</code> for example </p>

<pre><code>import time
import sys

if __name__ == '__main__':
    if  len(sys.argv) &lt; 2:
        print 'Err need filename'
        sys.exit()
    with open(sys.argv[1], 'r') as f:
        line = f.next()
        for line in f:
            jb, start, end =  line.rstrip().split(',')
            dt_format ='%Y-%m-%dT%H:%M:%S'
            start = time.strptime(start, dt_format)
            end = time.strptime(end, dt_format)
            delt = time.mktime(end) - time.mktime(start)
            m, s = divmod(delt, 60)
            h, m = divmod(m, 60)
            d, h = divmod(h, 24)
            if d !=0:
                print '{0},{1:d}-{2:02d}:{3:02d}:{4:02d}'.format(jb, int(d), int(h), int(m), int(s))
            else:
                print '{0},{2:02d}:{3:02d}:{4:02d}'.format(jb, int(d), int(h), int(m), int(s))
</code></pre>

<p>then run
<code>python ./util1.py f2.txt &gt; f2-1.txt</code>
this save output to f2-1.txt  </p>

<p>then </p>

<p><code>cp f1.txt f1_.txt</code></p>

<p>delete header row <code>Job,Start,End</code> from <code>f1_.txt</code></p>

<p><code>sort f1_.txt &gt; f1.sorted.txt</code></p>

<p><code>sort f2-1.txt &gt; f2-1.sorted.txt</code></p>

<p>and 
<code>diff -u f1.sorted.txt f2-1.sorted.txt</code></p>

python performance file io