Comparing the contents of very large files efficiently
<p>I need to compare two files of differing formats quickly and I'm not sure how to do it. I would very much appreciate it if someone could point me in the right direction.</p>

<p>I am working on CentOS 6 and I am most comfortable with Python (both Python 2 and Python 3 are available).</p>

<hr>

<h2>The problem</h2>

<p>I am looking to compare the contents two large files (quickly). The files, unfortunately, differ in content; I will need to modify the contents of one before I can compare them. They are not particularly well-organized, so I can't move linearly down each and compare line-by-line.</p>

<p>Here's an example of the files:</p>

<pre>
<i>File 1                    File 2</i>
Job,Time                  Job,Start,End
0123,3-00:00:00           0123,2016-01-01T00:00:00,2016-01-04T00:00:00
1111,05:30:00             1111,2016-01-01T00:00:00,2016-01-01T05:30:00
0000,00:00:05             9090.abc,2016-01-01T12:00:00,2016-01-01T22:00:00
9090.abc,10:00:00         0000,2015-06-01T00:00:00,2015-06-01T00:00:05
...                       ...
</pre>

<p>I would like to compare the contents of lines with the same "Job" field, like so:</p>

<pre>
<i>Job        File 1 Content   File 2 Content</i>
0123       3-00:00:00       2016-01-01T00:00:00,2016-01-04T00:00:00
1111       05:30:00         2016-01-01T00:00:00,2016-01-01T05:30:00
0000       00:00:05         2015-06-01T00:00:00,2015-06-01T00:00:05
9090.abc   10:00:00         2016-01-01T12:00:00,2016-01-01T22:00:00
...        ...              ...
</pre>

<p>I will be performing calculations on the <i>File 1 Content</i> and <i>File 2 Content</i> and comparing the two (for each line).</p>

<p>What is the most efficient way of doing this (matching lines)?</p>

<hr>

<p>The system currently in place loops through one file in its entirety for each line in the other (until a match is found). This process may take hours to complete, and the files are always growing. I am looking to make the process of comparing them as efficient as possible, but even marginal improvements in performance can have a drastic effect.</p>

<p>I appreciate any and all help.</p>

<p>Thank you!</p>

<p>If you can find a way to take advantage of hash tables your task will change from O(N^2) to O(N). The implementation will depend on exactly how large your files are and whether or not you have duplicate job IDs in file 2. I'll assume you don't have any duplicates. If you can fit file 2 in memory, just load the thing into pandas with job as the index. If you can't fit file 2 in memory, you can at least build a dictionary of {Job #: row # in file 2}. Either way, finding a match should be substantially faster.</p>

python performance file io