Comparing the contents of very large files efficiently
<p>I need to compare two files of differing formats quickly and I'm not sure how to do it. I would very much appreciate it if someone could point me in the right direction.</p>

<p>I am working on CentOS 6 and I am most comfortable with Python (both Python 2 and Python 3 are available).</p>

<hr>

<h2>The problem</h2>

<p>I am looking to compare the contents two large files (quickly). The files, unfortunately, differ in content; I will need to modify the contents of one before I can compare them. They are not particularly well-organized, so I can't move linearly down each and compare line-by-line.</p>

<p>Here's an example of the files:</p>

<pre>
<i>File 1                    File 2</i>
Job,Time                  Job,Start,End
0123,3-00:00:00           0123,2016-01-01T00:00:00,2016-01-04T00:00:00
1111,05:30:00             1111,2016-01-01T00:00:00,2016-01-01T05:30:00
0000,00:00:05             9090.abc,2016-01-01T12:00:00,2016-01-01T22:00:00
9090.abc,10:00:00         0000,2015-06-01T00:00:00,2015-06-01T00:00:05
...                       ...
</pre>

<p>I would like to compare the contents of lines with the same "Job" field, like so:</p>

<pre>
<i>Job        File 1 Content   File 2 Content</i>
0123       3-00:00:00       2016-01-01T00:00:00,2016-01-04T00:00:00
1111       05:30:00         2016-01-01T00:00:00,2016-01-01T05:30:00
0000       00:00:05         2015-06-01T00:00:00,2015-06-01T00:00:05
9090.abc   10:00:00         2016-01-01T12:00:00,2016-01-01T22:00:00
...        ...              ...
</pre>

<p>I will be performing calculations on the <i>File 1 Content</i> and <i>File 2 Content</i> and comparing the two (for each line).</p>

<p>What is the most efficient way of doing this (matching lines)?</p>

<hr>

<p>The system currently in place loops through one file in its entirety for each line in the other (until a match is found). This process may take hours to complete, and the files are always growing. I am looking to make the process of comparing them as efficient as possible, but even marginal improvements in performance can have a drastic effect.</p>

<p>I appreciate any and all help.</p>

<p>Thank you!</p>

<p>I was trying to develop something where you'd split one of the files into smaller files (say 100,000 records each) and keep a pickled dictionary of each file that contains all <code>Job_id</code> as a key and its line as a value. In a sense, an index for each database and you could use a hash lookup on each subfile to determine whether you wanted to read its contents.</p>

<p>However, you say that the file grows continually and each <code>Job_id</code> is unique. So, I would bite the bullet and run your current analysis once. Have a line counter that records how many lines you analysed for each file and write to a file somewhere. Then in future, you can use <code>linecache</code> to know what line you want to start at for your next analysis in both <code>file1</code> and <code>file2</code>; all previous lines have been processed so there's absolutely no point in scanning the whole content of that file again, just start where you ended in the previous analysis. </p>

<p>If you run the analysis at sufficiently frequent intervals, who cares if it's O(n^2) since you're processing, say, 10 records at a time and appending it to your combined database. In other words, the first analysis takes a long time but each subsequent analysis gets quicker and eventually <code>n</code> should converge on <code>1</code> so it becomes irrelevant.</p>

python performance file io