Comparing the contents of very large files efficiently
<p>I need to compare two files of differing formats quickly and I'm not sure how to do it. I would very much appreciate it if someone could point me in the right direction.</p>

<p>I am working on CentOS 6 and I am most comfortable with Python (both Python 2 and Python 3 are available).</p>

<hr>

<h2>The problem</h2>

<p>I am looking to compare the contents two large files (quickly). The files, unfortunately, differ in content; I will need to modify the contents of one before I can compare them. They are not particularly well-organized, so I can't move linearly down each and compare line-by-line.</p>

<p>Here's an example of the files:</p>

<pre>
<i>File 1                    File 2</i>
Job,Time                  Job,Start,End
0123,3-00:00:00           0123,2016-01-01T00:00:00,2016-01-04T00:00:00
1111,05:30:00             1111,2016-01-01T00:00:00,2016-01-01T05:30:00
0000,00:00:05             9090.abc,2016-01-01T12:00:00,2016-01-01T22:00:00
9090.abc,10:00:00         0000,2015-06-01T00:00:00,2015-06-01T00:00:05
...                       ...
</pre>

<p>I would like to compare the contents of lines with the same "Job" field, like so:</p>

<pre>
<i>Job        File 1 Content   File 2 Content</i>
0123       3-00:00:00       2016-01-01T00:00:00,2016-01-04T00:00:00
1111       05:30:00         2016-01-01T00:00:00,2016-01-01T05:30:00
0000       00:00:05         2015-06-01T00:00:00,2015-06-01T00:00:05
9090.abc   10:00:00         2016-01-01T12:00:00,2016-01-01T22:00:00
...        ...              ...
</pre>

<p>I will be performing calculations on the <i>File 1 Content</i> and <i>File 2 Content</i> and comparing the two (for each line).</p>

<p>What is the most efficient way of doing this (matching lines)?</p>

<hr>

<p>The system currently in place loops through one file in its entirety for each line in the other (until a match is found). This process may take hours to complete, and the files are always growing. I am looking to make the process of comparing them as efficient as possible, but even marginal improvements in performance can have a drastic effect.</p>

<p>I appreciate any and all help.</p>

<p>Thank you!</p>

<p>Parse each file and convert the data to <code>datetime.timedelta</code> objects.  Make a dictionary with the job number as the keys and timedelta object as the value(s):</p>

<pre><code>import operator, datetime, collections
def parse1(fp = 'job-file1.txt'):
    with open(fp) as f:
        next(f)
        for line in f:
            line = line.strip()
            job, job_length = line.split(',',1)
            if '-' in job_length:
                days, time = job_length.split('-')
                hours, minutes, seconds = time.split(':')
            else:
                days = 0
                hours, minutes, seconds = job_length.split(':')
            job_length = datetime.timedelta(days = int(days),
                                            hours = int(hours),
                                            minutes = int(minutes),
                                            seconds = int(seconds))
            yield (job, job_length)

fmt = '%Y-%m-%dT%H:%M:%S'
def parse2(fp = 'job-file2.txt'):
    with open(fp) as f:
        next(f)
        for line in f:
            line = line.strip()
            job, start, end = line.split(',')
            job_length = datetime.datetime.strptime(end, fmt) - datetime.datetime.strptime(start, fmt)
            yield (job, job_length)
</code></pre>

<p>Now you can either keep both timedelta objects and compare them later:</p>

<pre><code># {job_number:[timedelta1, timedelta2]
d = collections.defaultdict(list)

for key, value in parse1():
    d[key].append(value)

for key, value in parse2():
    d[key].append(value)
</code></pre>

<p>This lets you do something like:</p>

<pre><code>differences = {job:lengths for job,lengths in d.items() if not operator.eq(*lengths)}
print(differences)
</code></pre>

<p>Or you can just keep the difference between file1 and file2 job lengths as the value</p>

<pre><code>d = {key:value for key, value in parse1()}
for key, value in parse2():
    d[key] -= value
</code></pre>

<p>Then you would just check for differences with</p>

<pre><code>[job for job, difference in d.items() if difference.total_seconds() != 0]
</code></pre>

python performance file io