Filter a smaller file using another huge file
<p>I have a huge csv file with about 10^9 lines where each line has a pair of ids such as:</p>

<pre><code>IDa,IDb
IDb,IDa
IDc,IDd
</code></pre>

<p>Call this file1.  I have another much smaller csv file with about 10^6 lines in the same format. Call this file2. </p>

<p>I want to simply find the lines in file2 which contain at least one ID that exists somewhere in file1.</p>

<p>Is there a fast way to do this?  I don't mind if it is in awk, python or perl.</p>

<p>In perl,</p>

<pre><code>use strict;
use warnings;
use autodie;

# read file2
open my $file2, '&lt;', 'file2';
chomp( my @file2 = &lt;$file2&gt; );
close $file2;

# record file2 line numbers each id is found on
my %id;
for my $line_number (0..$#file2) {
    for my $id ( split /,/, $file2[$line_number] ) {
        push @{ $id{$id} }, $line_number;
    }
}

# look for those ids in file1
my @use_line;
open my $file1, '&lt;', 'file1';
while ( my $line = &lt;$file1&gt; ) {
    chomp $line;
    for my $id ( split /,/, $line ) {
        if ( exists $id{$id} ) {
            @use_line[ @{ $id{$id} } ] = @{ $id{$id} };
        }
    }
}
close $file1;

# print lines whose ids were found
print "$_\n" for @file2[ grep defined, @use_line ];
</code></pre>

python perl awk