Filter a smaller file using another huge file
<p>I have a huge csv file with about 10^9 lines where each line has a pair of ids such as:</p>

<pre><code>IDa,IDb
IDb,IDa
IDc,IDd
</code></pre>

<p>Call this file1.  I have another much smaller csv file with about 10^6 lines in the same format. Call this file2. </p>

<p>I want to simply find the lines in file2 which contain at least one ID that exists somewhere in file1.</p>

<p>Is there a fast way to do this?  I don't mind if it is in awk, python or perl.</p>

<p>Sample files:</p>

<pre><code>cat f1
IDa,IDb
IDb,IDa
IDc,IDd


cat f2
IDt,IDy
IDb,IDj
</code></pre>

<p>Awk solution:  </p>

<pre><code>awk -F, 'NR==FNR {a[$1]=$1;b[$2]=$2;next} ($1 in a)||($2 in b)' f1 f2
IDb,IDj
</code></pre>

<p>This will store first and second columns of file1 in array a and b. Then print those line if either first or second column is seen for second file.</p>

python perl awk