Python 3: How to compare two big files fastest?
<p>In <strong>"Big_file.txt</strong>", I want to extract UIDs of <strong>"User A"</strong> which is not duplicated with UIDs in "<strong>Small_file.txt</strong>". I wrote the following code but it seems it will never stop running. So, how to speed up the process ? Thank you very much :)</p>

<pre><code>import json

uid_available = []
linesB = []
for line in open('E:/Small_file.txt'):
    line = json.loads(line)
    linesB.append(hash(line['uid']))


for line in open('E:/Big_file.txt'):
    line = json.loads(line)
    if hash(line['uid']) not in linesB and line['user'] == 'User A':
        uid_available.append(line['uid'])
</code></pre>

<p>This is format of Big_file.txt (has 10 millions lines):</p>

<pre><code>{'uid': 111, 'user': 'User A'}
{'uid': 222, 'user': 'User A'}
{'uid': 333, 'user': 'User A'}
{'uid': 444, 'user': 'User B'}
{'uid': 555, 'user': 'User C'}
{'uid': 666, 'user': 'User C'}
</code></pre>

<p>This is format of Small_file.txt (has few millions lines):</p>

<pre><code>{'uid': 333, 'user': 'User A'}
{'uid': 444, 'user': 'User B'}
{'uid': 555, 'user': 'User C'}
</code></pre>

<p>The output I expect:</p>

<pre><code>111
222
</code></pre>

<p>looking up an item in a list takes <code>O(n)</code> time. If you use a <code>dict</code> or a <code>set</code> you can improve it to <code>O(1)</code>.</p>

<p>shortest modification you can do is :</p>

<pre><code>linesB = []
for line in open('E:/Small_file.txt'):
    line = json.loads(line)
    linesB.append(hash(line['uid']))
linesB = set(linesB)
</code></pre>

<p>or to do it right </p>

<pre><code>linesB = set()
for line in open('E:/Small_file.txt'):
    line = json.loads(line)
    linesB.add(hash(line['uid']))
</code></pre>

python json hash compare