How to remap ids to consecutive numbers quickly
<p>I have a large csv file with lines that looks like</p>

<pre><code>stringa,stringb
stringb,stringc
stringd,stringa
</code></pre>

<p>I  need to convert it so the ids are consecutively numbered from 0. In this case the following would work</p>

<pre><code>0,1
1,2
3,0
</code></pre>

<p>My current code looks like:</p>

<pre><code>import csv
names = {}
counter = 0
with open('foo.csv', 'rb') as csvfile:
     reader = csv.reader(csvfile)
     for row in reader:
         if row[0] in names:
             id1 =  row[0]
         else:
             names[row[0]] = counter
             id1 = counter
             counter += 1
         if row[1] in names:
             id2 = row[1]
         else:
             names[row[1]] = counter
             id2 = counter
             counter += 1
     print id1, id2
</code></pre>

<p>Python dicts use a lot of memory sadly and my input is large.</p>

<blockquote>
  <p>What can I do when the input is too large for the dict to fit in memory</p>
</blockquote>

<p>I would also be interested if there is a better/faster way to solve this problem in general.</p>

<p>You can use <code>factorize</code> if you want an array of id's: </p>

<pre><code>df = pd.read_csv(data, header=None, prefix='Col_')
print (pd.factorize(np.hstack(df.values)))

(array([0, 1, 1, 2, 3, 0]), array(['stringa', 'stringb', 'stringc', 'stringd'], dtype=object))
</code></pre>

<hr>

<p><strong>EDIT :</strong> (as per the comment)</p>

<p>You could take the slices of the tuple obtained after the <code>factorize</code> method and map accordingly to the entire <code>dataframe</code> by replacing one another as shown:</p>

<pre><code>num, letter = pd.factorize(np.hstack(df.values))

df.replace(to_replace=sorted(list(set(letter))), value=sorted(list(set(num))))

   Col_0  Col_1
0      0      1
1      1      2
2      3      0
</code></pre>

python pandas dataframe