python pandas/numpy quick way of replacing all values according to a mapping scheme
<p>let's say I have a huge panda data frame/numpy array where each element is a list of ordered values:</p>

<pre><code>sequences = np.array([12431253, 123412531, 12341234,12431253, 145345],
                     [5463456, 1244562, 23452],
                     [243524, 141234,12431253, 456367], 
                     [456345, 253451], 
                     [75635, 14145, 12346,12431253])
</code></pre>

<p>or,</p>

<pre><code>sequences = pd.DataFrame({'sequence': [[12431253, 123412531, 12341234,12431253, 145345],
                                      [5463456, 1244562, 23452],
                                      [243524, 141234, 456367,12431253],
                                      [456345, 253451],
                                      [75635, 14145, 12346,12431253]]})
</code></pre>

<p>and I want to replace them with another set of identifiers that start from 0, so I design a mapping like this:</p>

<pre><code>from compiler.ast import flatten
from sets import Set
mapping = pd.DataFrame({'v0': list(Set(flatten(sequences['sequence']))), 'v1': range(len(Set(flatten(sequences['sequence'])))})
</code></pre>

<p>......</p>

<p>so the result I was looking for:</p>

<pre><code>sequences = np.array([1, 2, 3,1, 4], [5, 6, 7], [8, 9, 10,1], [11, 12], [13, 14, 15,1])
</code></pre>

<p>how can I scale this up to a huge data frame/numpy of sequences ?</p>

<p>Thanks so much for any guidance! Greatly appreciated!</p>

<p><strong><em>Option 1</em></strong><br>
Using series definition</p>

<pre><code>stop = sequences.sequence.apply(np.size).cumsum()
start = end.shift().fillna(0).astype(int)
params = pd.concat([start, stop], axis=1, keys=['start', 'stop'])
params.apply(lambda x: list(np.arange(**x)), axis=1)

0    [0, 1, 2, 3]
1       [4, 5, 6]
2       [7, 8, 9]
3        [10, 11]
4    [12, 13, 14]
dtype: object
</code></pre>

python pandas numpy merge