Pandas - identifying dataframe values that start with value in a list
<p>Say I have the following dataframe:</p>

<pre><code>&gt;&gt;&gt; import pandas as pd
&gt;&gt;&gt; d=pd.DataFrame()
&gt;&gt;&gt; d['A']=['12345','12354','76','4']
&gt;&gt;&gt; d['B']=['4442','2345','33','5']
&gt;&gt;&gt; d['C']=['5553','4343','33','5']
&gt;&gt;&gt; d
       A     B     C
0  12345  4442  5553
1  12354  2345  4343
2     76    33    33
3      4     5     5
</code></pre>

<p>And say I have 3 values of interest:</p>

<pre><code>&gt;&gt;&gt; vals=['123','76']
</code></pre>

<p>I am interested in determining which values in my dataframe start with any of the values in my list. There are 3 cases in my example: (0,A) starts with 123; (1,A) starts with 123; and (2,A) starts with 76. </p>

<p>Is there a way I can do this without looping through each of my values?</p>

<p>If I were interested in matching values exactly I could just do:</p>

<pre><code>&gt;&gt;&gt; d.isin(vals)
       A      B      C
0  False  False  False
1  False  False  False
2   True  False  False
3  False  False  False
&gt;&gt;&gt; 
</code></pre>

<p>And if I was interested in whether the values start with 1 particular value I could do:</p>

<pre><code>&gt;&gt;&gt; d.applymap(lambda x:x.startswith('123'))
       A      B      C
0   True  False  False
1   True  False  False
2  False  False  False
3  False  False  False
&gt;&gt;&gt; 
</code></pre>

<p>But how can I combine these two to find any value that starts with any value in my list?</p>

<p>A bit complicated, but it seems the fastest solution. </p>

<p>You can use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.startswith.html" rel="nofollow"><code>str.startswith</code></a> which works only with Series, so use <code>list</code> comprehension and <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html" rel="nofollow"><code>concat</code></a> output. But need check more items in list <code>vals</code>, so use another list comprehension with <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_or.html" rel="nofollow"><code>numpy.logical_or</code></a> with <code>reduce</code> which works with <code>numpy array</code> -  convert output of <code>concat</code> and last create <code>Dataframe</code> with same column and index as original and data are output of <code>numpy.logical_or</code>:</p>

<pre><code>print ([pd.concat([d[col].str.startswith(i) for col in d], axis=1).values for i in vals])
[array([[ True, False, False],
       [ True, False, False],
       [False, False, False],
       [False, False, False]], dtype=bool), array([[False, False, False],
       [False, False, False],
       [ True, False, False],
       [False, False, False]], dtype=bool)]

print (np.logical_or.reduce(
      [pd.concat([d[col].str.startswith(i) for col in d], axis=1).values for i in vals]))
[[ True False False]
 [ True False False]
 [ True False False]
 [False False False]]

print(pd.DataFrame(np.logical_or.reduce(
      [pd.concat([d[col].strstartswith(i) for col in d], 
                 axis=1).values for i in vals]), 
                   index=d.index, columns=d.columns))

       A      B      C
0   True  False  False
1   True  False  False
2   True  False  False
3  False  False  False
</code></pre>

<p><strong>Timings</strong>:</p>

<pre><code>#[40000 rows x 3 columns]
d = pd.concat([d]*10000).reset_index(drop=True)


In [77]: %timeit (d.applymap(lambda x: any([x.startswith(v) for v in vals])))
1 loop, best of 3: 228 ms per loop

In [78]: %timeit (d.apply(lambda x: x.str.contains('|'.join(['^' + x for x in vals]))))
10 loops, best of 3: 147 ms per loop

In [79]: %timeit (d.astype(str).stack().str.match('^(?:{})'.format('|'.join(vals))).unstack())
10 loops, best of 3: 172 ms per loop

In [80]: %timeit (pd.DataFrame(np.logical_or.reduce([pd.concat([d[col].str.startswith(i) for col in d], axis=1).values for i in vals]), index=d.index, columns=d.columns))
10 loops, best of 3: 116 ms per loop
</code></pre>

python pandas