Convert Hex Column in Pandas Without Iterating
<p>I am trying to bin a Pandas dataframe in Python 3 in order to have more efficient grouping over a large dataset. Currently the performance bottleneck is in iterating over the dataframe using the .apply() method.</p>

<p>All entries within the column are in hex, so it seems like the pd.to_numeric function should do exactly what I want.</p>

<p>I've tried a variety of options, but so far nothing has worked.</p>

<pre><code>#  This sets all values to np.nan with coerced errors, 'Unable to parse string' with raise errors.
dataframe[bin] = pd.to_numeric(dataframe[to_bin], errors='coerce') % __NUM_BINS__ 

# Gives me "int() Cannot convert non-string with explicit base"
dataframe[bin] = int(dataframe[to_bin].astype(str), 16) % __NUM_BINS__

# Value Error: Invalid literal for int with base 10 'ffffffffff'
dataframe[bin] = dataframe.astype(np.int64) % __NUM_BINS__ 
</code></pre>

<p>Any suggestions? This seems like something that people would have to have tackled in the past.</p>

<p>After some assists from the comments above: a faster way to accomplish this is to use a generator function. That way it can deal with any exceptions if provided data that cannot be converted from hex.</p>

<pre><code>def bin_vals(lst):
    for item in lst:
        try:
             yield int(item, 16) % __NUM_BINS__
        except:
             yield __ERROR_BIN__ #whatever you store weird items in
</code></pre>

<p>Then in your conversion portion you would do the following:</p>

<pre><code>dataframe['binned_value'] = [bin for bin in bin_vals(df['val_to_bin'].tolist())]
</code></pre>

<p>That led to a substantial speedup from iterating through each row. It was also faster than the apply method which I had used originally.</p>

python pandas dataframe