Pandas Slow. Want first occurrence in DataFrame
<p>I have a DataFrame of <code>people</code>. One of the columns in this DataFrame is a <code>place_id</code>. I also have a DataFrame of places, where one of the columns is <code>place_id</code> and another is <code>weather</code>. For every person, I am trying to find the corresponding weather. Importantly, many people have the same <code>place_id</code>s. </p>

<p>Currently, my setup is this:</p>

<pre><code>def place_id_to_weather(pid):
    return place_df[place_df['place_id'] == pid]['weather'].item() 

person_df['weather'] = person_df['place_id'].map(place_id_to_weather)`
</code></pre>

<p>But this is untenably slow. I would like to speed this up. I suspect that I could achieve a speedup like this:</p>

<p>Instead of returning <code>place_df[...].item()</code>, which does a search for <code>place_id == pid</code> for that entire column and returns a series, and then grabbing the first item in that series, I really just want to curtail the search in <code>place_df</code> after the first match <code>place_df['place_id']==pid</code> has been found. After that, I don't need to search any further. How do I limit the search to first occurrences only?</p>

<p>Are there other methods I could use to achieve a speedup here? Some kind of join-type method?</p>

<p>The map function is your quickest method, the purpose of which is to avoid calling an entire dataframe to run some function repeatedly. This is what you ended up doing in your function i.e. calling an entire dataframe which is fine but not good doing it repeatedly. To tweak your code just a little will significantly speed up your process and only call the place_df dataframe once:</p>

<pre><code>person_df['weather'] = person_df['place_id'].map(dict(zip(place_df.place_id, place_df.weather)))
</code></pre>

python pandas