Fill MISSING values only in a dataframe (pandas)
<p>What I have in a dataframe:</p>

<pre><code>email    user_name    sessions    ymo
a@a.com    JD    1    2015-03-01
a@a.com    JD    2    2015-05-01
</code></pre>

<p>What I need:</p>

<pre><code>email    user_name    sessions    ymo
a@a.com    JD    0    2015-01-01
a@a.com    JD    0    2015-02-01
a@a.com    JD    1    2015-03-01
a@a.com    JD    0    2015-04-01
a@a.com    JD    2    2015-05-01
a@a.com    JD    0    2015-06-01
a@a.com    JD    0    2015-07-01
a@a.com    JD    0    2015-08-01
a@a.com    JD    0    2015-09-01
a@a.com    JD    0    2015-10-01
a@a.com    JD    0    2015-11-01
a@a.com    JD    0    2015-12-01
</code></pre>

<p><code>ymo</code> column are <code>pd.Timestamp</code>s:</p>

<pre><code>all_ymo

[Timestamp('2015-01-01 00:00:00'),
 Timestamp('2015-02-01 00:00:00'),
 Timestamp('2015-03-01 00:00:00'),
 Timestamp('2015-04-01 00:00:00'),
 Timestamp('2015-05-01 00:00:00'),
 Timestamp('2015-06-01 00:00:00'),
 Timestamp('2015-07-01 00:00:00'),
 Timestamp('2015-08-01 00:00:00'),
 Timestamp('2015-09-01 00:00:00'),
 Timestamp('2015-10-01 00:00:00'),
 Timestamp('2015-11-01 00:00:00'),
 Timestamp('2015-12-01 00:00:00')]
</code></pre>

<p>Unfortunately, this answer: <a href="http://stackoverflow.com/questions/31786881/adding-values-for-missing-data-combinations-in-pandas">Adding values for missing data combinations in Pandas</a> is not good as it creates duplicates for existing <code>ymo</code> values.</p>

<p>I tried something like this, but it is <em>extremely</em> slow:</p>

<pre><code>for em in all_emails:
    existent_ymo = fill_ymo[fill_ymo['email'] == em]['ymo']
    existent_ymo = set([pd.Timestamp(datetime.date(t.year, t.month, t.day)) for t in existent_ymo])
    missing_ymo = list(existent_ymo - all_ymo)
    multi_ind = pd.MultiIndex.from_product([[em], missing_ymo], names=col_names)
    fill_ymo = sessions.set_index(col_names).reindex(multi_ind, fill_value=0).reset_index()
</code></pre>

<ul>
<li>generate month beginning dates and <code>reindex</code>  </li>
<li><code>ffill</code> and <code>bfill</code> columns <code>['email', 'user_name']</code></li>
<li><code>fillna(0)</code> for column <code>'sessions'</code></li>
</ul>

<hr>

<pre><code>mbeg = pd.date_range('2015-01-31', periods=12, freq='M') - pd.offsets.MonthBegin()

df1 = df.set_index('ymo').reindex(mbeg)

df1[['email', 'user_name']] = df1[['email', 'user_name']].ffill().bfill()
df1['sessions'] = df1['sessions'].fillna(0).astype(int)

df1
</code></pre>

<p><a href="http://i.stack.imgur.com/FHOKR.png" rel="nofollow"><img src="http://i.stack.imgur.com/FHOKR.png" alt="enter image description here"></a></p>

python pandas