How to remove data from DataFrame permanently
<p>After reading CSV data file with:</p>

<pre><code>import pandas as pd  
df = pd.read_csv('data.csv')
print df.shape
</code></pre>

<p>I get DataFrame 99 rows (indexes) long:</p>

<pre><code>(99, 2)
</code></pre>

<p>To cleanup DataFrame I go ahead and apply dropna() method which reduces it to 33 rows:</p>

<pre><code>df = df.dropna()
print df.shape
</code></pre>

<p>which prints:</p>

<pre><code>(33, 2)
</code></pre>

<p>Now when I iterate the columns it prints out all 99 rows like they weren't dropped:</p>

<pre><code>for index, value in df['column1'].iteritems():
    print index
</code></pre>

<p>which gives me this:</p>

<pre><code>0
1
2
.
.
.
97
98
99
</code></pre>

<p>It appears the <code>dropna()</code> simply made the data "hidden". That hidden data returns back when I iterate DataFrame. How to assure the dropped data is removed from DataFrame instead just getting hidden?</p>

<p>You're being confused by the fact that the row labels have been preserved so the last row label is still <code>99</code>.</p>

<p>Example:</p>

<pre><code>In [2]:
df = pd.DataFrame({'a':[0,1,np.NaN, np.NaN, 4]})
df

Out[2]:
    a
0   0
1   1
2 NaN
3 NaN
4   4
</code></pre>

<p>After calling <code>dropna</code> the index row labels are preserved:</p>

<pre><code>In [3]:
df = df.dropna()
df

Out[3]:
   a
0  0
1  1
4  4
</code></pre>

<p>If you want to reset so that they are contiguous then call <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reset_index.html" rel="nofollow"><code>reset_index(drop=True)</code></a> to assign a new index:</p>

<pre><code>In [4]:
df = df.reset_index(drop=True)
df

Out[4]:
   a
0  0
1  1
2  4
</code></pre>

python pandas dataframe