How to optimize code that iterates on a big dataframe in Python
<p>I have a big pandas dataframe. It has thousands of columns and over a million rows. I want to calculate the difference between the max value and the min value row-wise. Keep in mind that there are many NaN values and some rows are all NaN values (but I still want to keep them!).</p>

<p>I wrote the following code. It works but it's time consuming:</p>

<pre><code>totTime = []
for index, row in date.iterrows():
    myRow = row.dropna()
    if len(myRow):
        tt = max(myRow) - min(myRow)
    else:
        tt = None
    totTime.append(tt)
</code></pre>

<p>Is there any way to optimize it? I tried with the following code but I get an error when it encounters all NaN rows:</p>

<pre><code>tt = lambda x: max(x.dropna()) - min(x.dropna())
totTime = date.apply(tt, axis=1)
</code></pre>

<p>Any suggestions will be appreciated!</p>

<p>I have the same problem about iterating. 2 points:</p>

<ol>
<li>Why don't you replace NaN values with 0? You can do it with this <code>df.replace(['inf','nan'],[0,0])</code>. It replaces inf and nan values.</li>
<li>Take a look at this <a href="http://stackoverflow.com/questions/7837722/what-is-the-most-efficient-way-to-loop-through-dataframes-with-pandas/7837947#7837947">This</a>. Maybe you can understand, I have a similar question about how to optimize the loop to calculate de difference between actual row with the previous one. </li>
</ol>

python pandas optimization dataframe