How to optimize code that iterates on a big dataframe in Python
<p>I have a big pandas dataframe. It has thousands of columns and over a million rows. I want to calculate the difference between the max value and the min value row-wise. Keep in mind that there are many NaN values and some rows are all NaN values (but I still want to keep them!).</p>

<p>I wrote the following code. It works but it's time consuming:</p>

<pre><code>totTime = []
for index, row in date.iterrows():
    myRow = row.dropna()
    if len(myRow):
        tt = max(myRow) - min(myRow)
    else:
        tt = None
    totTime.append(tt)
</code></pre>

<p>Is there any way to optimize it? I tried with the following code but I get an error when it encounters all NaN rows:</p>

<pre><code>tt = lambda x: max(x.dropna()) - min(x.dropna())
totTime = date.apply(tt, axis=1)
</code></pre>

<p>Any suggestions will be appreciated!</p>

<p>It is usually a bad idea to use a <code>python</code> <code>for</code> loop to iterate over a large <code>pandas.DataFrame</code> or a <code>numpy.ndarray</code>. You should rather use the available build in functions on them as they are optimized and in many cases actually not written in python but in a compiled language. In your case you should use the methods <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.max.html" rel="nofollow">pandas.DataFrame.max</a> and <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.min.html" rel="nofollow">pandas.DataFrame.min</a> that both give you an option <code>skipna</code> to skip <code>nan</code> values in your <code>DataFrame</code> without the need to actually drop them manually. Furthermore, you can choose a <code>axis</code> to minimize along. So you can specifiy <code>axis=1</code> to get the minimum along columns.</p>

<p>This will add up to something similar as what @EdChum just mentioned in the comments:</p>

<pre><code>data.max(axis=1, skipna=True) - data.min(axis=1, skipna=True)
</code></pre>

python pandas optimization dataframe