percentage completion of a long-running python task
<p>I have a python program that crunches a large dataset using Pandas. It currently takes about 15 minute to complete. I want to log (stdout &amp; send the metric to Datadog) about the progress of the task. Is there a way to get the %-complete of the task (or a function)? In the future, I might be dealing with larger datasets. The Python task that I am doing is a simple grouping of a large pandas data frame. Something like this:</p>

<pre><code>dfDict = {}
for cat in categoryList:
    df1 = df[df['category'] == cat]
    if len(df1.index) &gt; 0:
        df1[dateCol] = pd.to_datetime(df[dateCol])
        dfDict[cat] = df1
</code></pre>

<p>here, the categoryList has about 20000 items, and df is a large data frame having (say) a 5 million rows.</p>

<p>I am not looking for anything fancy (like progress-bars..). Just percentage complete value. Any ideas?</p>

<p>Thanks!</p>

<p>You can modify the following according to your needs.</p>

<pre><code>from time import sleep

for i in range(12):
    sleep(1)
    print("\r\t&gt; Progress\t:{:.2%}".format((i + 1)/12), end='')
</code></pre>

<p>What this basically does, is that it prevents <code>print()</code> from writing the default end character (<code>end=''</code>) and at the same time, it write a carriage return (<code>'\r'</code>) before anything else. In simple terms, you are overwriting the previous <code>print()</code> statement.</p>

python