percentage completion of a long-running python task
<p>I have a python program that crunches a large dataset using Pandas. It currently takes about 15 minute to complete. I want to log (stdout &amp; send the metric to Datadog) about the progress of the task. Is there a way to get the %-complete of the task (or a function)? In the future, I might be dealing with larger datasets. The Python task that I am doing is a simple grouping of a large pandas data frame. Something like this:</p>

<pre><code>dfDict = {}
for cat in categoryList:
    df1 = df[df['category'] == cat]
    if len(df1.index) &gt; 0:
        df1[dateCol] = pd.to_datetime(df[dateCol])
        dfDict[cat] = df1
</code></pre>

<p>here, the categoryList has about 20000 items, and df is a large data frame having (say) a 5 million rows.</p>

<p>I am not looking for anything fancy (like progress-bars..). Just percentage complete value. Any ideas?</p>

<p>Thanks!</p>

<p>the naive solution would be to just use the total amount of rows in your dataset and the index your are at, then calculate the progress:</p>

<pre><code>size = len(dataset)
for index, element in enumerate(dataset):
    print(index / size * 100)
</code></pre>

<p>This will only be somewhat reliable if every row takes around the same time to complete. Because you have a large dataset, it might average out over time, but if some rows take a millisecond, and another takes 10 minutes, the percentage will be garbage.</p>

<p>Also consider rounding the percentage to one decimal:</p>

<pre><code>size = len(dataset)
for index, element in enumerate(dataset):
    print(round(index / size * 100), 1)
</code></pre>

<p>Printing for every row might slow your task down significantly so consider this improvement:</p>

<pre><code>size       = len(dataset)
percentage = 0
for index, element in enumerate(dataset):
    new_percentage = round(index / size * 100), 1)
    if percentage != new_percentage:
        percentage = new_percentage
        print(percentage)
</code></pre>

<p>There are, of course, also modules for this:</p>

<p><a href="https://pypi.python.org/pypi/progressbar" rel="nofollow">progressbar</a></p>

<p><a href="https://pypi.python.org/pypi/progress" rel="nofollow">progress</a></p>

python