Efficiency when printing progress updates, print x vs if x%y==0: print x
<p>I am running an algorithm which reads an excel document by rows, and pushes the rows to a SQL Server, using Python. I would like to print some sort of progression through the loop. I can think of two very simple options and I would like to know which is more lightweight and why.
Option A:</p>

<pre><code>for x in xrange(1, sheet.nrows):
    print x
    cur.execute() # pushes to sql
</code></pre>

<p>Option B:</p>

<pre><code>for x in xrange(1, sheet.nrows):
    if x % some_check_progress_value == 0:
        print x
    cur.execute() # pushes to sql
</code></pre>

<p>I have a feeling that the second one would be more efficient but only for larger scale programs. Is there any way to calculate/determine this? </p>

<p>I am one of the developers of <a href="https://github.com/tqdm/tqdm" rel="nofollow">tqdm, a Python progress bar</a> that tries to be as efficient as possible while providing as many automated features as possible.</p>

<p>The biggest performance sink we had was indeed I/O: printing to the console/file/whatever.</p>

<p>But if your loop is tight (more than 100 iterations/second), then it's useless to print <em>every</em> update, you'd just as well print just 1/10 of the updates and the user would see no difference, while your bar would be 10 times less overhead (faster).</p>

<p>To fix that, at first we added a <code>mininterval</code> parameter which updated the display only every <code>x</code> seconds (which is by default 0.1 seconds, the human eye cannot really see anything faster than that). Something like that:</p>

<pre><code>import time

def my_bar(iterator, mininterval=0.1)
counter = 0
last_print_t = 0
for item in iterator:
    if (time.time() - last_print_t) &gt;= mininterval:
        last_print_t = time.time()
        print_your_bar_update(counter)
    counter += 1
</code></pre>

<p>This will mostly fix your issue as your bar will always have a constant display overhead which will be more and more negligible as you have bigger iterators.</p>

<p>If you want to go further in the optimization, <code>time.time()</code> is also an I/O operation and thus has a cost greater than simple Python statements. To avoid that, you want to minimize the calls you do to <code>time.time()</code> by introducing another variable: <code>miniters</code>, which is the minimum number of iterations you want to skip before even checking the time:</p>

<pre><code>import time

def my_bar(iterator, mininterval=0.1, miniters=10)
counter = 0
last_print_t = 0
last_print_counter = 0
for item in iterator:
    if (counter - last_print_counter) &gt;= miniters:
        if (time.time() - last_print_t) &gt;= mininterval:
            last_print_t = time.time()
            last_print_counter = counter
            print_your_bar_update(counter)
    counter += 1
</code></pre>

<p>You can see that <code>miniters</code> is similar to your Option B modulus solution, but it's better fitted as an added layer over time because time is more easily configured.</p>

<p>With these two parameters, you can manually finetune your progress bar to make it the most efficient possible for your loop.</p>

<p>However, <code>miniters</code> (or modulus) is tricky to get to work generally for everyone without manual finetuning, you need to make good assumptions and clever tricks to automate this finetuning. This is one of the major ongoing work we are doing on <code>tqdm</code>. Basically, what we do is that we try to calculate <code>miniters</code> to equal <code>mininterval</code>, so that time checking isn't even needed anymore. This automagic setting kicks in after <code>mininterval</code> gets triggered, something like that:</p>

<pre><code>from __future__ import division
import time

def my_bar(iterator, mininterval=0.1, miniters=10, dynamic_miniters=True)
counter = 0
last_print_t = 0
last_print_counter = 0
for item in iterator:
    if (counter - last_print_counter) &gt;= miniters:
        cur_time = time.time()
        if (cur_time - last_print_t) &gt;= mininterval:
            if dynamic_miniters:
                # Simple rule of three
                delta_it = counter - last_print_counter
                delta_t = cur_time - last_print_t
                miniters = delta_it * mininterval / delta_t
            last_print_t = cur_time
            last_print_counter = counter
            print_your_bar_update(counter)
    counter += 1
</code></pre>

<p>There are various ways to compute <code>miniters</code> automatically, but usually you want to update it to match <code>mininterval</code>.</p>

<p>If you are interested in digging more, you can check the <code>dynamic_miniters</code> internal parameters, <code>maxinterval</code> and an <a href="https://github.com/tqdm/tqdm/pull/251" rel="nofollow">experimental monitoring thread</a> of the <a href="https://github.com/tqdm/tqdm" rel="nofollow">tqdm project</a>.</p>

python algorithm performance python-2.7