Does python threading module speed up the execution time?
<p>I have a database containing employee information. I have 4000 employees. Each employee has an unique identification number.
I try to fetch employee information for each employee from the database using a python script. For 1 employee, the execution time for fetching info is 1 seconds. For 4000 employees, it makes <code>4000</code> seconds (67 minutes, who would like to wait that long?). 
The employee infos should be stored in a dictionary, in the following format:</p>

<pre><code>infos = {"ID1": ["info for employee 1"], "ID2": ["info for employee 2"], ... }
</code></pre>

<p>I'm thinking of doing the following to reduce the execution time:</p>

<ol>
<li>Get employee id numbers</li>
<li>Divide the id numbers into 10 groups</li>
<li>Start 10 threads simultaneously</li>
<li>Make each thread use 1 of 10 employee id groups and fetch those employees' info
from database into separate dictionaries</li>
<li>In the end, combine those 10 dictionaries</li>
</ol>

<p>Is it possible? Would this method reduce the execution time 10 times?</p>

<p>I think you're confusing <em>threading</em> with <em>concurrency</em>.</p>

<p>Threading is the act of simply using multiple threads of execution at the same time. This doesn't mean multiple actions are done simultaneously though... your processor still has to switch between the threads. This technique is useful when you're expecting to wait a long time for an operation to complete (like reading a large file) and you want other stuff to happen in the meantime (printing a progress bar, for example).</p>

<p>Concurrency is when you create multiple threads, and different threads get assigned to different cores in the processor. This is, of course, dependent on the hardware (you need a multicore processor to accomplish it).</p>

<p>Python's <code>threading</code> module allows for multiple threads to be created, sure; and in an ideal world it would automatically handle concurrency. But the default Python interpreter, CPython, does not allow for native concurrency due to the "Global Interpreter Lock". (See <a href="https://docs.python.org/3/library/threading.html" rel="nofollow">this page</a> just above the section 17.1.3 heading and <a href="http://stackoverflow.com/a/1294402/3377150">this SO post</a> for more information about GIL.) In fact, using <code>threading</code> like you suggest may even make your execution <em>slower</em> (see <a href="https://www.quantstart.com/articles/parallelising-python-with-threading-and-multiprocessing" rel="nofollow">here</a>).</p>

<p>I've not done concurrency in Python, but I would advise you to check out the <a href="https://docs.python.org/3/library/multiprocessing.html" rel="nofollow"><code>multiprocessing</code></a> module as a possible solution.</p>

python database multithreading