Python - Merge data from multiple thread instances
<p>I am currently working on a project that involves connecting two devices to a python script, retrieving data from them and outputting the data. </p>

<p>Code outline:</p>

<p>â¢   Scans for paired devices</p>

<p>â¢   Paired device found creates thread instance (Two devices connected = two thread instances )</p>

<p>â¢   Data is printed within the thread i.e. each instance has a separate 
bundle of data</p>

<p>Basically when two devices are connected two instances of my thread class is created. Each thread instance returns a different bundle of data.</p>

<p>My question is: Is there a way I can combine the two bundles of data into one bundle of data?</p>

<p>Any help on this is appreciated :)</p>

<p>I assume you are using the <code>threading</code> module.</p>

<h2>Threading in Python</h2>

<p>Python is not multithreaded for CPU activity. The interpreter still uses a GIL (Global Interpreter Lock) for most operations and therefore linearizing operations in a python script. Threading is good to do IO however, as other threads can be woken up while a thread waits for IO.</p>

<h2>Idea</h2>

<p>Because of the GIL we can just use a standard list to combine our data. The idea is to pass the same list or dictionary to every Thread we create using the <code>args</code> parameter. See <a href="https://docs.python.org/2/library/threading.html" rel="nofollow">pydoc for threading</a>.</p>

<p>Our simple implementation uses two Threads to show how it can be done. In real-world applications you probably use a Thread group or something similar..</p>

<h2>Implementation</h2>

<pre><code>def worker(data):
    # retrieve data from device
    data.append(1)
    data.append(2)
l = []
# Let's pass our list to the target via args.
a = Thread(target=worker, args=(l,))
b = Thread(target=worker, args=(l,))
# Start our threads
a.start()
b.start()
# Join them and print result
a.join()
b.join()
print(l)
</code></pre>

<h2>Further thoughts</h2>

<p>If you want to be 100% correct and don't rely on the GIL to linearize access to your list, you can use a simple mutex to lock and unlock or use the <a href="https://docs.python.org/2/library/queue.html#module-Queue" rel="nofollow">Queue module</a> which implements correct locking.</p>

<p>Depending on the nature of the data a dictionary might be more convenient to join data by certain keys.</p>

<h2>Other considerations</h2>

<p>Threads should be carefully considered. Alternatives such as <code>asyncio</code>, etc might be better suited.</p>

python multithreading