Concurrent download and processing of large files in python
<p>I have a list of URLs for large files to <strong>download</strong> (e.g. compressed archives), which I want to <strong>process</strong> (e.g. decompress the archives). </p>

<p>Both download and processing take a long time and processing is heavy on disk IO, so I want to have <strong>just one of each to run at a time</strong>. Since the two tasks take about the same time and do not compete for the same resources, I want to download the next file(s) while the last is being processed.</p>

<p>This is a variation of the <strong><a href="https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem" rel="nofollow">producer-consumer problem</a></strong>.</p>

<p>The situation is similar to <a href="http://stackoverflow.com/q/12474182/512111">reading and processing images</a> or <a href="http://stackoverflow.com/questions/37825218/fastest-way-to-read-and-process-100-000-urls-in-python">downloading loads of files</a>, but my downloader calls are not (yet) picklable, so I have not been able to use multiprocessing, and both tasks take about the same time.</p>

<p>Here is a dummy example, where both download and processing are blocking:</p>

<pre><code>import time
import posixpath

def download(urls):
    for url in urls:
        time.sleep(3)  # this is the download (more like 1000s) 
        yield posixpath.basename(url)

def process(fname):
    time.sleep(2)  # this is the processing part (more like 600s)

urls = ['a', 'b', 'c']
for fname in download(urls):
    process(fname)
    print(fname)
</code></pre>

<p>How could I make the two tasks concurrent? Can I use <code>yield</code> or <code>yield from</code> <a href="http://stackoverflow.com/questions/9708902/in-practice-what-are-the-main-uses-for-the-new-yield-from-syntax-in-python-3">in a smart way</a>, perhaps in combination with <a href="https://docs.python.org/2/library/collections.html#collections.deque" rel="nofollow"><code>deque</code></a>? Or must it be <a href="https://docs.python.org/3.4/library/asyncio.html" rel="nofollow"><code>asyncio</code></a> with <code>Future</code>?</p>

<p>I'd simply use <code>threading.Thread(target=process, args=(fname,))</code> and start a new thread for processing.</p>

<p>But before that, end last processing thread :</p>

<pre><code>t = None
for fname in download(urls):
    if t is not None: # wait for last processing thread to end
        t.join()
    t = threading.Thread(target=process, args=(fname,))
    t.start()
    print('[i] thread started for %s' % fname)
</code></pre>

<p>See <a href="https://docs.python.org/3/library/threading.html" rel="nofollow">https://docs.python.org/3/library/threading.html</a></p>

python concurrency yield coroutine yield-from