Convert iteration to map
<p>I've got the following code:</p>

<pre><code>@classmethod
def load(self):
    with open('yaml/instruments.yaml', 'r') as ymlfile:
        return {v['name']: Instrument(**v) for (v) in list(yaml.load_all(ymlfile))}
</code></pre>

<p>I'd like to load these in parallel using something like:</p>

<pre><code>return ThreadPoolExecutor.map(Instrument, list(yaml.load_all(ymlfile))
</code></pre>

<p>But I'm not quite sure how to get the parameters to pass.</p>

<p>Here's an example of instruments.yaml:</p>

<pre><code>---
    name: 'corn'
    #Trade December corn only
    multiplier: 5000
    contract_prefix: 'C'
    months_traded: [3, 5, 7, 9, 12]
    quandl: 'CHRIS/CME_C2'
    first_contract: 196003
    backtest_from: 199312
    trade_only: [12]
    contract_name_prefix: 'C'
    quandl_database: 'CME'
    slippage: 0.125 #half the spread
    ib_code: 'ZC'
</code></pre>

<p>How do I refactor my code as a map so I can use ThreadPoolExecutor?</p>

<p>Simple solution is to define a top level simple worker function for use in the executor:</p>

<pre><code>def make_instrument_pair(d):
    return d['name'], Instrument(**d)
</code></pre>

<p>Then change:</p>

<pre><code>@classmethod
def load(self):
    with open('yaml/instruments.yaml', 'r') as ymlfile:
        return {v['name']: Instrument(**v) for (v) in list(yaml.load_all(ymlfile))}
</code></pre>

<p>to:</p>

<pre><code>@classmethod
def load(self):
    with open('yaml/instruments.yaml') as ymlfile,\
         concurrent.futures.ThreadPoolExecutor(8) as executor:
        return dict(executor.map(make_instrument_pair, yaml.load_all(ymlfile)))
</code></pre>

<p>As I noted in my comments, this probably won't gain you anything; <a href="https://wiki.python.org/moin/GIL" rel="nofollow">the GIL</a> means that threads don't improve performance unless:</p>

<ol>
<li>The work is done in third party C extensions that explicitly release the GIL before doing a lot of C level work</li>
<li>The work is mostly I/O bound (or otherwise spends most of its time blocking in some way, whether it's sleeping, waiting on locks, etc.)</li>
</ol>

<p>Unless <code>Instrument</code> is really expensive to construct, even using <code>ProcessPoolExecutor</code> likely won't help; you need to do a meaningful amount of work in the tasks dispatched, or you're wasting more time on task management (and for processes, serialization and interprocess communication) than you gain in parallelism.</p>

python python-3.x