Error with OMP_NUM_THREADS when using dask distributed
<p>I am using <a href="http://distributed.readthedocs.io/en/latest/" rel="nofollow">distributed</a>, a framework to allow parallel computation. In this, my primary use case is with NumPy. When I include NumPy code that relies on <code>np.linalg</code>, I get an error with <code>OMP_NUM_THREADS</code>, which is related to the <a href="https://en.wikipedia.org/wiki/OpenMP" rel="nofollow">OpenMP library</a>.</p>

<p>An minimal example:</p>

<pre><code>from distributed import Executor
import numpy as np
e = Executor('144.92.142.192:8786')

def f(x, m=200, n=1000):
    A = np.random.randn(m, n)
    x = np.random.randn(n)
    #  return np.fft.fft(x)  # tested; no errors
    #  return np.random.randn(n)  # tested; no errors
    return A.dot(y).sum()  # tested; throws error below

s = [e.submit(f, x) for x in [1, 2, 3, 4]]
s = e.gather(s)
</code></pre>

<p>When I test with the linalg test, <code>e.gather</code> fails as each job throws the following error:</p>

<pre><code>OMP: Error #34: System unable to allocate necessary resources for OMP thread:
OMP: System error #11: Resource temporarily unavailable
OMP: Hint: Try decreasing the value of OMP_NUM_THREADS.
</code></pre>

<p>What should I set <code>OMP_NUM_THREADS</code> to?</p>

<h3>Short answer</h3>

<pre><code>export OMP_NUM_THREADS=1

or 

dask-worker --nthreads 1
</code></pre>

<h3>Explanation</h3>

<p>The <code>OMP_NUM_THREADS</code> environment variable controls the number of threads that many libraries, including the <code>BLAS</code> library powering <code>numpy.dot</code>, use in their computations, like matrix multiply.</p>

<p>The conflict here is that you have two parallel libraries that are calling each other, BLAS, and dask.distributed.  Each library is designed to use as many threads as there are logical cores available in the system.  </p>

<p>For example if you had eight cores then dask.distributed might run your function <code>f</code> eight times at once on different threads.  The <code>numpy.dot</code> function call within <code>f</code> would use eight threads per call, resulting in 64 threads running at once. </p>

<p>This is actually fine, you'll experience a performance hit but everything can run correctly, but it will be slower than if you use just eight threads at a time, either by limiting dask.distributed or by limiting BLAS.</p>

<p>Your system probably has <code>OMP_THREAD_LIMIT</code> set at some reasonable number like 16 to warn you of this event when it happens.</p>

python numpy cluster-computing dask