Pandas, Concurrent.Futures and the GIL
<p>I'm writing code using Pandas 0.18/Python 3.5 on an intel i3 (four cores).</p>

<p>I have read this:
<a href="https://www.continuum.io/content/pandas-releasing-gil" rel="nofollow">https://www.continuum.io/content/pandas-releasing-gil</a></p>

<p>I also have some work that is IO bound (parsing CSV files into dataframes).
I have to do a lot of calculation that is mostly multiplying dataframes.</p>

<p>My code is currently parallel using <code>concurrent.futures ThreadPoolExecutor</code>.</p>

<p>My question is:</p>

<ul>
<li>In general, should I be using threads to run pandas jobs in parallel, or does pandas make effective use of all cores without me having to explicitly tell it to? (in which case, I will execute my jobs serially).</li>
</ul>

<p>Best I can tell from reading the docs, pandas <a href="http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#releasing-the-gil" rel="nofollow">simply releases the GIL for certain operations</a>:</p>

<blockquote>
  <p>We are releasing the global-interpreter-lock (GIL) on some cython
  operations. This will allow other threads to run simultaneously during
  computation, potentially allowing performance improvements from
  multi-threading. Notably <code>groupby</code>, <code>nsmallest</code>, <code>value_counts</code> and some
  indexing operations benefit from this.</p>
</blockquote>

<p>All this means is that other threads can be executed by the Python interpreter while the calculations being one by pandas continue. It doesn't mean that pandas automatically scales the calculations across many threads. They sort of mention this in the docs as well:</p>

<blockquote>
  <p>Releasing of the GIL could benefit an application that uses threads
  for user interactions (e.g. QT), or performing multi-threaded
  computations.</p>
</blockquote>

<p>In order to get parallelization benefits, you need to actually be creating and executing multiple threads in your own code. So, you should continue using the <code>ThreadPoolExecutor</code> if you're trying to get parallel execution in your application.</p>

<p>Keep in mind that pandas is only releasing the GIL for <em>some</em> operations, so you may not get performance improvements with multiple threads if you're not calling any methods that actually release it.</p>

python multithreading pandas python-multithreading