C++ uses twice the memory when moving elements from one dequeue to another
<p>In my project, I use <a href="https://github.com/pybind/pybind11" rel="nofollow">pybind11</a> to bind C++ code to Python. Recently I have had to deal with very large data sets (70GB+) and encountered need to split data from one <code>std::deque</code> between multiple <code>std::deque</code>'s. Since my dataset is so large, I expect the split not to have much of memory overhead. Therefore I went for one pop - one push strategy, which in general should ensure that my requirements are met. </p>

<p>That is all in theory. In practice, my process got killed. So I struggled for past two days and eventually came up with following minimal example demonstrating the problem.</p>

<p>Generally the minimal example creates bunch of data in <code>deque</code> (~11GB), returns it to Python, then calls again to <code>C++</code> to move the elements. Simple as that. Moving part is done in executor.</p>

<p>The interesting thing is, that if I don't use executor, memory usage is as expected and also when limits on virtual memory by ulimit are imposed, the program really respects these limits and doesn't crash.</p>

<p><strong>test.py</strong></p>

<pre><code>from test import _test
import asyncio
import concurrent

async def test_main(loop, executor):
    numbers = _test.generate()
    # moved_numbers = _test.move(numbers) # This works!
    moved_numbers = await loop.run_in_executor(executor, _test.move, numbers) # This doesn't!

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    executor = concurrent.futures.ThreadPoolExecutor(1)

    task = loop.create_task(test_main(loop, executor))
    loop.run_until_complete(task)

    executor.shutdown()
    loop.close()
</code></pre>

<p><strong>test.cpp</strong></p>

<pre><code>#include &lt;deque&gt;
#include &lt;iostream&gt;
#include &lt;pybind11/pybind11.h&gt;
#include &lt;pybind11/stl.h&gt;

namespace py = pybind11;

PYBIND11_MAKE_OPAQUE(std::deque&lt;uint64_t&gt;);
PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr&lt;T&gt;);

template&lt;class T&gt;
void py_bind_opaque_deque(py::module&amp; m, const char* type_name) {
    py::class_&lt;std::deque&lt;T&gt;, std::shared_ptr&lt;std::deque&lt;T&gt;&gt;&gt;(m, type_name)
    .def(py::init&lt;&gt;())
    .def(py::init&lt;size_t, T&gt;());
}

PYBIND11_PLUGIN(_test) {
    namespace py = pybind11;
    pybind11::module m("_test");
    py_bind_opaque_deque&lt;uint64_t&gt;(m, "NumbersDequeue");

    // Generate ~11Gb of data.
    m.def("generate", []() {
        std::deque&lt;uint64_t&gt; numbers;
        for (uint64_t i = 0; i &lt; 1500 * 1000000; ++i) {
            numbers.push_back(i);
        }
        return numbers;
    });

    // Move data from one dequeue to another.
    m.def("move", [](std::deque&lt;uint64_t&gt;&amp; numbers) {
        std::deque&lt;uint64_t&gt; numbers_moved;

        while (!numbers.empty()) {
            numbers_moved.push_back(std::move(numbers.back()));
            numbers.pop_back();
        }
        std::cout &lt;&lt; "Done!\n";
        return numbers_moved;
    });

    return m.ptr();
}
</code></pre>

<p><strong>test/__init__.py</strong></p>

<pre><code>import warnings
warnings.simplefilter("default")
</code></pre>

<p><strong>Compilation</strong>:</p>

<pre><code>g++ -std=c++14 -O2 -march=native -fPIC -Iextern/pybind11 `python3.5-config --includes` `python3.5-config --ldflags` `python3.5-config --libs` -shared -o test/_test.so test.cpp
</code></pre>

<p><strong>Observations:</strong></p>

<ul>
<li>When the moving part is not done by executor, so we just call <code>moved_numbers = _test.move(numbers)</code>, all works as expected, memory usage showed by htop stays around <code>11Gb</code>, great!.</li>
<li>When moving part is done in executor, the program takes double the memory and crashes.</li>
<li><p>When limits on virtual memory are introduced (~15Gb), all works fine, which is probably the most interesting part.</p>

<p><code>ulimit -Sv 15000000 &amp;&amp; python3.5 test.py</code> >> <code>Done!</code>.</p></li>
<li><p>When we increase the limit the program crashes (150Gb > my RAM).</p>

<p><code>ulimit -Sv 150000000 &amp;&amp; python3.5 test.py</code> >> <code>[1]    2573 killed     python3.5 test.py</code></p></li>
<li><p>Usage of deque method <code>shrink_to_fit</code> doesn't help (And nor it should)</p></li>
</ul>

<p><strong>Used software</strong></p>

<pre><code>Ubuntu 14.04
gcc version 5.4.1 20160904 (Ubuntu 5.4.1-2ubuntu1~14.04)
Python 3.5.2
pybind11 latest release - v1.8.1
</code></pre>

<p><strong>Note</strong></p>

<p>Please note that this example was made merely to demonstrate the problem. Usage of <code>asyncio</code> and <code>pybind</code> is necessary for problem to occur.  </p>

<p>Any ideas on what might be going on are most welcomed.</p>

<p>The problem turned out to be caused by Data being created in one thread and then deallocated in another one. It is so because of malloc arenas in glibc <a href="https://siddhesh.in/posts/malloc-per-thread-arenas-in-glibc.html" rel="nofollow">(for reference see this)</a>. It can be nicely demonstrated by doing:</p>

<pre><code>executor1 = concurrent.futures.ThreadPoolExecutor(1)
executor2 = concurrent.futures.ThreadPoolExecutor(1)

numbers = await loop.run_in_executor(executor1, _test.generate)
moved_numbers = await loop.run_in_executor(executor2, _test.move, numbers)
</code></pre>

<p>which would take twice the memory allocated by <code>_test.generate</code> and</p>

<pre><code>executor = concurrent.futures.ThreadPoolExecutor(1)

numbers = await loop.run_in_executor(executor, _test.generate)
moved_numbers = await loop.run_in_executor(executor, _test.move, numbers)
</code></pre>

<p>which wound't.</p>

<p>This issue can be solved either by rewriting the code so it doesn't move the elements from one container to another (my case) or by setting environment variable <code>export MALLOC_ARENA_MAX=1</code> which will limit number of malloc arenas to 1. This however might have some performance implications involved (There is a good reason for having multiple arenas).</p>

python c++ out-of-memory python-asyncio pybind11