Data transfer between C++ and Python
<p>I would like to share memory between C++ and Python.</p>

<p>My problem:</p>

<ol>
<li>I am working with big data sets (up to 6 GB of RAM) in C++. All calculations are done in c++.</li>
<li>Then, I want to "paste" all my results to a Python program. But I can only write my data on disk, and then read that file from Python, which is not efficient. </li>
</ol>

<p>Is there any way to "map" memory corresponding to C++ variables so that I may access the data from Python? I don't want to copy 6GB of data onto a hard drive. </p>

<p><strong>First path</strong>: I think the more appropriate way for you to go is <a href="https://docs.python.org/3.4/library/ctypes.html" rel="nofollow">ctypes</a>. You can create a shared library, and then load the functions of the shared library in Python, and fill all the data containers you want in Python.</p>

<p>In Windows, you can create a DLL, and in Linux you can create a shared .so library.</p>

<p>Now this has the advantage that this will be independent of your Python version.</p>

<p><strong>Second path</strong>: I think it's less appropriate but you can get it to work, which is the <a href="https://docs.python.org/3.4/extending/extending.html" rel="nofollow">Python C Extension</a>. With this, you can call Python data containers (<code>PyObject</code>s) and fill them inside C.</p>

<p>However, the code you compile here will always need to be linked to Python libraries.</p>

<p><strong>Which one to use?</strong>: </p>

<ul>
<li>Use ctypes if you have some functions you want to call in C/C++, and then do the rest of the work in Python.</li>
<li>Use Python C Extension if you have some functions you want to call in Python, and you want to do the rest in C/C++.</li>
</ul>

<p>With both options, you can transfer huge blocks of memory between C++ and Python without necessarily involving any disk read/write operations.</p>

<p>Hope this helps.</p>

python c++ data-transfer