Initialization of very big vector in C++
<p>I created very big O(10M) floating point list in python. I would like to use this lookup table in my C++ project. What is the easiest and the most efficient way to transfer this array from python to C++. </p>

<p>My first idea was to generate c++ function, which is responsible for  initializations of such long vector and then compile it. 
The python code looks like above: </p>

<pre><code>def generate_initLookupTable_function():
    numbers_per_row = 100
    function_body = """
#include "PatBBDTSeedClassifier.h"

std::vector&lt;double&gt; PatBBDTSeedClassifier::initLookupTable()
{
   std::vector&lt;double&gt; indicesVector ={
    """
    row_nb = 1
    for bin_value in classifier._lookup_table[:,0]:
        function_body += "\t" + str(bin_value) +" , "
        if (row_nb % numbers_per_row) == 0:
            function_body += "\n"
        row_nb += 1

    function_body += """\n };
return indicesVector;
}
    """
    return function_body
</code></pre>

<p>The output file has size of 500 MB. And there is not possible to compile it (compilation terminated due to gcc crash):</p>

<pre><code>../src/PatBBDTSeedClassifier_lookupTable.cpp
lcg-g++-4.9.3: internal compiler error: Killed (program cc1plus)

0x409edc execute
../../gcc-4.9.3/gcc/gcc.c:2854
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
</code></pre>

<p>The another idea is to store python array into binary file and then read it in C++. But this is  tricky. I cannot properly read it.
I generate the table using such simple command: </p>

<pre><code>file = open("models/BBDT_lookuptable.dat", 'wb')
table = numpy.array(classifier._lookup_table[:,0])
table.tofile(file)
file.close()
</code></pre>

<p>Can you tell me how can I do it? I googled SO and I could't find andy sufficient answer. </p>

<p>Do you have any idea how can I deal with such big arrays. </p>

<p>I should have give you more detailed description of the problem. 
I use python to train the ML (sklearn) classifier and then I would like to deploy it in C++. Doe to timing issue (execution speed is a crucial part of my study) I use idea of <a href="http://arxiv.org/abs/1210.6861" rel="nofollow">bonsai boosted decision trees</a>. In this approach you transfer BDT into lookup table. </p>

<p>If you're using GNU tools it is rather easy to directly use <code>objcopy</code> to achieve that which is suggested by Jean-Francois; combining with python script of PM2Ring which writes a binary array, you can execute:</p>

<pre><code>objcopy -I binary test.data -B i386:x86-64 -O elf64-x86-64 testdata.o
</code></pre>

<p>(depending on your actual processor architecture, you might need to adjust). The command will create a new object named <code>testdata.o</code> with the following symbols:</p>

<pre><code>0000000000000100 D _binary_test_data_end
0000000000000100 A _binary_test_data_size
0000000000000000 D _binary_test_data_start
</code></pre>

<p>All these symbols will be visible as symbols with C linkage in the linked program. The <code>size</code> is not usable as such (it will be converted to an address as well), but the <code>*start</code> and <code>*end</code> can be used. Here is a minimal C++ program:</p>

<pre><code>#include &lt;iostream&gt;

extern "C" double _binary_test_data_start[];
extern "C" double _binary_test_data_end[0];

int main(void) {
    double *d = _binary_test_data_start;
    const double *end = _binary_test_data_end;

    std::cout &lt;&lt; (end - d) &lt;&lt; " doubles in total" &lt;&lt; std::endl;
    while (d &lt; end) {
        std::cout &lt;&lt; *d++ &lt;&lt; std::endl;
    }
}
</code></pre>

<p>The <code>_binary_test_data_end</code> will actually be just past the last element in the array <code>_binary_test_data_start</code>.</p>

<p>Compile + link this program with <code>g++ test.cc testdata.o -o program</code> (using the testdata.o from objcopy above).</p>

<p>Output (<code>cout</code> by default seems to truncate the decimals awkwardly): </p>

<pre><code>% ./a.out 
32 doubles in total
0
0.0625
0.125
0.1875
0.25
0.3125
0.375
0.4375
0.5
0.5625
0.625
0.6875
0.75
0.8125
0.875
0.9375
1
1.0625
1.125
1.1875
1.25
1.3125
1.375
1.4375
1.5
1.5625
1.625
1.6875
1.75
1.8125
1.875
1.9375
</code></pre>

<hr>

<p>You can also assign these values into a vector very easily; <code>std::vector&lt;double&gt;</code> accepts 2 iterators, where first points to the first element, and second to just one after; you can use the arrays here as they decay into pointers, and pointers can be used as iterators:</p>

<pre><code>std::vector&lt;double&gt; vec(_binary_test_data_start, _binary_test_data_end);
</code></pre>

<p>However, for big arrays, this is just needless copying. Also, using just the C array has the added benefit that it is <em>lazily loaded</em>; ELF executables are not read into memory, but they're paged in as needed; the binary array is loaded from file into RAM only as it is accessed.</p>

python c++ arrays