Initialization of very big vector in C++
<p>I created very big O(10M) floating point list in python. I would like to use this lookup table in my C++ project. What is the easiest and the most efficient way to transfer this array from python to C++. </p>

<p>My first idea was to generate c++ function, which is responsible for  initializations of such long vector and then compile it. 
The python code looks like above: </p>

<pre><code>def generate_initLookupTable_function():
    numbers_per_row = 100
    function_body = """
#include "PatBBDTSeedClassifier.h"

std::vector&lt;double&gt; PatBBDTSeedClassifier::initLookupTable()
{
   std::vector&lt;double&gt; indicesVector ={
    """
    row_nb = 1
    for bin_value in classifier._lookup_table[:,0]:
        function_body += "\t" + str(bin_value) +" , "
        if (row_nb % numbers_per_row) == 0:
            function_body += "\n"
        row_nb += 1

    function_body += """\n };
return indicesVector;
}
    """
    return function_body
</code></pre>

<p>The output file has size of 500 MB. And there is not possible to compile it (compilation terminated due to gcc crash):</p>

<pre><code>../src/PatBBDTSeedClassifier_lookupTable.cpp
lcg-g++-4.9.3: internal compiler error: Killed (program cc1plus)

0x409edc execute
../../gcc-4.9.3/gcc/gcc.c:2854
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
</code></pre>

<p>The another idea is to store python array into binary file and then read it in C++. But this is  tricky. I cannot properly read it.
I generate the table using such simple command: </p>

<pre><code>file = open("models/BBDT_lookuptable.dat", 'wb')
table = numpy.array(classifier._lookup_table[:,0])
table.tofile(file)
file.close()
</code></pre>

<p>Can you tell me how can I do it? I googled SO and I could't find andy sufficient answer. </p>

<p>Do you have any idea how can I deal with such big arrays. </p>

<p>I should have give you more detailed description of the problem. 
I use python to train the ML (sklearn) classifier and then I would like to deploy it in C++. Doe to timing issue (execution speed is a crucial part of my study) I use idea of <a href="http://arxiv.org/abs/1210.6861" rel="nofollow">bonsai boosted decision trees</a>. In this approach you transfer BDT into lookup table. </p>

<p>As you have noticed, the compiler crashes on such big data arrays.</p>

<p>What you can do besides of reading a binary file (since you don't want to do that) is to link with an assembly file. It still makes the executable self-sufficient and GAS is much more tolerant to big files. Here's an example of some asm file I generated using python and which assembles fine with classical <code>gcc</code>:</p>

<pre><code>.section .rodata
.globl FT
.globl FT_end
FT:
.byte  0x46,0x54,0x5f,0x43,0x46,0x0,0x0,0x0,0x0,0x0,0x0,0x3,0x43,0x4f,0x4d,0x50
.byte  0x32,0x30,0x31,0x0,0x3,0x88,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0
.byte  0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0
.byte  0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x28,0xe6,0x47,0x6,0x7,0x8,0x28,0x28
.byte  0x26,0x6,0x2a,0x6,0x6,0x40,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0
FT_end:
</code></pre>

<p>That technique allowed me to embed a 80 Megabyte binary file (1 million code lines) in an executable because I had no filesystem to read data files in that environment (QEMU)</p>

<p>Real test with the python code I finally could dig out:</p>

<p>Python code:</p>

<pre><code>floats = [0.12,0.45,0.34,4.567,22.7]

import struct
contents = struct.pack('f'*len(floats), *floats)

outbase = "_extref"
output_file = "data.s"

fw = open(output_file,"w")
fw.write(""".section .rodata
.globl {0}
.globl {0}_end
{0}:
""".format(outbase,outbase))
eof = False
current_offset = 0

while not eof:
    to_write = []
    if current_offset==len(contents):
        break
    if current_offset&lt;len(contents):
        fw.write(".byte  ")
        for i in range(0,16):
            if current_offset&lt;len(contents):
                to_write.append(hex(ord(contents[current_offset])))
                current_offset+=1
            else:
                eof = True
                break
        if len(to_write)&gt;0:
            fw.write(",".join(to_write)+"\n")

fw.write(outbase+"_end:\n")
fw.close()
</code></pre>

<p><code>test.cpp</code>: C++ code (C++11, I struggled for the pointer references to the asm part):</p>

<pre><code>#include &lt;iostream&gt;
#include &lt;vector&gt;

#include &lt;strings.h&gt;

extern const float extref;
extern const float extref_end;

int main()
{
    int size = (&amp;extref_end - &amp;extref);
    std::cout &lt;&lt; "nb_elements: " &lt;&lt; size &lt;&lt; std::endl;
    std::vector&lt;float&gt; v(size);
    memcpy(&amp;v[0],&amp;extref,sizeof(float)*size);

    for (auto it : v)
    {
      std::cout &lt;&lt; it &lt;&lt; std::endl;
    }
    return 0;

}
</code></pre>

<p>Python code generates a <code>data.s</code> file. Create an executable with:</p>

<pre><code>g++ -std=c++11  test.cpp data.s
</code></pre>

<p>run:</p>

<pre><code>nb_elements: 5
0.12
0.45
0.34
4.567
22.7
</code></pre>

<p>The main advantage of this method is that you can define as many symbols as you want, with the format that you want.</p>

python c++ arrays