Why doesn't the numpy-C api warn me about failed allocations?
<p>I've been writing a python extension that writes into a numpy array from C. During testing, I noticed that certain very large arrays would generate a segfault when I tried to access some of their elements. Specifically, the last line of the following code segment fails with a SEGFAULT:</p>

<pre><code>    // Size of buffer we will write to
    npy_intp buffer_len_alt = BUFFER_LENGTH;

    //
    PyArray_Descr * dtype;
    dtype = PyArray_DescrFromType(NPY_BYTE);
    PyObject* column = PyArray_Zeros(1, &amp;buffer_len_alt, dtype, 0);

    //Check that array creation succeeds
    if (column == NULL){
        // This exit point is not reached, so it looks like everything is OK
        return (PyObject *) NULL;
    }

    // Get the array's internal buffer so we can write to it
    output_buffer = PyArray_BYTES((PyArrayObject *)column);

    // Try writing to the buffer
    output_buffer[0] = 'x'; //No segfault
    output_buffer[((int) buffer_len_alt) - 1] = 'x'; //Segfault here
</code></pre>

<p>I checked and found that the error occurs only when I try to allocate an array of about 3GB (i.e. BUFFER_LENGTH is about 3*2^30). It's <a href="https://en.wikipedia.org/wiki/3_GB_barrier" rel="nofollow">not surprising</a> that an allocation of this size would fail, even if python is using it's custom allocator. What really concerns me is that <strong>numpy did not raise an error or otherwise indicate that the array creation did not go as planned</strong>.</p>

<p>I have already tried checking PyArray_ISCONTIGUOUS on the returned array, and using PyArray_GETCONTIGUOUS to ensure it is a single memory segment, but the SEGFAULT would still occur. NPY_ARRAY_DEFAULT creates contiguous arrays, so this shouldn't be necessary anyways.</p>

<p><strong>Is there some error flag I should be checking? How can I detect/prevent this situation in the future?</strong> Setting BUFFER_LENGTH to a smaller value obviously works, but this value is determined at runtime and I would like to know the exact bounds.</p>

<p><strong>EDIT</strong>:</p>

<p>As @DavidW pointed out, the error stems from casting buffer_len_alt to an int, since npy_intp can be a 64-bit number. Replacing the cast to int with a cast to 'unsigned long' fixes the problem for me</p>

<p>The issue (diagnosed in the comments) was actually with the array lookup rather than the allocation of the array.  Your code contained the line</p>

<pre><code>output_buffer[((int) buffer_len_alt) - 1] = 'x'
</code></pre>

<p>When <code>buffer_len_alt</code> (approx value 3000000000) was cast to an (32 bit) int (maximum value 2147483647) you ended up with an invalid address, probably a large negative number.</p>

<p>The solution is just to use</p>

<pre><code>output_buffer[buffer_len_alt - 1] = 'x'
</code></pre>

<p>(i.e. I don't see why you should need a cast at all).</p>

python c numpy python-c-api