Cython code runs 125x slower when compiled against python 2 vs python 3
<p>I have a big block of Cython code that is parsing <a href="http://ibis.org/interconnect_wip/touchstone_spec2_draft3.pdf" rel="nofollow">Touchstone files</a> that I want to work with Python 2 and Python 3.  I'm using very C-style parsing techniques for what I thought would be maximum efficiency, including manually malloc-ing and free-ing <code>char*</code> instead of using <code>bytes</code> so that I can avoid the GIL.  When compiled using</p>

<pre><code>python        3.5.2             0    anaconda
cython        0.24.1       py35_0    anaconda
</code></pre>

<p>I see speeds that I'm happy with, a moderate boost on small files (~20% faster) and a huge boost on large files (~2.5<strong>x</strong> faster).  When compiled against</p>

<pre><code>python        2.7.12            0    anaconda
cython        0.24.1       py27_0    anaconda
</code></pre>

<p>It runs about 125x slower (~17ms in Python 3 vs ~2.2s in Python 2).  It's the exact same code compiled in different environments using a pretty simple <code>setuputils</code> script.  I'm not currently using NumPy from Cython for any of the parsing or data storage.</p>

<pre><code>import cython
cimport cython

from cython cimport array
import array

from libc.stdlib cimport strtod, malloc, free
from libc.string cimport memcpy

ctypedef long long int64_t  # Really VS2008? Couldn't include this by default?

# Bunch of definitions and utility functions omitted

@cython.boundscheck(False)
cpdef Touchstone parse_touchstone(bytes file_contents, int num_ports):
    cdef:
        char c
        char* buffer = &lt;char*&gt; file_contents
        int64_t length_of_buffer = len(file_contents)
        int64_t i = 0

        # These are some cpdef enums
        FreqUnits freq_units
        Domain domain
        Format fmt
        double z0
        bint option_line_found = 0

        array.array data = array.array('d')
        array.array row = array.array('d', [0 for _ in range(row_size)])

    while i &lt; length_of_buffer:
        c = buffer[i]  # cdef char c
        if is_whitespace(c):
            i += 1
            continue

        if is_comment_char(c):
            # Returns the last index of the comment
            i = parse_comment(buffer, length_of_buffer)
            continue

        if not option_line_found and is_option_leader_char(c):
            # Returns the last index of the option line
            # assigns values of all references passed in
            i = parse_option_line(
                buffer, length_of_buffer, i,
                &amp;domain, &amp;fmt, &amp;z0, &amp;freq_units)
            if i &lt; 0:
                # Lots of boring code along the lines of
                # if i == some_int:
                #     raise Exception("message")
                # I did this so that only my top-level parse has to interact
                # with the interpreter, all the lower level functions have nogil
            option_line_found = 1

        if option_line_found:
            if is_digit(c):
                # Parse a float
                row[row_idx] = strtod(buffer + i, &amp;end_of_value)
                # Jump the cursor to the end of that float
                i = end_of_value - p - 1
                row_idx += 1
                if row_idx == row_size:
                    # append this row onto the main data array
                    data.extend(row)
                    row_idx = 0

        i += 1

    return Touchstone(num_ports, domain, fmt, z0, freq_units, data)
</code></pre>

<p>I've ruled out a few things, such as type casts.  I also tested where the code simply loops over the entire file doing nothing.  Either Cython optimized that away or it's just really fast because it causes <code>parse_touchstone</code> to not even show up in a <code>cProfile/pstats</code> report.  I determined that it's not just the comment, whitespace, and option line parsing (not shown is the significantly more complicated keyword-value parsing) after I threw in a print statement in the last <code>if row_idx == row_size</code> block to print out a status and discovered that it's taking about 0.5-1 second (guesstimate) to parse a row with 512 floating point numbers on it.  That really should not take so long, especially when using <code>strtod</code> to do the parsing.  I also checked parsing just 2 rows' worth of values then jumping out of the while loop and it told me that parsing the comments, whitespace, and option line took up about 800ms (1/3 of the overall time), and that was for 6 lines of text totaling less than 150 bytes.</p>

<p>Am I just missing something here?  Is there a small trick that would cause Cython code to run 3 orders of magnitude slower in Python 2 than Python 3?</p>

<p>(Note: I haven't shown the full code here because I'm not sure if I'm allowed to for legal reasons and because it's about 450 lines total)</p>

<p>The problem is with <code>strtod</code>, which is not <a href="http://www.ryanjuckett.com/programming/optimizing-atof-and-strtod/" rel="nofollow">optimized in VS2008</a>.  Apparently it internally calculates the length of the input string each time its called, and if you call it with a long string this will slow down your code considerably.  To circumvent this you have to write a wrapper around <code>strtod</code> to use only small buffers at a time (see the above link for one example of how to do this) or write your own <code>strtod</code> function.</p>

python python-2.7 cython