GAE Python code MUCH slower in production than locally
<p>In my Python GAE app, the following snippet of code is MUCH slower in production than when run locally.  The processing goes like this:</p>

<ol>
<li>A text file of about 1 MB is loaded in a POST.  Each line of the text file is an "item".</li>
<li>My code creates a list of items from the text file and checks for duplicates and validity (by comparing against a compiled RE).</li>
</ol>

<p>Here is the code: </p>

<pre><code>def process_items(self, text):
    item_list = text.split()
    item_set = set()
    n_valid = 0
    n_invalid = 0
    n_dups = 0
    out = ""
    for item in item_list:
        if item in item_set:
            n_dups += 1
            out += "DUPLICATE: %s\n" % item
        elif valid_item(item): # This compares against a compiled RE
            item_set.add(item)
            n_valid += 1
            out += "%s\n" % item
        else:
            n_invalid += 1
            out += "INVALID: %s\n" % item
    return out
</code></pre>

<p>When I run this on the local dev server, a 1MB file of 50,000 lines takes 5 seconds to process.</p>

<p>When I run this in production, the same file takes over a minute and the request times out.  The file upload only takes about a second so I know the bottle neck is the above code.</p>

<p>In the past, production code was about the same speed as my local code.  I don't think this code has changed, so I suspect there may have been a change on Google's end.</p>

<p>Any idea why this code is now much slower in production?
Anything I can do to make this code faster?  I need to return an annotated file to the user that indicates which lines are duplicates and which lines are invalid.</p>

<p>EDIT:</p>

<p>In response to mgilson's comment, I tried the following code, and it made a huge difference in execution time!  The processing that previously timed out after a minute now takes only about 5 seconds.  GAE is still slower than expected (even accounting the relatively slow server CPUs), but with the improved algorithm, it doesn't matter for me now.</p>

<pre><code>def process_items(self, text):
    item_list = text.split()
    item_set = set()
    n_valid = 0
    n_invalid = 0
    n_dups = 0
    for i, item in enumerate(item_list):
        item = item.strip()
        if item in item_set:
            n_dups += 1
            item_list[i] = "DUPLICATE: %s" % item
        elif valid_item(item): # This compares against a compiled RE
            item_set.add(item)
            n_valid += 1
            item_list[i] = item
        else:
            n_invalid += 1
            item_list[i] = "INVALID: %s" % item
    return "\n".join(item_list)
</code></pre>

<p>It's not at all unexpected that GAE production would run slower than locally -- Depending on your <a href="https://cloud.google.com/appengine/docs/about-the-standard-environment#instance_classes" rel="nofollow">instance class</a>, your production CPU can be throttled as low as 600MHz which is significantly slower than most developer computers.</p>

<p>One thing you <em>can</em> do to speed things up is to accumulate your results in a list (or yield them from a generator) and then use <code>str.join</code> to get the full result:</p>

<pre><code>def process_items(self, text):
    item_list = text.split()
    item_set = set()
    n_valid = 0
    n_invalid = 0
    n_dups = 0
    out = []
    for item in item_list:
        if item in item_set:
            n_dups += 1
            out.append("DUPLICATE: %s\n" % item)
        elif valid_item(item): # This compares against a compiled RE
            item_set.add(item)
            n_valid += 1
            out.append("%s\n" % item)
        else:
            n_invalid += 1
            out.append("INVALID: %s\n" % item)
    return "".join(out)
</code></pre>

python performance google-app-engine