Python script execution time increases when executed multiple time parallely
<p>I have a python script whose execution time is 1.2 second while it is being executed standalone.</p>

<p>But when I execute it 5-6 time parallely ( Am using postman to ping the url multiple times) the execution time shoots up.</p>

<p>Adding the breakdown of the time taken.</p>

<pre><code>1 run -&gt; ~1.2seconds
2 run -&gt; ~1.8seconds
3 run -&gt; ~2.3seconds
4 run -&gt; ~2.9seconds
5 run -&gt; ~4.0seconds
6 run -&gt; ~4.5seconds
7 run -&gt; ~5.2seconds
8 run -&gt; ~5.2seconds
9 run -&gt; ~6.4seconds
10 run -&gt; ~7.1seconds
</code></pre>

<p>Screenshot of top command(Asked in the comment):
<a href="http://i.stack.imgur.com/Ec3XC.png"><img src="http://i.stack.imgur.com/Ec3XC.png" alt="enter image description here"></a></p>

<p>This is a sample code:</p>

<pre><code>import psutil
import os
import time
start_time = time.time()
import cgitb
cgitb.enable()
import numpy as np
import MySQLdb as mysql
import cv2
import sys
import rpy2.robjects as robj
import rpy2.robjects.numpy2ri
rpy2.robjects.numpy2ri.activate()
from rpy2.robjects.packages import importr
R = robj.r
DTW = importr('dtw')

process= psutil.Process(os.getpid())
print " Memory Consumed after libraries load: "
print process.memory_info()[0]/float(2**20)

st_pt=4
# Generate our data (numpy arrays)
template = np.array([range(84),range(84),range(84)]).transpose()
query = np.array([range(2500000),range(2500000),range(2500000)]).transpose()


#time taken
print(" --- %s seconds ---" % (time.time() - start_time))
</code></pre>

<p>I also checked my memory consumption using <code>watch -n 1 free -m</code> and memory consumption also increases noticeably.</p>

<p>1) How do I make sure that the execution time of script remain constant everytime.</p>

<p>2) Can I load the libraries permanently so that the time taken by the script to load the libraries and the memory consumed can be minimized?</p>

<p>I made an enviroment and tried using </p>

<p><code>#!/home/ec2-user/anaconda/envs/test_python/</code></p>

<p>but it doesn't make any difference whatsoever.</p>

<p><strong>EDIT:</strong></p>

<p>I have AMAZON's EC2 server with 7.5GB RAM.</p>

<p>My php file with which am calling the python script.</p>

<pre><code>&lt;?php

    $response = array("error" =&gt; FALSE); 

    if($_SERVER['REQUEST_METHOD']=='GET'){

        $response["error"] = FALSE;
        $command =escapeshellcmd(shell_exec("sudo /home/ec2-user/anaconda/envs/anubhaw_python/bin/python2.7 /var/www/cgi-bin/dtw_test_code.py"));
        session_write_close();
        $order=array("\n","\\");
        $cleanData=str_replace($order,'',$command);
        $response["message"]=$cleanData;

    } else 
    {
        header('HTTP/1.0 400 Bad Request');
        $response["message"] = "Bad Request.";
    }
    echo json_encode($response);
?&gt;
</code></pre>

<p>Thanks</p>

<h2>Here's what we have:</h2>

<ul>
<li><p>EC2 instance  type is m3.large box which has only <strong>2</strong> vCPUs <a href="https://aws.amazon.com/ec2/instance-types/?nc1=h_ls" rel="nofollow">https://aws.amazon.com/ec2/instance-types/?nc1=h_ls</a></p></li>
<li><p>We need to run a CPU- and memory-hungry script which takes over a second to execute when CPU is not busy</p></li>
<li><p>You're building an API than needs to handle concurrent requests and running apache</p></li>
<li><p>From the screenshot I can conclude that:</p>

<ul>
<li><p>your CPUs are 100% utilized when 5 processes are run. Most likely they    would be 100% utilized even when fewer processes are run. So this is the bottleneck and no surprise that the more processes are run the more time is required â you CPU resources just get shared among concurrently running scripts.</p></li>
<li><p>each script copy eats about ~300MB of RAM so you have lots of spare RAM and it's not a bottleneck. The amount of free + buffers memory on your screenshot confirms that.</p></li>
</ul></li>
<li><p>The missing part is:</p>

<ol>
<li>are requests directly sent to your apache server or there's a balancer/proxy in front of it?</li>
<li>why do you need PHP in your example? There are plently of solutions available using python ecosystem only without a php wrapper ahead of it</li>
</ol></li>
</ul>

<h2>Answers to your questions:</h2>

<blockquote>
  <ol>
  <li>That's infeasible in general case</li>
  </ol>
</blockquote>

<p>The most you can do is to track your CPU usage and make sure its idle time doesn't drop below some empirical threshold â in this case your scripts would be run in more or less fixed amount of time. </p>

<p>To guarantee that you need to limit the number of  requests being processed concurrently. 
But if 100 requests are sent to your API concurrently you won't be able to handle them all in parallel! Only some of them will be handled in parallel while others waiting for their turn. But your server won't be knocked down trying to serve them all.</p>

<blockquote>
  <ol start="2">
  <li>Yes and no</li>
  </ol>
</blockquote>

<p><strong>No</strong> because unlikely can you do something in your present architecture when a new script is launched on every request through a php wrapper. BTW it's a very expensive operation to run a new script from scratch each time.</p>

<p><strong>Yes</strong> if a different solution is used. Here are the options:</p>

<ul>
<li><p>use a python-aware <em>pre-forking</em> webserver which will handle your requests directly. You'll spare CPU resources on python startup + you might utilize some preloading technics to share RAM among workers, i.e <a href="http://docs.gunicorn.org/en/stable/settings.html#preload-app" rel="nofollow">http://docs.gunicorn.org/en/stable/settings.html#preload-app</a>. You'd also need to limit the number of parallel workers to be run <a href="http://docs.gunicorn.org/en/stable/settings.html#workers" rel="nofollow">http://docs.gunicorn.org/en/stable/settings.html#workers</a> to adress your first requirement.</p></li>
<li><p>if you need <strong>PHP</strong> for some reason you might setup some <em>intermediary</em> between <strong>PHP</strong> script and python workers â i.e. a <em>queue</em>-like server.
Than simply run several instances of your <em>python</em> scripts which would wait for some request to be availble in the queue. Once it's available it would handle it and put the response back to the <em>queue</em> and php script would slurp it and return back to the client. But it's a more complex to build this that the first solution (if you can eliminate your PHP script of course) and more components would be involved.</p></li>
<li><p>reject the idea to handle such heavy requests concurrently, and instead assign each request a <strong>unique id</strong>, put the request into a <em>queue</em> and return this id to the client immediately. The request will be picked up by an offline handler and put back into the <em>queue</em> once it's finished. It will be client's responsibility to poll your API for readiness of this particular request</p></li>
<li><p>1st and 2nd combined â handle requests in <strong>PHP</strong> and request another HTTP server (or any other TCP server) handling your preloaded .py-scripts</p></li>
</ul>

python time