Python script execution time increases when executed multiple time parallely
<p>I have a python script whose execution time is 1.2 second while it is being executed standalone.</p>

<p>But when I execute it 5-6 time parallely ( Am using postman to ping the url multiple times) the execution time shoots up.</p>

<p>Adding the breakdown of the time taken.</p>

<pre><code>1 run -&gt; ~1.2seconds
2 run -&gt; ~1.8seconds
3 run -&gt; ~2.3seconds
4 run -&gt; ~2.9seconds
5 run -&gt; ~4.0seconds
6 run -&gt; ~4.5seconds
7 run -&gt; ~5.2seconds
8 run -&gt; ~5.2seconds
9 run -&gt; ~6.4seconds
10 run -&gt; ~7.1seconds
</code></pre>

<p>Screenshot of top command(Asked in the comment):
<a href="http://i.stack.imgur.com/Ec3XC.png"><img src="http://i.stack.imgur.com/Ec3XC.png" alt="enter image description here"></a></p>

<p>This is a sample code:</p>

<pre><code>import psutil
import os
import time
start_time = time.time()
import cgitb
cgitb.enable()
import numpy as np
import MySQLdb as mysql
import cv2
import sys
import rpy2.robjects as robj
import rpy2.robjects.numpy2ri
rpy2.robjects.numpy2ri.activate()
from rpy2.robjects.packages import importr
R = robj.r
DTW = importr('dtw')

process= psutil.Process(os.getpid())
print " Memory Consumed after libraries load: "
print process.memory_info()[0]/float(2**20)

st_pt=4
# Generate our data (numpy arrays)
template = np.array([range(84),range(84),range(84)]).transpose()
query = np.array([range(2500000),range(2500000),range(2500000)]).transpose()


#time taken
print(" --- %s seconds ---" % (time.time() - start_time))
</code></pre>

<p>I also checked my memory consumption using <code>watch -n 1 free -m</code> and memory consumption also increases noticeably.</p>

<p>1) How do I make sure that the execution time of script remain constant everytime.</p>

<p>2) Can I load the libraries permanently so that the time taken by the script to load the libraries and the memory consumed can be minimized?</p>

<p>I made an enviroment and tried using </p>

<p><code>#!/home/ec2-user/anaconda/envs/test_python/</code></p>

<p>but it doesn't make any difference whatsoever.</p>

<p><strong>EDIT:</strong></p>

<p>I have AMAZON's EC2 server with 7.5GB RAM.</p>

<p>My php file with which am calling the python script.</p>

<pre><code>&lt;?php

    $response = array("error" =&gt; FALSE); 

    if($_SERVER['REQUEST_METHOD']=='GET'){

        $response["error"] = FALSE;
        $command =escapeshellcmd(shell_exec("sudo /home/ec2-user/anaconda/envs/anubhaw_python/bin/python2.7 /var/www/cgi-bin/dtw_test_code.py"));
        session_write_close();
        $order=array("\n","\\");
        $cleanData=str_replace($order,'',$command);
        $response["message"]=$cleanData;

    } else 
    {
        header('HTTP/1.0 400 Bad Request');
        $response["message"] = "Bad Request.";
    }
    echo json_encode($response);
?&gt;
</code></pre>

<p>Thanks</p>

<p>The ec2 cloud does not guarantee 7.5gb of free memory on the server.  This would mean that the VM performance is severely impacted like you are seeing where the server has less than 7.5gb of physical free ram.  Try reducing the amount of memory the server thinks it has.</p>

<p>This form of parallel performance is very expensive.  Typically with 300mb requirement, the ideal would be a script which is long running, and re-uses the memory for multiple requests.  The Unix fork function allows a shared state to be re-used.  The <code>os.fork</code> gives this in python, but may not be compatible with your libraries.</p>

python time