Python script execution time increases when executed multiple time parallely
<p>I have a python script whose execution time is 1.2 second while it is being executed standalone.</p>

<p>But when I execute it 5-6 time parallely ( Am using postman to ping the url multiple times) the execution time shoots up.</p>

<p>Adding the breakdown of the time taken.</p>

<pre><code>1 run -&gt; ~1.2seconds
2 run -&gt; ~1.8seconds
3 run -&gt; ~2.3seconds
4 run -&gt; ~2.9seconds
5 run -&gt; ~4.0seconds
6 run -&gt; ~4.5seconds
7 run -&gt; ~5.2seconds
8 run -&gt; ~5.2seconds
9 run -&gt; ~6.4seconds
10 run -&gt; ~7.1seconds
</code></pre>

<p>Screenshot of top command(Asked in the comment):
<a href="http://i.stack.imgur.com/Ec3XC.png"><img src="http://i.stack.imgur.com/Ec3XC.png" alt="enter image description here"></a></p>

<p>This is a sample code:</p>

<pre><code>import psutil
import os
import time
start_time = time.time()
import cgitb
cgitb.enable()
import numpy as np
import MySQLdb as mysql
import cv2
import sys
import rpy2.robjects as robj
import rpy2.robjects.numpy2ri
rpy2.robjects.numpy2ri.activate()
from rpy2.robjects.packages import importr
R = robj.r
DTW = importr('dtw')

process= psutil.Process(os.getpid())
print " Memory Consumed after libraries load: "
print process.memory_info()[0]/float(2**20)

st_pt=4
# Generate our data (numpy arrays)
template = np.array([range(84),range(84),range(84)]).transpose()
query = np.array([range(2500000),range(2500000),range(2500000)]).transpose()


#time taken
print(" --- %s seconds ---" % (time.time() - start_time))
</code></pre>

<p>I also checked my memory consumption using <code>watch -n 1 free -m</code> and memory consumption also increases noticeably.</p>

<p>1) How do I make sure that the execution time of script remain constant everytime.</p>

<p>2) Can I load the libraries permanently so that the time taken by the script to load the libraries and the memory consumed can be minimized?</p>

<p>I made an enviroment and tried using </p>

<p><code>#!/home/ec2-user/anaconda/envs/test_python/</code></p>

<p>but it doesn't make any difference whatsoever.</p>

<p><strong>EDIT:</strong></p>

<p>I have AMAZON's EC2 server with 7.5GB RAM.</p>

<p>My php file with which am calling the python script.</p>

<pre><code>&lt;?php

    $response = array("error" =&gt; FALSE); 

    if($_SERVER['REQUEST_METHOD']=='GET'){

        $response["error"] = FALSE;
        $command =escapeshellcmd(shell_exec("sudo /home/ec2-user/anaconda/envs/anubhaw_python/bin/python2.7 /var/www/cgi-bin/dtw_test_code.py"));
        session_write_close();
        $order=array("\n","\\");
        $cleanData=str_replace($order,'',$command);
        $response["message"]=$cleanData;

    } else 
    {
        header('HTTP/1.0 400 Bad Request');
        $response["message"] = "Bad Request.";
    }
    echo json_encode($response);
?&gt;
</code></pre>

<p>Thanks</p>

<p>1) You really can't ensure the execution will take always the same time, but at least you can avoid performance degradation by using a "locking" strategy like the ones described in <a href="http://raspberrypi.stackexchange.com/a/22006">this answer</a>.</p>

<p>Basically you can test if the lockfile exists, and if so, put your program to sleep a certain amount of time, then try again. </p>

<p>If the program does not find the lockfile, it creates it, and delete the lockfile at the end of its execution.</p>

<p>Please note: in the below code, when the script fails to get the lock for a certain number of <code>retries</code>, it will exit (but this choice is really up to you).</p>

<p>The following code exemplifies the use of a file as a "lock" against parallel executions of the same script.</p>

<pre><code>import time
import os
import sys

lockfilename = '.lock'
retries = 10
fail = True

for i in range(retries):
    try:
        lock = open(lockfilename, 'r')
        lock.close()
        time.sleep(1)
    except Exception:
        print('Got after {} retries'.format(i))
        fail = False
        lock = open(lockfilename, 'w')
        lock.write('Locked!')
        lock.close()
        break

if fail:
    print("Cannot get the lock, exiting.")
    sys.exit(2)

# program execution...
time.sleep(5)
# end of program execution

os.remove(lockfilename)
</code></pre>

<p>2) This would mean that different python instances share the same memory pool and I think it's not feasible.</p>

python time