Make 2 functions run at the same time and in parallel?
<p>I have an array</p>

<pre><code>myArray = array(url1,url2,...,url90)
</code></pre>

<p>I want to execute this commande 3 times in parallel</p>

<p><code>scrapy crawl mySpider -a links=url</code></p>

<p>and each time with 1 url,</p>

<pre><code>scrapy crawl mySpider -a links=url1
scrapy crawl mySpider -a links=url2
scrapy crawl mySpider -a links=url3
</code></pre>

<p>and when the first one finish his job, he will get the other url like</p>

<pre><code>scrapy crawl mySpider -a links=url4
</code></pre>

<p>I read <a href="http://stackoverflow.com/questions/21519431/r-run-2-different-blocks-of-code-at-the-same-time-in-parallel">this question</a>, and <a href="http://stackoverflow.com/questions/2957116/make-2-functions-run-at-the-same-time">this one</a>  and I try this:</p>

<pre><code>import threading
from threading import Thread

def func1(url):

    scrapy crawl mySpider links=url

if __name__ == '__main__':
    myArray = array(url1,url2,...,url90)
    for(url in myArray):
        Thread(target = func1(url)).start()
</code></pre>

<p>When you write <code>target = func1(url)</code> you actually runnig <code>func1</code> and passing result to <code>Thread</code> (not a reference do the function). This means functions are run on the loop not in the seperate thread.</p>

<p>You need to rewrite it like that:</p>

<pre><code>if __name__ == '__main__':
    myArray = array(url1,url2,...,url90)
    for(url in myArray):
        Thread(target=func1, args=(url,))).start()
</code></pre>

<p>Then you are telling Thread to run <code>func1</code> with arguments <code>(url,)</code></p>

<p>Also you should wait for Threads to finish after the loop, otherwise your program with terminate just after starting all the threads.</p>

<p>EDIT:
and if you want only 3 threads to be run on the same time you may want to use ThreadPool:</p>

<pre><code>if __name__ == '__main__':
    from multiprocessing.pool import ThreadPool

    pool = ThreadPool(processes=3)
    pool.map(func, myArray)
</code></pre>

python parallel-processing scrapy ipython-parallel