Code runs slow but multiple python IDEs makes it faster
<p>I have a simple webcrawler which crawls ~4 (specific) sites per second. If I run the script in tow different python IDE's I can double the speed because both programs runs the code with ~4 crawls per second. 
Why does my code runs slower then It could be? Or is it faster because I use a silly way to make my script use multithreading/multiprocessing by using two different IDE's at the same time?</p>

<p>I use Python 3.5.2</p>

<p>It's all about threading, as explained in the comments.</p>

<p>Now there is a solution for your program :</p>

<pre><code>import requests
import threading

class Crawler:
    def __init__(self, url):
        self.url = url
        self.session = requests.Session()

    def crawl(self):
        page = self.session.get(url)
        # do your crawling here

with open('urls.txt', 'r') as f: # assuming you use a file with a list of urls
    for url in f:
        while(threading.active_count() &gt; 20): # Use 20 threads
            time.sleep(0.1)
        c = Crawler(url)
        t = threading.Thread(target = c.crawl())
</code></pre>

python multithreading performance multiprocessing