Crawl a large number of web pages
<p>I want to use Scrappy to crawl a large selection of web pages. Because I have to use proxy, and the proxy is bad, lots of time is wasted on changing IPs. How can I use multi-threading to speed this along?</p>

<p>(Ps: I use a HttpProxyMiddleware.py to get proxyIP from a redis database.</p>

<pre><code>    proxy_config = settings.get("PROXY_CONFIG")
    self.r =redis.Redis(host=proxy_config.get('redis_host'),
    port=proxy_config.get("redis_port", 6379))
    self.ips_key = proxy_config.get('ip_key')
</code></pre>

<p>there are lots of ips in it. but part of them do not work. I set the timeout = 5s , so lots operation of changing ips waste lots of time.</p>

<p>because scrapy is use twisted, its work stream is<br>
spider.py(generate request)->  HttpProxyMiddleware.py(add proxy to request,check response to see if the ip is working )->  spider.py(parse() process response)</p>

<p>At first ,I try to use multi-threading to speed this. the result shows that all threading depend on the same one customized-middleware "HttpProxyMiddleware.py". as i known this is just a Class and it is not a singleton. I don't how scrapy to implement it. so I have to create multiple  HttpProxyMiddleware , as HttpProxyMiddleware1.py HttpProxyMiddleware2.py ...... , and I also create mutilple spider.py as spider1.py ,spider2.py...... each spider use one HttpProxyMiddleware correspondingly. And it work. but look bad. I ask google for help, I get the answer is that use reactor(Twisted), and i used it :</p>

<pre><code>    from twisted.internet import reactor
    reactor.suggestThreadPoolSize(30)
</code></pre>

<p>but it not work. Maybe my usage is wrong. so my question is how to use reactor or else mutiple-thread method to speed this along?</p>

<pre><code>    HttpProxyMiddleware.py extends scrapy's downloadmiddleware
    class HttpProxyMiddleware(object):{
     def process_request(self, request, spider):
         #add proxy to request

     def process_response(self, request, response, spider):
         #check response to decide to change ips or not    
   }
</code></pre>

<p>At last, I am a newcomer here, if my description of questions is not clear, please point it out. I will correct it immediately. Thank you, I appreciate every assistance.</p>

<p>Use lots of proxies.</p>

<p>For instance for a similar project, I've installed tor.</p>

<p>You can run multiple instances of tor, thus having multiple dedicated proxies available.</p>

<p>Run an instance on 127.0.0.1:9050, another on 127.0.0.1:9051, and so on and so forth.</p>

<p>Script the launch of all these instances, and also script their restart (so you get a different exit node, hence another IP).</p>

<p>Now, have your scrapy script taking the proxy adress as a parameter (<code>argv</code>)</p>

<p>And have a controller script running everything like so :</p>

<pre><code># pseudo code
while 1:
    tor_restart.sh # restart all tor nodes
    for i between 0 and n:
        python scrapy_script.py 9050+i &amp;
    while there is a scrapy_script.py in the process list, sleep.
</code></pre>

<p>Look at this :</p>

<p><a href="http://tor.stackexchange.com/questions/327/how-may-i-run-multiple-tor-relay-instances-in-a-single-linux-machine">http://tor.stackexchange.com/questions/327/how-may-i-run-multiple-tor-relay-instances-in-a-single-linux-machine</a></p>

python scrapy