Python: how to make parallelize processing
<p>I need to divide the task to 8 processes.
I use <code>multiprocessing</code> to do that.
I try to describe my task:
I have dataframe and there are column with urls. Some urls have a captcha and I try to use proxies from other file to get page from every url.
It takes a lot of time and I want to divide that. I want to open first url with one proxy, secong url with another proxy etc. I can't use <code>map</code> or <code>zip</code>, because length of list with proxies is smaller. 
urls looks like</p>

<pre><code>['https://www.avito.ru/moskva/avtomobili/bmw_x5_2016_840834845', 'https://www.avito.ru/moskva/avtomobili/bmw_1_seriya_2016_855898883', 'https://www.avito.ru/moskva/avtomobili/bmw_3_seriya_2016_853351780', 'https://www.avito.ru/moskva/avtomobili/bmw_3_seriya_2016_856641142', 'https://www.avito.ru/moskva/avtomobili/bmw_3_seriya_2016_856641140', 'https://www.avito.ru/moskva/avtomobili/bmw_3_seriya_2016_853351780', 'https://www.avito.ru/moskva/avtomobili/bmw_3_seriya_2016_856641134', 'https://www.avito.ru/moskva/avtomobili/bmw_3_seriya_2016_856641141']
</code></pre>

<p>and proxies looks like </p>

<pre><code>['http://203.223.143.51:8080', 'http://77.123.18.56:81', 'http://203.146.189.61:80', 'http://113.185.19.130:80', 'http://212.235.226.133:3128', 'http://5.39.89.84:8080']
</code></pre>

<p>My code:</p>

<pre><code>def get_page(url):
m = re.search(r'avito.ru\/[a-z]+\/avtomobili\/[a-z0-9_]+$', url)
if m is not None:
    url = 'https://www.' + url
    print url
    proxy = pd.read_excel('proxies.xlsx')
    proxies = proxy.proxy.values.tolist()
    for i, proxy in enumerate(proxies):
        print "Trying HTTP proxy %s" % proxy
        try:
            result = urllib.urlopen(url, proxies={'http': proxy}).read()
            if 'ÐÑ Ð¾Ð±Ð½Ð°ÑÑÐ¶Ð¸Ð»Ð¸, ÑÑÐ¾ Ð·Ð°Ð¿ÑÐ¾ÑÑ, Ð¿Ð¾ÑÑÑÐ¿Ð°ÑÑÐ¸Ðµ Ñ Ð²Ð°ÑÐµÐ³Ð¾ IP-Ð°Ð´ÑÐµÑÐ°, Ð¿Ð¾ÑÐ¾Ð¶Ð¸ Ð½Ð° Ð°Ð²ÑÐ¾Ð¼Ð°ÑÐ¸ÑÐµÑÐºÐ¸Ðµ' in result:
                raise Exception
            else:
                page = page.read()
                soup = BeautifulSoup(page, 'html.parser')
                price = soup.find('span', itemprop="price")
                print price
        except:
            print "Trying next proxy %s in 10 seconds" % proxy
            time.sleep(10)

if __name__ == '__main__':
    pool = Pool(processes=8)
    pool.map(get_page, urls)
</code></pre>

<p>My code takes 8 urls and try open it with one proxy. How can I change algorithm to open 8 urls with 8 different proxies?</p>

<p>Something like that might help:</p>

<pre><code>def get_page(url):
m = re.search(r'avito.ru\/[a-z]+\/avtomobili\/[a-z0-9_]+$', url)
if m is not None:
    url = 'https://www.' + url
    print url
    proxy = pd.read_excel('proxies.xlsx')
    proxies = proxy.proxy.values.tolist()
    for i, proxy in enumerate(proxies):
        thread.start_new_thread( run, (proxy,i ) )

def run(proxy,i):
    print "Trying HTTP proxy %s" % proxy
    try:
        result = urllib.urlopen(url, proxies={'http': proxy}).read()
        if 'ÐÑ Ð¾Ð±Ð½Ð°ÑÑÐ¶Ð¸Ð»Ð¸, ÑÑÐ¾ Ð·Ð°Ð¿ÑÐ¾ÑÑ, Ð¿Ð¾ÑÑÑÐ¿Ð°ÑÑÐ¸Ðµ Ñ Ð²Ð°ÑÐµÐ³Ð¾ IP-Ð°Ð´ÑÐµÑÐ°, Ð¿Ð¾ÑÐ¾Ð¶Ð¸ Ð½Ð° Ð°Ð²ÑÐ¾Ð¼Ð°ÑÐ¸ÑÐµÑÐºÐ¸Ðµ' in result:
                raise Exception
        else:
            page = page.read()
            soup = BeautifulSoup(page, 'html.parser')
            price = soup.find('span', itemprop="price")
            print price
    except:
        print "Trying next proxy %s in 10 seconds" % proxy
            time.sleep(10)

if __name__ == '__main__':
    pool = Pool(processes=8)
    pool.map(get_page, urls)
</code></pre>

python multithreading proxy multiprocessing