Multiprocessing and Selenium Python
<p>I have 3 drivers (Firefox browsers) and I want them to <code>do something</code> in a list of websites.</p>

<p>I have a worker defined as:</p>

<pre><code>def worker(browser, queue):
    while True:
        id_ = queue.get(True)
        obj = ReviewID(id_)
        obj.search(browser)
        if obj.exists(browser):
            print(obj.get_url(browser))
        else:
            print("Nothing")
</code></pre>

<p>So the worker will just acces to a queue that contains the ids and use the browser to do something.</p>

<p>I want to have a pool of workers so that as soon as a worker has finished using the browser to do something on the website defined by id_, then it immediately starts to work using the same browser to do something on the next id_ found in queue. I have then this:</p>

<pre><code>pool = Pool(processes=3)  # I want to have 3 drivers
manager = Manager()
queue = manager.Queue()
# Define here my workers in the pool
for id_ in ids:
    queue.put(id_)
for i in range(3):
    queue.put(None)
</code></pre>

<p>Here I have a problem, I don't know how to define my workers so that they are in the pool. To each driver I need to assign a worker, and all the workers share the same queue of ids. Is this possible? How can I do it?</p>

<p>Another idea that I have is to create a queue of browsers so that if a driver is doing nothing, it is taken by a worker, along with an id_ from the queue in order to perform a new process. But I'm completely new to multiprocessing and actually don't know how to write this.</p>

<p>I appreciate your help.</p>

<p>You could try instantiating the browser in the worker:</p>

<pre><code>def worker(queue):
    browser = webdriver.Chrome()
    try:
        while True:
            id_ = queue.get(True)
            obj = ReviewID(id_)
            obj.search(browser)
            if obj.exists(browser):
                print(obj.get_url(browser))
            else:
                print("Nothing")
    finally:
        brower.quit()
</code></pre>

python selenium python-multiprocessing