Many Selenium browsers doing the same task (python)
<p>I have a list of ids that identify websites. I need to do this:</p>

<pre><code>Open a browser
open a main website that allows a query
for id in ids:
    -search in the main website the id,
    so that I get the website corresponding to the id with the browser
    -load the website corresponding to id with the browser
    -do some stuff with the info in the website
</code></pre>

<p>Each of the loops can take 10 seconds. If I have many ids this is of course a problem.</p>

<p>I noticed that the loops are independent. This is, if I divide the ids list in 2, write 2 python scripts that look like the box above and run them from different terminals I will spend half the time. Is there a way to do the loops simultaneously (say, 6 loops at the same time) in different browsers? (actually all of them Firefox, but different windows).</p>

<p>What I want to do is to create a queue of ids, and have a number (say 6) of Firefox windows with the main website. The first id in the queue will go to the first window, and the loop starts. The second id... etc.... when a loop is done (say window number 3 finishes its loop earlier than every other window), then that window is available for a next loop, so the next id in the queue will be handled by that window.</p>

<p>Sorry if my language is not very technical. Thanks for your help.</p>

<p>If you think that splitting it in two parts will help speed up things then you can do this (furthering the split technique).</p>

<p>1) Write a function that takes as argument the id of a website.</p>

<p>2) Write a for loop which calls the def again and again sending the nenxt id.</p>

<p>Something like this:</p>

<pre><code>def check(id):
    open("id")

ids=[id1, id2, ...]

for id in ids:
    check(id)
</code></pre>

python selenium parallel-processing