Running loop synchronously in python
<p>I have a block of code, that crawls an infinite height website <code>(Like FACEBOOK)</code>. </p>

<p>The Python selenium script asks the page javascript to go to the bottom of the page in order to load page further down. But eventually it happens that the loop runs asynchronously and the website's rate-limiter blocks the script.</p>

<p>I need the page to wait for the page to load first and then continue, but i failed in doing that. </p>

<p>The following things are what i have tried till now.</p>

<p>The code goes as follows : </p>

<pre><code>while int(number_of_news) != int(len(news)) :
    driver.execute_script("window.scrollTo(document.body.scrollHeight/2, document.body.scrollHeight);")
    news = driver.find_elements_by_class_name("news-text")
    print(len(news))
</code></pre>

<p>The output is something like</p>

<p><a href="http://i.stack.imgur.com/uTRaL.png" rel="nofollow"><img src="http://i.stack.imgur.com/uTRaL.png" alt="enter image description here"></a></p>

<p>Which i interpreted as the loop being executed multiple times when the value is <code>43, 63... and so on</code>.</p>

<p>I also tried making it recursive, but the result is still the same. The recursive code is as follows :</p>

<pre><code>def call_news(_driver, _news, _number_of_news):
    _driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    _news = driver.find_elements_by_class_name("news-text")
    print(len(_news))
    if int(len(_news)) != int(number_of_news) :
        call_news(_driver, _news, _number_of_news)
    else :
        return _news
</code></pre>

<p>Any kind of tip is appreciated.</p>

<p>You can set the <code>page_load_timeout</code> to make the driver wait for the page to load</p>

<pre><code>driver.set_page_load_timeout(10)
</code></pre>

<p>Another option is to wait for the number of elements to change</p>

<pre><code>current_number_of_news = 0
news = []
while int(number_of_news) != int(len(news)) :
    driver.execute_script("window.scrollTo(document.body.scrollHeight/2, document.body.scrollHeight);")
    while (current_number_of_news == len(news)) :
        news = driver.find_elements_by_class_name("news-text")
    current_number_of_news = len(news)
    print(len(news))
</code></pre>

python selenium asynchronous selenium-webdriver synchronous