Http requests freezes after severel requests
<p>Okay, here is my code:</p>

<pre><code>from lxml import html
from lxml import etree
from selenium import webdriver
import calendar
import math
import urllib
import progressbar
import requests
</code></pre>

<p><em>Using selenium</em></p>

<pre><code>path_to_driver = '/home/vladislav/Shit/geckodriver'
browser = webdriver.Firefox(executable_path = path_to_driver)
</code></pre>

<p><em>Create a dict, where i store data and create progressbars</em></p>

<pre><code>DataDict = {}
barY = progressbar.ProgressBar(max_value=progressbar.UnknownLength)
barM = progressbar.ProgressBar(max_value=progressbar.UnknownLength)
barW = progressbar.ProgressBar(max_value=progressbar.UnknownLength)
</code></pre>

<p><em>Forming parameters in a loop, constructing a url from them and send a <code>browser.get</code> request</em></p>

<pre><code>for year in (range(2014,2016)):
        barY.update(year)
        for month in range(1,13):
            barM.update(month)
            weeks = math.ceil(calendar.monthrange(year,month)[1]/4)
            for week in range(weeks):
                barW.update(week)
                if (week &gt; 2):
                    start_day = 22
                    end_day = calendar.monthrange(year,month)[1]
                else:
                    start_day =7*week + 1
                    end_day = 7*(week + 1)
                start_date = str(year) + '-' + str(month).zfill(2) +'-' + str(start_day).zfill(2)
                end_date = str(year) + '-' +str(month).zfill(2) + '-' + str(end_day).zfill(2)
                params = {'end-date': end_date, 'start-date': start_date}
                url = 'http://www.finam.ru/profile/moex-akcii/aeroflot/news'
                url = url + ('&amp;' if urllib.parse.urlparse(url).query else '?') + urllib.parse.urlencode(params)
</code></pre>

<h1>The request itself</h1>

<pre><code>            browser.get(url)

                try: 
                    news_list = browser.find_element_by_class_name('news-list')
                    news_list_text = news_list.text
                    news_list_text = news_list_text.split('\n')
                    for i in range(int(len(news_list_text)/2)):
                        DataDict.update({news_list_text[2*i]:news_list_text[2*i+1]})
                    print("Found! Adding news to the dictionary!")
                except:
                    pass
</code></pre>

<h1>But after 2-4 requests it just freezes:(</h1>

<p>Whats the problem?
<a href="http://i.stack.imgur.com/F23eu.png" rel="nofollow"><img src="http://i.stack.imgur.com/F23eu.png" alt="enter image description here"></a></p>

<p>Okay, the problem was in an advertising banner, which appeared after several requests. Solution is just to wait (<code>time.sleep</code>), untill the banner disapeares, and the send request again!:</p>

<pre><code> try:
                browser.get(url)
                try: 
                    news_list = browser.find_element_by_class_name('news-list')
                    news_list_text = news_list.text
                    news_list_text = news_list_text.split('\n')
                    for i in range(int(len(news_list_text)/2)):
                        DataDict.update({news_list_text[2*i]:news_list_text[2*i+1]})
                    #print("Found! Adding news to the dictionary!")
                except:
                    pass
                time.sleep(10)
            except:
                print("perchaps this shitty AD?")
                try:
                    news_list = browser.find_element_by_class_name('news-list')
                    news_list_text = news_list.text
                    news_list_text = news_list_text.split('\n')
                    for i in range(int(len(news_list_text)/2)):
                        DataDict.update({news_list_text[2*i]:news_list_text[2*i+1]})
                    #print("Found! Adding news to the dictionary!")
                except:
                    pass
</code></pre>

python html selenium