How to wait for the site to return the data using Beautifulsoup4
<p>I wrote a script using beautifulsoup4 , the script basically brings the list of ciphers from the table present on a web page. The problem is my python script doesn't wait for the returned content of the web page and either breaks or says 'list index out of range'. The code is as follows: </p>

<pre><code>ssl_lab_url = 'https://www.ssllabs.com/ssltest/analyze.html?d='+site
req  = requests.get(ssl_lab_url)
data = req.text
soup = BeautifulSoup(data)
 print CYELLOW+"Now Bringing in the LIST of cipher gathered from SSL LABS for "+str(ssl_lab_url)+CEND
        for i in tqdm(range(10000)):
           sleep(0.01)
           table = soup.find_all('table',class_='reportTable', limit=5)[-1]
           data = [ str(td.text.split()[0]) for td in table.select("td.tableLeft")]
        print CGREEN+str(data)+CEND
        time.sleep(1)
</code></pre>

<p>It sometimes return NOTHING in <code>data</code>  or says :</p>

<pre><code>Traceback (most recent call last):
  File "multiple_scan_es.py", line 79, in &lt;module&gt;
    scan_cipher_ssl(list_url )
  File "multiple_scan_es.py", line 62, in scan_cipher_ssl
    table = soup.find_all('table',class_='reportTable', limit=5)[-1]
IndexError: list index out of range
</code></pre>

<p>I need to wait here , how to do so ?</p>

<p>If the data isn't present in the original HTML page but is returned from JS code in the background, consider using a headless browser, such as PhantomJS, with Selenium. <a href="https://realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/" rel="nofollow">Here's an example</a>.</p>

python beautifulsoup