Get visible content of a page using selenium and BeautifulSoup
<p>I want to retrieve all visible content of a web page. Let say for example <a href="https://dukescript.com/best/practices/2015/11/23/dynamic-templates.html" rel="nofollow">this</a> webpage. I am using a headless firefox browser remotely with selenium.</p>

<p>The script I am using looks like this</p>

<pre><code>driver = webdriver.Remote('http://0.0.0.0:xxxx/wd/hub', desired_capabilities)
driver.get(url)
dom = BeautifulSoup(driver.page_source, parser)

f = dom.find('iframe', id='dsq-app1')
driver.switch_to_frame('dsq-app1')
s = driver.page_source
f.replace_with(BeautifulSoup(s, 'html.parser'))

with open('out.html', 'w') as fe:
    fe.write(dom.encode('utf-8'))
</code></pre>

<p>This is supposed to load the page, parse the dom, and then replace the iframe with id <code>dsq-app1</code> with it's visible content. If I execute those commands one by one via my python command line it works as expected. I can then see the paragraphs with all the visible content. When instead I execute all those commands at once, either by executing the script or by pasting all this snippet in my interpreter, it behaves differently. The paragraphs are missing, the content still exists in json format, but it's not what I want.</p>

<p>Any idea why this may happening? Something to do with <code>replace_with</code> maybe?</p>

<p>To add to Or Duan's answer I provide what I ended up doing. The problem of finding whether a page or parts of a page have loaded completely is an intricate one. I tried to use implicit and explicit waits but again I ended up receiving half-loaded frames. My workaround is to check the <code>readyState</code> of the original document and the readyState of iframes.</p>

<p>Here is a sample function</p>

<pre><code>def _check_if_load_complete(driver, timeout=10):
    elapsed_time = 1
    while True:
        if (driver.execute_script('return document.readyState') == 'complete' or
                elapsed_time == timeout):
            break
        else:
            sleep(0.0001)
        elapsed_time += 1
</code></pre>

<p>then I used that function right after I changed the focus of the driver to the iframe</p>

<pre><code>driver.switch_to_frame('dsq-app1')
_check_if_load_complete(driver, timeout=10)
</code></pre>

python html selenium beautifulsoup