Get visible content of a page using selenium and BeautifulSoup
<p>I want to retrieve all visible content of a web page. Let say for example <a href="https://dukescript.com/best/practices/2015/11/23/dynamic-templates.html" rel="nofollow">this</a> webpage. I am using a headless firefox browser remotely with selenium.</p>

<p>The script I am using looks like this</p>

<pre><code>driver = webdriver.Remote('http://0.0.0.0:xxxx/wd/hub', desired_capabilities)
driver.get(url)
dom = BeautifulSoup(driver.page_source, parser)

f = dom.find('iframe', id='dsq-app1')
driver.switch_to_frame('dsq-app1')
s = driver.page_source
f.replace_with(BeautifulSoup(s, 'html.parser'))

with open('out.html', 'w') as fe:
    fe.write(dom.encode('utf-8'))
</code></pre>

<p>This is supposed to load the page, parse the dom, and then replace the iframe with id <code>dsq-app1</code> with it's visible content. If I execute those commands one by one via my python command line it works as expected. I can then see the paragraphs with all the visible content. When instead I execute all those commands at once, either by executing the script or by pasting all this snippet in my interpreter, it behaves differently. The paragraphs are missing, the content still exists in json format, but it's not what I want.</p>

<p>Any idea why this may happening? Something to do with <code>replace_with</code> maybe?</p>

<p>Sounds like the dom elements are not yet loaded when your code try to reach them. </p>

<p>Try to <a href="http://stackoverflow.com/questions/7781792/selenium-waitforelement">wait</a> for the elements to be fully loaded and just then replace.</p>

<p>This works for your when you run it command by command because then you let the driver load all the elements before you execute more commands. </p>

python html selenium beautifulsoup