Get visible content of a page using selenium and BeautifulSoup
<p>I want to retrieve all visible content of a web page. Let say for example <a href="https://dukescript.com/best/practices/2015/11/23/dynamic-templates.html" rel="nofollow">this</a> webpage. I am using a headless firefox browser remotely with selenium.</p>

<p>The script I am using looks like this</p>

<pre><code>driver = webdriver.Remote('http://0.0.0.0:xxxx/wd/hub', desired_capabilities)
driver.get(url)
dom = BeautifulSoup(driver.page_source, parser)

f = dom.find('iframe', id='dsq-app1')
driver.switch_to_frame('dsq-app1')
s = driver.page_source
f.replace_with(BeautifulSoup(s, 'html.parser'))

with open('out.html', 'w') as fe:
    fe.write(dom.encode('utf-8'))
</code></pre>

<p>This is supposed to load the page, parse the dom, and then replace the iframe with id <code>dsq-app1</code> with it's visible content. If I execute those commands one by one via my python command line it works as expected. I can then see the paragraphs with all the visible content. When instead I execute all those commands at once, either by executing the script or by pasting all this snippet in my interpreter, it behaves differently. The paragraphs are missing, the content still exists in json format, but it's not what I want.</p>

<p>Any idea why this may happening? Something to do with <code>replace_with</code> maybe?</p>

<p>Try to get the Page Source after detecting the required ID/CSS_SELECTOR/CLASS or LINK.</p>

<p>You can always use explicit wait of Selenium WebDriver.</p>

<pre><code>from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Remote('http://0.0.0.0:xxxx/wd/hub', desired_capabilities)
driver.get(url)
f = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.ID,idName) 
# here 10 is time for which script will try to find given id
# provide the id name
dom = BeautifulSoup(driver.page_source, parser)

f = dom.find('iframe', id='dsq-app1')
driver.switch_to_frame('dsq-app1')
s = driver.page_source
f.replace_with(BeautifulSoup(s, 'html.parser'))

with open('out.html', 'w') as fe:
    fe.write(dom.encode('utf-8'))
</code></pre>

<p>Correct me if this not work</p>

python html selenium beautifulsoup