HTML snapshots using the selenium webdriver?
<p>I am trying to capture all the visible content of a page as text. Let's say <a href="https://dukescript.com/best/practices/2015/11/23/dynamic-templates.html" rel="nofollow">that</a> one for example.</p>

<p>If I store the page source then I won't be capturing the comments section because it's loaded using javascript. </p>

<p>Is there a way to take HTML snapshots with selenium webdriver?
(Preferably expressed using the python wrapper)</p>

<p>Regardless of whether or not the HTML of the page is generated using JavaScript, you will still be able to capture it using <code>driver.page_source</code>.</p>

<p>I imagine the reason you haven't been able to capture the source of the comments section in your example is because it's contained in an iframe - In order to capture the html source for content within a frame/iframe you'll need to first switch focus to that particular frame followed by calling <code>driver.page_source</code>.</p>

python selenium-webdriver web-crawler