HTML snapshots using the selenium webdriver?
<p>I am trying to capture all the visible content of a page as text. Let's say <a href="https://dukescript.com/best/practices/2015/11/23/dynamic-templates.html" rel="nofollow">that</a> one for example.</p>

<p>If I store the page source then I won't be capturing the comments section because it's loaded using javascript. </p>

<p>Is there a way to take HTML snapshots with selenium webdriver?
(Preferably expressed using the python wrapper)</p>

<p>This code will take a screenshot of the entire page:</p>

<pre><code>from selenium import webdriver

driver = webdriver.Firefox()
driver.get('https://dukescript.com/best/practices/2015/11/23/dynamic-templates.html')
driver.save_screenshot('screenshot.png')
driver.quit()
</code></pre>

<p>however, if you just want a screenshot of a specific element, you could use this:</p>

<pre><code>def get_element_screenshot(element: WebElement) -&gt; bytes:
    driver = element._parent
    ActionChains(driver).move_to_element(element).perform()  # focus
    src_base64 = driver.get_screenshot_as_base64()
    scr_png = b64decode(src_base64)
    scr_img = Image(blob=scr_png)

    x = element.location["x"]
    y = element.location["y"]
    w = element.size["width"]
    h = element.size["height"]
    scr_img.crop(
        left=math.floor(x),
        top=math.floor(y),
        width=math.ceil(w),
        height=math.ceil(h))
    return scr_img.make_blob()
</code></pre>

<p>Where the WebElement is the Element you're chasing. of course, this method requires you to import <code>from base64 import b64decode</code> and <code>from wand.image import Image</code> to handle the cropping.</p>

python selenium-webdriver web-crawler