python Get HTML from JS using selenium
<p>I'm trying to get the div HTML from <a href="https://www.workday.com/en-us/company/careers/open-positions.html#?q=" rel="nofollow">https://www.workday.com/en-us/company/careers/open-positions.html#?q=</a>.</p>

<p>But the div listing job posts is loaded from <code>granite.min.js</code> based on network XHR.</p>

<pre><code>from selenium import webdriver
from bs4 import BeautifulSoup
from pprint import pprint


path_to_chromedriver = "/Users/RichWin/Documents/chromedriver.exe"
browser = webdriver.Chrome(executable_path=path_to_chromedriver)

driver = browser.get('https://www.workday.com/en-us/company/careers/open-positions.html#?q=')

elem = driver.find_element_by_id('template-content')

soup = BeautifulSoup(elem.get_text, "html.parser")

for tag in soup.find_all('div'):
    pprint(tag)
</code></pre>

<p>Can anyone help me?</p>

<p>Ok, so your code has a couple of problems.</p>

<p>a) you need to wait for the <code>template-content</code> div to load its content. In the code bellow I use <a href="http://selenium-python.readthedocs.io/waits.html" rel="nofollow"><code>implicitly_wait</code></a> to wait 30 seconds.
<br>
b) <code>find_element_by_id</code> doesn't return <code>HTML</code> but a <code>Selenium</code> object. Therefore you cannot pass it to <code>BeautifulSoup</code> for parsing.</p>

<pre><code>from pprint import pprint
from bs4 import BeautifulSoup
from selenium import webdriver


url = 'https://www.workday.com/en-us/company/careers/open-positions.html#?q='
path_to_chromedriver = "/Users/RichWin/Documents/chromedriver.exe"

browser = webdriver.Chrome(executable_path=path_to_chromedriver)
browser.implicitly_wait(30)
browser.get(url)

elem = browser.find_element_by_id('template-content')
elem_html = elem.get_attribute('innerHTML')

soup = BeautifulSoup(elem_html, "html.parser")
for tag in soup.find_all('div'):
    pprint(tag)

browser.quit()
</code></pre>

javascript python selenium