How to get elements of webpage that load after initial webpage?
<p>I'm trying to download stock option data from Yahoo Finance (<a href="https://finance.yahoo.com/quote/GOOG/options?p=GOOG" rel="nofollow">here's Google as an example</a>) with <code>requests.get</code>, which doesn't seem to be downloading everything. I'm trying to get the dropdown of dates with an XPath but even <code>//option</code> doesn't return anything even though Chrome DevTools says there are 13 instances!</p>

<p>I expect this has something to do with the fact that the parts of the site that actually matter are being loaded <em>after</em> all the navigation bars and such, and I don't know how to get all of it. Could you please suggest a method for getting the text of each item in the date dropdown menu?</p>

<p>If you open the dev console and refresh the page again (caches might need to be purged), you can see some requests with type <code>xhr</code>.</p>

<p>They are usually initiated by JavaScript programs and will load some data besides those provided by HTML body.</p>

<p>That's what you can look into.</p>

python html python-3.x xpath web-scraping