Getting all HTML from requests.get()
<p>I just started with web scraping with Python and hit the wall.
I am using the requests library to get the HTML code from a website. For example, the Google search result website: "<a href="https://www.google.com/?gws_rd=ssl#q=ball" rel="nofollow">https://www.google.com/?gws_rd=ssl#q=ball</a>"</p>

<p>When I hit <kbd>F12</kbd> and check the HTML, it looks different than with:</p>

<pre><code>site = requests.get("https://www.google.com/?gws_rd=ssl#q=ball")
print(site.text)
</code></pre>

<p>with <code>requests.get</code>, text is much shorter and not all information is visible (it starts with <code>!doctype</code>, however).
Because of that I am unable to work with this HTML.</p>

<p>Can you tell me where the mistake is?</p>

<hr>

<p>This is actually an exercise from the book "Automate the boring stuff with Python". The task is to search for some item Google and then find few first results with HTML locators. I cannot do it because when I use <code>requests.get()</code> I cannot see any objects for links in the HTML code.</p>

<p>The HTML you see using the browser's development tools is what the browser is currently working with. This includes any changes performed via Javascript. The data you are getting when using Requests is before any Javascript has operated on the page. (Note that Requests doesn't process Javascript so you will be unable to acquire a javascript processed page using just Requests.)</p>

<p>If you're specifically looking to scrape Google Search, use a url like <a href="https://www.google.com/search?q=test" rel="nofollow">https://www.google.com/search?q=test</a>. This particular url is for Google's non-javascript site. Keep in mind that Google (and most other sites) doesn't appreciate scraping so you may run into other issues when doing so.</p>

python html python-requests