Python urllib is not extracting reader comments from a website
<p>I am trying to extract reader comments from the following page with the code shown below. But the output html <em>test.html</em> does not contain any comments from the page. How do I get this information with Python? </p>

<p><a href="http://www.theglobeandmail.com/opinion/it-doesnt-matter-who-won-the-debate-america-has-already-lost/article32314064/comments/" rel="nofollow">http://www.theglobeandmail.com/opinion/it-doesnt-matter-who-won-the-debate-america-has-already-lost/article32314064/comments/</a></p>

<pre><code>from bs4 import BeautifulSoup
import urllib
import urllib.request
import urllib.parse

req =urllib.request.Request('http://www.theglobeandmail.com/opinion/it-doesnt-matter-who-won-the-debate-america-has-already-lost/article32314064/comments/')
response = urllib.request.urlopen(req)
the_page = response.read()

soup = BeautifulSoup(the_page, 'html.parser')
f = open('test.html', 'w')
f.write(soup.prettify())
f.close()
</code></pre>

<p>Thanks! </p>

<p>The comments are retrieved using an ajax requests which you can mimic:</p>

<p><a href="https://i.stack.imgur.com/fqp4Y.png" rel="nofollow"><img src="https://i.stack.imgur.com/fqp4Y.png" alt="enter image description here"></a></p>

<p>You can see there are numerous parameters but what is below is enough to get a result, I will leave it to you to figure out how you can influence the results:</p>

<pre><code>from json import loads
from urllib.request import  urlopen
from urllib.parse import urlencode

data = {"categoryID":"Production",
    "streamID":"32314064",
    "APIKey":"2_oNjjtSC8Qc250slf83cZSd4sbCzOF4cCiqGIBF8__5dWzOJY_MLAoZvds76cHeQD",
    "callback" :"foo",}
r = urlopen("http://comments.us1.gigya.com/comments.getComments", data=urlencode(data).encode("utf-8"))
json_dcts = loads(r.read().decode("utf-8"))["comments"]

print(json_dcts)
</code></pre>

<p>That gives you a list of dicts that hold all the comments, upvotes, negvotes etc.. If you wanted to parse the key it is in the url of inside one of th scripts <code>src='https://cdns.gigya.com/js/socialize.js?apiKey=2_oNjjtSC8Qc250slf83cZSd4sbCzOF4cCiqGIBF8__5dWzOJY_MLAoZvds76cHeQD'</code>, the  <em>streamID</em> is in your original url.</p>

python web-scraping urllib