Scraping Pantip Forum using BeautifulSoup
<p>I'm trying to scrape some forum posts from <a href="http://pantip.com/tag/Isuzu" rel="nofollow">http://pantip.com/tag/Isuzu</a></p>

<p>One such page is <a href="http://pantip.com/topic/35647305" rel="nofollow">http://pantip.com/topic/35647305</a></p>

<p>I want to get each post text along with its author and timestamp into a csv file.</p>

<p>I'm using Beautiful Soup, but admittedly I'm a complete beginner at python and web scraping. The code that I have right now gets the required fields, but only for the first post. I need information for all posts on that thread. I tried <em>soup.find_all()</em> and <em>soup.select()</em>, but I'm not getting the desired results.</p>

<p>Here's the code I'm using:</p>

<pre><code>from bs4 import BeautifulSoup
import urllib2 

print "Reading URL..."
url = urllib2.urlopen("http://pantip.com/topic/35647305")
content = url.read()
soup = BeautifulSoup(content, "html.parser")

print "Finding desired HTML..."
table = soup.select("abbr.timeago")

print "\nScraped HTML is:"
print table

text = BeautifulSoup(str(table).strip(),"html.parser").get_text().encode("utf-8").replace("\n", "")
print "\nScraped text is:\n" + text
</code></pre>

<p>Any clues as to what I'm doing wrong would be deeply appreciated. Also, any suggestions as to how this could be done in a better, cleaner way are welcome.</p>

<p>As mentioned, I'm a beginner, so please don't mind any stupid mistakes. :-)</p>

<p>Thanks!</p>

<p>The comments are rendered using an Ajax request:</p>

<pre><code>import requests
from bs4 import BeautifulSoup

params = {"tid": "35647305", # in the url
          "type": "3"}

with requests.Session() as s:    
    s.headers.update({"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36",
                         "X-Requested-With": "XMLHttpRequest"})
    r = (s.get("http://pantip.com/forum/topic/render_comments", params=params))
    data = r.json() # data["comments"] contains what you want
</code></pre>

<p>Which will give you all the data. So all you need is to pass the <em>tid</em>  from each url and update the tid in the params dict.</p>

python html web-scraping beautifulsoup