Web Scraping with Javascript Contents using Python PyQt
<p>I am now performing a task of scraping content systematically from a course list which seems to be rendered by javascript. I followed some scripts using PyQt4 on the web but failed (which I copied below). More precisely, the script works at some websites with javascript which loads content with clicking on its specific link. However, the following website (ouhk, the link I copied below in the script) does not seem to carry link for directing users to specific content, namely Programme Information, Programme Structure and Fee, etc. Instead, it uses tag containers and FTP for storing and loading information (that I found from its source code). </p>

<p>I am wondering if there is anyway to modify the following script so that I can scrape those content by using PyQt4, or I have to look for other ways to achieve this purpose?</p>

<pre><code>import sys  
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html 

from bs4 import BeautifulSoup
#import urllib.request
#from urllib.parse import urljoin


#Take this class for granted.Just use result of rendering.
class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  

  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit()  

url = 'http://www.ouhk.edu.hk/wcsprd/Satellite?pagename=OUHK/tcSchSing2014&amp;c=C_LIPACE&amp;cid=1450268562831&amp;lang=eng&amp;sch=LIP'  
r = Render(url)  
result = r.frame.toHtml()
print result
</code></pre>

<p>Look into using selenium Library! I have scraped multiple websites with this library. People state that it is slow, but for my purposes it works great.</p>

<p>Also if your kinda new to web scraping, look into what Xpaths are for scraping elements that would otherwise be difficult to get to. With Xpath, all you need to do in a chrome browser is right clickt he page, insspect element, unfold all the tags, and then right click the tag you want to scrape and click copy Xpath, then you can paste the path into modules in the selenium library. Really simple, heres a link for selenium information. </p>

<p><a href="http://selenium-python.readthedocs.io/" rel="nofollow">http://selenium-python.readthedocs.io/</a></p>

javascript python web-scraping pyqt