Beautifulsoup - scraping everything but table data
<p>Hi I'm new to python and currently trying to download data from a table on a website (<a href="http://www.pa.org.mt/AppList?ReceivedDate=2016-8-31" rel="nofollow">http://www.pa.org.mt/AppList?ReceivedDate=2016-8-31</a>)</p>

<p>I've tried many different solutions but everything I try keeps returning an empty list. I read that the problem might be that the table is loaded using Javascript however when I switch off Javascript the table remains, and I can obviously see the data I want when I view the source code. </p>

<p>I am using python 2.7</p>

<p>When I run this code:</p>

<pre><code>from bs4 import BeautifulSoup
import urllib2

url = 'http://www.pa.org.mt/AppList?ReceivedDate=2016-8-31
page = urllib2.urlopen(url)
soup = BeautifulSoup(page, 'html.parser')
print soup
</code></pre>

<p>Where the table should be I get:</p>

<pre><code>&lt;link href="appsearch/main.css" rel="stylesheet" type="text/css" /&gt;
        &lt;TABLE id="Table1" cellSpacing="1" cellPadding="1" width="100%" border="0"&gt;
            &lt;TR&gt;
                &lt;TD align=center&gt;

                    &lt;br /&gt;
                    &lt;br /&gt;
                &lt;/TD&gt;
            &lt;/TR&gt;
            &lt;TR&gt;
                &lt;TD&gt;
                        &lt;/TD&gt;
            &lt;/TR&gt;
        &lt;/TABLE&gt;
</code></pre>

<p>When I view the page source code however I can see the information I want (I've copied and pasted a small part of it </p>

<pre><code>&lt;TD align=center&gt;
         &lt;p align='center' class='H1'&gt;&lt;u&gt;Planning Authority      Applications Received (Planning Applications Outside Development Zone)&lt;/u&gt;&lt;/p&gt;&lt;p align='center'&gt;Result For Date 2016-8-31&lt;/p&gt;&lt;p align='center'&gt;Result output on 03/09/2016 23:23:29&lt;/p&gt;&lt;strong&gt;&lt;i&gt;Disclaimer&lt;/strong&gt;: The information ....in accordance with the Development Planning Act.&lt;/i&gt;
                    &lt;br /&gt;
                    &lt;br /&gt;
                &lt;/TD&gt;
            &lt;/TR&gt;
            &lt;TR&gt;
                &lt;TD&gt;
                        &lt;table class='formTable'&gt;&lt;tr&gt;&lt;td class='sectionHeading' colspan=2&gt;Application Details&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;table class='formTable'&gt;&lt;TR&gt;&lt;td class='sectionHeading'&gt;Case Number&lt;/td&gt;&lt;td class='sectionHeading'&gt;Location&lt;/td&gt;&lt;td class='sectionHeading'&gt;Proposal&lt;/td&gt;&lt;td class='sectionHeading'&gt;Applicant&lt;/td&gt;&lt;td class='sectionHeading'&gt;Architect&lt;/td&gt;&lt;td class='sectionHeading'&gt;Case Category&lt;/td&gt;&lt;td class='sectionHeading'&gt;Local Council&lt;/td&gt;&lt;/tr&gt;&lt;TR&gt;&lt;td class='fieldData'&gt;&lt;a href='SearchPA?Systemkey=166837&amp;CaseFullRef=PA/05054/1
</code></pre>

<p>I would be great if you could give me any suggestions or point me in the direction of any material that might help me out. </p>

<p>As I said earlier, I'm new to both python and stackoverflow so my apologies if a similar question has already been answered or if I haven't given the right information. </p>

<p>Thanks</p>

<p>If you clear your cache and go directly to <em><a href="http://www.pa.org.mt/appsreceived?month=01/08/2016" rel="nofollow">http://www.pa.org.mt/appsreceived?month=01/08/2016</a></em> you see no data at all just like you see in your own output:</p>

<p><a href="http://i.stack.imgur.com/2Wo7T.png" rel="nofollow"><img src="http://i.stack.imgur.com/2Wo7T.png" alt="enter image description here"></a></p>

<p>You need to use a session and visit the page preceding the page you want first:</p>

<pre><code>import  requests
head = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"}

with requests.Session() as s:
    s.headers.update(head)
    s.get("http://www.pa.org.mt/appsreceived?CaseType=PA&amp;Category=PAI")

    r2 = (s.get("http://www.pa.org.mt/AppList?ReceivedDate=2016-8-31"))
    print(r2.content)
</code></pre>

<p>Now the next problem, the html is broken so html.parse will not get what you want:</p>

<pre><code>In [4]: with requests.Session() as s:
   ...:         s.headers.update(head)
   ...:         r= s.get("http://www.pa.org.mt/appsreceived?CaseType=PA&amp;Category=PAI")
   ...:         page = (s.get("http://www.pa.org.mt/AppList?ReceivedDate=2016-8-31").content)
   ...:         soup = BeautifulSoup(page, 'html.parser')
   ...:         print(soup.select_one("#Table1"))
   ...:     
&lt;table border="0" cellpadding="1" cellspacing="1" id="Table1" width="100%"&gt;
&lt;tr&gt;
&lt;td align="center"&gt;
&lt;p align="center" class="H1"&gt;&lt;u&gt;Planning Authority Applications Received (Planning Applications Within Development Zone)&lt;/u&gt;&lt;/p&gt;&lt;p align="center"&gt;Result For Date 2016-8-31&lt;/p&gt;&lt;p align="center"&gt;Result output on 04/09/2016 01:56:44&lt;/p&gt;&lt;strong&gt;&lt;i&gt;Disclaimer&lt;/i&gt;&lt;/strong&gt;: The information below has been extracted from an on-line database and is meant only for your general guidance.The Planning Authority disclaims any responsibility for any inaccuracies there may be on this site. If you wish to verify the correctness of any information then you are advised to contact us directly. Furtheremore, in the event of any discrepancies between the information contained on this site and official printed communication then the latter is to prevail, in accordance with the Development Planning Act.&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;
</code></pre>

<p><em>lxml</em> or <em>html5lib</em> will, I won't add the output as it is quite large but using either parser will give you the full table data.</p>

python html css beautifulsoup