BeautifulSoup unable to parse Goa University site
<p>I am working on a parsing project which require me to parse educational website. While doing so, my code is unable to parse <a href="http://unigoa.ac.in" rel="nofollow"><strong>University of Goa</strong></a> site. It does not return as expected.
My code:</p>

<pre class="lang-py prettyprint-override"><code>from bs4 import BeautifulSoup
import requests

hdrs = {'User-Agent': 'Mozilla / 5.0 (X11 Linux x86_64) AppleWebKit / 537.36 (\
    KHTML, like Gecko) Chrome / 52.0.2743.116 Safari / 537.36'}    

r = requests.get(url, verify=True, headers=hdrs)
result = BeautifulSoup(r.content)
print(result)
</code></pre>

<p>It prints:</p>

<pre><code>&lt;html&gt;&lt;head&gt;&lt;script type="text/javascript"&gt;
    document.location="https://www.unigoa.ac.in/result_redirect.php";
&lt;/script&gt;
&lt;/head&gt;&lt;/html&gt;
</code></pre>

<p>instead of raw html parsed tree. I tried passing explicity parser <code>lxml</code> and <code>html5lib</code> to BeautifulSoup but it also does not work as expected. Kindly help me.
Thanks in advance.</p>

<p>You need to create a session then parse and use the redirect url:</p>

<pre><code>with requests.Session() as s:
    s.headers.update(hdrs)
    r = s.get("https://www.unigoa.ac.in")
    result = BeautifulSoup(r.content)
    redirect = result.find("script").text.split("=")[1].strip('";\r\n')
    r2 = s.get(redirect)
    print(r2.text)
</code></pre>

<p><em>r2.text</em> will give you the html you see on the home page.</p>

python python-2.7 python-3.x parsing beautifulsoup