web-scrapping hidden href using python
<p>IÂ´m using python to get all the possible href from the following webpage: </p>

<p><a href="http://www.congresovisible.org/proyectos-de-ley/" rel="nofollow">http://www.congresovisible.org/proyectos-de-ley/</a></p>

<p>example these two</p>

<pre><code>href="ppor-medio-de-la-cual-se-dictan-medidas-para-defender-el-acceso-de-los-usuarios-del-sistema-de-salud-a-medicamentos-de-calidad-eficacia-y-seguridad-acceso-de-los-usuarios-del-sistema-de-salud-a-medicamentos/8683"&gt;

href="ppor-medio-del-cual-el-congreso-de-la-republica-facultado-por-el-numeral-17-del-articulo-150-de-la-constitucion-politica-de-colombia-y-en-aras-de-facilitar-la-paz-decreta-otorgar-amnistia-e-indulto-a-los-miembros-del-grupo-armado-organizado-al-margen-de-la-ley-farc-ep/8682"&gt;
</code></pre>

<p>and at the end have a list with all possible href in that page.</p>

<p>However, by clicking in ver todos ("see all") there are more hrefs. But if you check the source page, even if you add /#page=4 or whatever page to the url, the total hrefs remain the same (actually the page source doesn't change). How could I get all those hidden hrefs? </p>

<blockquote>
  <p>Prenote: I assume you use Python 3+.</p>
</blockquote>

<p>What happens is, you click "See All", it requests an API, takes data, dumps into view. This is all AJAX process.</p>

<p>The hard and complicated way is to use, Selenium, but there is no need actually. With a little bit debug on browser, you can see <a href="http://www.congresovisible.org/proyectos-de-ley/search/proyectos-de-ley/?q=%20&amp;page=1" rel="nofollow">where it loads the data</a>.</p>

<p>This is page one. <code>q</code> is probably search query, <code>page</code> is exactly which page. 5 element per page. You can request it via <code>urllib</code> or <code>requests</code> and parse it with <code>json</code> package into a dict.</p>

<hr>

<h1>A Simple Demonstration</h1>

<p>I wanted to try it myself and it seems server we get the data from needs a <code>User-Agent</code> header to process, otherwise, it simply throws <code>403</code> (Forbidden). I am trying on Python 3.5.1.</p>

<pre><code>from urllib.request import urlopen, Request
import json

# Creating headers as dict, to pass User-Agent. I am using my own User-Agent here.
# You can use the same or just google it.
# We need to use User-Agent, otherwise, server does not accept request and returns 403.
headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36 OPR/39.0.2256.48"
}

# Creating a Request object.
# See, we pass headers, below.
req = Request("http://www.congresovisible.org/proyectos-de-ley/search/proyectos-de-ley/?q=%20&amp;page=1", headers=headers)

# Getting a response
res = urlopen(req)

# The thing is, it returns binary, we need to convert it to str in order to pass it on json.loads function.
# This is just a little bit complicated.
data_b = res.read()
data_str = data_b.decode("utf-8")

# Now, this is the magic.
data = json.loads(data_str)

print(data)

# Now you can manipulate your data. :)
</code></pre>

<h3>For Python 2.7</h3>

<ul>
<li>You can use <code>urllib2</code>. <code>urllib2</code> does not seperate into packages like it does in Python 3. So, all you have to do <code>from urllib2 import Request, urlopen</code>.</li>
</ul>

javascript python web-scraping href