Webscraping using Python - Link is unchanged with form input
<p>I planned to retrieve the historical data from the open web available. From the link:</p>

<p><a href="https://www.entsoe.eu/db-query/consumption/mhlv-a-specific-country-for-a-specific-day" rel="nofollow">https://www.entsoe.eu/db-query/consumption/mhlv-a-specific-country-for-a-specific-day</a></p>

<p>Ideally, I am trying to change the country, day,month,year using the inputs from a Pandas Dataframe and retrieve the results (energy consumption here in this web page) and store back to excel. </p>

<p>I am trying with different web scrappers and one information is doubtful to me about the possibilities.  </p>

<p>It is : when I manually change the Country, day, month,year and when retriving results, the web link remains unchanged. Is it possible to achieve my goal with this web link.</p>

<p>Thank you for your time. </p>

<p>First of all, you need to understand what happens when you click "Send" button. A POST request is sent to the same URL with parameters corresponding to values you've selected on the form. You can see this request in the browser developer tools - "Network" tab. Now, you need to simulate this request in your code (I'll use the awesome <a href="http://docs.python-requests.org/en/master/" rel="nofollow"><code>requests</code> package</a> below)</p>

<p>The other problem is that if you inspect what you get in the response to that POST request, you would not find the same <code>table</code> element with the desired data as you would see in the browser. This is because the <code>table</code> is dynamically generated from the <code>myData</code> javascript variable "sitting" in on of the <code>script</code> elements. Since nor <code>BeautifiulSoup</code>, nor <code>requests</code> is a browser and cannot execute JavaScript, you need to extract the <code>myData</code> value from the script.</p>

<p>Here is a working code that would get you the desired data in the "archived" scope for 01/01/2009:</p>

<pre><code>import re
from ast import literal_eval
from pprint import pprint

import requests
from bs4 import BeautifulSoup


url = "https://www.entsoe.eu/db-query/consumption/mhlv-a-specific-country-for-a-specific-day"
data = {
    "opt_period": "2",
    "opt_Country": "3",
    "opt_Day": "1",
    "opt_Month": "1",
    "opt_Year": "2009",
    "opt_Response": "1",
    "send": "send"
}
with requests.Session() as session:
    session.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'}
    # visit the page
    session.get(url)

    # make a POST request
    response = session.post(url, data=data)
    soup = BeautifulSoup(response.content, 'html.parser')

    # find the desired script
    pattern = re.compile(r"var myData = (.*?);", re.MULTILINE | re.DOTALL)
    script = soup.find("script", text=pattern)

    # extract the data from the script
    match = pattern.search(script.get_text())
    data = match.group(1).strip()
    data = literal_eval(data)

    pprint(data)
</code></pre>

<p>Prints a Python list of lists:</p>

<pre><code>[['AT',
  '2009-01-01',
  6277,
  6002,
  5649,
  5230,
  5034,
  5038,
  4858,
  5127,
  5342,
  5747,
  6100,
  6373,
  6325,
  6210,
  6129,
  6160,
  6588,
  7007,
  7058,
  6887,
  6586,
  6137,
  6494,
  5974]]
</code></pre>

<p>Also note how we are <a class='doc-link' href="http://stackoverflow.com/documentation/python/1792/web-scraping-with-python/8152/maintaining-web-scraping-session-with-requests#t=201610061906003569883">maintaining a web-scraping session</a> in this example.</p>

python web-scraping beautifulsoup scrapy mechanize