Can I scrape the raw data from highcharts.js?
<p>I want to scrape the data from a page that shows a graph using <code>highcharts.js</code>, and thus I finished to parse all the pages to get to the <a href="http://www.worldweatheronline.com/brussels-weather-averages/be.aspx" rel="nofollow">following page</a>. However, the last page, the one that displays the dataset, uses <code>highcharts.js</code> to show the graph, which it seems to be near impossible to access to the raw data.</p>

<p>I use Python 3.5 with BeautifulSoup.</p>

<p>Is it still possible to parse it? If so how can I scrape it?</p>

<p>The data is in a script tag. You can get the script tag using bs4 and a regex. You could also extract the data using a regex but I like using <a href="https://github.com/scrapinghub/js2xml" rel="nofollow">/js2xml</a> to parse js functions into a xml tree:</p>

<pre><code>from bs4 import BeautifulSoup
import requests
import re
import js2xml

soup = BeautifulSoup(requests.get("http://www.worldweatheronline.com/brussels-weather-averages/be.aspx").content, "html.parser")
script = soup.find("script", text=re.compile("Highcharts.Chart")).text
# script = soup.find("script", text=re.compile("precipchartcontainer")).text if you want precipitation data
parsed = js2xml.parse(script)
print js2xml.pretty_print(parsed)
</code></pre>

<p>That gives you:</p>

<pre><code>&lt;program&gt;
  &lt;functioncall&gt;
    &lt;function&gt;
      &lt;identifier name="$"/&gt;
    &lt;/function&gt;
    &lt;arguments&gt;
      &lt;funcexpr&gt;
        &lt;identifier/&gt;
        &lt;parameters/&gt;
        &lt;body&gt;
          &lt;var name="chart"/&gt;
          &lt;functioncall&gt;
            &lt;function&gt;
              &lt;dotaccessor&gt;
                &lt;object&gt;
                  &lt;functioncall&gt;
                    &lt;function&gt;
                      &lt;identifier name="$"/&gt;
                    &lt;/function&gt;
                    &lt;arguments&gt;
                      &lt;identifier name="document"/&gt;
                    &lt;/arguments&gt;
                  &lt;/functioncall&gt;
                &lt;/object&gt;
                &lt;property&gt;
                  &lt;identifier name="ready"/&gt;
                &lt;/property&gt;
              &lt;/dotaccessor&gt;
            &lt;/function&gt;
            &lt;arguments&gt;
              &lt;funcexpr&gt;
                &lt;identifier/&gt;
                &lt;parameters/&gt;
                &lt;body&gt;
                  &lt;assign operator="="&gt;
                    &lt;left&gt;
                      &lt;identifier name="chart"/&gt;
                    &lt;/left&gt;
                    &lt;right&gt;
                      &lt;new&gt;
                        &lt;dotaccessor&gt;
                          &lt;object&gt;
                            &lt;identifier name="Highcharts"/&gt;
                          &lt;/object&gt;
                          &lt;property&gt;
                            &lt;identifier name="Chart"/&gt;
                          &lt;/property&gt;
                        &lt;/dotaccessor&gt;
                        &lt;arguments&gt;
                          &lt;object&gt;
                            &lt;property name="chart"&gt;
                              &lt;object&gt;
                                &lt;property name="renderTo"&gt;
                                  &lt;string&gt;tempchartcontainer&lt;/string&gt;
                                &lt;/property&gt;
                                &lt;property name="type"&gt;
                                  &lt;string&gt;spline&lt;/string&gt;
                                &lt;/property&gt;
                              &lt;/object&gt;
                            &lt;/property&gt;
                            &lt;property name="credits"&gt;
                              &lt;object&gt;
                                &lt;property name="enabled"&gt;
                                  &lt;boolean&gt;false&lt;/boolean&gt;
                                &lt;/property&gt;
                              &lt;/object&gt;
                            &lt;/property&gt;
                            &lt;property name="colors"&gt;
                              &lt;array&gt;
                                &lt;string&gt;#FF8533&lt;/string&gt;
                                &lt;string&gt;#4572A7&lt;/string&gt;
                              &lt;/array&gt;
                            &lt;/property&gt;
                            &lt;property name="title"&gt;
                              &lt;object&gt;
                                &lt;property name="text"&gt;
                                  &lt;string&gt;Average Temperature (Â°c) Graph for Brussels&lt;/string&gt;
                                &lt;/property&gt;
                              &lt;/object&gt;
                            &lt;/property&gt;
                            &lt;property name="xAxis"&gt;
                              &lt;object&gt;
                                &lt;property name="categories"&gt;
                                  &lt;array&gt;
                                    &lt;string&gt;January&lt;/string&gt;
                                    &lt;string&gt;February&lt;/string&gt;
                                    &lt;string&gt;March&lt;/string&gt;
                                    &lt;string&gt;April&lt;/string&gt;
                                    &lt;string&gt;May&lt;/string&gt;
                                    &lt;string&gt;June&lt;/string&gt;
                                    &lt;string&gt;July&lt;/string&gt;
                                    &lt;string&gt;August&lt;/string&gt;
                                    &lt;string&gt;September&lt;/string&gt;
                                    &lt;string&gt;October&lt;/string&gt;
                                    &lt;string&gt;November&lt;/string&gt;
                                    &lt;string&gt;December&lt;/string&gt;
                                  &lt;/array&gt;
                                &lt;/property&gt;
                                &lt;property name="labels"&gt;
                                  &lt;object&gt;
                                    &lt;property name="rotation"&gt;
                                      &lt;number value="270"/&gt;
                                    &lt;/property&gt;
                                    &lt;property name="y"&gt;
                                      &lt;number value="40"/&gt;
                                    &lt;/property&gt;
                                  &lt;/object&gt;
                                &lt;/property&gt;
                              &lt;/object&gt;
                            &lt;/property&gt;
                            &lt;property name="yAxis"&gt;
                              &lt;object&gt;
                                &lt;property name="title"&gt;
                                  &lt;object&gt;
                                    &lt;property name="text"&gt;
                                      &lt;string&gt;Temperature (Â°c)&lt;/string&gt;
                                    &lt;/property&gt;
                                  &lt;/object&gt;
                                &lt;/property&gt;
                              &lt;/object&gt;
                            &lt;/property&gt;
                            &lt;property name="tooltip"&gt;
                              &lt;object&gt;
                                &lt;property name="enabled"&gt;
                                  &lt;boolean&gt;true&lt;/boolean&gt;
                                &lt;/property&gt;
                              &lt;/object&gt;
                            &lt;/property&gt;
                            &lt;property name="plotOptions"&gt;
                              &lt;object&gt;
                                &lt;property name="spline"&gt;
                                  &lt;object&gt;
                                    &lt;property name="dataLabels"&gt;
                                      &lt;object&gt;
                                        &lt;property name="enabled"&gt;
                                          &lt;boolean&gt;true&lt;/boolean&gt;
                                        &lt;/property&gt;
                                      &lt;/object&gt;
                                    &lt;/property&gt;
                                    &lt;property name="enableMouseTracking"&gt;
                                      &lt;boolean&gt;false&lt;/boolean&gt;
                                    &lt;/property&gt;
                                  &lt;/object&gt;
                                &lt;/property&gt;
                              &lt;/object&gt;
                            &lt;/property&gt;
                            &lt;property name="series"&gt;
                              &lt;array&gt;
                                &lt;object&gt;
                                  &lt;property name="name"&gt;
                                    &lt;string&gt;Average High Temp (Â°c)&lt;/string&gt;
                                  &lt;/property&gt;
                                  &lt;property name="color"&gt;
                                    &lt;string&gt;#FF8533&lt;/string&gt;
                                  &lt;/property&gt;
                                  &lt;property name="data"&gt;
                                    &lt;array&gt;
                                      &lt;number value="6"/&gt;
                                      &lt;number value="8"/&gt;
                                      &lt;number value="11"/&gt;
                                      &lt;number value="14"/&gt;
                                      &lt;number value="19"/&gt;
                                      &lt;number value="21"/&gt;
                                      &lt;number value="23"/&gt;
                                      &lt;number value="23"/&gt;
                                      &lt;number value="19"/&gt;
                                      &lt;number value="15"/&gt;
                                      &lt;number value="9"/&gt;
                                      &lt;number value="6"/&gt;
                                    &lt;/array&gt;
                                  &lt;/property&gt;
                                &lt;/object&gt;
                                &lt;object&gt;
                                  &lt;property name="name"&gt;
                                    &lt;string&gt;Average Low Temp (Â°c)&lt;/string&gt;
                                  &lt;/property&gt;
                                  &lt;property name="color"&gt;
                                    &lt;string&gt;#4572A7&lt;/string&gt;
                                  &lt;/property&gt;
                                  &lt;property name="data"&gt;
                                    &lt;array&gt;
                                      &lt;number value="2"/&gt;
                                      &lt;number value="2"/&gt;
                                      &lt;number value="4"/&gt;
                                      &lt;number value="6"/&gt;
                                      &lt;number value="10"/&gt;
                                      &lt;number value="12"/&gt;
                                      &lt;number value="14"/&gt;
                                      &lt;number value="14"/&gt;
                                      &lt;number value="11"/&gt;
                                      &lt;number value="8"/&gt;
                                      &lt;number value="5"/&gt;
                                      &lt;number value="2"/&gt;
                                    &lt;/array&gt;
                                  &lt;/property&gt;
                                &lt;/object&gt;
                              &lt;/array&gt;
                            &lt;/property&gt;
                          &lt;/object&gt;
                        &lt;/arguments&gt;
                      &lt;/new&gt;
                    &lt;/right&gt;
                  &lt;/assign&gt;
                &lt;/body&gt;
              &lt;/funcexpr&gt;
            &lt;/arguments&gt;
          &lt;/functioncall&gt;
        &lt;/body&gt;
      &lt;/funcexpr&gt;
    &lt;/arguments&gt;
  &lt;/functioncall&gt;
&lt;/program&gt;
</code></pre>

<p>So to get all the data:</p>

<pre><code>In [28]: from bs4 import BeautifulSoup  
In [29]: import requests
In [30]: import re    
In [31]: import js2xml    
In [32]: from itertools import repeat    
In [33]: from pprint import pprint as pp
In [34]: soup = BeautifulSoup(requests.get("http://www.worldweatheronline.com/brussels-weather-averages/be.aspx").content, "html.parser")

In [35]: script = soup.find("script", text=re.compile("Highcharts.Chart")).text

In [36]: parsed = js2xml.parse(script)

In [37]: data = [d.xpath(".//array/number/@value") for d in parsed.xpath("//property[@name='data']")]

In [38]: categories = parsed.xpath("//property[@name='categories']//string/text()")

In [39]: output =  list(zip(repeat(categories), data))    
In [40]: pp(output)
[(['January',
   'February',
   'March',
   'April',
   'May',
   'June',
   'July',
   'August',
   'September',
   'October',
   'November',
   'December'],
  ['6', '8', '11', '14', '19', '21', '23', '23', '19', '15', '9', '6']),
 (['January',
   'February',
   'March',
   'April',
   'May',
   'June',
   'July',
   'August',
   'September',
   'October',
   'November',
   'December'],
  ['2', '2', '4', '6', '10', '12', '14', '14', '11', '8', '5', '2'])]
</code></pre>

<p>Like I said you could just use a regex but <em>js2xml</em> I find is more reliable as erroneous spaces etc.. won't break it.</p>

javascript python highcharts web-scraping beautifulsoup