How to scrape charts from a website with python?
<p><strong>EDIT:</strong> </p>

<p>So I have save the script codes below to a text file but using re to extract the data still doesn't return me anything. My code is: </p>

<pre><code>file_object = open('source_test_script.txt', mode="r")
soup = BeautifulSoup(file_object, "html.parser")
pattern = re.compile(r"^var (chart[0-9]+) = new Highcharts.Chart\(({.*?})\);$", re.MULTILINE | re.DOTALL)
scripts = soup.find("script", text=pattern)
profile_text = pattern.search(scripts.text).group(1)
profile = json.loads(profile_text)

print profile["data"], profile["categories"]
</code></pre>

<hr>

<p>I would like to extract the chart's data from a website. The following is the source code of the chart. </p>

<pre><code>  &lt;script type="text/javascript"&gt;
    jQuery(function() {

    var chart1 = new Highcharts.Chart({

          chart: {
             renderTo: 'chart1',
              defaultSeriesType: 'column',
            borderWidth: 2
          },
          title: {
             text: 'Productions'
          },
          legend: {
            enabled: false
          },
          xAxis: [{
             categories: [1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016],

          }],
          yAxis: {
             min: 0,
             title: {
             text: 'Productions'
          }
          },

          series: [{
               name: 'Productions',
               data: [1,1,0,1,6,4,9,15,15,19,24,18,53,42,54,53,61,36]
               }]
       });
    });

    &lt;/script&gt;
</code></pre>

<p>There are several charts like that from the website, called "chart1", "chart2", etc. I would like to extract the following data: the categories line and the data line, for each chart: </p>

<pre><code>categories: [1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016]

data: [1,1,0,1,6,4,9,15,15,19,24,18,53,42,54,53,61,36]
</code></pre>

<p>I tried the code as you suggested but kept getting this:</p>

<pre><code>"Failed to parse chart1: while parsing a flow mapping
  in "&lt;unicode string&gt;", line 29, column 16:
          tooltip: {
                   ^
expected ',' or '}', but got '{'" 
</code></pre>

<p>My entire code: </p>

<pre><code>import json
import urllib2
import re
import sys
import yaml
from selenium.webdriver import Chrome as Browser
from bs4 import BeautifulSoup


URL = ("http://citec.repec.org/p/c/pcl20.html")
browser=Browser()
browser.get(URL)
source = browser.page_source

#file_object = open('source_test.txt', mode="r")
soup = BeautifulSoup(source, "html.parser")

pattern = re.compile(r"var (chart[0-9]+) = new Highcharts.Chart\(({.*?})\);", re.MULTILINE | re.DOTALL | re.UNICODE)

charts = {}

# find every &lt;script&gt; tag in the source using beautifulsoup
for tag in soup.find_all('script'):

    # tabs are special in yaml so remove them first
    script = tag.text.replace('\t', '')

    # find each object declaration
    for name, obj_declaration in pattern.findall(script):
        try:
            # parse the javascript declaration
            charts[name] = yaml.safe_load(obj_declaration)
        except Exception, e:
            print "Failed to parse {0}: {1}".format(name, e)

# extract the data you want
for name in charts:
    print "## {0} ##".format(name);
    print "categories:", charts[name]['xAxis'][0]['categories']
    print "data:", charts[name]['series'][0]['data']
    print
</code></pre>

python graph screen-scraping