Extracting JS variable information inside script tag
<p>I'm retrieving an HTML page from a URL and want to extract information from a <code>script</code> tag within that HTML. I am specifically looking for this particular <code>script</code> tag:</p>

<pre><code>&lt;script type="text/javascript"&gt;
    var zomato = zomato || {};
    zomato.menuPages = [{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/4bab50546bf3314e25dea4310ddf524e.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/4bab50546bf3314e25dea4310ddf524e.jpg","filename":"4bab50546bf3314e25dea4310ddf524e.jpg","url_master":"menus_original\/705\/51705\/4bab50546bf3314e25dea4310ddf524e.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/4bab50546bf3314e25dea4310ddf524e.jpg","data_center":"sng","menu_type":"FOOD","title":"FOOD","menu_type_class":"FOOD","real_menu_type":"FOOD","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344370},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/0a284792e41edbb5ba5bbc7b0cde26db.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/0a284792e41edbb5ba5bbc7b0cde26db.jpg","filename":"0a284792e41edbb5ba5bbc7b0cde26db.jpg","url_master":"menus_original\/705\/51705\/0a284792e41edbb5ba5bbc7b0cde26db.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/0a284792e41edbb5ba5bbc7b0cde26db.jpg","data_center":"sng","menu_type":"FOOD","title":"FOOD","menu_type_class":"FOOD","real_menu_type":"FOOD","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344371},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/6ff338c3891bca1cc61574e9864b15ae.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/6ff338c3891bca1cc61574e9864b15ae.jpg","filename":"6ff338c3891bca1cc61574e9864b15ae.jpg","url_master":"menus_original\/705\/51705\/6ff338c3891bca1cc61574e9864b15ae.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/6ff338c3891bca1cc61574e9864b15ae.jpg","data_center":"sng","menu_type":"FOOD","title":"FOOD","menu_type_class":"FOOD","real_menu_type":"FOOD","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344365},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/ff5a5ea0945782ad1d82102461a39b52.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/ff5a5ea0945782ad1d82102461a39b52.jpg","filename":"ff5a5ea0945782ad1d82102461a39b52.jpg","url_master":"menus_original\/705\/51705\/ff5a5ea0945782ad1d82102461a39b52.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/ff5a5ea0945782ad1d82102461a39b52.jpg","data_center":"sng","menu_type":"FOOD","title":"FOOD","menu_type_class":"FOOD","real_menu_type":"FOOD","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344366},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/3cb04e221c4db345ceb41b638d9faa6a.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/3cb04e221c4db345ceb41b638d9faa6a.jpg","filename":"3cb04e221c4db345ceb41b638d9faa6a.jpg","url_master":"menus_original\/705\/51705\/3cb04e221c4db345ceb41b638d9faa6a.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/3cb04e221c4db345ceb41b638d9faa6a.jpg","data_center":"sng","menu_type":"FOOD","title":"FOOD","menu_type_class":"FOOD","real_menu_type":"FOOD","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344367},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/733759862d474dfd8e710fa08e78849b.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/733759862d474dfd8e710fa08e78849b.jpg","filename":"733759862d474dfd8e710fa08e78849b.jpg","url_master":"menus_original\/705\/51705\/733759862d474dfd8e710fa08e78849b.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/733759862d474dfd8e710fa08e78849b.jpg","data_center":"sng","menu_type":"FOOD","title":"FOOD","menu_type_class":"FOOD","real_menu_type":"FOOD","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344368},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/69144be9b82cbba9adcc9de35003522d.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/69144be9b82cbba9adcc9de35003522d.jpg","filename":"69144be9b82cbba9adcc9de35003522d.jpg","url_master":"menus_original\/705\/51705\/69144be9b82cbba9adcc9de35003522d.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/69144be9b82cbba9adcc9de35003522d.jpg","data_center":"sng","menu_type":"FOOD","title":"FOOD","menu_type_class":"FOOD","real_menu_type":"FOOD","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344369},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/9dfd7dcc0e45639acbde792781012e0d.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/9dfd7dcc0e45639acbde792781012e0d.jpg","filename":"9dfd7dcc0e45639acbde792781012e0d.jpg","url_master":"menus_original\/705\/51705\/9dfd7dcc0e45639acbde792781012e0d.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/9dfd7dcc0e45639acbde792781012e0d.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344483},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/b89c707ff99087cd8098ddaf3b5f1346.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/b89c707ff99087cd8098ddaf3b5f1346.jpg","filename":"b89c707ff99087cd8098ddaf3b5f1346.jpg","url_master":"menus_original\/705\/51705\/b89c707ff99087cd8098ddaf3b5f1346.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/b89c707ff99087cd8098ddaf3b5f1346.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344484},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/439bf88da8bfce35ba44c6f206360a90.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/439bf88da8bfce35ba44c6f206360a90.jpg","filename":"439bf88da8bfce35ba44c6f206360a90.jpg","url_master":"menus_original\/705\/51705\/439bf88da8bfce35ba44c6f206360a90.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/439bf88da8bfce35ba44c6f206360a90.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344485},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/854abd602c815f84dcaa2fdea1c22f81.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/854abd602c815f84dcaa2fdea1c22f81.jpg","filename":"854abd602c815f84dcaa2fdea1c22f81.jpg","url_master":"menus_original\/705\/51705\/854abd602c815f84dcaa2fdea1c22f81.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/854abd602c815f84dcaa2fdea1c22f81.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344486},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/7670f299fd8f065252b94665df390790.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/7670f299fd8f065252b94665df390790.jpg","filename":"7670f299fd8f065252b94665df390790.jpg","url_master":"menus_original\/705\/51705\/7670f299fd8f065252b94665df390790.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/7670f299fd8f065252b94665df390790.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344487},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/f308c376afe08aed9b4ccf38eb0d6652.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/f308c376afe08aed9b4ccf38eb0d6652.jpg","filename":"f308c376afe08aed9b4ccf38eb0d6652.jpg","url_master":"menus_original\/705\/51705\/f308c376afe08aed9b4ccf38eb0d6652.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/f308c376afe08aed9b4ccf38eb0d6652.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344488},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/c734ed73e8e5f15f2e3ef9e287bf86f7.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/c734ed73e8e5f15f2e3ef9e287bf86f7.jpg","filename":"c734ed73e8e5f15f2e3ef9e287bf86f7.jpg","url_master":"menus_original\/705\/51705\/c734ed73e8e5f15f2e3ef9e287bf86f7.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/c734ed73e8e5f15f2e3ef9e287bf86f7.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344489},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/0cbe5c590f3d5312238de6b00cc9b0a9.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/0cbe5c590f3d5312238de6b00cc9b0a9.jpg","filename":"0cbe5c590f3d5312238de6b00cc9b0a9.jpg","url_master":"menus_original\/705\/51705\/0cbe5c590f3d5312238de6b00cc9b0a9.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/0cbe5c590f3d5312238de6b00cc9b0a9.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344490},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/3db4f82866e075bb1852990a0cdbe30a.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/3db4f82866e075bb1852990a0cdbe30a.jpg","filename":"3db4f82866e075bb1852990a0cdbe30a.jpg","url_master":"menus_original\/705\/51705\/3db4f82866e075bb1852990a0cdbe30a.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/3db4f82866e075bb1852990a0cdbe30a.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344477},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/1e0df9160c02273466e96239eae1a555.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/1e0df9160c02273466e96239eae1a555.jpg","filename":"1e0df9160c02273466e96239eae1a555.jpg","url_master":"menus_original\/705\/51705\/1e0df9160c02273466e96239eae1a555.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/1e0df9160c02273466e96239eae1a555.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344478},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/0d7fd654ec9f090883fa428df0f1ebb2.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/0d7fd654ec9f090883fa428df0f1ebb2.jpg","filename":"0d7fd654ec9f090883fa428df0f1ebb2.jpg","url_master":"menus_original\/705\/51705\/0d7fd654ec9f090883fa428df0f1ebb2.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/0d7fd654ec9f090883fa428df0f1ebb2.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344479},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/da16fc1d8d9641581fca258cbcb99f80.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/da16fc1d8d9641581fca258cbcb99f80.jpg","filename":"da16fc1d8d9641581fca258cbcb99f80.jpg","url_master":"menus_original\/705\/51705\/da16fc1d8d9641581fca258cbcb99f80.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/da16fc1d8d9641581fca258cbcb99f80.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344480},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/5336b40a11d4486db5e3a4bcfb0e9ae8.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/5336b40a11d4486db5e3a4bcfb0e9ae8.jpg","filename":"5336b40a11d4486db5e3a4bcfb0e9ae8.jpg","url_master":"menus_original\/705\/51705\/5336b40a11d4486db5e3a4bcfb0e9ae8.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/5336b40a11d4486db5e3a4bcfb0e9ae8.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344481},{"url":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/c96e340be834ecf086536234a56e7626.jpg","href":"https:\/\/b.zmtcdn.com\/data\/menus\/705\/51705\/c96e340be834ecf086536234a56e7626.jpg","filename":"c96e340be834ecf086536234a56e7626.jpg","url_master":"menus_original\/705\/51705\/c96e340be834ecf086536234a56e7626.jpg","path_master":"\/home\/foodie\/zomato_data\/menus_original\/705\/51705\/c96e340be834ecf086536234a56e7626.jpg","data_center":"sng","menu_type":"BAR","title":"BAR","menu_type_class":"BAR","real_menu_type":"BAR","is_salt_special_menu":0,"start_date":"","consumer_upload":0,"start_date_formatted":"","end_date":"","end_date_formatted":"","id":129344482}];
    zomato.menuTypes = ["DEFAULT","FOOD","BAR","DELIVERY","SPECIAL","TAKEAWAY","INTERNAL"];
    zomato.currentMenuPage = 1;
&lt;/script&gt;
</code></pre>

<p>That list goes on for quite a while. I am using <a href="/questions/tagged/beautifulsoup" class="post-tag" title="show questions tagged &#39;beautifulsoup&#39;" rel="tag">beautifulsoup</a>. This:</p>

<pre><code>soup.find_all('script')[14] 
</code></pre>

<p>gives me the exact <code>script</code> tag I want. However, once I have done that, I am not sure how to parse further.</p>

<p>Is there a way I can access <code>zomato.menuPages</code> as a python list and then access its elements? If there are no decent Python solutions, maybe something in JS?</p>

<p>Well, a way to do that is to make use of <strong>regular expressions</strong> which of course is not the most easy thing to do but comes in handy sometimes. So what I tried is the following:</p>

<pre><code>#!/usr/bin/env python

from BeautifulSoup import BeautifulSoup
import requests  
import re 

# I didn't post the url for typical reasons 
url = "the_url"

r = requests.get(url)
response = r.text
soup = BeautifulSoup(response)
x = soup.findAll(name = 'script')[14] 


# use regular expression
values = re.findall(r'zomato..*?=\s*(.*?);', str(x), re.DOTALL | re.MULTILINE)
</code></pre>

<p>So what this regular expression will do in this case is to give you 
a list consisted of 4 elements - for example the 2nd element will be 
<code>zomato.menuPages</code> as you asked for.
Then you can process <code>zomato.menuPages</code> a bit more, like:</p>

<pre><code>k = ''.join(values[1])
w = k[1:-1]
list = w.split("{",21)
print list[1]
</code></pre>

<p>Then you could try to convert the items of the <code>list</code> from string -> dictionary to be able to parse them more easily(using json or ast maybe).</p>

<p>Also you could parse each of these values(I refer to the values list) using another regular expression.
Also using the <code>groupdict</code> function from the regex module you could create a dictionary from each element of the list based on certain regex rules.
Well, I hope it will be of some help!</p>

javascript python beautifulsoup html-parsing