Extract AJAX content and java scripted content using scrapy
<p>I am trying to crawling this <a href="http://www.freeindex.co.uk/profile(the-main-event-management-company)_266537.htm" rel="nofollow">site</a> and want to extract contact number which is inside the call button.</p>

<p>How can I implement this code??</p>

<p>Seems like a simple AJAX request is being made to retrieve html string with the phone numbers:</p>

<p><a href="http://i.stack.imgur.com/HNQ7y.png" rel="nofollow"><img src="http://i.stack.imgur.com/HNQ7y.png" alt="enter image description here"></a></p>

<pre><code>import re
import scrapy

class MySpider(scrapy.Spider):
    name = 'sophone'
    start_urls = [
        'http://www.freeindex.co.uk/profile(the-main-event-management-company)_266537.htm'
    ]

    def parse(self, response):
        # item id can be extracted from url
        item_id = re.findall("(\d+)\.htm", response.url)[0]
        # phone api can be made using this id
        url = 'http://www.freeindex.co.uk/customscripts' \
              '/popup_view_tel_details.asp?id={}'.format(item_id)
        yield scrapy.Request(url, self.parse_phone)

    def parse_phone(self, response):
        from scrapy.shell import inspect_response
        inspect_response(response, self)
</code></pre>

python scrapy web-crawler