Use scrapy with selenium for dynamic pages
<p>I want to crawl this pages for all the exibitors:</p>

<p><a href="https://greenbuildexpo.com/Attendee/Expohall/Exhibitors" rel="nofollow">https://greenbuildexpo.com/Attendee/Expohall/Exhibitors</a></p>

<p>But scrapy doesn't load the content, what I'm doing now it using selenium to load the page and search the links with scrapy:</p>

<pre><code>url = 'https://greenbuildexpo.com/Attendee/Expohall/Exhibitors'

driver_1 = webdriver.Firefox()
driver_1.get(url)
content = driver_1.page_source

response = TextResponse(url='',body=content,encoding='utf-8')
print len(set(response.xpath('//*[contains(@href,"Attendee/")]//@href').extract()))
</code></pre>

<p>The site doesn't seem to make any new request when "next" button is pressed so I was hoping to get all links at one, but I'm only getting 43 links with that code. They should be around 500.</p>

<p>Now I'm trying to crawl the page by pressing the "next" button:</p>

<pre><code>for i in range(10):
    xpath = '//*[@id="pagingNormalView"]/ul/li[15]'
    driver_1.find_element_by_xpath(xpath).click()
</code></pre>

<p>but I got an error:</p>

<pre><code>File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"xpath","selector":"//*[@id=\"pagingNormalView\"]/ul/li[15]"}
Stacktrace:
</code></pre>

<p>You don't need <code>selenium</code> for that, there is an XHR request to get all exhibitors, simulate it, demo from the <a href="https://doc.scrapy.org/en/latest/topics/shell.html" rel="nofollow">Scrapy Shell</a>:</p>

<pre><code>$ scrapy shell https://greenbuildexpo.com/Attendee/Expohall/Exhibitors
In [1]: fetch("https://greenbuildexpo.com/Attendee/ExpoHall/GetAllExhibitors")
2016-10-13 12:45:46 [scrapy] DEBUG: Crawled (200) &lt;GET https://greenbuildexpo.com/Attendee/ExpoHall/GetAllExhibitors&gt; (referer: None)

In [2]: import json

In [3]: data = json.loads(response.body)

In [4]: len(data["Data"])
Out[4]: 541

# printing booth number for demonstration purposes
In [5]: for item in data["Data"]:
   ...:     print(item["BoothNumber"])
   ...:  
2309
2507
...
1243
2203
943
</code></pre>

python selenium xpath scrapy