Get HTML file after Javascript executed with Scrapy + Splash
<p>I want to crawl page that includes javascript using Scrapy and Splash.</p>

<p>In page, <code>&lt;script type = text/javascript&gt; JS_FUNCTIONS(generate html content) &lt;/script&gt;</code> exists, so I tried to get html file after running JS_FUNCTIONS like below.</p>

<pre><code>import scrapy
from scrapy_splash import SplashRequest

class FooSpider(scrapy.Spider):
    name = 'foo'
    start_urls = ["http://foo.com"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, args={'wait': 0.5})

    def parse(self, response):
        file_name = response.url.split("//")[-1]
        with open(filename, 'wb') as f:
            f.write(response.body)
</code></pre>

<p>when I execute command <code>scrapy crawl foo</code>, it returns html file that still including <code>&lt;script type = text/javascript&gt; JS_FUNCTIONS(generate html content) &lt;/script&gt;</code> and doesn't contain html content, which should be generated by JS_FUNCTIONS.</p>

<p>How can I get html file including contents that is generated by javascript?</p>

<p>Thanks.</p>

<p>Maybe try executing with the following lua code:</p>

<pre><code>lua_code = """
function main(splash)
  local url = splash.args.url
  assert(splash:go(url))
  assert(splash:wait(0.5))
  return {
    html = splash:html(),
  }
end
                    """

SplashRequest(url,self.parse, args={'lua_source': lua_code}, endpoint='execute')
</code></pre>

python scrapy web-crawler scrapy-splash