Python, Scrapy problems when parsing tables in Earning Reports
<p>I am trying to parse some data from the table (the balance sheet) under every earning report. Here I use AMD as an example, but not limited to AMD.</p>

<p>Here is <a href="http://www.marketwired.com/press-release/amd-reports-2016-second-quarter-results-nasdaq-amd-2144535.htm" rel="nofollow">the link</a></p>

<p>The problem I have now is that I cannot get any reading - my spider always returns EMPTY result. I used <code>scrapy shell "http://example.com"</code> to test my xpath, which I directly copied from Google Chrome Inspector, and it still didn't work.</p>

<p>Here is my xpath (Chrome browser provided):</p>

<pre><code>//*[@id="newsroom-copy"]/div[2]/div[8]/table/tbody/tr[9]/td[4]/text()
</code></pre>

<p>Here is my code:</p>

<pre><code>import scrapy

class ESItem(scrapy.Item):
    Rev = scrapy.Field()

class ESSpider(scrapy.Spider):
    name = "es"
    start_urls = [
        'http://www.marketwired.com/press-release/amd-reports-2016-second-quarter-results-nasdaq-amd-2144535.htm',
    ]

    def parse(self, response):
        item = ESItem()
        for earning in response.xpath('//*[@id="newsroom-copy"]/div[2]/div[8]/table/tbody'):
            item['Rev'] = earning.xpath('tr[9]/td[4]/text()').extract_first()
            yield item
</code></pre>

<p>I am looking for retrieving the "revenue numbers" from the table on the bottom of the report.</p>

<p>Thanks!</p>

<p>I run my code by using this command:</p>

<pre><code>scrapy runspider ***.py -o ***.json
</code></pre>

<p>Code runs fine, no error, just didn't return what I really look for.</p>

<p><strong>UPDATE</strong>: I kind of figure out something... I have to remove that "tbody" tag from the XPATH, which I don't understand... Can anyone explain this a little bit please?</p>

<p>The html provided by the inspect tool in chrome is the result of the browser interpretation of the actual code that it is sent by the server to your browser.</p>

<p>The <code>tbody</code> tag is a prime example. If you view the page source of a website you'll see a structure like this</p>

<pre><code>&lt;table&gt;
    &lt;tr&gt;
        &lt;td&gt;&lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;
</code></pre>

<p>Now if you inspect the page this happens</p>

<pre><code>&lt;table&gt;
    &lt;tbody&gt;
        &lt;tr&gt;
            &lt;td&gt;&lt;/td&gt;
        &lt;/tr&gt;
    &lt;/tbody&gt;
&lt;/table&gt;
</code></pre>

<p>What scrapy gets is the page source and not the "inspector" so whenever you try to select something in a page make sure it exists on the page source.</p>

<p>Another example of this is when you try to select some element that is generated by javascript while the page is being loaded. Scrapy won't get this either so you'll need to use something else to interpret it like scrapy-splash or selenium.</p>

<p>As a side note, take the time to learn xpath and css selectors. It's a time saver when you know how to query elements just right.</p>

<pre><code>//*[@id='newsroom-copy']/div[2]/div[8]/table/tr[9]/td[4]/text()
</code></pre>

<p>is equivalent to</p>

<pre><code>//table/tr[td/text()='Net revenue']/td[4]/text()
</code></pre>

<p>See how much nicer it looks?</p>

python python-2.7 scrapy scrapy-spider