extract div tag entries that are part of a table using python scrapy
<p>Im trying to extract some data on a webpage using python scrapy. I don't know enough HTML/CSS to know if this is well formatted, but it doesn't appear to be. The target information I am interested in has a pattern as shown below. A Table contains a set of entries (Name, Year, Int1, Int2) that I am interested in extracting. But these are not in the standard TD tags, instead they are part of DIV tags. here's an example:</p>

<pre><code>&lt;table width='100%'&gt;
&lt;tr&gt;
&lt;td width='50%'&gt;
&lt;div style='width: 10px; float: left'&gt;&amp;nbsp;&lt;/div&gt;
     &lt;div style='width: 232px; float: left'&gt;Mr. Richard D. Hanson&lt;/div&gt;
     &lt;div style='width: 40px; float: left'&gt;1989&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;1&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;27&lt;/div&gt;&lt;/td&gt;&lt;td width='50%'&gt;&lt;div style='width: 10px; float: left'&gt;&amp;nbsp;&lt;/div&gt;
     &lt;div style='width: 232px; float: left'&gt;Alison G. Mills, CPA&lt;/div&gt;
     &lt;div style='width: 40px; float: left'&gt;1989&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;8&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;12&lt;/div&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td width='50%'&gt;&lt;div style='width: 10px; float: left'&gt;&amp;nbsp;&lt;/div&gt;
     &lt;div style='width: 232px; float: left'&gt;Mr. Timothy D. Harrell&lt;/div&gt;
     &lt;div style='width: 40px; float: left'&gt;1989&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;28&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;28&lt;/div&gt;&lt;/td&gt;&lt;td width='50%'&gt;&lt;div style='width: 10px; float: left'&gt;&amp;nbsp;&lt;/div&gt;
     &lt;div style='width: 232px; float: left'&gt;Debora R. Mitchell, PhD&lt;/div&gt;
     &lt;div style='width: 40px; float: left'&gt;1989&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;20&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;21&lt;/div&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td width='50%'&gt;&lt;div style='width: 10px; float: left'&gt;&amp;nbsp;&lt;/div&gt;
&lt;div style='width: 232px; float: left'&gt;Mr. Tim J. Scoggins&lt;/div&gt;
     &lt;div style='width: 40px; float: left'&gt;1989&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;1&lt;/div&gt;
     &lt;div style='width: 88px; float: left; text-align: right'&gt;9&lt;/div&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/table&gt;
</code></pre>

<p>Here's what I have tried so far using the Scrapy Shell</p>

<p>Attempt 1:</p>

<p>This works, but then I need to co-relate the entries - ie get the Year and Int1 and Int2 for each Name that is accessed below</p>

<pre><code>&gt;&gt;&gt; response.xpath('//div[@style="width: 232px; float: left"]/text()').extract()
[u'Mr. Richard D. Hanson', u'Alison G. Mills, CPA', u'Mr. Timothy D. Harrell', u'Debora R. Mitchell, PhD', u'Mr. Tim J. Scoggins']
</code></pre>

<p>Attempt 2: 
In this attempt I am hoping to make one call to then iterate over each entry and store it in a dictionary. Unfortunately, Im not sure whats happening here</p>

<pre><code>&gt;&gt;&gt; response.xpath('//table[@width="100%"]/tr/td[@width="50%"]/div[@style="width: 10px; float: left"]/text()').extract()
[u'\xa0', u'\xa0', u'\xa0', u'\xa0', u'\xa0']
</code></pre>

<p>Any ideas?</p>

<p>You can get the texts of every inner <code>div</code> and then <a href="http://stackoverflow.com/q/9671224/771848">split the extracted list into chunks</a>:</p>

<pre><code>In [1]: data = response.xpath("//table/tr/td/div/text()").extract() 
In [2]: [data[x+1:x+5] for x in xrange(0, len(data), 5)]
Out[2]: 
[[u'Mr. Richard D. Hanson', u'1989', u'1', u'27'],
 [u'Alison G. Mills, CPA', u'1989', u'8', u'12'],
 [u'Mr. Timothy D. Harrell', u'1989', u'28', u'28'],
 [u'Debora R. Mitchell, PhD', u'1989', u'20', u'21'],
 [u'Mr. Tim J. Scoggins', u'1989', u'1', u'9']]
</code></pre>

python xpath scrapy