scrapy NBA schedule not collating correctly
<p>Trying to get a simple web scrape up and running. The goal is to dump the dt gm tm and ntv classes into csv - eventually. Here it is json for clarity. One step at a time.</p>

<p>here's the spider:</p>

<pre><code>import scrapy

class QuotesSpider(scrapy.Spider):
    name = "schedule"
    start_urls = [
        'http://www.nba.com/schedules/national_tv_schedule/',
    ]

    def parse(self, response):
        for game in response.css('td'):
            yield {
                'date': game.css('td.dt::text').extract(),
                'time': game.css('td.tm::text').extract(),
            }
</code></pre>

<p>really simple - but spits out like so: (truncated for brevity)</p>

<pre><code>[
{"date": ["Sat, Oct 1", " ", "Sun, Oct 2", "Mon, Oct 3", " ", " ", " ", " ", " ", " "], "time": ["7:30 pm", "8:00 pm", "8:00 pm", "2:30 pm", "8:00 pm", "8:00 pm", "8:30 pm", "9:00 pm", "10:00 pm", "10:00 pm", "7:00 pm", "7:00 pm", "8:00 pm", "8:00 pm", "10:00 pm", "10:30 pm", "2:30 pm", "7:00 pm", "10:00 pm", "10:30 pm", "7:00 pm", "7:00 pm", "7:30 pm", "7:30 pm", "8:00 pm", "10:30 pm", "10:00 pm"]},
{"date": [], "time": []},
{"date": [], "time": []},
{"date": [], "time": []},
{"date": [], "time": []},
{"date": [], "time": []},
{"date": ["Sat, Oct 1"], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["7:30 pm"]},
{"date": [], "time": []},
{"date": [" "], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["8:00 pm"]},
{"date": [], "time": []},
{"date": ["Sun, Oct 2"], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["8:00 pm"]},
{"date": [], "time": []},
{"date": ["Mon, Oct 3"], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["2:30 pm"]},
{"date": [], "time": []},
{"date": [" "], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["8:00 pm"]},
{"date": [], "time": []},
{"date": [" "], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["8:00 pm"]},
{"date": [], "time": []},
{"date": [" "], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["8:30 pm"]},
{"date": [], "time": []},
{"date": [" "], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["9:00 pm"]},
{"date": [], "time": []},
{"date": [" "], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["10:00 pm"]},
{"date": [], "time": []},
{"date": [" "], "time": []},
{"date": [], "time": []},
{"date": [], "time": ["10:00 pm"]},
{"date": [], "time": []}
]
</code></pre>

<p>The first dict has the right data in the right order, but not collated. The following dicts have not matched up the data in the first dict correctly. I tried a while statement to take out newlines but was unsuccessful.</p>

<p>Any suggestions? I built this using the Scrapy tutorial. I know that I will eventually need to insert the correct dates.</p>

<p>You probably want to be more specific on the table and rows you select.</p>

<p>Take a look at the HTML for the start of the schedule:</p>

<pre><code>   &lt;div id="scheduleMain" style="margin:0 5px 0 0!important;"&gt;
   &lt;table border="0" cellpadding="0" cellspacing="0" class="genSchedTable tvindex"&gt;
      &lt;tr class="header"&gt;
         &lt;td colspan="4"&gt;NATIONAL TV SCHEDULE - 2016-17&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr class="title"&gt;
         &lt;td class="date"&gt;Date&lt;/td&gt;
         &lt;td class="game"&gt;Teams&lt;/td&gt;
         &lt;td class="time"&gt;Time (ET)&lt;/td&gt;
         &lt;td class="natTV"&gt;Network&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
         &lt;td class="dt"&gt;Mon, Oct 3&lt;/td&gt;
         &lt;td class="gm"&gt;&lt;a href="/thunder"&gt;Oklahoma City&lt;/a&gt; @ &lt;a href="/real_madrid"&gt;Real Madrid&lt;/a&gt;&lt;br&gt;Preseason

         &lt;/td&gt;
         &lt;td class="tm"&gt;2:30 pm&lt;/td&gt;
         &lt;td class="ntv"&gt;&lt;img border="0" src="http://i.cdn.turner.com/nba/nba/images/shrinkee_NBATV.gif"&gt;&lt;img border="0" src="http://i.cdn.turner.com/nba/nba/images/shrinkee_NBAC.gif"&gt;&lt;/td&gt;
      &lt;/tr&gt;
      ...
</code></pre>

<p>You can see that the table you're after is inside a <code>&lt;div id="scheduleMain"&gt;</code>.</p>

<p>And you should select table rows ( <code>&lt;tr&gt;</code>), not table cells in your for loop, and in each loop iteration, select the cells for time and date:</p>

<pre><code>def parse(self, response):
    for game in response.css('#scheduleMain &gt; table tr:nth-child(n+3)'):
        yield {
            'date': game.css('td.dt::text').extract(),
            'time': game.css('td.tm::text').extract(),
        }
</code></pre>

<p><code>tr:nth-child(n+3)</code> is used to select rows 3, 4, 5... (the 1st 2 rows are headers)</p>

python scrapy