Scrapy csv file has uniform empty rows?
<p>here is the spider:</p>

<pre><code>import scrapy
from danmurphys.items import DanmurphysItem

class MySpider(scrapy.Spider):
    name = 'danmurphys'
    allowed_domains = ['danmurphys.com.au']
    start_urls = ['https://www.danmurphys.com.au/dm/navigation/navigation_results_gallery.jsp?params=fh_location%3D%2F%2Fcatalog01%2Fen_AU%2Fcategories%3C%7Bcatalog01_2534374302084767_2534374302027742%7D%26fh_view_size%3D120%26fh_sort%3D-sales_value_30_days%26fh_modification%3D&amp;resetnav=false&amp;storeExclusivePage=false']


    def parse(self, response):        
        urls = response.xpath('//h2/a/@href').extract()
        for url in urls:            
            request = scrapy.Request(url , callback=self.parse_page)      
            yield request

    def parse_page(self , response):
        item = DanmurphysItem()
        item['brand'] = response.xpath('//span[@itemprop="brand"]/text()').extract_first().strip()
        item['name'] = response.xpath('//span[@itemprop="name"]/text()').extract_first().strip()
        item['url'] = response.url     
        return item
</code></pre>

<p>and here is the items :   </p>

<pre><code>import scrapy
class DanmurphysItem(scrapy.Item):  
    brand = scrapy.Field()
    name = scrapy.Field()
    url = scrapy.Field()
</code></pre>

<p>when I run the spider with this command :</p>

<pre><code>scrapy crawl danmurphys -o output.csv
</code></pre>

<p>the output is like this :
<a href="http://i.stack.imgur.com/p2AAL.png" rel="nofollow"><img src="http://i.stack.imgur.com/p2AAL.png" alt="enter image description here"></a></p>

<p>This output shows the typical symptom of csv file handle opened using <code>"w"</code> mode on windows (to fix Python 3 compatibility maybe) but omitting <code>newline</code>.</p>

<p>While this has no effect on Linux/Unix based systems, on Windows, 2 carriage return chars are issued, inserting a fake blank line after every data line.</p>

<pre><code>with open("output.csv","w") as f:
     cr = csv.writer(f)
</code></pre>

<p>correct way of doing it (python 3):</p>

<pre><code>with open("output.csv","w",newline='') as f:  # python 3
     cr = csv.writer(f)
</code></pre>

<p>(in python 2, setting <code>"wb"</code> as open mode fixes it)</p>

<p>If the file is created by a program you cannot or do not want to modify, you can always post-process the file as follows:</p>

<pre><code>with open("output.csv","rb") as f:
   with open("output_fix.csv","w") as f2:
       f2.write(f.read().decode().replace("\r","")) # python 3
       f2.write(f.read().replace("\r","")) # python 2
</code></pre>

python scrapy