python webscraping and write data into csv
<p>I'm trying to save all the data(i.e all pages) in single csv file but this code only save the final page data.Eg Here  url[] contains 2 urls. the final csv only contains the 2nd url data.
I'm clearly doing something wrong in the loop.but i dont know what.
And also this page contains 100 data points. But this code only write first 44 rows.
please help this issue.............</p>

<pre><code>from bs4 import BeautifulSoup
import requests
import csv
url = ["http://sfbay.craigslist.org/search/sfc/npo","http://sfbay.craigslist.org/search/sfc/npo?s=100"]
for ur in url:
    r = requests.get(ur)
    soup = BeautifulSoup(r.content)
    g_data = soup.find_all("a", {"class": "hdrlnk"})
    gen_list=[]
    for row in g_data:
       try:
            name = row.text
       except:
            name=''
       try:
            link = "http://sfbay.craigslist.org"+row.get("href")
       except:
            link=''
       gen=[name,link]
       gen_list.append(gen)

with open ('filename2.csv','wb') as file:
    writer=csv.writer(file)
    for row in gen_list:
        writer.writerow(row)
</code></pre>

<p>the gen_list is being initialized again inside your loop that runs over the urls.</p>

<pre><code>gen_list=[]
</code></pre>

<p>Move this line outside the for loop.</p>

<pre><code>...
url = ["http://sfbay.craigslist.org/search/sfc/npo","http://sfbay.craigslist.org/search/sfc/npo?s=100"]
gen_list=[]
for ur in url:
...
</code></pre>

python python-2.7 csv web-scraping beautifulsoup