Writing multiple rows into CSV file
<p>I'm trying to write multiple rows in to a CSV file using python and I've been working on this code for a while to piece together how to do this. My goal here is simply to use the oxford dictionary website, and web-scrape the year and words created for each year into a csv file. I want each row to start with the year I'm searching for and then list all the words across horizontally. Then, I want to be able to repeat this for multiple years.</p>

<p>Here's my code so far:</p>

<pre><code>import requests
import re 
import urllib2
import os
import csv

year_search = 1550
subject_search = ['Law'] 

path = '/Applications/Python 3.5/Economic'
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

user_agent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
header = {'User-Agent':user_agent}
request = urllib2.Request('http://www.oed.com/', None, header)
f = opener.open(request)  
data = f.read()
f.close()
print 'database first access was successful'

resultPath = os.path.join(path, 'OED_table.csv')
htmlPath = os.path.join(path, 'OED.html')
outputw = open(resultPath, 'w')
outputh = open(htmlPath, 'w')
request = urllib2.Request('http://www.oed.com/search?browseType=sortAlpha&amp;case-insensitive=true&amp;dateFilter='+str(year_search)+'&amp;nearDistance=1&amp;ordered=false&amp;page=1&amp;pageSize=100&amp;scope=ENTRY&amp;sort=entry&amp;subjectClass='+str(subject_search)+'&amp;type=dictionarysearch', None, header)
page = opener.open(request)
urlpage = page.read()
outputh.write(urlpage)
new_word = re.findall(r'&lt;span class=\"hwSect\"&gt;&lt;span class=\"hw\"&gt;(.*?)&lt;/span&gt;', urlpage)
print str(new_word)
outputw.write(str(new_word))
page.close()
outputw.close()
</code></pre>

<p>This outputs my string of words that were identified for the year 1550. Then I tried to make code write to a csv file on my computer, which it does, but I want to do two things that I'm messing up here:</p>

<ol>
<li>I want to be able to insert multiple rows into this
and </li>
<li>I want to have the year show up in the first spot</li>
</ol>

<p>Next part of my code:</p>

<pre><code>with open('OED_table.csv', 'w') as csvfile:
    fieldnames = ['year_search']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    writer.writerow({'year_search': new_word})
</code></pre>

<p>I was using the <code>csv</code> module's <a href="https://docs.python.org/3/library/csv.html" rel="nofollow">online documentation</a> as a reference for the second part of the code. </p>

<p>And just to clarify, I included the first part of the code in order to give perspective.</p>

<p>You really shouldn't parse html with a regex. That said, here's how to modify your code to produce a csv file of all the words found. </p>

<p>Note: for unknown reasons the list of result word varies in length from one execution to the next.</p>

<pre><code>import csv
import os
import re
import requests
import urllib2

year_search = 1550
subject_search = ['Law']

path = '/Applications/Python 3.5/Economic'
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

user_agent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
header = {'User-Agent':user_agent}

# commented out because not used
#request = urllib2.Request('http://www.oed.com/', None, header)
#f = opener.open(request)
#data = f.read()
#f.close()
#print 'database first access was successful'

resultPath = os.path.join(path, 'OED_table.csv')
htmlPath = os.path.join(path, 'OED.html')
request = urllib2.Request(
    'http://www.oed.com/search?browseType=sortAlpha&amp;case-insensitive=true&amp;dateFilter='
    + str(year_search)
    + '&amp;nearDistance=1&amp;ordered=false&amp;page=1&amp;pageSize=100&amp;scope=ENTRY&amp;sort=entry&amp;subjectClass='
    + str(subject_search)
    + '&amp;type=dictionarysearch', None, header)
page = opener.open(request)

with open(resultPath, 'wb') as outputw, open(htmlPath, 'w') as outputh:
    urlpage = page.read()
    outputh.write(urlpage)

    new_words = re.findall(
        r'&lt;span class=\"hwSect\"&gt;&lt;span class=\"hw\"&gt;(.*?)&lt;/span&gt;', urlpage)
    print new_words
    csv_writer = csv.writer(outputw)
    for word in new_words:
        csv_writer.writerow([year_search, word])
</code></pre>

<p>Here's the contents of the <code>OED_table.csv</code> file when it works:</p>

<pre class="lang-none prettyprint-override"><code>1550,above bounden
1550,accomplice
1550,baton
1550,civilist
1550,garnishment
1550,heredity
1550,maritime
1550,municipal
1550,nil
1550,nuncupate
1550,perjuriously
1550,rank
1550,semi-
1550,torture
1550,unplace
</code></pre>

python csv web-scraping