Create new headers when writing a csv using python
<p>IÂ´m web scrapping different webpages and for each webpage IÂ´m writing each row of the csv file</p>

<pre><code>import csv
fieldnames=["Title", "Author", "year"]
counter=1
for webpage of webpages:
    if counter==1:
        f = open('file.csv', 'wb')  
        my_writer = csv.DictWriter(f, fieldnames)
        my_writer.writeheader()
        f.close()

    something where I get the information (title, author and year) for each webpage

    variables={ele:"NA" for ele in fieldnames}
    variables['Title']=title        
    variables['Author']=author
    variables['year']=year


    with open('file.csv', 'a+b') as f:
    header = next(csv.reader(f))
    dict_writer = csv.DictWriter(f, header)
    dict_writer.writerow(variables) 
    counter+=1
</code></pre>

<p>However, there could be more than one author (so author after web scrapping is actually a list) so I would like to have in the headers of the csv file: author1, author2, author3, etc. But I don't know what would be the maximum number of authors. So in the loop I would like to edit the header and start adding author2,author3 etc depending if in that row is necessary to create more authors.</p>

<p>It could be something like:</p>

<pre><code>def write_to_csv(file_name, records, fieldnames=None):

    import csv
    from datetime import datetime

    with open('/tmp/' + file_name, 'w') as csvfile:
        if not fieldnames:
            fieldnames = records[0].keys()
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames,   extrasaction='ignore')
        writer.writeheader()
        for row in records:
            writer.writerow(row)

def scrap():
    for webpage of webpages:
        webpage_data = [{'title':'','author1':'foo','author2':'bar'}] #sample data
        write_to_csv(webpage[0].title+'csv', webpage_data,webpage_data[0].keys())
</code></pre>

<p>I`m assuming:</p>

<ul>
<li>Data will be consistent for the same webpage, but differ the next webpage in loop</li>
<li>webpage data is a list of dictionaries, having values mapped to keys</li>
<li>the above code is based on Python 3</li>
</ul>

<p>So in the loop, we`ll just get the data, and pass the relevant fieldnames and the values to another function, so be able to write it to csv.</p>

python csv