Write multiple rows from dict using csv
<p><strong>Update</strong>: I do not want to use <code>pandas</code> because I have a list of dict's and want to write each one to disk as they come in (part of webscraping workflow).</p>

<p>I have a dict that I'd like to write to a csv file. I've come up with a solution, but I'd like to know if there's a more <code>pythonic</code> solution available. Here's what I envisioned (but doesn't work):</p>

<pre><code>import csv
test_dict = {"review_id": [1, 2, 3, 4],
             "text": [5, 6, 7, 8]}

with open('test.csv', 'w') as csvfile:
    fieldnames = ["review_id", "text"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(test_dict)
</code></pre>

<p>Which would ideally result in:</p>

<pre><code>review_id text
        1    5
        2    6
        3    7
        4    8 
</code></pre>

<p>The code above doesn't seem to work that way I'd expect it to and throws a value error. So, I've  turned to following solution (which does work, but seems verbose).</p>

<pre><code>with open('test.csv', 'w') as csvfile:
    fieldnames = ["review_id", "text"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    response = test_dict
    cells = [{x: {key: val}} for key, vals in response.items()
             for x, val in enumerate(vals)]
    rows = {}
    for d in cells:
        for key, val in d.items():
          if key in rows:
            rows[key].update(d.get(key, None))
          else:
            rows[key] = d.get(key, None)     
    for row in [val for _, val in rows.items()]:
        writer.writerow(row)
</code></pre>

<p>Again, to reiterate what I'm looking for: the block of code directly above works (i.e., produces the desired result mentioned early in the post), but seems verbose. So, is there a more <code>pythonic</code> solution?</p>

<p>Thanks!</p>

<p>If you don't mind using a 3rd-party package, you could do it with <a href="http://pandas.pydata.org/" rel="nofollow"><code>pandas</code></a>.</p>

<pre><code>import pandas as pd
pd.DataFrame(test_dict).to_csv('test.csv', index=False)
</code></pre>

<p><strong>update</strong></p>

<p>So, you have several dictionaries and all of them seems to come from a scraping routine. </p>

<pre><code>import pandas as pd

test_dict = {"review_id": [1, 2, 3, 4],
             "text": [5, 6, 7, 8]}
pd.DataFrame(test_dict).to_csv('test.csv', index=False)

list_of_dicts = [test_dict, test_dict]
for d in list_of_dicts:
    pd.DataFrame(d).to_csv('test.csv', index=False, mode='a', header=False)
</code></pre>

<p>This time, you would be appending to the file and without the header.</p>

<p>The output is:</p>

<pre><code>review_id,text
1,5
2,6
3,7
4,8
1,5
2,6
3,7
4,8
1,5
2,6
3,7
4,8
</code></pre>

python csv