Write multiple rows from dict using csv
<p><strong>Update</strong>: I do not want to use <code>pandas</code> because I have a list of dict's and want to write each one to disk as they come in (part of webscraping workflow).</p>

<p>I have a dict that I'd like to write to a csv file. I've come up with a solution, but I'd like to know if there's a more <code>pythonic</code> solution available. Here's what I envisioned (but doesn't work):</p>

<pre><code>import csv
test_dict = {"review_id": [1, 2, 3, 4],
             "text": [5, 6, 7, 8]}

with open('test.csv', 'w') as csvfile:
    fieldnames = ["review_id", "text"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(test_dict)
</code></pre>

<p>Which would ideally result in:</p>

<pre><code>review_id text
        1    5
        2    6
        3    7
        4    8 
</code></pre>

<p>The code above doesn't seem to work that way I'd expect it to and throws a value error. So, I've  turned to following solution (which does work, but seems verbose).</p>

<pre><code>with open('test.csv', 'w') as csvfile:
    fieldnames = ["review_id", "text"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    response = test_dict
    cells = [{x: {key: val}} for key, vals in response.items()
             for x, val in enumerate(vals)]
    rows = {}
    for d in cells:
        for key, val in d.items():
          if key in rows:
            rows[key].update(d.get(key, None))
          else:
            rows[key] = d.get(key, None)     
    for row in [val for _, val in rows.items()]:
        writer.writerow(row)
</code></pre>

<p>Again, to reiterate what I'm looking for: the block of code directly above works (i.e., produces the desired result mentioned early in the post), but seems verbose. So, is there a more <code>pythonic</code> solution?</p>

<p>Thanks!</p>

<p>You have two different problems in your question:</p>

<ol>
<li>Create a csv file from a dictionary where the values are containers and not primitives.</li>
</ol>

<p>For the first problem, the solution is generally to transform the container type into a primitive type.  The most common method is creating a json-string.  So for example:</p>

<pre><code>&gt;&gt;&gt; import json
&gt;&gt;&gt; x = [2, 4, 6, 8, 10]
&gt;&gt;&gt; json_string = json.dumps(x)
&gt;&gt;&gt; json_string
'[2, 4, 6, 8, 10]'
</code></pre>

<p>So your data conversion might look like:</p>

<pre><code>import json


def convert(datadict):
    '''Generator which converts a dictionary of containers into a dictionary of json-strings.

    args:
        datadict(dict): dictionary which needs conversion

    yield:
        tuple: key and string
    '''
    for key, value in datadict.items():
        yield key, json.dumps(value)


def dump_to_csv_using_dict(datadict, fields=None, filepath=None, delimiter=None):
    '''Dumps a datadict value into csv

    args:
        datadict(list): list of dictionaries to dump
        fieldnames(list): field sequence to use from the dictionary [default: sorted(datadict.keys())]
        filepath(str): filepath to save to  [default: 'tmp.csv']
        delimiter(str): delimiter to use in csv [default: '|']
    '''
    fieldnames = sorted(datadict.keys()) if fields is None else fields
    filepath = 'tmp.csv' if filepath is None else filepath
    delimiter = '|' if not delimiter else delimiter
    with open(filepath, 'w') as csvfile:
       writer = csv.DictWriter(csvfile, fieldnames, restval='', extrasaction='ignore', delimiter=delimiter)
       writer.writeheader()
       for each_dict in datadict:
           writer.writerow(each_dict)
</code></pre>

<p>So the naive conversion looks like this:</p>

<pre><code># Conversion code
test_data = {
    "review_id": [1, 2, 3, 4],
         "text": [5, 6, 7, 8]}
}

converted_data = dict(convert(test_data))
data_list = [converted_data]
dump_to_csv(data_list)
</code></pre>

<ol start="2">
<li>Create a final value that is actually some sort of a merging of two disparate data sets.</li>
</ol>

<p>To do this, you need to find a way to combine data from different keys.  This is not an easy problem to generically solve.</p>

<p>That said, it's easy to combine two lists with zip.</p>

<pre><code>&gt;&gt;&gt; x = [2, 4, 6]
&gt;&gt;&gt; y = [1, 3, 5]
&gt;&gt;&gt; zip(y, x)
[(1, 2), (3, 4), (5, 6)]
</code></pre>

<p>In addition, in the event that your lists are not the same size, python's itertools package provides a method, izip_longest, which will yield back the full zip even if one list is shorter than another.  Note izip_longest returns a generator.</p>

<pre><code>from itertools import izip_longest
&gt;&gt;&gt; x = [2, 4]
&gt;&gt;&gt; y = [1, 3, 5]
&gt;&gt;&gt; z = izip_longest(y, x, fillvalue=None)  # default fillvalue is None
&gt;&gt;&gt; list(z)  # z is a generator
[(1, 2), (3, 4), (5, None)]
</code></pre>

<p>So we could add another function here:</p>

<pre><code>from itertoops import izip_longest

def combine(data, fields=None, default=None):
    '''Combines fields within data

    args:
        data(dict): a dictionary with lists as values
        fields(list): a list of keys to combine [default: all fields in random order]
        default: default fill value [default: None]
    yields:
        tuple: columns combined into rows
    '''
    fields = data.keys() if field is None else field
    columns = [data.get(field) for field in fields]
    for values in izip_longest(*columns, fillvalue=default):
        yield values
</code></pre>

<p>And now we can use this to update our original conversion.</p>

<pre><code>def dump_to_csv(data, filepath=None, delimiter=None):
    '''Dumps list into csv

    args:
        data(list): list of values to dump
        filepath(str): filepath to save to  [default: 'tmp.csv']
        delimiter(str): delimiter to use in csv [default: '|']
    '''
    fieldnames = sorted(datadict.keys()) if fields is None else fields
    filepath = 'tmp.csv' if filepath is None else filepath
    delimiter = '|' if not delimiter else delimiter
    with open(filepath, 'w') as csvfile:
       writer = csv.writer(csvfile, delimiter=delimiter)
       for each_row in data:
           writer.writerow(each_dict)

# Conversion code
test_data = {
    "review_id": [1, 2, 3, 4],
         "text": [5, 6, 7, 8]}
}

combined_data = combine(test_data)
data_list = [combined_data]
dump_to_csv(data_list)    
</code></pre>

python csv