Merging two dictionaries together and adding None to the desired location
<p>I've scraped some <a href="http://www.richmond.com/data-center/salaries-local-government-employees-2014/?Agency=&amp;Last_Name=&amp;First_Name=&amp;Job_Title=PRINCIPAL&amp;appSession=82100926574322760625479516407609655632811717398403303845564132296054295099051816928601026788234447191565075848437884339109744304&amp;RecordID=29541&amp;PageID=8&amp;PrevPageID=2&amp;cpipage=1&amp;CPIsortType=&amp;CPIorderBy=&amp;cbCurrentRecordPosition=1&amp;Mod0LinkToDetails=True" rel="nofollow">data</a> and due to the way the website was structured I put the data into two dictionaries. </p>

<pre><code>&gt;&gt;&gt;pprint(dict(data))
{u'Additional compensation': [u'$32,241'],
 u'Agency': [u'Chesterfield County Schools', u'City of Richmond Schools'],
 u'Bonuses or other allowances': [u'$12,500'],
 u'COMMENTS': [u'$28,088 - Board Paid Annuity; $4,153 - Excess Health Benefit Contribution;',
               u''],
 u'Full Name': [u'Marcus J. Newsome', u'Dana T. Bedden'],
 u'Total Compensation': [u'$282,258', u'']}

&gt;&gt;&gt;pprint(dict(data2))
{u'Base Salary': [u'$229,758', u'$234,068'],
 u'COMMENTS': [u'12,500 CAR ALLOWANCE, 40,000 DEFFERRED COMPENSATION'],
 u'Deferred compensation': [u'$40,000'],
 u'Job Title': [u'SUPERINTENDENT', u'SUPERINTENDENT'],
 u'Total Compensation': [u'$266,309'],
 u'Work location': [u'Office Of Superintendent']}
</code></pre>

<p>I've combined the data into one master dictionary and I've attempted to put it into a csv file. </p>

<pre><code>for d in data2, data:
    for k, v in d.iteritems():
        master_data[k].append(v)

with open('test2.csv', 'wb') as f:
    writer = csv.writer(f)
    writer.writerows(zip(*([k] + master_data[k] for k in sorted(master_data))))
</code></pre>

<p>The problem is that only the first person's (<code>Marcus J. Newsome</code>) information is exported to the csv. I think this is due to fact that there are keys/values (for example <code>Additional compensation</code>) that belong to <code>Dana T. Bedden</code> that aren't present in <code>Marcus J Newsome</code>'s data. </p>

<p>To address this I've tried adding <code>None</code> to the positions to fix this problem. </p>

<pre><code>for d in data2, data:
    master_data.update((k, [None, master_data[k]]) for k in master_data if k not in d)

&gt;&gt;&gt;pprint(dict(master_data))
{u'Additional compensation': [None, [[u'$32,241']]],
 u'Agency': [None,
             [[u'Chesterfield County Schools', u'City of Richmond Schools']]],
 u'Base Salary': [None, [[u'$229,758', u'$234,068']]],
 u'Bonuses or other allowances': [None, [[u'$12,500']]],
 u'COMMENTS': [[u'12,500 CAR ALLOWANCE, 40,000 DEFFERRED COMPENSATION'],
               [u'$28,088 - Board Paid Annuity; $4,153 - Excess Health Benefit Contribution;',
                u'']],
 u'Deferred compensation': [None, [[u'$40,000']]],
 u'Full Name': [None, [[u'Marcus J. Newsome', u'Dana T. Bedden']]],
 u'Job Title': [None, [[u'SUPERINTENDENT', u'SUPERINTENDENT']]],
 u'Total Compensation': [[u'$266,309'], [u'$282,258', u'']],
 u'Work location': [None, [[u'Office Of Superintendent']]]}
</code></pre>

<p>Unfortunately, this doesn't seem to work the way I want it to. Ultimately I would like to have my output look like this:</p>

<p><strong>Desired Output</strong></p>

<pre><code>{u'Additional compensation': [[None, [u'$32,241']]],
 u'Agency': [[u'Chesterfield County Schools'], [u'City of Richmond Schools']]],
 u'Base Salary': [[u'$229,758'], [u'$234,068']]],
 u'Bonuses or other allowances': [[u'$12,500'], None]],
 u'COMMENTS': [[u'12,500 CAR ALLOWANCE, 40,000 DEFFERRED COMPENSATION'],
               [u'$28,088 - Board Paid Annuity; $4,153 - Excess Health Benefit Contribution;',
                u'']],
 u'Deferred compensation': [[u'$40,000'], None]],
 u'Full Name': [[u'Marcus J. Newsome'], [u'Dana T. Bedden']]],
 u'Job Title': [[u'SUPERINTENDENT'], [u'SUPERINTENDENT']]],
 u'Total Compensation': [[u'$266,309'], [u'$282,258', u'']],
 u'Work location': [None, [u'Office Of Superintendent']]]}
</code></pre>

<p>Anyone have any ideas?</p>

<p>It'd be much better to change the way you store scraped data.</p>

<p>Pseudocode:</p>

<pre><code>data = []
for row in table:
    person = get_data_from_row(row)
    person.update(get_data_from_person_page(row))
    data.append(person)
</code></pre>

<p>Then you can use <a href="https://docs.python.org/3/library/csv.html#csv.DictWriter" rel="nofollow"><code>csv.DictWriter</code></a> without any complex data manipulation:</p>

<pre><code>with open('data.csv', 'w') as f:
    fieldnames = data[0].keys()
    writer = csv.DictWriter(f, fieldnames)
    writer.writeheader()
    for row in data:
        writer.writerow(row)
</code></pre>

python python-2.7 csv dictionary