Data extraction: Creating dictionary of dictionaries with lists in python
<p>I have data similar to the following in a file:</p>

<pre><code>Name, Age, Sex, School, height, weight, id

Joe, 10, M, StThomas, 120, 20, 111

Jim, 9, M, StThomas, 126, 22, 123

Jack, 8, M, StFrancis, 110, 15, 145

Abel, 10, F, StFrancis, 128, 23, 166
</code></pre>

<p>The actual data might be 100 columns and a million rows.</p>

<p>What I am trying to do is create a dict in the following pattern:</p>

<pre><code>school_data = {'StThomas': {'weight':[20,22], 'height': [120,126]},
               'StFrancis': {'weight':[15,23], 'height': [110,128]} }
</code></pre>

<p>Things I tried:</p>

<ol>
<li><p>Trial 1: (very expensive in terms of computation)</p>

<pre><code>school_names  = []
for lines in read_data[1:]:
    data = lines.split('\t')
    school_names.append(data[3])

school_names = set(school_names)

for lines in read_data[1:]:
    for school in schools:
        if school in lines:
            print lines
</code></pre></li>
<li><p>Trial 2:</p>

<pre><code>for lines in read_data[1:]:
    data = lines.split('\t')
    school_name = data[3]
    height = data[4]
    weight = data[5]
    id = data [6]
    x[id] = {school_name: (weight, height)}
</code></pre></li>
</ol>

<p>The above two are methods in which I tried to proceed but did not get closer to the solution.</p>

<p>The easiest way to do this within the standard library is using existing tools, <a href="https://docs.python.org/3/library/csv.html#csv.DictReader" rel="nofollow"><code>csv.DictReader</code></a> and <a href="https://docs.python.org/3/library/collections.html#collections.defaultdict" rel="nofollow"><code>collections.defaultdict</code></a>:</p>

<pre><code>from collections import defaultdict
from csv import DictReader

data = defaultdict(lambda: defaultdict(list))  #Â *

with open(datafile) as file_:
    for row in DictReader(file_):
        data[row[' School'].strip()]['height'].append(int(row[' height']))
        data[row[' School'].strip()]['weight'].append(int(row[' weight']))
</code></pre>

<p>Note that the spaces in e.g. <code>' School'</code> and the <code>.strip()</code> are necessary because of the spaces in the header row of your input file. Result:</p>

<pre><code>&gt;&gt;&gt; data
defaultdict(&lt;function &lt;lambda&gt; at 0x10261c0c8&gt;, {'StFrancis': defaultdict(&lt;type 'list'&gt;, {'weight': [15, 23], 'height': [110, 128]}), 'StThomas': defaultdict(&lt;type 'list'&gt;, {'weight': [20, 22], 'height': [120, 126]})})
&gt;&gt;&gt; data['StThomas']['height']
[120, 126]
</code></pre>

<p>Alternatively, if you're planning to do further analysis, look into something like <a href="http://pandas.pydata.org/" rel="nofollow"><code>pandas</code></a> and its <code>DataFrame</code> data structure.</p>

<p>* <em>see <a href="http://stackoverflow.com/q/8419401/3001761">Python defaultdict and lambda</a> if this seems weird</em> </p>

python dictionary