python, smart way of storing repeating entries
<p>I've log files containing all sort of junk as well as useful data.
I'm extracting some information by matching certain patterns and able to get in following format while reading the file line by line in python and applying some if statements </p>

<pre><code>job id: job#33ABC
 Bin 1:30.86
 Bin2: 30.86

job id: job#44BC
 Bin1: 27.22
 Bin2: 8.53
 Bin3: 35.75

job id: job#65A
 Bin2: 17.135075
 Bin4: 17.135120

job id: job#P17
 Bin 3: 7.328211
 Bin 4: 15.918724
</code></pre>

<p>Now the issue is the same log set repeats with same job ids (with different values)</p>

<pre><code>job id: job#33ABC
Bin1: 99
Bin2: 1099
...
...

...
</code></pre>

<p>If there is some smart way of writing in a file/csv in a tabular format such that by only looking at job_id, it presents all Bin1, Bin2 jobs per set, Right now it is basically lot of duplication, same job comes again with a different set of values like job#33ABC keep coming 10,11 times with different values</p>

<pre><code>job_id        bin 1, bin2, bin3,bin4   
                 set#1                       set #2             set #3
job#33ABC     30.86, 30.86, 0, 0         30.86, 30.86, 0,0  
job#44BC      27.22, 8.53, 35.75, 0      0,0,0,34.56
....
...
</code></pre>

<p>I'm reading that log line by line</p>

<pre><code>  for line in input_file:
     if job_name in line:
        &lt;extract job_name logic&gt;  
        print job_name[0]
    if 'bin1 matches'
         bin1[0]=&lt;all logic&gt;
         print "bin1", 
         bin1[0] 
        ..
        ...
</code></pre>

<p>UPDATE
I tried using a dictionary like</p>

<pre><code>records{}
for line in input_file:
     if job_name in line:
        &lt;extract job_name logic&gt;  
        print job_name[0]

    if 'bin1 matches'
         bin1[0]=&lt;all logic&gt;
         print "bin1", 
         bin1[0] 
         records[job_name[0]]=records.get(job_name[0],[])+[bin1[0]] 

    if 'bin2 matches'
        ...     
        ..
        records[job_name[0]]=records.get(job_name[0],[])+[bin2[0]]   

for key, value in records.items():
      writer.writerow([key, value])
</code></pre>

<p>but it is presenting in following format;</p>

<pre><code>    33ABC, " ['30.86','30.86','99.0','1099' ]
</code></pre>

<p>My question is how can i identify and present like
         33ABC, " ['30.86','30.86',,'99.0','1099',, ] Since there has to be 4 job bins, right now it is taking all values as one large list, instead of breaking into 4, 4 bins or is there any way of doing that in current logic?</p>

<p>Putting the issue of IO aside, you could use a <a href="https://docs.python.org/3/library/collections.html#defaultdict-objects" rel="nofollow"><code>defaultdict</code></a> for easy logistics. Or a <code>defaultdict</code> of <code>defaultdict</code>s to be precise.</p>

<p>Your outer dict can have keys corresponding to job names. The value for each job name is a dict itself, with keys corresponding to bin names. So for each job name and key name you try to append an element to the value of the inner dict. If the bin had come up by then, the value gets appended. If the bin is new, the default empty list is used, and the first value is appended (<code>defaultdict</code>s are only needed to avoid testing for existing keys all the time):</p>

<pre><code>from collections import defaultdict
logs = defaultdict(lambda: defaultdict(list))

# simulate the following input stream:
#
# job1:
#    bin1: val1
#    bin1: val2
#    bin2: val3
#
# job2:
#    bin2: val4
#
# job1:
#    bin1: val5

logs['job1']['bin1'].append('val1')
logs['job1']['bin1'].append('val2')
logs['job1']['bin2'].append('val3')
logs['job2']['bin2'].append('val4')
logs['job1']['bin1'].append('val5')

# see what we've got, converted to a non-default dict for prettiness
print({k:dict(logs[k]) for k in logs})
</code></pre>

<p>This will return</p>

<pre><code>{'job1': {'bin1': ['val1', 'val2', 'val5'], 'bin2': ['val3']}, 'job2': {'bin2': ['val4']}}
</code></pre>

<p>You can see that values are collected across job and bin names. You can perform the writing according to your taste, but I don't think a CSV makes sense in this context. You probably have to use a printer of your own.</p>

<p>You just have to loop over <code>logs</code> first, this will give you the unique job names (the keys), then loop over each bin for the given job, then print the lists of values:</p>

<pre><code>for lkey in logs:
    # lkey is a job name, logs[lkey] is a defaultdict
    print(lkey)
    for bkey in logs[lkey]:
         # logs[lkey][bkey] is a list of values which you can print         
         print(bkey)
         print(logs[lkey][bkey])
</code></pre>

<p>The output from the above is</p>

<pre><code>job1
bin1
['val1', 'val2', 'val5']
bin2
['val3']
job2
bin2
['val4']
</code></pre>

<p>which is what we'd expect.</p>

python file dictionary text