Python3, nested dict comparison (recursive?)
<p>I'm writing a program to take a .csv file and create 'metrics' for ticket closure data. Each ticket has one or more time entries; the goal is to grab the 'delta' (ie - time difference) for <code>open</code> -> <code>close</code> and <code>time_start</code> -> <code>time_end</code> on a PER TICKET basis; these are not real variables, they're just for the purpose of this question.</p>

<p>So, say we have ticket 12345 that has 3 time entries like so:</p>

<p><code>ticket: 12345 
open: 2016-09-26 00:00:00.000 close: 2016-09-27 00:01:00.000 
time_start: 2016-09-26 00:01:00.000 time_end: 2016-09-26 00:02:00.000
ticket: 12345 
open: 2016-09-26 00:00:00.000 close: 2016-09-27 00:01:00.000 
time_start: 2016-09-26 00:01:00.000 time_end: 2016-09-26 00:02:00.000
ticket: 12345 
open: 2016-09-26 00:00:00.000 close: 2016-09-27 00:01:00.000 
time_start: 2016-09-26 00:01:00.000 time_end: 2016-09-27 00:02:00.000</code></p>

<p>I'd like to have the program display ONE entry for this, adding up the 'deltas', like so:</p>

<p><code>ticket: 12345
Delta open/close ($total time from open to close):
Delta start/end: ($total time of ALL ticket time entries added up)</code></p>

<p>Here's what I have so far; </p>

<p>.csv example: </p>

<pre><code>Ticket #,Ticket Type,Opened,Closed,Time Entry Day,Start,End
737385,Software,2016-09-06 12:48:31.680,2016-09-06 15:41:52.933,2016-09-06 00:00:00.000,1900-01-01 15:02:00.417,1900-01-01 15:41:00.417
737318,Hardware,2016-09-06 12:20:28.403,2016-09-06 14:35:58.223,2016-09-06 00:00:00.000,1900-01-01 14:04:00.883,1900-01-01 14:35:00.883
737296,Printing/Scan/Fax,2016-09-06 11:37:10.387,2016-09-06 13:33:07.577,2016-09-06 00:00:00.000,1900-01-01 13:29:00.240,1900-01-01 13:33:00.240
737273,Software,2016-09-06 10:54:40.177,2016-09-06 13:28:24.140,2016-09-06 00:00:00.000,1900-01-01 13:17:00.860,1900-01-01 13:28:00.860
737261,Software,2016-09-06 10:33:09.070,2016-09-06 13:19:41.573,2016-09-06 00:00:00.000,1900-01-01 13:05:00.113,1900-01-01 13:15:00.113
737238,Software,2016-09-06 09:52:57.090,2016-09-06 14:42:16.287,2016-09-06 00:00:00.000,1900-01-01 12:01:00.350,1900-01-01 12:04:00.350
737238,Software,2016-09-06 09:52:57.090,2016-09-06 14:42:16.287,2016-09-06 00:00:00.000,1900-01-01 14:36:00.913,1900-01-01 14:42:00.913
737220,Password,2016-09-06 09:28:16.060,2016-09-06 11:41:16.750,2016-09-06 00:00:00.000,1900-01-01 11:30:00.303,1900-01-01 11:36:00.303
737197,Hardware,2016-09-06 08:50:23.197,2016-09-06 14:02:18.817,2016-09-06 00:00:00.000,1900-01-01 13:48:00.530,1900-01-01 14:02:00.530
736964,Internal,2016-09-06 01:02:27.453,2016-09-06 05:46:00.160,2016-09-06 00:00:00.000,1900-01-01 06:38:00.917,1900-01-01 06:45:00.917
</code></pre>

<p>class Time_Entry.py:</p>

<pre><code>#! /usr/bin/python
from datetime import *

class Time_Entry:

def __init__(self, ticket_no, time_entry_day, opened, closed, start, end):
    self.ticket_no = ticket_no
    self.time_entry_day = time_entry_day
    self.opened = opened
    self.closed = closed
    self.start = datetime.strptime(start, '%Y-%m-%d %H:%M:%S.%f')
    self.end = datetime.strptime(end, '%Y-%m-%d %H:%M:%S.%f')
    self.total_open_close_delta = 0
    self.total_start_end_delta = 0

def open_close_delta(self, topen, tclose):
    open_time = datetime.strptime(topen, '%Y-%m-%d %H:%M:%S.%f')
    if tclose != '\\N':
        close_time = datetime.strptime(tclose, '%Y-%m-%d %H:%M:%S.%f')
        self.total_open_close_delta = close_time - open_time

def start_end_delta(self, tstart, tend):
    start_time = datetime.strptime(tstart, '%Y-%m-%d %H:%M:%S.%f')
    end_time = datetime.strptime(tend, '%Y-%m-%d %H:%M:%S.%f')
    start_end_delta = (end_time - start_time).seconds
    self.total_start_end_delta += start_end_delta
    return (self.total_start_end_delta)

def add_start_end_delta(self, delta):
    self.total_start_end_delta += delta

def display(self):
    print('Ticket #: %7.7s Start: %-15s End: %-15s Delta: %-10s' % (self.ticket_no, self.start.time(), self.end.time(), self.total_start_end_delta))
</code></pre>

<p>Which is called by metrics.py:</p>

<pre><code>#! /usr/bin/python

import csv
import pprint
from Time_Entry import *

file = '/home/jmd9qs/userdrive/metrics.csv'

# setup CSV, load up a list of dicts
reader = csv.DictReader(open(file))
dict_list = []

for line in reader:
    dict_list.append(line)

def load_tickets(ticket_list):
    for i, key in enumerate(ticket_list):
        ticket_no = key['Ticket #']
        time_entry_day = key['Time Entry Day']
        opened = key['Opened']
        closed = key['Closed']
        start = key['Start']
        end = key['End']

        time_entry = Time_Entry(ticket_no, time_entry_day, opened, closed, start, end)
        time_entry.open_close_delta(opened, closed)
        time_entry.start_end_delta(start, end)

        for h, key2 in enumerate(ticket_list):
            ticket_no2 = key2['Ticket #']
            time_entry_day2 = key2['Time Entry Day']
            opened2 = key2['Opened']
            closed2 = key2['Closed']
            start2 = key2['Start']
            end2 = key2['End']
            time_entry2 = Time_Entry(ticket_no2, time_entry_day2, opened2, closed2, start2, end2)

            if time_entry.ticket_no == time_entry2.ticket_no and i != h:
                # add delta and remove second time_entry from dict (no counting twice)
                time_entry2_delta = time_entry2.start_end_delta(start2, end2)
                time_entry.add_start_end_delta(time_entry2_delta)
                del dict_list[h]
    time_entry.display()

load_tickets(dict_list)
</code></pre>

<p>This seems to work OK so far; however, I get multiple lines of output per ticket instead of one with the 'deltas' added. FYI the way the program displays output is different from my example, which is intentional. See example below:</p>

<pre><code>Ticket #:  738388 Start: 15:24:00.313000 End: 15:35:00.313000 Delta: 2400      
Ticket #:  738388 Start: 16:30:00.593000 End: 16:40:00.593000 Delta: 1260      
Ticket #:  738381 Start: 15:40:00.763000 End: 16:04:00.767000 Delta: 1440      
Ticket #:  738357 Start: 13:50:00.717000 End: 14:10:00.717000 Delta: 1200      
Ticket #:  738231 Start: 11:16:00.677000 End: 11:21:00.677000 Delta: 720       
Ticket #:  738203 Start: 16:15:00.710000 End: 16:31:00.710000 Delta: 2160      
Ticket #:  738203 Start: 09:57:00.060000 End: 10:02:00.060000 Delta: 1560      
Ticket #:  738203 Start: 12:26:00.597000 End: 12:31:00.597000 Delta: 900       
Ticket #:  738135 Start: 13:25:00.880000 End: 13:50:00.880000 Delta: 2040      
Ticket #:  738124 Start: 07:56:00.117000 End: 08:31:00.117000 Delta: 2100      
Ticket #:  738121 Start: 07:47:00.903000 End: 07:52:00.903000 Delta: 300       
Ticket #:  738115 Start: 07:15:00.443000 End: 07:20:00.443000 Delta: 300       
Ticket #:  737926 Start: 06:40:00.813000 End: 06:47:00.813000 Delta: 420       
Ticket #:  737684 Start: 18:50:00.060000 End: 20:10:00.060000 Delta: 13380     
Ticket #:  737684 Start: 13:00:00.560000 End: 13:08:00.560000 Delta: 8880      
Ticket #:  737684 Start: 08:45:00        End: 10:00:00        Delta: 9480      
</code></pre>

<p>Note that there are a few tickets with more than one entry, which is what I don't want.</p>

<p>Any notes on style, convention, etc. are also welcome as I'm trying to be more 'Pythonic'</p>

<p>The problem here is that with a nested loop like the one you implemented you double-examine the same ticket. Let me explain it better:</p>

<pre><code>ticket_list = [111111, 111111, 666666, 777777] # lets simplify considering the ids only

# I'm trying to keep the same variable names
for i, key1 in enumerate(ticket_list): # outer loop

    cnt = 1

    for h, key2 in enumerate(ticket_list): # inner loop
        if key1 == key2 and i != h:
            print('&gt;&gt; match on i:', i, '- h:', h)
            cnt += 1

    print('Found', key1, cnt, 'times')
</code></pre>

<p>See how it double counts the <code>111111</code></p>

<pre><code>&gt;&gt; match on i: 0 - h: 1
Found 111111 2 times
&gt;&gt; match on i: 1 - h: 0
Found 111111 2 times
Found 666666 1 times
Found 777777 1 times
</code></pre>

<p>That's because you will match the <code>111111</code> both when the inner loop examines the first position and the outer the second (<code>i: 0, h: 1</code>), and again when the outer is on the second position and the inner is on the first (<code>i: 1, h: 0</code>).</p>

<hr>

<h2>A proposed solution</h2>

<p>A better solution for your problem is to group the entries of the same ticket together and then sum your deltas. <a href="https://docs.python.org/2/library/itertools.html#itertools.groupby" rel="nofollow"><code>groupby</code></a> is ideal for your task. Here I took the liberty to rewrite some code:</p>

<p>Here I modified the constructor in order to accept the dictionary itself. It makes passing the parameters later less messy. I also removed the methods to add the deltas, later we'll see why.  </p>

<pre><code>import csv
import itertools
from datetime import *

class Time_Entry(object):

    def __init__(self, entry):
        self.ticket_no = entry['Ticket #']
        self.time_entry_day = entry['Time Entry Day']
        self.opened = datetime.strptime(entry['Opened'], '%Y-%m-%d %H:%M:%S.%f')
        self.closed = datetime.strptime(entry['Closed'], '%Y-%m-%d %H:%M:%S.%f')
        self.start = datetime.strptime(entry['Start'], '%Y-%m-%d %H:%M:%S.%f')
        self.end = datetime.strptime(entry['End'], '%Y-%m-%d %H:%M:%S.%f')
        self.total_open_close_delta = (self.closed - self.opened).seconds
        self.total_start_end_delta = (self.end - self.start).seconds


    def display(self):
        print('Ticket #: %7.7s Start: %-15s End: %-15s Delta: %-10s' % (self.ticket_no, self.start.time(), self.end.time(), self.total_start_end_delta))
</code></pre>

<p>Here we load the data using <a href="https://docs.python.org/2/tutorial/datastructures.html#list-comprehensions" rel="nofollow">list comprehensions</a>, the final output will be a the list of <code>Time_Entry</code> objects:</p>

<pre><code>with open('metrics.csv') as ticket_list:
    time_entry_list = [Time_Entry(line) for line in csv.DictReader(ticket_list)]

print(time_entry_list)
# [&lt;Time_Entry object at 0x101142f60&gt;, &lt;Time_Entry object at 0x10114d048&gt;, &lt;Time_Entry object at 0x1011fddd8&gt;, ... ]
</code></pre>

<p>In the nested-loop version instead you kept rebuilding the <code>Time_Entry</code> inside the inner loop, which means for 100 entries you end up initializing 10000 temporary variables! Creating a list "outside" instead allows us to initialize each <code>Time_Entry</code> only once.</p>

<p>Here comes the magic: we can use the <code>groupby</code> in order to collect all the objects with the same <code>ticket_no</code> in the same list:</p>

<pre><code>sorted(time_entry_list, key=lambda x: x.ticket_no)
ticket_grps = itertools.groupby(time_entry_list, key=lambda x: x.ticket_no)

tickets = [(id, [t for t in tickets]) for id, tickets in ticket_grps]
</code></pre>

<p>The final result in <code>ticket</code> is a list tuples with the ticket id in the first position, and the list of associated <code>Time_Entry</code> in the last:</p>

<pre><code>print(tickets)
# [('737385', [&lt;Time_Entry object at 0x101142f60&gt;]),
#  ('737318', [&lt;Time_Entry object at 0x10114d048&gt;]),
#  ('737238', [&lt;Time_Entry object at 0x1011fdd68&gt;, &lt;Time_Entry object at 0x1011fde80&gt;]),
#  ...]
</code></pre>

<p>So finally we can iterate over all the tickets, and using again a list comprehension we can build a list containing only the deltas so we can sum them together. You can see why we removed the old method to update the deltas, since now we simply store their value for the single entry and then sum them externally.</p>

<p>Here is your result:</p>

<pre><code>for ticket in tickets:
    print('ticket:', ticket[0])
    # extract list of deltas and then sum
    print('Delta open / close:', sum([entry.total_open_close_delta for entry in ticket[1]]))
    print('Delta start / end:', sum([entry.total_start_end_delta for entry in ticket[1]]))
    print('(found {} occurrences)'.format(len(ticket[1])))
    print()
</code></pre>

<p>Output:</p>

<pre><code>ticket: 736964
Delta open / close: 17012
Delta start / end: 420
(found 1 occurrences)

ticket: 737197
Delta open / close: 18715
Delta start / end: 840
(found 1 occurrences)

ticket: 737220
Delta open / close: 7980
Delta start / end: 360
(found 1 occurrences)

ticket: 737238
Delta open / close: 34718
Delta start / end: 540
(found 2 occurrences)

ticket: 737261
Delta open / close: 9992
Delta start / end: 600
(found 1 occurrences)

ticket: 737273
Delta open / close: 9223
Delta start / end: 660
(found 1 occurrences)

ticket: 737296
Delta open / close: 6957
Delta start / end: 240
(found 1 occurrences)

ticket: 737318
Delta open / close: 8129
Delta start / end: 1860
(found 1 occurrences)

ticket: 737385
Delta open / close: 10401
Delta start / end: 2340
(found 1 occurrences)
</code></pre>

<p>At the end of the story: list comprehensions can be super-useful, they allows you to do a lot of stuff with a super-compact syntax. Also the python standard library contains a lot of ready-to-use tools that can really come to your aid, so get familiar!</p>

python list csv dictionary recursion