Efficient iteration through nested python dictionaries
<p>I have a dataset for which values determined to meet specific criteria is used to perform probability calculations as part of a summation. Currently, I hold the data in nested dictionaries to simplify the process of deterministic processing. </p>

<p>The algorithm I'm using proves to be very expensive and after a while overwhelms the memory.</p>

<p>The psudocode for the processing is as follows:</p>

<pre><code>for businessA in business : # iterate over 77039 values 
    for businessB in business : # iterate over 77039 values
        if businessA != businessB :
            for rating in business[businessB] : # where rating is 1 - 5
                for review in business[businessB][rating] :
                    user = reviewMap[review]['user'];
                    if user in business[businessA]['users'] :
                        for users in business[businessA]['users'] :
                            # do something
                # do probability
                # a print is here
</code></pre>

<p>How can I write the above more effectively to maintain accurate probability summation for each businessA?</p>

<hr>

<p><strong>EDIT</strong> including source code - here, businessA and businessB are in seperate dictionaries, however it is of note that they hold the same businessIDs (bid) in each. It is just a change of what the value is for each key:value pair.</p>

<pre><code>def crossMatch(TbidMap) :
    for Tbid in TbidMap :
        for Lbid in LbidMap :
            # Ensure T and L aren't the same business
            if Tbid != Lbid :
                # Get numer of reviews at EACH STAR rate for L
                for stars in LbidMap[Lbid] :
                    posTbid = 0;
                    # For each review check if user rated the Tbid
                    for Lreview in LbidMap[Lbid][stars] :
                        user = reviewMap[Lreview]['user'];
                        if user in TbidMap[Tbid] :
                            # user rev'd Tbid, get their Trid &amp; see if gave Tbid pos rev
                            for Trid in TbidMap[Tbid][user] :
                                Tstar = reviewMap[Trid]['stars'];
                                if Tstar in pos_list :
                                    posTbid += 1;
                    #probability calculations happen here
</code></pre>

<p>There are over 5 billion combinations of companies in your dataset, which is really going to stress the memory out. I think you're storing all results into memory; instead, I would do interim dumps to a database and free up your containers. This is a sketch of the approach as I have no real data to test on, and it might be easier to respond to your difficulties as you encounter them. Ideally there would be an interim container for nested lists so that you could use <code>executemany</code> but this is so heavily nested with abbreviated names and no test data that it's difficult to follow.</p>

<pre><code>import sqlite3

def create_interim_mem_dump(cursor, connection):

    query = """CREATE TABLE IF NOT EXISTS ratings(
            Tbid TEXT,
            Lbid TEXT,
            posTbid TEXT)
            """
    cursor.execute(query)
    connection.commit()


def crossMatch(TbidMap, cursor, connection) :
    for Tbid in TbidMap :
        for Lbid in LbidMap :
            # Ensure T and L aren't the same business
            if Tbid != Lbid :
                # Get numer of reviews at EACH STAR rate for L
                for stars in LbidMap[Lbid] :
                    posTbid = 0;
                    # For each review check if user rated the Tbid
                    for Lreview in LbidMap[Lbid][stars] :
                        user = reviewMap[Lreview]['user'];
                        if user in TbidMap[Tbid] :
                            # user rev'd Tbid, get their Trid &amp; see if gave Tbid pos rev
                            for Trid in TbidMap[Tbid][user] :
                                Tstar = reviewMap[Trid]['stars'];
                                if Tstar in pos_list :
                                    posTbid += 1;   
                    query = """INSERT INTO ratings (Tbid, Lbid, posTbid) 
                            VALUES (?, ?, ?)"""
                    cursor.execute(query, (Tbid, Lbid, posTbid))
        connection.commit()



if __name__ == '__main__':
    conn = sqlite3.connect('collated_ratings.db')
    c = conn.cursor()

    create_db = create_interim_mem_dump(c, conn)
    your_data = 'Some kind of dictionary into crossMatch()'
    c.close()
    conn.close()
</code></pre>

python performance dictionary