Size issues with Python shelve module
<p>I want to store a few dictionaries using the shelve module, however, I am running into a problem with the size. I use Python 3.5.2 and the latest shelve module.</p>

<p>I have a list of words and I want to create a map from the bigrams (character level) to the words. The structure will look something like this:</p>

<pre><code>'aa': 'aardvark', 'and', ...
'ab': 'absolute', 'dab', ...
...
</code></pre>

<p>I read in a large file consisting of approximately 1.3 million words. So the dictionary gets pretty large. This is the code:</p>

<pre><code>self.bicharacters // part of class
def _create_bicharacters(self):
    '''
    Creates a bicharacter index for calculating Jaccard coefficient.
    '''
    with open('wordlist.txt', encoding='ISO-8859-1') as f:
        for line in f:

            word = line.split('\t')[2]

            for i in range(len(word) - 1):
                bicharacter = (word[i] + word[i+1])

                if bicharacter in self.bicharacters:
                    get = self.bicharacters[bicharacter]
                    get.append(word)
                    self.bicharacters[bicharacter] = get
                else:
                    self.bicharacters[bicharacter] = [word]
</code></pre>

<p>When I ran this code using a regular Python dictionary, I did not run into issues, but I can't spare those kinds of memory resources due to the rest of the program also having quite a large memory footprint. </p>

<p>So I tried using the shelve module. However, when I run the code above using shelve the program stops after a while due to no more memory on disk, the shelve db that was created was around 120gb, and it had still not read even half the 1.3M word list from the file. What am I doing wrong here?</p>

<p>The problem here is not so much the number of keys, but that each key references a list of words.</p>

<p>While in memory as one (huge) dictionary, this isn't that big a problem as the words are simply shared between the lists; each list is simply a sequence of references to other objects and here many of those objects are the same, as only one string per word needs to be referenced.</p>

<p>In <code>shelve</code>, however, each value is pickled and stored separately, meaning that a <em>concrete copy</em> of the words in a list will have to be stored <em>for each value</em>. Since your setup ends up adding a given word to a large number of lists, this multiplies your data needs rather drastically.</p>

<p>I'd switch to using a SQL database here. Python comes with bundled with <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow"><code>sqlite3</code></a>. If you create one table for individual words, and second table for each possible bigram, and a third that simply links between the two (a many-to-many mapping, linking bigram row id to word row id), this can be done very efficiently. You can then do very efficient lookups as SQLite is quite adept managing memory and indices for you.</p>

python dictionary shelve