Python memory allocation with pandas and pickle
<p>I am running a python script which can be roughly summed (semi-psuedo-coded) as follows:</p>

<pre><code>import pandas as pd
for json_file in json_files:
    with open(json_file,'r') as fin:
        data = fin.readlines()
    data_str = '[' + ','.join(x.strip() for x in data) + ']'
    df = pd.read_json(data_str)
    df.to_pickle('%s.pickle' % json_file)
    del df, data, data_str
</code></pre>

<p>The process works iteratively creating data frames, saving them each to a unique file. However, my memory seems to get used up during the process, as if <code>del df, data, data_str</code> does not free up memory (originally, I did not include the <code>del</code> statement in the code, but I hoped that adding it would resolve the issue -- it did not). During each iteration, approximately the same amount of data is being read into the data frame, approximately 3% of my available memory; as the process iterates, each iteration a there is a reported 3% bump in <code>%MEM</code> (from <code>ps u | grep [p]ython</code> in my terminal), and eventually my memory is swamped and the process is killed. My question is how should I change my code/approach so that at each iteration, the memory from the previous iteration is freed? </p>

<p>To note, I'm running Ubuntu 16.04 with Python 3.5.2 via Anaconda. </p>

<p>Thanks in advance for your direction. </p>

<p>In python automatic garbage collection deallocates the variable (pandas DataFrame are also just another object in terms of python). There are different garbage collection strategies that can be tweaked (requires significant learning).</p>

<p>You can manually trigger the garbage collection using</p>

<pre><code>import gc
gc.collect()
</code></pre>

<p>But frequent calls to garbage collection is discouraged as it is a costly operation and may affect performance.</p>

<p><a href="http://www.digi.com/wiki/developer/index.php/Python_Garbage_Collection" rel="nofollow">Reference</a></p>

python memory-management memory-leaks