Pandas dataframe pivot - Memory Error
<p>I have a dataframe <code>df</code> with the following structure:</p>

<pre><code>        val          newidx    Code
Idx                             
0       1.0      1220121127    706
1       1.0      1220121030    706
2       1.0      1620120122    565
</code></pre>

<p>It has 1000000 lines. 
In total we have 600 unique <code>Code</code> value and 200000 unique <code>newidx</code> values.</p>

<p>If I perform the following operation</p>

<pre><code>df.pivot_table(values='val', index='newidx', columns='Code', aggfunc='max')
</code></pre>

<p>I get a <code>MemoryError</code> . but this sounds strange as the size of the resulting dataframe should be sustainable: 200000x600.</p>

<p>How much memory requires such operation? Is there a way to fix this memory error?</p>

<p>Try to see if this fits in your memory:</p>

<pre><code>df.groupby(['newidx', 'Code'])['val'].max().unstack()
</code></pre>

<p><code>pivot_table</code> is unfortunately very memory intensive as it may make multiple copies of data.</p>

<hr>

<p>If the <code>groupby</code> does not work, you will have to split your DataFrame into smaller pieces. Try not to assign multiple times. For example, if reading from csv:</p>

<pre><code>df = pd.read_csv('file.csv').groupby(['newidx', 'Code'])['val'].max().unstack()
</code></pre>

<p>avoids multiple assignments.</p>

python pandas dataframe