Memory management in numpy arrays,python
<p>I get a memory error when processing very large(>50Gb) file (problem: RAM memory gets full).</p>

<p>My solution is: I would like to read only 500 kilo bytes of data once and process( and delete it from memory and go for next 500 kb). Is there any other better solution? or If this solution seems better , how to do it with numpy array?</p>

<p>It is just 1/4th the code(just for an idea)</p>

<pre><code>  import h5py
  import numpy as np
  import sys
  import time
  import os
  hdf5_file_name = r"test.h5"
  dataset_name = 'IMG_Data_2'

  file = h5py.File(hdf5_file_name,'r+')
  dataset = file[dataset_name]
  data = dataset.value
  dec_array = data.flatten() 
  ........
</code></pre>

<p>I get memory error at this point itsef as it trys to put in all the data to memory.</p>

<h2>Quick answer</h2>

<ul>
<li><a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html" rel="nofollow">Numpuy.memmap</a> allows presenting a large file on disk as a numpy array. Don't know if it allows mapping files larger than RAM+swap though. Worth a shot.</li>
<li>[Presentation about out-of-memory work with Python] (<a href="http://hilpisch.com/TPQ_Out_of_Memory_Analytics.html" rel="nofollow">http://hilpisch.com/TPQ_Out_of_Memory_Analytics.html</a>)</li>
</ul>

<h2>Longer answer</h2>

<p>A key question is how much RAM you have (&lt;10GB, >10GB) and what kind of processing you're doing (need to look at each element in the dataset once or need to look at the whole dataset at once).</p>

<p>If it's &lt;10GB and need to look once, then your approach seems like the most decent one. It's a standard way to deal with datasets which are larger than main memory. What I'd do is increase the size of a chunk from 500kb to something closer to the amount of memory you have - perhaps half of physical RAM, but anyway, something in the GB range, but not large enough to cause swapping to disk and interfere with your algorithm. A nice optimisation would be to hold two chunks in memory at one time. One is being processes, while the other is being loaded in parallel from disk. This works because loading stuff from disk is relatively expensive, but it doesn't require much CPU work - the CPU is basically waiting for data to load. It's harder to do in Python, because of the GIL, but numpy and friends should not be affected by that, since they release the GIL during math operations. The <code>threading</code> package might be useful here.</p>

<p>If you have low RAM AND need to look at the whole dataset at once (perhaps when computing some quadratic-time ML algorithm, or even doing random accesses in the dataset), things get more complicated, and you probably won't be able to use the previous approach. Either upgrade your algorithm to a linear one, or you'll need to implement some logic to make the algorithms in numpy etc work with data on disk directly rather than have it in RAM.</p>

<p>If you have >10GB of RAM, you might let the operating system do the hard work for you and increase swap size enough to capture all the dataset. This way everything is loaded into virtual memory, but only a subset is loaded into physical memory, and the operating system handles the transitions between them, so everything looks like one giant block of RAM. How to increase it is OS specific though.</p>

python arrays numpy memory