Memory error on large Shapefile in Python
<pre><code>import shapefile
data = shapefile.Reader("data_file.shp")
shapes = data.shapes()
</code></pre>

<p>My problem is that getting the shapes from the Shapefile reader gives me an exception <code>MemoryError</code> when using <a href="https://pypi.python.org/pypi/pyshp" rel="nofollow">Pyshp</a>.</p>

<p>The <code>.shp</code> file is quite large, at 1.2 gB. But I am using ony 3% of my machine's 32gB, so I don't understand it.</p>

<p>Is there any other approach that I can take? Can process the file in chunks in Python? Or use some tool to spilt the file into chinks, then process each of them individually?</p>

<p>Quoting from <a href="http://stackoverflow.com/questions/5537618/memory-errors-and-list-limits">this answer</a> by thomas:</p>

<blockquote>
  <p>The <code>MemoryError</code> exception that you are seeing is the direct result of running out of available RAM. This could be caused by either the 2GB per program limit imposed by Windows (<a href="http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx#memory_limits" rel="nofollow">32bit programs</a>), or lack of available RAM on your computer. (This <a href="http://stackoverflow.com/questions/4285185/python-memory-limit">link</a> is to a previous question). You should be able to extend the 2GB by using 64bit copy of Python, provided you are using a 64bit copy of windows.</p>
</blockquote>

<p>So try a 64bit copy of Python or provide more detail about your platform and Python versions.</p>

python shapefile pyshp