Memory error on large Shapefile in Python
<pre><code>import shapefile
data = shapefile.Reader("data_file.shp")
shapes = data.shapes()
</code></pre>

<p>My problem is that getting the shapes from the Shapefile reader gives me an exception <code>MemoryError</code> when using <a href="https://pypi.python.org/pypi/pyshp" rel="nofollow">Pyshp</a>.</p>

<p>The <code>.shp</code> file is quite large, at 1.2 gB. But I am using ony 3% of my machine's 32gB, so I don't understand it.</p>

<p>Is there any other approach that I can take? Can process the file in chunks in Python? Or use some tool to spilt the file into chinks, then process each of them individually?</p>

<p>Although I haven't been able to test it, Pyshp should be able to read it regardless of the file size or memory limits. Creating the <code>Reader</code> instance doesn't load the entire file, only the header information. </p>

<p>It seems the problem here is that you used the <code>shapes()</code> method, which reads all shape information into memory at once. This usually isn't a problem, but it is with files this big. As a general rule you should instead use the <code>iterShapes()</code> method which reads each shape one by one. </p>

<pre><code>import shapefile
data = shapefile.Reader("data_file.shp")
for shape in data.iterShapes():
    # do something...
</code></pre>

python shapefile pyshp