Cache file handle to netCDF files in python
<p>Is there a way to cache python file handles? I have a function which takes a netCDF file path as input, opens it, extracts some data from the netCDF file and closes it. It gets called a lot of times, and the overhead of opening the file each time is high.</p>

<p>How can I make it faster by maybe caching the file handle? Perhaps there is a python library to do this</p>

<p>Yes, you can use following python libraries:</p>

<ul>
<li><a href="https://pypi.python.org/pypi/dill" rel="nofollow">dill</a> (required)</li>
<li><a href="https://pypi.python.org/pypi/python-memcached" rel="nofollow">python-memcached</a> (optional)</li>
</ul>

<p>Let's follow the example. You have two files:</p>

<pre><code># save.py - it puts deserialized file handler object to memcached
import dill
import memcache            


mc = memcache.Client(['127.0.0.1:11211'], debug=0)
file_handler = open('data.txt', 'r')
mc.set("file_handler", dill.dumps(file_handler))
print 'saved!'   
</code></pre>

<p>and </p>

<pre><code># read_from_file.py - it gets deserialized file handler object from memcached, 
#                     then serializes it and read lines from it
import dill
import memcache


mc = memcache.Client(['127.0.0.1:11211'], debug=0)
file_handler = dill.loads(mc.get("file_handler"))
print file_handler.readlines() 
</code></pre>

<p>Now if you run:</p>

<pre><code>python save.py
python read_from_file.py
</code></pre>

<p>you can get what you want. </p>

<p><strong>Why it works?</strong></p>

<p>Because you didn't close the file (<code>file_handler.close()</code>), so object still exist in memory (has not been garbage collected, because of <a href="https://docs.python.org/2/library/weakref.html" rel="nofollow">weakref</a>) and you can use it. Even in different process.</p>

<p><strong>Solution</strong></p>

<pre><code>import dill
import memcache


mc = memcache.Client(['127.0.0.1:11211'], debug=0)
serialized = mc.get("file_handler")
if serialized:
    file_handler = dill.loads(serialized)
else:
    file_handler = open('data.txt', 'r')
    mc.set("file_handler", dill.dumps(file_handler))
print file_handler.readlines() 
</code></pre>

python netcdf