Data buffering/storage - Python
<p>I am writing an embedded application that reads data from a set of sensors and uploads to a central server. This application is written in Python and runs on a Rasberry Pi unit. </p>

<p>The data needs to be collected every 1 minute, however, the Internet connection is unstable and I need to buffer the data to a non volatile storage (SD-card) etc. whenever there is no connection. The buffered data should be uploaded as and when the connection comes back.</p>

<p>Presently, I'm thinking about storing the buffered data in a SQLite database and writing a cron job that can read the data from this database continuously and upload.</p>

<p>Is there a python module that can be used for such feature? </p>

<blockquote>
  <p>Is there a python module that can be used for such feature? </p>
</blockquote>

<p>I'm not aware of any readily available module, however it should be quite straight forward to build one. Given your requirement:</p>

<blockquote>
  <p>the Internet connection is unstable and I need to buffer the data to a non volatile storage (SD-card) etc. whenever there is no connection. The buffered data should be uploaded as and when the connection comes back.</p>
</blockquote>

<p>The algorithm looks something like this (pseudo code):</p>

<pre><code># buffering module
data = read(sensors)
db.insert(data)

# upload module
# e.g. scheduled every 5 minutes via cron
data = db.read(created &gt; last_successful_upload)
success = upload(data)
if success:
    last_successful_upload = max(data.created)
</code></pre>

<p>The key is to seperate the buffering and uploading concerns. I.e. when reading data from the sensor don't attempt to immediately upload, always upload from the scheduled module. This keeps the two modules simple and stable.</p>

<p>There are a few edge cases however that you need to concern yourself with to make this work reliably:</p>

<ol>
<li>insert data while uploading is in progress</li>
<li>SQLlite doesn't support being accessed from multiple processes well</li>
</ol>

<p>To solve this, you might want to consider another database, or create multiple SQLite databases or even flat files for each batch of uploads. </p>

python offline buffering