Data buffering/storage - Python
<p>I am writing an embedded application that reads data from a set of sensors and uploads to a central server. This application is written in Python and runs on a Rasberry Pi unit. </p>

<p>The data needs to be collected every 1 minute, however, the Internet connection is unstable and I need to buffer the data to a non volatile storage (SD-card) etc. whenever there is no connection. The buffered data should be uploaded as and when the connection comes back.</p>

<p>Presently, I'm thinking about storing the buffered data in a SQLite database and writing a cron job that can read the data from this database continuously and upload.</p>

<p>Is there a python module that can be used for such feature? </p>

<p>If you mean a module to work with SQLite database, check out <a href="http://www.sqlalchemy.org/" rel="nofollow">SQLAlchemy</a>.</p>

<p>If you mean a module which can do what cron does, check out <a href="https://docs.python.org/3/library/sched.html" rel="nofollow">sched</a>, a python event scheduler.</p>

<p>However, this looks like a perfect place to implemet a task queue --using a dedicated task broker (rabbitmq, redis, zeromq,..), or python's <a href="https://docs.python.org/3/library/threading.html" rel="nofollow">threads</a> and <a href="https://docs.python.org/2/library/queue.html" rel="nofollow">queues</a>. In general, you want to submit an upload task, and worker thread will pick it up and execute, while the task broker handles retries and failures. All this happens asynchronously, without blocking your main app.</p>

<p>UPD: Just to clarify, you don't need the database if you use a task broker, because a task broker stores the tasks for you. </p>

python offline buffering