Problems Using the Berkeley DB Transactional Processing
<p>I'm writing a set of programs that have to operate on a common database, possibly concurrently. For the sake of simplicity (for the user), I didn't want to require the setup of a database server. Therefore I setteled on Berkeley DB, where one can just fire up a program and let it create the DB if it doesn't exist.</p>

<p>In order to let programs work concurrently on a database, one has to use the transactional features present in the 5.x release (here I use python3-bsddb3 6.1.0-1+b2 with libdb5.3 5.3.28-12): the documentation clearly says that it can be done. However I quickly ran in trouble, even with some basic tasks :</p>

<ul>
<li>Program 1 initializes records in a table </li>
<li>Program 2 has to scan the records previously added by program 1 and updates them with additional data.</li>
</ul>

<p>To speed things up, there is an index for said additional data. When program 1 creates the records, the additional data isn't present, so the pointer to that record is added to the index under an empty key. Program 2 can then just quickly seek to the not-yet-updated records.</p>

<p>Even when not run concurrently, the record updating program crashes after a few updates. First it complained about insufficient space in the mutex area. I had to resolve this with an obscure DB_CONFIG file and then run db_recover.</p>

<p>Next, again after a few updates it complained '<em>Cannot allocate memory -- BDB3017 unable to allocate space from the buffer cache</em>'. db_recover and relaunching the program did the trick, only for it to crash again with the same error a few records later.</p>

<p>I'm not even mentioning concurrent use: when one of the programs is launched while the other is running, they almost instantly crash with deadlock, panic about corrupted segments and ask to run recover. I made many changes so I went throug a wide spectrum of errors which often yield irrelevant matches when searched for. I even rewrote the db calls to use lmdb, which in fact works quite well and is really quick, which tends to indicate my program logic isn't at fault. Unfortunately it seems the datafile produced by lmdb is quite sparse, and quickly grew to unacceptable sizes.</p>

<p>From what I said, it seems that maybe some resources are being leaked somewhere. I'm hesitant to rewrite all this directly in C to check if the problem can come from the Python binding.</p>

<p>I can and I will update the question with code, but for the moment ti is long enough. I'm looking for people who have used the transactional stuff in BDB, for similar uses, which could point me to some of the gotchas.</p>

<p>Thanks</p>

<p>RPM (see <a href="http://rpm5.org" rel="nofollow">http://rpm5.org</a>) uses Berkeley DB in transactional mode. There's a fair number of gotchas, depending on what you are attempting.</p>

<p>You have already found DB_CONFIG: you MUST configure the sizes for mutexes and locks, the defaults are invariably too small.</p>

<p>Needing to run db_recover while developing is quite painful too. The best fix (imho) is to automate recovery while opening by checking the return code for DB_RUNRECOVERY, and then reopening the dbenv with DB_RECOVER.</p>

<p>Deadlocks are usually design/coding errors: run db_stat -CA to see what is deadlocked (or what locks are held) and adjust your program. "Works with lmdv" isn't sufficient to claim working code ;-)</p>

<p>Leaks can be seen with either valgrind and/or BDB compilation with -fsanitize:address. Note that valgrind will report false uninitializations unless you use overrides and/or compile BDB to initialize.</p>

python python-3.x berkeley-db