Match Making using GAE + ndb
<p>I have a game in which users contact a server to find a user of their level who wants to play a game. Here is the basic architecture of a game request.</p>

<p><a href="http://i.stack.imgur.com/vMIHi.jpg" rel="nofollow"><img src="http://i.stack.imgur.com/vMIHi.jpg" alt="enter image description here"></a></p>

<p>I am using <code>ndb</code> to store a waiting queue for each user level in the Google DataStore.</p>

<p>I am accessing these queues by their keys to ensure strong consistency (per <a href="https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/#eventual-consistency-in-cloud-datastore" rel="nofollow">this article</a>). The entities are stored in the queue using a repeated (list of) <a href="https://cloud.google.com/appengine/docs/python/ndb/entity-property-reference#types" rel="nofollow">LocalStructuredProperty</a>.</p>

<p>Questions:</p>

<ol>
<li>An entity is deleted from a waiting queue because it is matched to a request. The transaction is committed but not yet applied. That same entity is matched with another request and deleted. Will this throw an error?</li>
<li>These strongly consistent accesses are limited to ~1 write/sec. Is there a better architecture that would eliminate this constraint?</li>
</ol>

<p>One thing I've considered for the latter question is to maintain multiple queues (whose number grows and shrinks with demand).</p>

<p>1.- If you do the entity get and the post inside a transaction, then the same entity can not be matched for a game and therefore no error and it remains consistent.</p>

<p>2.- The 1 write per second is sthe limit for transactions inside the same entity group. If you need more, you can shard the queue entity.</p>

<p>You can use a dedicated memcache or a redis instance to avoid contention. This are much faster than the datastore.</p>

<p>See how these guys use tree nodes to do the match making:
<a href="https://www.youtube.com/watch?v=9nWyWwY2Onc" rel="nofollow">https://www.youtube.com/watch?v=9nWyWwY2Onc</a></p>

python google-app-engine google-cloud-datastore app-engine-ndb