Efficiently retrieve data (all in one batch ideally) with mongengine in Python 3
<p>Let's say I have class User which inherits from the Document class (I am using Mongoengine). Now, I want to retrieve all users signed up after some timestamp. Here is the method I am using:</p>

<pre><code>def get_users(cls, start_timestamp):
    return cls.objects(ts__gte=start_timestamp)
</code></pre>

<p>1000 documents are returned in 3 seconds. This is extremely slow. I have done similar queries in SQL in a couple of miliseconds. I am new to MongoDB and No-SQL in general, so I guess I am doing something terribly wrong.</p>

<p>I suspect the retrieval is slow because it is done in several batches. I read somewhere that for PyMongo the batch size is 101, but I do not know if that is same for Mongoengine. </p>

<p>Can I change the batch size, so I could get all documents at once. I will know approximately how much data will be retrieved in total. </p>

<p>Any other suggestions are very welcome.</p>

<p>Thank you! </p>

<p>As you suggest there is no way that it should take 3 seconds to run this query. However, the issue is not going to be the performance of the pymongo driver, some things to consider:</p>

<ul>
<li>Make sure that the <code>ts</code> field is included in the <a href="http://mongoengine-odm.readthedocs.io/guide/defining-documents.html#indexes" rel="nofollow">indexes</a> for the user collection</li>
<li>Mongoengine does some aggressive de-referencing so if the 1000 returned user documents have one or more <code>ReferenceField</code> then each of those results in additional queries. <a href="http://mongoengine-odm.readthedocs.io/guide/querying.html#query-efficiency-and-performance" rel="nofollow">There are ways to avoid this.</a></li>
<li>Mongoengine provides <a href="http://mongoengine-odm.readthedocs.io/guide/querying.html#aggregation" rel="nofollow">a direct interface</a> to the pymongo method for the <a href="https://docs.mongodb.com/manual/aggregation/#aggregation-pipeline" rel="nofollow">mongodb aggregation framework</a> this is <strong>by far the most efficient way to query mongodb</strong></li>
<li>mongodb <a href="https://groups.google.com/forum/#!topic/mongodb-announce/HWfqcK1T034" rel="nofollow">recently released</a> an official python ODM <a href="https://github.com/mongodb/pymodm" rel="nofollow">pymodm</a> in part to provide better default performance than mongoengine</li>
</ul>

python mongodb mongoengine