Using a generator to iterate over a large collection in Mongo
<p>I have a collection with 500K+ documents which is stored on a single node mongo. Every now and then my pymongo cursor.find() fails as it times out.</p>

<p>While I could set the <code>find</code> to ignore timeout, I do not like that approach. Instead, I tried a generator (adapted from <a href="http://stackoverflow.com/questions/9786736/how-to-read-through-collection-in-chunks-by-1000">this</a> answer and <a href="http://code.activestate.com/recipes/137270-use-generators-for-fetching-large-db-record-sets/" rel="nofollow">this</a> link):</p>

<pre><code>def mongo_iterator(self, cursor, limit=1000):
        skip = 0
        while True:
            results = cursor.find({}).sort("signature", 1).skip(skip).limit(limit)

            try:
                results.next()

            except StopIteration:
                break

            for result in results:
                yield result

            skip += limit
</code></pre>

<p>I then call this method using:</p>

<pre><code>ref_results_iter = self.mongo_iterator(cursor=latest_rents_refs, limit=50000)
for ref in ref_results_iter:
    results_latest1.append(ref)
</code></pre>

<p>The problem:
My iterator does not return the same number of results. The issue is that next() advances the cursor. So for every call I lose one element...</p>

<p>The question: 
Is there a way to adapt this code so that I can check if next exists? Pymongo 3x does not provide hasNext() and 'alive' check <a href="http://api.mongodb.com/python/current/api/pymongo/cursor.html" rel="nofollow">is not guaranteed</a> to return false.</p>

<p>Why not use </p>

<pre><code>for result in results:
    yield result
</code></pre>

<p>The for loop should handle <code>StopIteration</code> for you.</p>

python mongodb pymongo pymongo-3.x