Difficulty with document batch import, pymongo
<p>I'm having a much more difficult time than I thought I would importing multiple documents from Mongo into RAM in batch. I am writing an application to communicate with a MongoDB via <code>pymongo</code> that currently has 2GBs, but in the near future could grow to over 1TB. Because of this, batch reading a limited number of records into RAM at a time is important for scalability. </p>

<p>Based on <a href="http://stackoverflow.com/questions/9786736/how-to-read-through-collection-in-chunks-by-1000]">this post</a> and <a href="https://api.mongodb.com/python/2.7.2/api/pymongo/cursor.html" rel="nofollow">this documentation</a> I thought this would be about as easy as: </p>

<pre><code>HOST = MongoClient(MONGO_CONN)
DB_CONN = HOST.database_name
collection = DB_CONN.collection_name
cursor = collection.find()
cursor.batch_size(1000) 
next_1K_records_in_RAM = cursor.next()
</code></pre>

<p>This isn't working for me, however. Even though I have a Mongo collection populated with >200K BSON objects, this reads them in one at a time as single dictionaries, e.g. <code>{_id : ID1, ...}</code> instead of what I'm looking for, which is an error of dictionaries representing multiple documents in my collections, e.g. <code>[{_id : ID1, ...}, {_id : ID2, ...}, ..., {_id: ID1000, ...}]</code>. </p>

<p>I wouldn't expect this to matter, but I'm on python 3.5 instead of 2.7. </p>

<p>As this example references a secure, remote data source this isn't a reproducible example. Apologies for that. If you have a suggestion for how the question can be improved please let me know. </p>

<ul>
<li>Python version is irrelevant here, nothing to do with your output.</li>
<li>Batch_size defines only how many documents mongoDB returns in a single
trip to DB (under some limitations: <a href="http://api.mongodb.com/python/current/api/pymongo/cursor.html" rel="nofollow">see here
here</a> ) </li>
<li>collection.find always returns an iterator/cursor or None. Batching does its job transparently)
(the later if no documents are found)</li>
<li><p>To examine returned documents you
have to iterate through the cursor i.e.</p>

<p><code>For document in cursor:
   print (document)
</code></p>

<p>or if you want a list of the documents: <code>list(cursor)</code></p>

<ul>
<li>Remember to do a <code>cursor.rewind()</code> if you need to revisit the documents</li>
</ul></li>
</ul>

python mongodb python-3.x pymongo