Performing an operation on every document in a MongoDB instance
<p>I have a mongoDB collection with 1.5 million documents, all of which have the same fields, and I want to take the contents of Field A (which is unique in every document) and perform <code>f(A)</code> on it, then create and populate Field B. Pseudocode in Python:</p>

<pre><code>for i in collection.find():
    x = i**2
    collection.update(i,x) #update i with x
</code></pre>

<p>NOTE: I am aware that the update code is probably wrong, but unless it affects the speed of operation, I chose to leave it there for the sake of simplicity</p>

<p>The problem is, this code is really really slow, primarily because it can run through 1000 documents in about a second, then the server cuts off the cursor for about a minute,then it allows another 1000. I'm wondering if there is any way to optimize this operation, or if I'm stuck with this slow bottleneck.</p>

<p>Additional notes:</p>

<ol>
<li><p>I have adjusted <code>batch_size</code> as an experiment, it is faster, but it's not efficient, and still takes hours</p></li>
<li><p>I am also aware that SQL could probably do this faster, there are other reasons I am using an noSQL DB that are not relevant to this problem</p></li>
<li><p>The instance is running locally so for all intents and purposes, there is not network latency</p></li>
<li><p>I have seen <a href="http://stackoverflow.com/questions/21716314/performance-tuning-mongodb-query-update">this</a> question, but it's answer doesn't really address my problem</p></li>
</ol>

<p>Database clients tend to be extremely abstracted from actual database activity, so observed delay behaviors can be deceptive. It's likely that you are actually hammering the database in that time, but the activity is all hidden from the Python interpreter.</p>

<p>That said, there are a couple things you can do to make this lighter.</p>

<p>1) Put an index on the property <code>A</code> that you're basing the update on. This will allow it to return much faster.</p>

<p>2) Put a projection operator on your <code>find</code> call:</p>

<pre><code>for doc in collection.find(projection=['A']):
</code></pre>

<p>That will ensure that you only return the fields you need to, and if you've properly indexed the unique <code>A</code> property, will ensure your results are drawn entirely from the very speedy index.</p>

<p>3) Use an update operator to ensure you only have to send the new field back. Rather than send the whole document, send back the dictionary:</p>

<pre><code>{'$set': {'B': a**2}}
</code></pre>

<p>which will create the field <code>B</code> in each document without affecting any of the other content.</p>

<p>So, the whole block will look like this:</p>

<pre><code>for doc in collection.find(projection=['A', '_id']):
    collection.update(filter={'_id': doc['_id']},
                      update={'$set': {'B': doc['A']**2}})
</code></pre>

<p>That should cut down substantially on the work that Mongo has to do, as well as (currently irrelevant to you) network traffic.</p>

python mongodb performance optimization