Performing an operation on every document in a MongoDB instance
<p>I have a mongoDB collection with 1.5 million documents, all of which have the same fields, and I want to take the contents of Field A (which is unique in every document) and perform <code>f(A)</code> on it, then create and populate Field B. Pseudocode in Python:</p>

<pre><code>for i in collection.find():
    x = i**2
    collection.update(i,x) #update i with x
</code></pre>

<p>NOTE: I am aware that the update code is probably wrong, but unless it affects the speed of operation, I chose to leave it there for the sake of simplicity</p>

<p>The problem is, this code is really really slow, primarily because it can run through 1000 documents in about a second, then the server cuts off the cursor for about a minute,then it allows another 1000. I'm wondering if there is any way to optimize this operation, or if I'm stuck with this slow bottleneck.</p>

<p>Additional notes:</p>

<ol>
<li><p>I have adjusted <code>batch_size</code> as an experiment, it is faster, but it's not efficient, and still takes hours</p></li>
<li><p>I am also aware that SQL could probably do this faster, there are other reasons I am using an noSQL DB that are not relevant to this problem</p></li>
<li><p>The instance is running locally so for all intents and purposes, there is not network latency</p></li>
<li><p>I have seen <a href="http://stackoverflow.com/questions/21716314/performance-tuning-mongodb-query-update">this</a> question, but it's answer doesn't really address my problem</p></li>
</ol>

<p>Maybe you should do your updates in multiple threads. I think it may be better  to load data in one thread, split it into multiple parts and pass that parts to parallel worker threads that will perform updates. It will be faster.</p>

<p>EDIT:</p>

<p>I suggest you doing paginated queries.
Python pseudocode: </p>

<pre><code>count = collection.count()
page_size = 20
i = 0;
while(i &lt; count):
    for row in collection.find().limit(pageSize).skip(i):
        x = i**2
        collection.update(i, x);
    i += page_size
</code></pre>

python mongodb performance optimization