Mongodb aggregate out of memory
<p>I'm using mongodb aggregate to sample documents from a large collection.</p>

<p><a href="https://docs.mongodb.com/manual/reference/operator/aggregation/sample/" rel="nofollow">https://docs.mongodb.com/manual/reference/operator/aggregation/sample/</a></p>

<p>After making several consecutive calls, I see the memory of mongodb climbing up, and after around the 12th call, it crashes with OutOfMemory error.</p>

<p>How can I tell Mongodb to free up the memory after it has finished processing a query?</p>

<p>The reason you are asking this is because you don't know how the <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/sample/" rel="nofollow"><code>$sample</code></a> operator works. As mentioned in the <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/sample/#behavior" rel="nofollow">documentation</a>, </p>

<blockquote>
  <p>In order to get N random documents:</p>
  
  <ul>
  <li><p>If N is greater than or equal to 5% of the total documents in the collection, $sample performs a collection scan, performs a sort, and then select the top N documents. As such, the $sample stage is subject to the <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/sort/#sort-memory-limit" rel="nofollow">sort memory restrictions</a>.</p></li>
  <li><p>If N is less than 5% of the total documents in the collection,
      If using WiredTiger Storage Engine, $sample uses a pseudo-random cursor over the collection to sample N documents.
      If using MMAPv1 Storage Engine, $sample uses the _id index to randomly select N documents.</p></li>
  </ul>
</blockquote>

<p>So I think the number of random documents you want to get is greater than 5%. What you need is set <code>allowDiskUse</code> to <code>True</code>.</p>

<pre><code>collection.aggregate(pipeline, allowDiskUse=True)
</code></pre>

python mongodb pymongo aggregation-framework