Quickly count the number of objects in bson document
<p>I'd like to calculated the number of documents stored in a mongodb bson file without having to import the file into the db via mongo restore.  </p>

<p>The best I've been able to come up with in python is </p>

<pre><code>bson_doc = open('./archive.bson','rb')
it = bson.decode_file_iter(bson_doc)
total = sum(1 for _ in it)
print(total)
</code></pre>

<p>This works in theory, but is slow in practice when bson documents are large. Anyone have a quicker approach to counting the number of documents in a bson document without doing a full decode?</p>

<p>I am currently using the python 2.7 and pymongo. 
<a href="https://api.mongodb.com/python/current/api/bson/index.html" rel="nofollow">https://api.mongodb.com/python/current/api/bson/index.html</a></p>

<p>I don't have a file at hand to try, but I believe there's a way - if you'll parse the data by hand.</p>

<p>The <a href="https://github.com/mongodb/mongo-python-driver/blob/0b34f9702ca8bed45792a53287d33a2292b99152/bson/__init__.py#L855" rel="nofollow">source for <code>bson.decode_file_iter</code></a> (sans the docstring) goes like this:</p>

<pre><code>_UNPACK_INT = struct.Struct("&lt;i").unpack

def decode_file_iter(file_obj, codec_options=DEFAULT_CODEC_OPTIONS):
    while True:
        # Read size of next object.
        size_data = file_obj.read(4)
        if len(size_data) == 0:
            break  # Finished with file normaly.
        elif len(size_data) != 4:
            raise InvalidBSON("cut off in middle of objsize")
        obj_size = _UNPACK_INT(size_data)[0] - 4
        elements = size_data + file_obj.read(obj_size)
        yield _bson_to_dict(elements, codec_options)
</code></pre>

<p>I presume, the time-consuming operation is <code>_bson_to_dict</code> call - and you don't need one.</p>

<p>So, all you need is to read the file - get the int32 value with the next document's size and skip it. Then count how many documents you've encountered doing this.</p>

<p>So, I believe, this function should do the trick:</p>

<pre><code>import struct
import os
from bson.errors import InvalidBSON

def count_file_documents(file_obj):
    """Counts how many documents provided BSON file contains"""
    cnt = 0
    while True:
        # Read size of next object.
        size_data = file_obj.read(4)
        if len(size_data) == 0:
            break  # Finished with file normaly.
        elif len(size_data) != 4:
            raise InvalidBSON("cut off in middle of objsize")
        obj_size = struct.Struct("&lt;i").unpack(size_data)[0] - 4
        # Skip the next obj_size bytes
        file_obj.seek(obj_size, os.SEEK_CUR)
        cnt += 1
    return cnt
</code></pre>

<p>(I haven't tested the code, though. Don't have MongoDB at hand.)</p>

python mongodb bson