How can I determine when Bloomd will scale the bloom filter?
<p>I'm using Bloomd and its scalable bloom filter to store/check billions of urls for our broad crawler. It was working very good for first 1-1.5 billion urls and it has been using around 16 GB of memory but it seems that more than 2 billion urls will be added to it soon and I would like to understand when Bloomd will try to scale filter to 32 GB (and we will upgrade our server memory to 64 GB or more).</p>

<p>The "info" command provides some data but I'm not sure which key represents what and how can I understand how many url's I can add to it before it scales up.</p>

<p>Here is my "info" command results</p>

<pre><code>START
capacity 5461000000
checks 5893888032
check_hits 5400239954
check_misses 493648078
in_memory 1
page_ins 7
page_outs 6
probability 0.000100
sets 493648075
set_hits 493648016
set_misses 59
size 1859303638
storage 17205844037
END
</code></pre>

<p>Also I'll appreciate if someone knows better approach than using Scalable Bloom Filters for that kind of massive URL sets.</p>

<p>Your filter is using only 34% of its capacity (size/capacity = 1859303638/5461000000).</p>

python web-scraping web-crawler bloom-filter