Optimal bulk query size for multiSearch in ElasticSearch
<p>I am trying to query elasticsearch with multisearch , but it doesn't seems to improve the time a lot.</p>

<p>For approx <strong>70k queries</strong> , time taken by different bulk_sizes are :</p>

<p>For <strong>single search</strong> for every item time taken = <strong>2611s</strong></p>

<p>For <strong>multisearch (bulksize=1000)</strong> time taken = <strong>2400s</strong> </p>

<p>For <strong>multisearch (bulksize=10)</strong> time taken = <strong>2326s</strong></p>

<p>So, I need to know </p>

<p>a) Is this the correct way to do MultiSearch ?</p>

<p>b) What is the optimal Bulk Size for Multi Search ?</p>

<p>Here'e my code :</p>

<pre><code>search_arr = []

        for k in range(i,i+BULK_SIZE):
            search_arr.append({'index':'test'})
            search_arr.append({"query": {"match": {"title": title[k]}}, "size": 5})
        request =''
        for each in search_arr:
            request += '%s \n' %json.dumps(each)

        resp = es.msearch(body=request)
</code></pre>

<p>The number of concurrent searches is limited by the Search Thread Pool. </p>

<p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html" rel="nofollow">https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html</a></p>

<blockquote>
  <p>For count/search operations. 
  Thread pool type is fixed with a size of int((# of available_processors * 3) / 2) + 1, queue_size of 1000.</p>
</blockquote>

<p>This means if you have single processor, then you will have 2 concurrent searches, and rest of the requests will go into the queue and will be processed as and when the threads become available again.  </p>

python algorithm elasticsearch