ElasticSearch indexing with 450K documents - performance
<p>We have ElasticSearch (1.5) on AWS (t2.micro, 2 instances with 10GB SSD storage each) and a MySQL with ~450K fairly big/complex entities. </p>

<p>I'm using python to read from MySql, serialize to JSON and PUT to ElasticSearch. There are 10 threads working simultaneously, each PUTing bulk of 1000 documents at the time.</p>

<p>Total of ~450K (1.3GB) documents, it takes around 20min to process and send  to ElasticSearch.</p>

<p>Problem is that only around 85% of them get indexed and rest are lost. When I reduce number of documents to ~100K they all get indexed.</p>

<p>Looking at ElasticSearch AWS monitor I can see CPU getting up to 100% while indexing, but it doesnt give any errors. </p>

<p>What is the best way to find out bottle here? I want it fast but cant afford losing any documents.</p>

<p>EDIT.
I've run it again checking output of /_cat/thread_pool?v every few minutes. Indexed 390805 out of 441400. Out of thread_pool bellow:</p>

<pre><code>host             ip            bulk.active bulk.queue bulk.rejected index.active index.queue index.rejected search.active search.queue search.rejected 
&lt;host&gt; x.x.x.x           1         22            84            0           0              0             0            0               0 
&lt;host&gt; x.x.x.x           1         11            84            0           0              0             0            0               0 
&lt;host&gt; x.x.x.x           1         29            84            0           0              0             0            0               0 
&lt;host&gt; x.x.x.x           1         13            84            0           0              0             0            0               0 
&lt;host&gt; x.x.x.x           0          0            84            0           0              0             0            0               0 
&lt;host&gt; x.x.x.x           1         17            84            0           0              0             0            0               0 
&lt;host&gt; x.x.x.x           0          0            84            0           0              0             0            0               0 
</code></pre>

<p>EDIT 2</p>

<pre><code>host             ip            bulk.active bulk.queue bulk.rejected index.active index.queue index.rejected search.active search.queue search.rejected 
&lt;host&gt; x.x.x.x           0          0            84            0           0              0             0            0               0 
</code></pre>

<p>EDIT 3 </p>

<pre><code>$ curl https://xxxxx.es.amazonaws.com/_cat/thread_pool?v&amp;h=id,host,ba,bs,bq,bqs,br,ââbl,bc,bmi,bma
[1] 15896
host             ip            bulk.active bulk.queue bulk.rejected index.active index.queue index.rejected search.active search.queue search.rejected 
&lt;host&gt; x.x.x.x           0          0            84            0           0              0             0            0               0 
</code></pre>

<p>^^ copy/paste of what I'm getting back</p>

<p>EDIT 4 </p>

<pre><code>$ curl 'https://xxxxx.es.amazonaws.com/_cat/thread_pool?v&amp;h=id,host,ba,bs,bq,bqs,brââ,ââbl,bc,bmi,bma'
&lt;html&gt;&lt;body&gt;&lt;h1&gt;400 Bad request&lt;/h1&gt;
Your browser sent an invalid request.
&lt;/body&gt;&lt;/html&gt;
</code></pre>

<p>still nothing</p>

<p>EDIT 5</p>

<pre><code>id   host             ba bs bq bqs br bmi bma bl br    bc 
n6Ad &lt;host&gt;  0  1  0  50 84   1   1  1 84 25821 
</code></pre>

<p>some mysterious way it worked when I changed order of params</p>

<p>I suspect EC2 is your bottleneck.  Based on <a href="http://stackoverflow.com/questions/28984106/whats-is-cpu-credit-balance-in-ec2">the way burstable instances are allocated CPU</a>, a <code>t2.micro</code> accrues 6 cpu credits an hour.  </p>

<p>So in your first hour up, your Elasticsearch nodes will be able to run one vCPU at 100% for a <em>maximum of 6 minutes</em> before being "capped" at an unspecified lower resource allocation (assume it is less than the instance quota of 10% of a vCPU).</p>

<p>Elasticsearch is most likely to be CPU-bound during indexing.  If the indexing process is sending bulk requests faster than an instance can ingest (due to being CPU-bound... within quota, or outside of quota <em>after those first 6 minutes of bursting</em>), those requests will queue.  Once the queue is saturated, Elasticsearch will begin rejecting requests.</p>

<blockquote>
  <p>This may help explain why you're able to index 100k documents without issue <em>(under 6min?)</em>, while your full collection (~450k) encounters difficulty.</p>
</blockquote>

<hr>

<p>If your cluster is CPU-bound during indexing, and bulk requests are being rejected, you'll want to either:</p>

<ol>
<li><p>Increase the compute resources available to your cluster nodes during indexing </p>

<p><strong>OR</strong></p></li>
<li><p>Throttle your indexer to keep pace with your cluster ingestion capacity.</p></li>
</ol>

<hr>

<p>You could build an indexer that was more resilient to smaller node-types, by running your <code>thread_pool</code> request to check how many bulk requests were in queue (perhaps as a percentage of the total queue size), before deciding to fire the next bulk indexing request.</p>

python mysql amazon-web-services elasticsearch