ElasticSearch bulk update: organizing JSON using python script
<p>So I am trying to index some public Amazon data set (products) into my ElasticSearch.</p>

<p>I have a very large JSON file for data (9.9 Gigabytes). I have splitted the file into various smaller files (for memory's sake), and now each file has the following structure:</p>

<pre><code>{"asin": "0001048791", "salesRank": {"Books": 6334800}, "imUrl": "http://ecx.images-amazon.com/images/I/51MKP0T4DBL.jpg", "categories": [["Books"]], "title": "The Crucible: Performed by Stuart Pankin, Jerome Dempsey &amp;amp; Cast"}
{"asin": "0000143561", "categories": [["Movies &amp; TV", "Movies"]], "description": "3Pack DVD set - Italian Classics, Parties and Holidays.", "title": "Everyday Italian (with Giada de Laurentiis), Volume 1 (3 Pack): Italian Classics, Parties, Holidays", "price": 12.99, "salesRank": {"Movies &amp; TV": 376041}, "imUrl": "http://g-ecx.images-amazon.com/images/G/01/x-site/icons/no-img-sm._CB192198896_.gif", "related": {"also_viewed": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC", "B002I5GNVU", "B000RBU4BM"], "buy_after_viewing": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC"]}}
{"asin": "0000037214", "related": {"also_viewed": ["B00JO8II76", "B00DGN4R1Q", "B00E1YRI4C"]}, "title": "Purple Sequin Tiny Dancer Tutu Ballet Dance Fairy Princess Costume Accessory", "price": 6.99, "salesRank": {"Clothing": 1233557}, "imUrl": "http://ecx.images-amazon.com/images/I/31mCncNuAZL.jpg", "brand": "Big Dreams", "categories": [["Clothing, Shoes &amp; Jewelry", "Girls"], ["Clothing, Shoes &amp; Jewelry", "Novelty, Costumes &amp; More", "Costumes &amp; Accessories", "More Accessories", "Kids &amp; Baby"]]}
</code></pre>

<p>Products are JSON objects, arranged one-in-a-line. </p>

<p>Now I want to use ElasticSearch _bulk update to index all this data.</p>

<p>Now since each document in ElasticSearch requires a header (Correct?), I have written a python script to create a new file with appropriate format.</p>

<p>The shell script looks like this:</p>

<pre><code>#!/bin/sh

# 0. Some constants to re-define to match your environment
ES_HOST=localhost:9200
JSON_FILE_IN=/home/aksarora/amazon-sample/parts/newaa
JSON_FILE_OUT=/home/aksarora/amazon-sample/parts_parsed/newaa.json

# 1. Python code to transform your JSON file
PYTHON="import json,sys;
out = open('$JSON_FILE_OUT', 'w');
with open('$JSON_FILE_IN', 'r') as json_in:
    docs = [json.loads(line) for line in json_in]
    for doc in docs:
        out.write('%s\n' % json.dumps({\"index\": {}}));
        out.write('%s\n' % json.dumps(doc, indent=0).replace('\n', ''));
"

# 2. run the Python script from step 1
python3 -c "$PYTHON"

# 3. use the output file from step 2 in the curl command
curl -s -XPOST $ES_HOST/amazon/products/_bulk --data-binary @$JSON_FILE_OUT 
</code></pre>

<p>But when I run this, I get the following error:</p>

<pre><code>Traceback (most recent call last):
  File "&lt;string&gt;", line 4, in &lt;module&gt;
  File "&lt;string&gt;", line 4, in &lt;listcomp&gt;
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 42 (char 41)
{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"parse_exception","reason":"Failed to derive xcontent"},"status":400}
</code></pre>

<p>Any idea what I am doing wrong?
Thanks.</p>

<p>I don't get the same results when I run the following which tries to reproduce the problem:</p>

<pre><code>import json
import sys

json_in =(
    """{"asin": "0001048791", "salesRank": {"Books": 6334800}, "imUrl": "http://ecx.images-amazon.com/images/I/51MKP0T4DBL.jpg", "categories": [["Books"]], "title": "The Crucible: Performed by Stuart Pankin, Jerome Dempsey &amp;amp; Cast"}""",
    """{"asin": "0000143561", "categories": [["Movies &amp; TV", "Movies"]], "description": "3Pack DVD set - Italian Classics, Parties and Holidays.", "title": "Everyday Italian (with Giada de Laurentiis), Volume 1 (3 Pack): Italian Classics, Parties, Holidays", "price": 12.99, "salesRank": {"Movies &amp; TV": 376041}, "imUrl": "http://g-ecx.images-amazon.com/images/G/01/x-site/icons/no-img-sm._CB192198896_.gif", "related": {"also_viewed": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC", "B002I5GNVU", "B000RBU4BM"], "buy_after_viewing": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC"]}}""",
    """{"asin": "0000037214", "related": {"also_viewed": ["B00JO8II76", "B00DGN4R1Q", "B00E1YRI4C"]}, "title": "Purple Sequin Tiny Dancer Tutu Ballet Dance Fairy Princess Costume Accessory", "price": 6.99, "salesRank": {"Clothing": 1233557}, "imUrl": "http://ecx.images-amazon.com/images/I/31mCncNuAZL.jpg", "brand": "Big Dreams", "categories": [["Clothing, Shoes &amp; Jewelry", "Girls"], ["Clothing, Shoes &amp; Jewelry", "Novelty, Costumes &amp; More", "Costumes &amp; Accessories", "More Accessories", "Kids &amp; Baby"]]}""",
)

out = sys.stdout  # send output to screen
docs = [json.loads(line) for line in json_in]  # assume one object per line
for doc in docs:
    out.write('%s\n' % json.dumps({"index": {}}))
    out.write('%s\n' % json.dumps(doc))
</code></pre>

<p>Output:</p>

<pre class="lang-none prettyprint-override"><code>{"index": {}}
{"asin": "0001048791", "salesRank": {"Books": 6334800}, "imUrl": "http://ecx.images-amazon.com/images/I/51MKP0T4DBL.jpg", "categories": [["Books"]], "title": "The Crucible: Performed by Stuart Pankin, Jerome Dempsey &amp;amp; Cast"}
{"index": {}}
{"asin": "0000143561", "description": "3Pack DVD set - Italian Classics, Parties and Holidays.", "title": "Everyday Italian (with Giada de Laurentiis), Volume 1 (3 Pack): Italian Classics, Parties, Holidays", "price": 12.99, "imUrl": "http://g-ecx.images-amazon.com/images/G/01/x-site/icons/no-img-sm._CB192198896_.gif", "related": {"also_viewed": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC", "B002I5GNVU", "B000RBU4BM"], "buy_after_viewing": ["B0036FO6SI", "B000KL8ODE", "000014357X", "B0037718RC"]}, "salesRank": {"Movies &amp; TV": 376041}, "categories": [["Movies &amp; TV", "Movies"]]}
{"index": {}}
{"asin": "0000037214", "title": "Purple Sequin Tiny Dancer Tutu Ballet Dance Fairy Princess Costume Accessory", "price": 6.99, "imUrl": "http://ecx.images-amazon.com/images/I/31mCncNuAZL.jpg", "related": {"also_viewed": ["B00JO8II76", "B00DGN4R1Q", "B00E1YRI4C"]}, "salesRank": {"Clothing": 1233557}, "brand": "Big Dreams", "categories": [["Clothing, Shoes &amp; Jewelry", "Girls"], ["Clothing, Shoes &amp; Jewelry", "Novelty, Costumes &amp; More", "Costumes &amp; Accessories", "More Accessories", "Kids &amp; Baby"]]}
</code></pre>

python json elasticsearch