/bin/sh: line 62: to: command not found
<p>I have a python code in which I am calling a shell command. The part of the code where I did the shell command is:</p>

<pre><code>try:
    def parse(text_list):
        text = '\n'.join(text_list)
        cwd = os.getcwd()
        os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
        synnet_output = subprocess.check_output(["echo '%s' | syntaxnet/demo.sh 2&gt;/dev/null"%text], shell = True)
        os.chdir(cwd)
        return synnet_output
except Exception as e:
    sys.stdout.write(str(e))
</code></pre>

<p>Now, when i run this code on a local file with some sample input (I did <code>cat /home/sree/example.json | python parse.py</code>) it works fine and I get the required output. But I am trying to run the code with an input on my HDFS (the same <code>cat</code> command but input file path is from HDFS) which contains exactly the same type of json entries and it fails with an error:</p>

<pre><code>/bin/sh: line 62: to: command not found
list index out of range
</code></pre>

<p>I read similar questions on Stack Overflow and the solution was to include a Shebang line for the shell script that is being called. I do have the shebang line <code>#!/usr/bin/bash</code> in <code>demo.sh</code> script.</p>

<p>Also, <code>which bash</code> gives <code>/usr/bin/bash</code>.</p>

<p>Someone please elaborate.</p>

<p>There was a problem with some special characters appearing in the text string that i was inputting to <code>demo.sh</code>. I solved this by storing <code>text</code> into a temporary file and sending the contents of that file to <code>demo.sh</code>.</p>

<p>That is: </p>

<pre><code>try:
    def parse(text_list):
        text = '\n'.join(text_list)
        cwd = os.getcwd()
        with open('/tmp/data', 'w') as f:
            f.write(text)
        os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
        synnet_output = subprocess.check_output(["cat /tmp/data | syntaxnet/demo.sh 2&gt;/dev/null"%text], shell = True)
        os.chdir(cwd)
        return synnet_output
except Exception as e:
    sys.stdout.write(str(e))
</code></pre>

python json shell hadoop subprocess