Storm missing resources folder when deploying JAR topology
<p>I'm trying to deploy a topology that uses a MultiLang Bolt (written in Python):</p>

<pre><code>builder.setBolt("avro-parser", new AvroBolt(), 3).shuffleGrouping("main-kafka-spout");
builder.setBolt("nlp-analyzer", new NLPBolt("/python/analyzer/audio_parser.py"), 2).shuffleGrouping("avro-parser");
</code></pre>

<p>I can submit the topology without problems, but checking the log files I see this error:</p>

<pre><code>java.lang.RuntimeException: Error when launching multilang subprocess

    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:89) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.task.ShellBolt.prepare(ShellBolt.java:131) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.executor$fn__7953$fn__7966.invoke(executor.clj:792) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.util$async_loop$fn__625.invoke(util.clj:482) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_91]
Caused by: java.io.IOException: Cannot run program "/usr/bin/python" (in directory "/var/lib/storm/supervisor/stormdist/sintonea-topology-main-22-1472285031/resources"): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(Unknown Source) ~[?:1.8.0_91]
    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:82) ~[storm-core-1.0.1.jar:1.0.1]
    ... 5 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_91]
    at java.lang.UNIXProcess.&lt;init&gt;(Unknown Source) ~[?:1.8.0_91]
    at java.lang.ProcessImpl.start(Unknown Source) ~[?:1.8.0_91]
    at java.lang.ProcessBuilder.start(Unknown Source) ~[?:1.8.0_91]
    at org.apache.storm.utils.ShellProcess
</code></pre>

<p>Of course, I've noticed that I don't have a "resources" folder in the specified log trace:</p>

<blockquote>
  <p>/var/lib/storm/supervisor/stormdist/sintonea-topology-main-22-1472285031/resources</p>
</blockquote>

<p>If you do a "ls" command in that directory, you'll see it:</p>

<pre><code>stormcode.ser  stormconf.ser  stormjar.jar
</code></pre>

<p>Why storm is not creating a resources folder in that directory? </p>

<p>I have two resources folders inside src/main/resources: </p>

<ul>
<li>schema: contains avro scheme files</li>
<li>python: contains the python code</li>
</ul>

<p>These folders are copied into the JAR in the root directory, instead of a /resources folder.</p>

<p>How is that possible? I have an AvroBolt that is using an schema in this way and it's not giving any problem (tested with a DummyBolt instead of MultiLang Bolt):</p>

<pre><code>_schema = parser.parse(getClass().getResourceAsStream("/schema/caller_request.avsc"));
</code></pre>

<p><strong>EDIT</strong></p>

<p>I've moved my resources folder to </p>

<blockquote>
  <p>baseDirectory.value / "multilang"</p>
</blockquote>

<p>After reading this:</p>

<p><a href="http://storm.apache.org/releases/current/Creating-a-new-Storm-project.html" rel="nofollow">http://storm.apache.org/releases/current/Creating-a-new-Storm-project.html</a></p>

<p>Now the resource folder is created and copied, but I'm still having the same issue:</p>

<pre><code>Serializer Exception:
/usr/bin/python: can't open file '/resources/python/analyzer/audio_parser.py': [Errno 2] No such file or directory
</code></pre>

<p><strong>EDIT 2</strong></p>

<p>I've found a workaround (create two resources folders at different levels): </p>

<ol>
<li>Create a multilang/resources in the parent directory of src (one level up). Copy the python folder inside: {base-directory}/multilang/resources/python/*.py</li>
<li>Leave intact the folder src/main/schema: {base-directory}/src/main/resources/schema/caller_request.avsc.</li>
</ol>

<p>And added this to my build.sbt:</p>

<pre><code>unmanagedResourceDirectories in Compile += { baseDirectory.value / "multilang" }
unmanagedClasspath in Compile += baseDirectory.value / "multilang"
</code></pre>

<p>And set up my ShellBot as (ignoring the word "resources" or "/" in path):</p>

<pre><code>NLPBolt nlpBolt = new NLPBolt("python/analyzer/audio_parser.py");
</code></pre>

<p>And:</p>

<pre><code>_schema = parser.parse(getClass().getResourceAsStream("/schema/caller_request.avsc"));
</code></pre>

<p>It seems to work, but I don't like this solution. Any thoughts?</p>

<p><strong>EDIT 3</strong></p>

<p>By the way, this code:</p>

<pre><code>import storm
#from nltk.stem.snowball import SnowballStemmer
from es_tagger import SpanishTagger

config = utils.load_json('python/analyzer/data/config.json')

class AudioParserBolt(storm.BasicBolt):
</code></pre>

<p>Produces the following error:</p>

<pre><code>Serializer Exception:
Traceback (most recent call last):
  File "python/analyzer/audio_parser.py", line 27, in &lt;module&gt;
    class AudioParserBolt(storm.BasicBolt):
AttributeError: 'module' object has no attribute 'BasicBolt'
</code></pre>

<p>So, this is basically what I did to get it working:</p>

<ol>
<li>Create folder ${basedir}/multilang/resources --> copy python code inside.</li>
<li>Remove python code from ${basedir}/src/main/resources and leave only the Avro schemes.</li>
<li>Add to the build.sbt file:</li>
</ol>

<blockquote>
  <p>unmanagedResourceDirectories in Compile += { baseDirectory.value / "multilang" }
      unmanagedClasspath in Compile += baseDirectory.value / "multilang"</p>
</blockquote>

<ol start="4">
<li>Download storm.py from <a href="https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py" rel="nofollow">https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py</a> and copy/paste it into ${basedir}/multilang/resources/python folder.</li>
<li>In the Python code, comment or remove any print instruction (provokes Java exceptions parsing the tuples because print writes into standard output).</li>
<li>In case your Python process is a bit slow (mine is an NLP process and requires a little bit to setup the first time), tell your topology to wait a little more:</li>
</ol>

<blockquote>
  <p>config.put(Config.TOPOLOGY_SUBPROCESS_TIMEOUT_SECS, );</p>
</blockquote>

<ol start="7">
<li>And a lot of trial and error =)</li>
</ol>

<p>Any time you need to use an external file in your Python code, like a configuration file, call it like this (the parent directory will be the multilang/resources we created before):</p>

<blockquote>
  <p>config = utils.load_json('python/analyzer/data/config.json')</p>
</blockquote>

java python sbt apache-storm