How to pip install a celery task module
<p>I have a django REST API setup on one machine (currently in test on local machine but will be on a web server eventually). Let's call this machine "client". I also have a computing server to run CPU-intensive tasks that requires a long execution time. Let's call this machine "run-server".</p>

<p>"run-server" runs a celery worker connected to a local rabbitmq server. The worker currently is in a git module with this structure:</p>

<pre><code>proj/
    client.py
    cmd.sh
    requirements.txt
    tasks.py
</code></pre>

<p>The whole thing runs in a virtualenv for what it's worth. The <code>cmd.sh</code> basically executes <code>celery multi start workername -A tasks -l info</code> on "run-server". The <code>client.py</code> is a cli script that can submit a tasks to the "run-server" manually from the shell from any machine (i.e. the "client"). </p>

<p>I want to run the equivalent of the client script from a django setup <em>without</em> having to copy the <code>tasks.py</code> and <code>client.py</code> code in the django repository. Ideally I would <code>pip install proj</code> from the django code and import proj to use it just like the client script does.</p>

<p><strong><em>How can I package proj to achieve that?</em></strong></p>

<p>I am used to package my own python module with a structure roughly looking like:</p>

<pre><code>proj/
    bin/
        proj
    proj/
        __init__.py
        __main__.py
        script.py
    setup.py
    requirements.txt
</code></pre>

<p>I managed to make it work on my own. The structure above just works. Instead of <code>celery multi start workername -A tasks -l info</code>, you simply replace with <code>celery multi start workername -A proj.tasks -l info</code> and everything works. The same version of the module have to be installed within django and as the worker because the job queuein is done via duck-typing (i.e. the path and names must match)</p>

python django rabbitmq celery