Package only binary compiled .so files of a python library compiled with Cython
<p>I have a package named <code>mypack</code> which inside has a module <code>mymod.py</code>, and 
the <code>__init__.py</code>.
For some reason that is not in debate, I need to package this module compiled 
(nor .py or .pyc files are allowed). That is, the <code>__init__.py</code> is the only
source file allowed in the distributed compressed file.</p>

<p>The folder structure is:</p>

<pre><code>. 
â  
âââ mypack
â   âââ __init__.py
â   âââ mymod.py
âââ setup.py
</code></pre>

<p>I find that Cython is able to do this, by converting each .py file in a .so library
that can be directly imported with python. </p>

<p>The question is: how the <code>setup.py</code> file must be in order to allow an easy packaging and installation?</p>

<p>The target system has a virtualenv where the package must be installed with 
whatever method that allows easy install and uninstall (easy_install, pip, etc are all
welcome).  </p>

<p>I tried all that was at my reach. I read <code>setuptools</code> and <code>distutils</code> documentation, 
all stackoverflow related questions,
and tried with  all kind of commands (sdist, bdist, bdist_egg, etc), with lots 
of combinations of setup.cfg and MANIFEST.in file entries.</p>

<p>The closest I got was with the below setup file, that would subclass the bdist_egg 
command in order to remove also .pyc files, but that is breaking the installation.</p>

<p>A solution that  installs "manually" the files in the venv is 
also good, provided that all ancillary files that are included in a proper
installation are covered (I need to run <code>pip freeze</code> in the venv and see
<code>mymod==0.0.1</code>).</p>

<p>Run it with:</p>

<pre><code>python setup.py bdist_egg --exclude-source-files
</code></pre>

<p>and (try to) install it with </p>

<pre><code>easy_install mymod-0.0.1-py2.7-linux-x86_64.egg
</code></pre>

<p>As you may notice, the target is linux 64 bits with python 2.7.</p>

<pre><code>from Cython.Distutils import build_ext
from setuptools import setup, find_packages
from setuptools.extension import Extension
from setuptools.command import bdist_egg
from setuptools.command.bdist_egg import  walk_egg, log 
import os

class my_bdist_egg(bdist_egg.bdist_egg):

    def zap_pyfiles(self):
        log.info("Removing .py files from temporary directory")
        for base, dirs, files in walk_egg(self.bdist_dir):
            for name in files:
                if not name.endswith('__init__.py'):
                    if name.endswith('.py') or name.endswith('.pyc'):
                        # original 'if' only has name.endswith('.py')
                        path = os.path.join(base, name)
                        log.info("Deleting %s",path)
                        os.unlink(path)

ext_modules=[
    Extension("mypack.mymod", ["mypack/mymod.py"]),
]

setup(
  name = 'mypack',
  cmdclass = {'build_ext': build_ext, 
              'bdist_egg': my_bdist_egg },
  ext_modules = ext_modules,
  version='0.0.1',
  description='This is mypack compiled lib',
  author='Myself',
  packages=['mypack'],
)
</code></pre>

<p>This was exactly the sort of problem <a href="http://lucumr.pocoo.org/2014/1/27/python-on-wheels/" rel="nofollow"><strong>the Python wheels format</strong></a> â <a href="https://www.python.org/dev/peps/pep-0427/" rel="nofollow">described in PEP 427</a> â was developed to address.</p>

<p>Wheels are a replacement for Python eggs (which were/are problematic for a bunch of reasons) â <a href="https://packaging.python.org/installing/" rel="nofollow">they are supported by <code>pip</code></a>, can contain architecture-specific private binaries (here is <a href="https://github.com/lepture/python-wheels/blob/master/.travis.yml" rel="nofollow">one example of such an arrangement</a>) and are accepted generally by the Python communities who have stakes in these kind of things.</p>

<p>Here is one <code>setup.py</code> snippet from the <a href="http://lucumr.pocoo.org/2014/1/27/python-on-wheels/" rel="nofollow">aforelinked</a> <em>Python on Wheels</em> article, showing how one sets up a binary distribution:</p>

<pre><code>import os
from setuptools import setup
from setuptools.dist import Distribution

class BinaryDistribution(Distribution):
    def is_pure(self):
        return False

setup(
    ...,
    include_package_data=True,
    distclass=BinaryDistribution,
)
</code></pre>

<p>â¦ in leu of the older (but probably somehow still canonically supported) <code>setuptools</code> classes you are using. Itâs very straightforward to make Wheels for your distribution purposes, as outlined â as I recall from experience, either the <code>wheel</code> modulesâ build process is somewhat cognizant of <code>virtualenv</code>, or itâs very easy to use one within the other.</p>

<p>In any case, trading in the <code>setuptools</code> egg-based APIs for wheel-based tooling should save you some serious pain, I should think.</p>

python cython setuptools distutils setup.py