How can I document click commands using Sphinx?
<p>(Note: this is being asked to share knowledge, not to look for help)</p>

<p><a href="http://click.pocoo.org/5/" rel="nofollow"><code>click</code></a> is a popular Python library for developing CLI applications with. <a href="http://www.sphinx-doc.org/en/stable/" rel="nofollow"><code>sphinx</code></a> is a popular library for documenting Python packages with. <a href="https://github.com/pallets/click/issues/127" rel="nofollow">One problem that some have faced</a> is integrating these two tools so that they can generate Sphinx documentation for their click-based commands.</p>

<p>I ran into this problem recently. I decorated some of my functions with <code>click.command</code> and <code>click.group</code>, added docstrings to them and then generated HTML documentation for them using Sphinx's <code>autodoc</code> extension. What I found is that it omitted all documentation and argument descriptions for these commands because they had been converted into <code>Command</code> objects by the time autodoc got to them.</p>

<p>How can I modify my code to make the documentation for my commands available to both the end user when they run <code>--help</code> on the CLI, and also to people browsing the Sphinx-generated documentation?</p>

<p><strong>Decorating command containers</strong></p>

<p>One possible solution to this problem that I've recently discovered and seems to work would be to start off defining a decorator that can be applied to classes. The idea is that the programmer would define commands as private members of a class, and the decorator creates a public function member of the class that's based on the command's callback. For example, a class <code>Foo</code> containing a command <code>_bar</code> would gain a new function <code>bar</code> (assuming <code>Foo.bar</code> does not already exist).</p>

<p>This operation leaves the original commands as they are, so it shouldn't break existing code. Because these commands are private, they should be omitted in generated documentation. The functions based on them, however, should show up in documentation on account of being public.</p>

<pre><code>def ensure_cli_documentation(cls):
    """
    Modify a class that may contain instances of :py:class:`click.BaseCommand`
    to ensure that it can be properly documented (e.g. using tools such as Sphinx).

    This function will only process commands that have private callbacks i.e. are
    prefixed with underscores. It will associate a new function with the class based on
    this callback but without the leading underscores. This should mean that generated
    documentation ignores the command instances but includes documentation for the functions
    based on them.

    This function should be invoked on a class when it is imported in order to do its job. This
    can be done by applying it as a decorator on the class.

    :param cls: the class to operate on
    :return: `cls`, after performing relevant modifications
    """
    for attr_name, attr_value in dict(cls.__dict__).items():
        if isinstance(attr_value, click.BaseCommand) and attr_name.startswith('_'):
            cmd = attr_value
            try:
                # noinspection PyUnresolvedReferences
                new_function = copy.deepcopy(cmd.callback)
            except AttributeError:
                continue
            else:
                new_function_name = attr_name.lstrip('_')
                assert not hasattr(cls, new_function_name)
                setattr(cls, new_function_name, new_function)

    return cls
</code></pre>

<p><strong>Avoiding issues with commands in classes</strong></p>

<p>The reason that this solution assumes commands are inside classes is because that's how most of my commands are defined in the project I'm currently working on - I load most of my commands as plugins contained within subclasses of <code>yapsy.IPlugin.IPlugin</code>. If you want to define the callbacks for commands as class instance methods, you may run into a problem where click doesn't supply the <code>self</code> parameter to your command callbacks when you try to run your CLI. This can be solved by currying your callbacks, like below:</p>

<pre><code>class Foo:
    def _curry_instance_command_callbacks(self, cmd: click.BaseCommand):
        if isinstance(cmd, click.Group):
            commands = [self._curry_instance_command_callbacks(c) for c in cmd.commands.values()]
            cmd.commands = {}
            for subcommand in commands:
                cmd.add_command(subcommand)

        try:
            if cmd.callback:
                cmd.callback = partial(cmd.callback, self)

            if cmd.result_callback:
                cmd.result_callback = partial(cmd.result_callback, self)
        except AttributeError:
            pass

        return cmd
</code></pre>

<p><strong>Example</strong></p>

<p>Putting this all together:</p>

<pre><code>from functools import partial

import click
from click.testing import CliRunner
from doc_inherit import class_doc_inherit


def ensure_cli_documentation(cls):
    """
    Modify a class that may contain instances of :py:class:`click.BaseCommand`
    to ensure that it can be properly documented (e.g. using tools such as Sphinx).

    This function will only process commands that have private callbacks i.e. are
    prefixed with underscores. It will associate a new function with the class based on
    this callback but without the leading underscores. This should mean that generated
    documentation ignores the command instances but includes documentation for the functions
    based on them.

    This function should be invoked on a class when it is imported in order to do its job. This
    can be done by applying it as a decorator on the class.

    :param cls: the class to operate on
    :return: `cls`, after performing relevant modifications
    """
    for attr_name, attr_value in dict(cls.__dict__).items():
        if isinstance(attr_value, click.BaseCommand) and attr_name.startswith('_'):
            cmd = attr_value
            try:
                # noinspection PyUnresolvedReferences
                new_function = cmd.callback
            except AttributeError:
                continue
            else:
                new_function_name = attr_name.lstrip('_')
                assert not hasattr(cls, new_function_name)
                setattr(cls, new_function_name, new_function)

    return cls


@ensure_cli_documentation
@class_doc_inherit
class FooCommands(click.MultiCommand):
    """
    Provides Foo commands.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._commands = [self._curry_instance_command_callbacks(self._calc)]

    def list_commands(self, ctx):
        return [c.name for c in self._commands]

    def get_command(self, ctx, cmd_name):
        try:
            return next(c for c in self._commands if c.name == cmd_name)
        except StopIteration:
            raise click.UsageError('Undefined command: {}'.format(cmd_name))

    @click.group('calc', help='mathematical calculation commands')
    def _calc(self):
        """
        Perform mathematical calculations.
        """
        pass

    @_calc.command('add', help='adds two numbers')
    @click.argument('x', type=click.INT)
    @click.argument('y', type=click.INT)
    def _add(self, x, y):
        """
        Print the sum of x and y.

        :param x: the first operand
        :param y: the second operand
        """
        print('{} + {} = {}'.format(x, y, x + y))

    @_calc.command('subtract', help='subtracts two numbers')
    @click.argument('x', type=click.INT)
    @click.argument('y', type=click.INT)
    def _subtract(self, x, y):
        """
        Print the difference of x and y.

        :param x: the first operand
        :param y: the second operand
        """
        print('{} - {} = {}'.format(x, y, x - y))

    def _curry_instance_command_callbacks(self, cmd: click.BaseCommand):
        if isinstance(cmd, click.Group):
            commands = [self._curry_instance_command_callbacks(c) for c in cmd.commands.values()]
            cmd.commands = {}
            for subcommand in commands:
                cmd.add_command(subcommand)

        if cmd.callback:
            cmd.callback = partial(cmd.callback, self)

        return cmd


@click.command(cls=FooCommands)
def cli():
    pass


def main():
    print('Example: Adding two numbers')
    runner = CliRunner()
    result = runner.invoke(cli, 'calc add 1 2'.split())
    print(result.output)

    print('Example: Printing usage')
    result = runner.invoke(cli, 'calc add --help'.split())
    print(result.output)


if __name__ == '__main__':
    main()
</code></pre>

<p>Running <code>main()</code>, I get this output:</p>

<pre class="lang-none prettyprint-override"><code>Example: Adding two numbers
1 + 2 = 3

Example: Printing usage
Usage: cli calc add [OPTIONS] X Y

  adds two numbers

Options:
  --help  Show this message and exit.


Process finished with exit code 0
</code></pre>

<p>Running this through Sphinx, I can view the documentation for this in my browser:</p>

<p><a href="http://i.stack.imgur.com/aa10A.png" rel="nofollow"><img src="http://i.stack.imgur.com/aa10A.png" alt="Sphinx documentation"></a></p>

python documentation command-line-interface python-sphinx python-click