collect some specific classes in python module
<p>I have a packages contains classes <code>C1, C2, CInvalid3, C4</code>, which are defined in different sub modules.</p>

<pre><code>my_package/
    __init__.py
    sub_package1/
        __init__.py
        sub_module1.py # contains C1, C2
    sub_package2/
        __init__.py
        sub_module2.py # contains CInvalid3, C4
</code></pre>

<p>Given a string containing some class's name, I want to initialize the corresponding class. For example,</p>

<pre><code>some_valid_class_name = 'C1'
some_valid_class = all_valid_classes[some_valid_class_name]()
</code></pre>

<p>Currently I create a module <code>all_classes.py</code> under <code>my_package</code>, which contains</p>

<pre><code>all_valid_classes = {} # I want the dict contain C1, C2 and C4

def this_is_a_valid_class(cls):
    all_valid_classes[cls.__name__] = cls
    return cls
</code></pre>

<p>And in <code>sub_module1.py</code> and <code>sub_module2.py</code>, I use</p>

<pre><code>from my_package.all_classes import this_is_a_valid_class

@this_is_a_valid_class
class C1(object):
    pass
</code></pre>

<p>and add</p>

<pre><code>from .sub_package1 import sub_module1
from .sub_package2 import sub_module2
</code></pre>

<p>to <code>my_package/__init__.py</code>.</p>

<p>Is there any better way to handle this? I import all things in <code>my_package/__init__.py</code>. When I add new <code>sub_package3.sub_module3.C99</code>, I need to remember to add <code>from .sub_package3 import sub_module3</code> to <code>my_package/__init__.py</code>. I don't think these are good ideas.</p>

<p>To get what you're looking for, what you can do is to <em>mark</em> valid classes with a class attribute:</p>

<pre><code>class C1(object):
  _valid = True
</code></pre>

<p>And then dynamically import all modules in your package, storing valid classes separately in your <code>all_valid_classes</code> dictionary during the process.</p>

<p>The key thing is that you have to import the modules of all the classes you want to use no matter what, so you can do it manually (as you're doing now) or dynamically (finding the filenames of the modules within the package and importing them).</p>

<p>This snippet shows the concept (please note that it is really ugly and <strong>only intended as a proof-of-concept</strong>, I assembled it in a rush!). It would go in your <code>my_module/__init__.py</code> file:</p>

<pre><code>import os
import fnmatch
from importlib import import_module
from types import TypeType

# dictionary with all valid classes
all_valid_classes = {}

# walk directory and load modules dynamically
this_path = os.path.dirname(__file__)
modules = []
for root, dirnames, filenames in os.walk(this_path):
    for filename in fnmatch.filter(filenames, '*.py'):

        # avoid this file importing itself
        full_path = os.path.join(root, filename)
        if full_path == os.path.join(this_path, '__init__.py'):
            continue

        # convert the filename to the module name
        trimmed_path = full_path[len(this_path) + 1:-3]
        module_name = trimmed_path.replace('/', '.')

        # import the module
        module = import_module('%s.%s' % (__name__, module_name))

        # find valid classes (classes with a '_valid = True' attribute
        for item_name in dir(module):
            item = getattr(module, item_name)
            if type(item) == TypeType:
                try:
                    valid_attr = getattr(item, '_valid')
                    if valid_attr is True:
                        all_valid_classes[item_name] = item
                except AttributeError:
                    pass
</code></pre>

<p>After importing <code>my_module</code>, <code>all_valid_classes</code> will contain <code>C1</code>, <code>C2</code> and <code>C4</code> (provided that they are marked with <code>_valid = True</code>).</p>

<p>ps. Note, however, that I would personally do the imports manually as you're doing already, it is more pythonic and clean in my opinion.</p>

python python-2.7 python-3.x oop python-3.4