How to subscribe to a list of multiple kafka wildcard patterns using kafka-python?
<p>I'm subscribing to Kafka using a pattern with a wildcard, as shown below. The wildcard represents a dynamic customer id.</p>

<pre><code>consumer.subscribe(pattern='customer.*.validations')
</code></pre>

<p>This works well, because I can pluck the customer Id from the topic string. But now I need to expand on the functionality to listen to a similar topic for a slightly different purpose. Let's call it <code>customer.*.additional-validations</code>. The code needs to live in the same project because so much functionality is shared, but I need to be able to take a different path based on the type of queue.</p>

<p>In the <a href="http://kafka-python.readthedocs.io/en/master/#kafkaconsumer" rel="nofollow">Kafka documentation</a> I can see that it is possible to subscribe to an array of topics. However these are hard-coded strings. Not patterns that allow for flexibility.</p>

<pre><code>&gt;&gt;&gt; # Deserialize msgpack-encoded values
&gt;&gt;&gt; consumer = KafkaConsumer(value_deserializer=msgpack.loads)
&gt;&gt;&gt; consumer.subscribe(['msgpackfoo'])
&gt;&gt;&gt; for msg in consumer:
...     assert isinstance(msg.value, dict)
</code></pre>

<p>So I'm wondering if it is possible to somehow do a combination of the two? Kind of like this (non-working):</p>

<pre><code>consumer.subscribe(pattern=['customer.*.validations', 'customer.*.additional-validations'])
</code></pre>

<p>In the KafkaConsumer code, it supports list of topics, or a pattern,</p>

<p><a href="https://github.com/dpkp/kafka-python/blob/68c8fa4ad01f8fef38708f257cb1c261cfac01ab/kafka/consumer/group.py#L717" rel="nofollow">https://github.com/dpkp/kafka-python/blob/68c8fa4ad01f8fef38708f257cb1c261cfac01ab/kafka/consumer/group.py#L717</a></p>

<pre><code>   def subscribe(self, topics=(), pattern=None, listener=None):
        """Subscribe to a list of topics, or a topic regex pattern
        Partitions will be dynamically assigned via a group coordinator.
        Topic subscriptions are not incremental: this list will replace the
        current assignment (if there is one).
</code></pre>

<p>So you can create a regex, with OR condition using <code>|</code>, that should work as subscribe to multiple dynamic topics regex, as it internally uses <code>re</code> module for matching.</p>

<p><code>(customer.*.validations)|(customer.*.additional-validations)</code></p>

python apache-kafka python-kafka