Dstream output to Rabbitmq
<p>I am using spark 2.0 (python). I am trying to send the output stream to Rabbitmq. Following is my implemetation, </p>

<pre><code>def sendPartition(iter):
    connection = pika.BlockingConnection(pika.URLParameters('amqp://user:pass@ip:5672/'))
    channel = connection.channel()
    for record in iter:
        channel.basic_publish(exchange='ExchangePostingUpdate',
                              routing_key='PostingUpdate',
                              body=json.dumps(record)
                              )

    channel.close()

data_stream.foreachRDD(lambda rdd: rdd.foreachPartition(lambda partition:sendPartition(partition,rabbitconnection)))
</code></pre>

<p>It is working fine. But as we see here it is estabilshing connection and closing it for every partition. Is there any better way to do this ?</p>

<p>i tried to estabilsh a connection and pass to <code>sendPartition(iter,conn)</code> but i did not work. </p>

<p>What you have right now is probably close to the optimum in a general case. Python workers don't share state so you cannot reuse connections between partitions in the same batch.</p>

<p>It should be possible to reuse connections for each worker between batches by creating a singleton wrapping a connection pool (there are well established patterns you can use like Borg pattern or using module as a singleton) but this will work only under certain assumptions:</p>

<ul>
<li><code>spark.python.worker.reuse</code> is set to true.</li>
<li>Disabled dynamic allocation (<code>spark.dynamicAllocation.enabled</code>).</li>
</ul>

<p>If resources are created dynamically and / or not reused keeping persistent pool won't be helpful.</p>

python apache-spark rabbitmq spark-streaming