HttpIO - Consuming external resources in a Dataflow transform
<p>I'm trying to write a custom <code>Source</code> using the Python Dataflow SDK to read JSON data in parallel from a REST endpoint.</p>

<p>E.g. for a given set of IDs, I need to retrieve the data from:
<code>
https://foo.com/api/results/1
https://foo.com/api/results/2
...
https://foo.com/api/results/{maxID}
</code></p>

<p>The key features I need are monitoring &amp; rate limiting : even though I need parallelism (either thread/process-based or using async/coroutines), I also need to make sure my job stays "polite" towards the API endpoint - effectively avoiding involuntary DDoS.</p>

<p>Using <a href="https://github.com/GoogleCloudPlatform/psq" rel="nofollow">psq</a>, I should be able to implement some kind of rate-limit mechanism, but then I'd lose the ability to monitor progress &amp; ETA using the <a href="https://cloud.google.com/dataflow/pipelines/dataflow-monitoring-intf" rel="nofollow">Dataflow Service Monitoring</a></p>

<p>It seems that, although they work well together, monitoring isn't unified between Google Cloud Dataflow and Google Cloud Pub/Sub (which uses Google Stackdriver Monitoring)</p>

<p><strong>How should I go about building a massively-parallel HTTP consumer workflow which implements rate limiting and has web-based monitoring ?</strong> </p>

<p>Dataflow does not currently have a built-in way of doing global rate-limiting, but you can use the Source API to do this.  The key concept is that each split of a Source will be processed by a single thread at most, so you can implement local rate limiting separately for each split.</p>

<p>This solution does not use Pub/Sub at all, so you can exclusively use the Dataflow Monitoring UI.  If you want to set up alerts based on particular events in your pipeline, you could do something like <a href="http://stackoverflow.com/questions/35068351/how-do-i-set-alerts-based-on-events-in-my-dataflow-pipelines">this</a></p>

python google-cloud-dataflow google-cloud-pubsub airflow