How to stream POST data into Python requests?
<p>I'm using the Python <code>requests</code> library to send a POST request. The part of the program that produces the POST data can <strong>write</strong> into an arbitrary file-like object (output stream).</p>

<p>How can I make these two parts fit?</p>

<p>I would have expected that <code>requests</code> provides a streaming interface for this use case, but it seems it doesn't. It only accepts as <code>data</code> argument a file-like object from which it <strong>reads</strong>. It doesn't provide a file-like object into which I can <strong>write</strong>.</p>

<p>Is this a fundamental issue with the Python HTTP libraries?</p>

<h1>Ideas so far:</h1>

<p>It seems that the simplest solution is to <code>fork()</code> and to let the requests library communicate with the POST data producer throgh a <strong>pipe</strong>.</p>

<p>Is there a better way?</p>

<p>Alternatively, I could try to complicate the POST data producer. However, that one is parsing one XML stream (from stdin) and producing a new XML stream to used as POST data. Then I have the same problem in reverse: The XML serializer libraries want to <strong>write</strong> into a file-like object, I'm not aware of any possibility that an XML serializer provides a file-like object from which other can <strong>read</strong>.</p>

<p>I'm also aware that the cleanest, classic solution to this is coroutines, which are somewhat available in Python through generators (<code>yield</code>). The POST data could be streamed through (<code>yield</code>) instead of a file-like object and use a pull-parser.</p>

<p>However, is possible to make <code>requests</code> accept an iterator for POST data? And is there an XML serializer that can readily be used in combination with <code>yield</code>?</p>

<p>Or, are there any wrapper objects that turn writing into a file-like object into a generator, and/or provide a file-like object that wraps an iterator?</p>

<p>The only way of connecting a data producer that requires a push interface for its data sink with a data consumer that requires a pull interface for its data source is through an intermediate buffer. Such a system can be operated only by running the producer and the consumer in "parallel" - the producer fills the buffer and the consumer reads from it, each of them being suspended as necessary. Such a parallelism can be simulated with <em>cooperative multitasking</em>, where the producer yields the control to the consumer when the buffer is full, and the consumer returns the control to the producer when the buffer gets empty. By taking the generator approach you will be building a custom-tailored cooperative multitasking solution for your case, which will hardly end up being simpler compared to the easy pipe-based approach, where the responsibility of scheduling the producer and the consumer is entirely with the OS.</p>

python xml http python-requests generator