How to stream POST data into Python requests?
<p>I'm using the Python <code>requests</code> library to send a POST request. The part of the program that produces the POST data can <strong>write</strong> into an arbitrary file-like object (output stream).</p>

<p>How can I make these two parts fit?</p>

<p>I would have expected that <code>requests</code> provides a streaming interface for this use case, but it seems it doesn't. It only accepts as <code>data</code> argument a file-like object from which it <strong>reads</strong>. It doesn't provide a file-like object into which I can <strong>write</strong>.</p>

<p>Is this a fundamental issue with the Python HTTP libraries?</p>

<h1>Ideas so far:</h1>

<p>It seems that the simplest solution is to <code>fork()</code> and to let the requests library communicate with the POST data producer throgh a <strong>pipe</strong>.</p>

<p>Is there a better way?</p>

<p>Alternatively, I could try to complicate the POST data producer. However, that one is parsing one XML stream (from stdin) and producing a new XML stream to used as POST data. Then I have the same problem in reverse: The XML serializer libraries want to <strong>write</strong> into a file-like object, I'm not aware of any possibility that an XML serializer provides a file-like object from which other can <strong>read</strong>.</p>

<p>I'm also aware that the cleanest, classic solution to this is coroutines, which are somewhat available in Python through generators (<code>yield</code>). The POST data could be streamed through (<code>yield</code>) instead of a file-like object and use a pull-parser.</p>

<p>However, is possible to make <code>requests</code> accept an iterator for POST data? And is there an XML serializer that can readily be used in combination with <code>yield</code>?</p>

<p>Or, are there any wrapper objects that turn writing into a file-like object into a generator, and/or provide a file-like object that wraps an iterator?</p>

<p><code>request</code> does take an iterator or generator as <code>data</code> argument, the details are described in <a href="http://docs.python-requests.org/en/master/user/advanced/#chunk-encoded-requests" rel="nofollow">Chunk-Encoded Requests</a>. The transfer encoding needs to be chunked in this case because the data size is not known beforehand.</p>

<p>Here is a very simle example that uses a <a href="https://docs.python.org/3/library/queue.html#queue.Queue" rel="nofollow"><code>queue.Queue</code></a> and can be used as a file-like object for writing:</p>

<pre><code>import requests
import queue
import threading

class WriteableQueue(queue.Queue):

    def write(self, data):
        # An empty string would be interpreted as EOF by the receiving server
        if data:
            self.put(data)

    def __iter__(self):
        return iter(self.get, None)

    def close(self):
        self.put(None)

# quesize can be limited in case producing is faster then streaming
q = WriteableQueue(100)

def post_request(iterable):
    r = requests.post("http://httpbin.org/post", data=iterable)
    print(r.text)

threading.Thread(target=post_request, args=(q,)).start()

# pass the queue to the serializer that writes to it ...    
q.write(b'1...')
q.write(b'2...')

# closing ends the request
q.close()
</code></pre>

python xml http python-requests generator