How to use yaml.load_all with fileinput.input?
<p>Without resorting to <code>''.join</code>, is there a Pythonic way to use PyYAML's <code>yaml.load_all</code> with <code>fileinput.input()</code> for easy streaming of multiple documents from multiple sources?</p>

<p>I'm looking for something like the following (non-working example):</p>

<pre><code># example.py
import fileinput

import yaml

for doc in yaml.load_all(fileinput.input()):
    print(doc)
</code></pre>

<p>Expected output:</p>

<pre><code>$ cat &gt;pre.yaml &lt;&lt;&lt;'--- prefix-doc'
$ cat &gt;post.yaml &lt;&lt;&lt;'--- postfix-doc'
$ python example.py pre.yaml - post.yaml &lt;&lt;&lt;'--- hello'
prefix-doc
hello
postfix-doc
</code></pre>

<p>Of course, <code>yaml.load_all</code> expects either a string, bytes, or a file-like object and <code>fileinput.input()</code> is none of those things, so the above example does not work.</p>

<p>Actual output:</p>

<pre><code>$ python example.py pre.yaml - post.yaml &lt;&lt;&lt;'--- hello'
...
AttributeError: FileInput instance has no attribute 'read'
</code></pre>

<p>You can make the example work with <code>''.join</code>, but that's cheating. I'm looking for a way that does not read the entire stream into memory at once.</p>

<p>We might rephrase the question as <em>Is there some way to emulate a string, bytes, or file-like object that proxies to an underlying iterator of strings?</em> However, I doubt that <code>yaml.load_all</code> actually needs the entire file-like interface, so that phrasing would ask for more than is strictly necessary.</p>

<p>Ideally I'm looking for the minimal adapter that would support something like this:</p>

<pre><code>for doc in yaml.load_all(minimal_adapter(fileinput.input())):
    print(doc)
</code></pre>

<p>Your <code>minimal_adapter</code> should take a <code>fileinput.FileInput</code> as a parameter and return an object which <code>load_all</code> can use. <code>load_all</code> either takes as an argument a string, but that would require concatenating the input, or it expects the argument to have a <code>read()</code> method.</p>

<p>Since your minimal_adapter needs to preserve some state, I find it clearest/easiest to implement it as an instance of a class that has a <code>__call__</code> method, and have that method return the instance and store its argument for future use. Implemented that way, the class should also have a <code>read()</code> method, as this will be called after handing the instance to <code>load_all</code>:</p>

<pre><code>import fileinput
import ruamel.yaml


class MinimalAdapter:
    def __init__(self):
        self._fip = None
        self._buf = None  # storage of read but unused material, maximum one line

    def __call__(self, fip):
        self._fip = fip  # store for future use
        self._buf = ""
        return self

    def read(self, size):
        if len(self._buf) &gt;= size:
            # enough in buffer from last read, just cut it off and return
            tmp, self._buf = self._buf[:size], self._buf[size:]
            return tmp
        for line in self._fip:
            self._buf += line
            if len(self._buf) &gt; size:
                break
        else:
            # ran out of lines, return what we have
            tmp, self._buf = self._buf, ''
            return tmp
        tmp, self._buf = self._buf[:size], self._buf[size:]
        return tmp


minimal_adapter = MinimalAdapter()

for doc in ruamel.yaml.load_all(minimal_adapter(fileinput.input())):
    print(doc)
</code></pre>

<p>With this, running your example invocation exactly gives the output that you want. </p>

<p>This is probably only more memory efficient for larger files. The <code>load_all</code> tries to read 1024 byte blocks at a time (easily found out by putting a print statement in <code>MinimalAdapter.read()</code>) and <code>fileinput</code> does some buffering as well (use <code>strace</code> if your interested to find out how it behaves).</p>

<hr>

<p><sub>This was done using <a href="https://pypi.python.org/pypi/ruamel.yaml" rel="nofollow">ruamel.yaml</a> a YAML 1.2 parser, of which I am the author. This should work for PyYAML, of which ruamel.yaml is a derived superset, as well.</sub></p>

python pyyaml