csv.reader read from Requests stream: iterator should return strings, not bytes
<p>I'm trying to stream response to <code>csv.reader</code> using <code>requests.get(url, stream=True)</code>  to handle quite big data feeds. My code worked fine with <code>python2.7</code>. Here's code:</p>

<pre><code>response = requests.get(url, stream=True)
ret = csv.reader(response.iter_lines(decode_unicode=True), delimiter=delimiter, quotechar=quotechar,
    dialect=csv.excel_tab)
for line in ret:
    line.get('name')
</code></pre>

<p>Unfortunately after migration to python3.6 I got an following error: </p>

<pre><code>_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
</code></pre>

<p>I was trying to find some wrapper/decorator that would covert result of <code>response.iter_lines()</code> iterator from bytes to string, but no luck with that.
I already tried to use <code>io</code> package and also <code>codecs</code>. Using <code>codecs.iterdecode</code> doesn't split data in lines, it's just split probably by <code>chunk_size</code>,
and in this case <code>csv.reader</code> is complaining in following way:</p>

<pre><code>_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
</code></pre>

<p>I'm guessing you could wrap this in a <code>genexp</code> and feed decoded lines to it:</p>

<pre><code>from contextlib import closing

with closing(requests.get(url, stream=True)) as r:
    f = (line.decode('utf-8') for line in r.iter_lines())
    reader = csv.reader(f, delimiter=',', quotechar='"')
    for row in reader:
        print(row)
</code></pre>

<p>Using some sample data in <code>3.5</code> this shuts up <code>csv.reader</code>, every line fed to it is first <code>decoded</code> in the genexp. Also, I'm using <a href="https://docs.python.org/3/library/contextlib.html#contextlib.closing" rel="nofollow"><code>closing</code></a> from <a href="https://docs.python.org/3/library/contextlib.html" rel="nofollow"><code>contextlib</code></a> as is <a href="http://docs.python-requests.org/en/master/user/advanced/#body-content-workflow" rel="nofollow"><em>generally suggested</em></a> to automatically <code>close</code> the responce.</p>

python django python-3.x csv python-requests