How to decode binary file with " for index, line in enumerate(file)"?
<p>I am opening up an extremely large binary file I am opening in Python 3.5 in <code>file1.py</code>:</p>

<pre><code>with open(pathname, 'rb') as file:
    for i, line in enumerate(file):
        # parsing here
</code></pre>

<p>However, I naturally get an error because I am reading the file in binary mode and then creating a list of bytes. Then with a for loop, you are comparing string to bytes and here the code fails. </p>

<p>If I was reading in individual lines, I would do this:</p>

<pre><code>with open(fname, 'rb') as f:
    lines = [x.decode('utf8').strip() for x in f.readlines()]
</code></pre>

<p>However, I am using <code>for index, lines in enumerate(file):</code>. What is the correct approach in this case? Do I decode the next objects? </p>

<p>Here is the actual code I am running: </p>

<pre><code>with open(bam_path, 'rb') as file:
    for i, line in enumerate(file):
        line_data=pd.DataFrame({k.strip():v.strip()
            for k,_,v in (e.partition(':')
                for e in line.split('\t'))}, index=[i])
</code></pre>

<p>And here is the error: </p>

<pre><code>Traceback (most recent call last):                                                                                                
  File "file1.py", line 18, in &lt;module&gt;                                                                                        
    for e in line.split('\t'))}, index=[i])                                                                                       
TypeError: a bytes-like object is required, not 'str' 
</code></pre>

<p>You could feed a generator with the decoded lines to <code>enumerate</code>:</p>

<pre><code>for i, line in enumerate(l.decode(errors='ignore') for l in f):
</code></pre>

<p>Which does the trick of yielding every line in <code>f</code> after decoding it. I've added <code>errors='ignore'</code> due to the fact that opening with <code>r</code> failed with an unknown start byte.</p>

<p>As an aside, you could just replace all string literals with byte literals when operating on <code>bytes</code>, i.e: <code>partition(b':')</code>, <code>split(b'\t')</code> and do your work using <code>bytes</code> (pretty sure pandas works fine with them).</p>

python python-3.x binary decode enumerate