Reading a binary file with memoryview
<p>I read a large file in the code below which has a special structure - among others two blocks that need be processed at the same time. Instead of seeking back and forth in the file I load these two blocks wrapped in <code>memoryview</code> calls</p>

<pre><code>with open(abs_path, 'rb') as bsa_file:
    # ...
    # load the file record block to parse later
    file_records_block = memoryview(bsa_file.read(file_records_block_size))
    # load the file names block
    file_names_block = memoryview(bsa_file.read(total_file_name_length))
    # close the file
file_records_index = names_record_index = 0
for folder_record in folder_records:
    name_size = struct.unpack_from('B', file_records_block, file_records_index)[0]
    # discard null terminator below
    folder_path = struct.unpack_from('%ds' % (name_size - 1),
        file_records_block, file_records_index + 1)[0]
    file_records_index += name_size + 1
    for __ in xrange(folder_record.files_count):
        file_name_len = 0
        for b in file_names_block[names_record_index:]:
            if b != '\x00': file_name_len += 1
            else: break
        file_name = unicode(struct.unpack_from('%ds' % file_name_len,
            file_names_block,names_record_index)[0])
        names_record_index += file_name_len + 1
</code></pre>

<p>The file is correctly parsed, but as it's my first use of the mamoryview interface I am not sure I do it right. The file_names_block is composed as seen by null terminated c strings.</p>

<ol>
<li>Is my trick <code>file_names_block[names_record_index:]</code> using the memoryview magic or do I create some n^2 slices ? Would I need to use <code>islice</code> here ?</li>
<li>As seen I just look for the null byte manually and then proceed to <code>unpack_from</code>. But I read in <a href="http://stackoverflow.com/a/20024532/281545">How to split a byte string into separate bytes in python</a> that I can use <code>cast()</code> (docs ?) on the memory view - any way to use that (or another trick) to split the view in bytes ? Could I just call <code>split('\x00')</code> ? Would this preserve the memory efficiency ?</li>
</ol>

<p>I would appreciate insight on the one right way to do this (in python 2).</p>

<p>A <code>memoryview</code> is not going to give you any advantages when it comes to null-terminated strings as they have no facilities for anything but fixed-width data. You may as well use <code>bytes.split()</code> here instead:</p>

<pre><code>file_names_block = bsa_file.read(total_file_name_length)
file_names = file_names_block.split(b'\00')
</code></pre>

<p>Slicing a <code>memoryview</code> doesn't use extra memory (other than the view parameters), but if using a cast you do produce new native objects for the parsed memory region the moment you try to access elements in the sequence.</p>

<p>You can still use the <code>memoryview</code> for the <code>file_records_block</code> parsing; those strings are prefixed by a length giving you the opportunity to use slicing. Just keep slicing bytes of the memory view as you process <code>folder_path</code> values, there's no need to keep an index:</p>

<pre><code>for folder_record in folder_records:
    name_size = file_records_block[0]  # first byte is the length, indexing gives the integer
    folder_path = file_records_block[1:name_size].tobytes()
    file_records_block = file_records_block[name_size + 1:]  # skip the null
</code></pre>

<p>Because the <code>memoryview</code> was sourced from a <code>bytes</code> object, indexing will give you the integer value for a byte, <code>.tobytes()</code> on a given slice gives you a new <code>bytes</code> string for that section, and you can then continue to slice to leave the remainder for the next loop.</p>

python python-2.7 binaryfiles memoryview