Regex on bytestring in Python 3
<p>I am using RegEx to match BGP messages in a byte string. An example byte string is looking like this:</p>

<p><code>b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x13\x04\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x13\x04'</code></p>

<p>\xff (8 times) is used as "magic marker" to start a single message. Now I want to split the messages to parse each of them.</p>

<pre><code>messages = re.split(b'\xff{8}', payload)
</code></pre>

<p>Matching works fine but I got some empty fields in my messages array.</p>

<pre><code>b'' 
b''
b'001304'
b''
b''
b'001304'
</code></pre>

<p>Can someone explain this behavior? Why are there two empty fields between each (correct splitted) message. In larger byte strings sometimes there is just one empty field between each messages.</p>

<p>I think you want to match 8 occurrences of <code>\xff</code>, not just 8 trailing <code>f</code>s (e.g. <code>\xfffffffff</code>):</p>

<pre><code>messages = re.split(b'(?:\xff){8}', payload)
                      ^^^    ^
</code></pre>

<p>Also, there are just more than one 8 consecutive <code>\xff</code>s in your string on end. You might want to use</p>

<pre><code>messages = re.split(b'(?:(?:\xff){8})+', payload)
</code></pre>

<p>However, that will still result in having an empty first element if the match is found at the start of the data. You may remove the part at the beginning before splitting:</p>

<pre><code>messages = re.split(b'(?:(?:\xff){8})+', re.sub(b'^(?:(?:\xff){8})+', b'', payload))
</code></pre>

<p><strong>HOWEVER</strong>, the best idea is to just remove the empty elements with a list comprehension or with <code>Filter</code> (<a href="http://stackoverflow.com/questions/39265923/regex-on-bytestring-in-python-3/39265986?noredirect=1#comment65915808_39265986">kudos for testing goes to you</a>):</p>

<pre><code>messages = [x for x in re.split(b'(?:\xff){8}', payload) if x]
# Or, the fastest way here as per the comments
messages = list(filter(None, messages))
</code></pre>

<p><strong>See an updated <a href="https://ideone.com/RxJ4Ln" rel="nofollow">Python 3 demo</a></strong></p>

python regex python-3.x bytestring