Reusing compiled regular expressions in python


<p>I have a program in Perl that uses a regular expression to store supported file extensions. It reuses this regex through the code. Each file extension has a description since the regular expression has the 'x' flag. I cannot figure out how to port this to python (2.7).</p>

<h3>The original Perl</h3>

<pre class="lang-perl prettyprint-override"><code>use strict;

my @files = ('foo.abc','foo.ABC','foo.mi','foo.txt','foo.ma','foo.iff','foo.avi');

my $exts = qr/abc|mi|avi|ma|iff|tga/;

foreach my $f (sort @files) {
    if ($f =~ m/^([^.]+\.$exts)/) {
        print "file matches: $f\n";
    } 
    else {
        print "file does not match: $f\n";
    }
}
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file does not match: foo.ABC
file matches: foo.abc
file matches: foo.avi
file matches: foo.iff
file matches: foo.ma
file matches: foo.mi
file does not match: foo.txt
</code></pre>

<p>This works just as well when I add whitespace with the <code>/x</code> modifier</p>

<pre class="lang-perl prettyprint-override"><code>$exts = qr/
    abc  (?# alembic )
    |mi  (?# mentalray )
    |avi (?# windows video )
    |ma  (?# maya ascii )
    |iff (?# amiga bitmap )
    |tga (?# targa bitmap )
/ix;

foreach my $f (sort @files) {

    if ( $f =~ m/^([^.]+\.$exts )/ ) {
        print "file matches: $f\n";
    }
    else {
        print "file does not match: $f\n";
    }
}
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file matches: foo.ABC
file matches: foo.abc
file matches: foo.avi
file matches: foo.iff
file matches: foo.ma
file matches: foo.mi
file does not match: foo.txt
</code></pre>

<p>Python supports compiled regular expressions, and you can use them as components of other regular expressions</p>

<h3>Python</h3>

<pre class="lang-python prettyprint-override"><code>import re

files = [ 'foo.abc','foo.ABC','foo.mi','foo.txt','foo.ma','foo.iff','foo.avi' ]

exts = re.compile(r'(?:abc|mi|avi|ma|iff|tga)')

for f in sorted(files):
    m = re.search(r'^([^.]+\.{EXTS})'.format(EXTS=exts.pattern),f)
    if m:
        print 'file matches: {0}'.format(f)
    else:
        print 'file does not match: {0}'.format(f)
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file does not match: foo.ABC
file matches: foo.abc
file matches: foo.avi
file matches: foo.iff
file matches: foo.ma
file matches: foo.mi
file does not match: foo.txt
'''
</code></pre>

<p>But as soon as I use <code>re.VERBOSE</code>, the regex fails</p>

<pre class="lang-python prettyprint-override"><code>exts = re.compile(r'''(?:
                     abc   # alembic
                    |mi    # mentalray
                    |avi   # windows video
                    |ma    # maya ascii
                    |iff   # amiga bitmap
                    |tga   # targa bitmap
                    )''', re.IGNORECASE + re.VERBOSE)

for f in sorted(files):
    m = re.search(r'^([^.]+\.{EXTS})'.format(EXTS=exts.pattern),f)
    if m:
        print 'file matches: {0}'.format(f)
    else:
        print 'file does not match: {0}'.format(f)
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file does not match: foo.ABC
file does not match: foo.abc
file does not match: foo.avi
file does not match: foo.iff
file does not match: foo.ma
file does not match: foo.mi
file does not match: foo.txt
</code></pre>

<p>My actual code has more than 50 extensions, with comments about what they are, so I would really like to support this.</p>

<p>I've searched all the "nested regular expression" posts I could find, but all of them are string hacks. No <em>actual</em> regular expression nesting that I can find. </p>

<p>Can Python do this?</p>

<p>You are doing this completely wrong. First of all the <code>.pattern</code> attribute is just <strong>a string</strong>. So it's 100% useless calling <code>re.compile</code> and then extracting the initial string used to obtain the regex object to pass to <code>re.search</code>:</p>

<pre><code>&gt;&gt;&gt; regex = re.compile(r'''(
...     verbose #lol
...     | pattern  #rofl
... )
... ''', re.VERBOSE)
&gt;&gt;&gt; regex.match('verbose')  # finds the match!
&lt;_sre.SRE_Match object; span=(0, 7), match='verbose'&gt;
&gt;&gt;&gt; re.search(regex.pattern, 'verbose')  # does not find the match!
&gt;&gt;&gt;
</code></pre>

<p>As you can see the <code>pattern</code> attribute is just the <strong>initial</strong> string used to build the regex object:</p>

<pre><code>&gt;&gt;&gt; regex.pattern
'(\n    verbose #lol\n    | pattern  #rofl\n)\n'
&gt;&gt;&gt; type(regex.pattern)
&lt;class 'str'&gt;
</code></pre>

<p>So by passing this into <code>re.search</code> you make <code>re.search</code> <em>recompile</em> it, and since <code>re.search</code> doesn't have the <code>re.VERBOSE</code> flag it compiles it with a different meaning:</p>

<pre><code>&gt;&gt;&gt; re.search(regex.pattern, 'verbose', re.VERBOSE)
&lt;_sre.SRE_Match object; span=(0, 7), match='verbose'&gt;
</code></pre>

<p>Also, I'd do this instead:</p>

<pre><code>exts = [
    'abc',   # extension abc blah blah
    'cde',   # extension cde blah blah
]
exts_pattern = '(?:{})'.format('|'.join(re.escape(extension) for extension in exts))

regex = re.compile(r'^([^.]+\.{}'.format(exts_pattern), re.IGNORECASE)
</code></pre>

<p>Or similar. I.e. you keep the various extensions as a <code>list</code> and put whatever python comments you want, and when you build the regex object using <code>compile</code> you iterate over those. This makes adding an extension easier, and also it's probably useful to have such list anyways.</p>

<hr>

<p>And to answer your final question: no python <code>re</code> module does not support "regex nesting" in any way. You must provide a <em>string</em> pattern which gets compiled into a regex object.</p>

python regex python-2.7 perl