Reusing compiled regular expressions in python


<p>I have a program in Perl that uses a regular expression to store supported file extensions. It reuses this regex through the code. Each file extension has a description since the regular expression has the 'x' flag. I cannot figure out how to port this to python (2.7).</p>

<h3>The original Perl</h3>

<pre class="lang-perl prettyprint-override"><code>use strict;

my @files = ('foo.abc','foo.ABC','foo.mi','foo.txt','foo.ma','foo.iff','foo.avi');

my $exts = qr/abc|mi|avi|ma|iff|tga/;

foreach my $f (sort @files) {
    if ($f =~ m/^([^.]+\.$exts)/) {
        print "file matches: $f\n";
    } 
    else {
        print "file does not match: $f\n";
    }
}
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file does not match: foo.ABC
file matches: foo.abc
file matches: foo.avi
file matches: foo.iff
file matches: foo.ma
file matches: foo.mi
file does not match: foo.txt
</code></pre>

<p>This works just as well when I add whitespace with the <code>/x</code> modifier</p>

<pre class="lang-perl prettyprint-override"><code>$exts = qr/
    abc  (?# alembic )
    |mi  (?# mentalray )
    |avi (?# windows video )
    |ma  (?# maya ascii )
    |iff (?# amiga bitmap )
    |tga (?# targa bitmap )
/ix;

foreach my $f (sort @files) {

    if ( $f =~ m/^([^.]+\.$exts )/ ) {
        print "file matches: $f\n";
    }
    else {
        print "file does not match: $f\n";
    }
}
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file matches: foo.ABC
file matches: foo.abc
file matches: foo.avi
file matches: foo.iff
file matches: foo.ma
file matches: foo.mi
file does not match: foo.txt
</code></pre>

<p>Python supports compiled regular expressions, and you can use them as components of other regular expressions</p>

<h3>Python</h3>

<pre class="lang-python prettyprint-override"><code>import re

files = [ 'foo.abc','foo.ABC','foo.mi','foo.txt','foo.ma','foo.iff','foo.avi' ]

exts = re.compile(r'(?:abc|mi|avi|ma|iff|tga)')

for f in sorted(files):
    m = re.search(r'^([^.]+\.{EXTS})'.format(EXTS=exts.pattern),f)
    if m:
        print 'file matches: {0}'.format(f)
    else:
        print 'file does not match: {0}'.format(f)
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file does not match: foo.ABC
file matches: foo.abc
file matches: foo.avi
file matches: foo.iff
file matches: foo.ma
file matches: foo.mi
file does not match: foo.txt
'''
</code></pre>

<p>But as soon as I use <code>re.VERBOSE</code>, the regex fails</p>

<pre class="lang-python prettyprint-override"><code>exts = re.compile(r'''(?:
                     abc   # alembic
                    |mi    # mentalray
                    |avi   # windows video
                    |ma    # maya ascii
                    |iff   # amiga bitmap
                    |tga   # targa bitmap
                    )''', re.IGNORECASE + re.VERBOSE)

for f in sorted(files):
    m = re.search(r'^([^.]+\.{EXTS})'.format(EXTS=exts.pattern),f)
    if m:
        print 'file matches: {0}'.format(f)
    else:
        print 'file does not match: {0}'.format(f)
</code></pre>

<h3>output</h3>

<pre class="lang-none prettyprint-override"><code>file does not match: foo.ABC
file does not match: foo.abc
file does not match: foo.avi
file does not match: foo.iff
file does not match: foo.ma
file does not match: foo.mi
file does not match: foo.txt
</code></pre>

<p>My actual code has more than 50 extensions, with comments about what they are, so I would really like to support this.</p>

<p>I've searched all the "nested regular expression" posts I could find, but all of them are string hacks. No <em>actual</em> regular expression nesting that I can find. </p>

<p>Can Python do this?</p>

<p>It is a myth that Perl will interpolate one compiled regular expression into another. If you write this</p>

<pre><code>my $exts = qr/ abc | mi | avi | ma | iff | tga /x;

if ( $f =~ /^([^.]+\.$exts)/ ) {
    ...
}
</code></pre>

<p>then within <code>$f =~ /^([^.]+\.$exts)/</code>, the contents of the regex pattern is evaluated in a <em>double-quote context</em>. That means Perl will <em>stringify</em> <code>$exts</code> to something like <code>(?^x: abc | mi | avi | ma | iff | tga )</code> (the exact result depends on what Perl pragma are in place) and <em>interpolate</em> that string before compiling the pattern</p>

<p>So the regex match is actually doing this</p>

<pre><code>$f =~ /^([^.]+\.(?^x: abc | mi | avi | ma | iff | tga ))/
</code></pre>

<p>which is clearly correct, because the <code>/x</code> modifier is enabled within the expression</p>

<p>The difference from Python is <strong><em>only</em></strong> that Python is not as careful about what an <code>re</code> object returns by its <code>pattern</code> or <code>__str__</code> methods, so they can't be injected as substrings into other patterns</p>

<p>As far as I know, the <code>pattern</code> method just returns the original regex string that was compiled to create the object. That makes it much like using C <code>#define</code> symbols: you must be very careful with parentheses, either in the definition of the original or in its invocation</p>

python regex python-2.7 perl