Using regex, extract quoted strings that may contain nested quotes
<p>I have the following string:</p>

<pre class="lang-none prettyprint-override"><code>'Well, I've tried to say "How Doth the Little Busy Bee," but it all came different!' Alice replied in a very melancholy voice. She continued, 'I'll try again.'
</code></pre>

<p>Now, I wish to extract the following quotes:</p>

<pre class="lang-none prettyprint-override"><code>1. Well, I've tried to say "How Doth the Little Busy Bee," but it all came different!
2. How Doth the Little Busy Bee,
3. I'll try again.
</code></pre>

<p>I tried the following code but I'm not getting what I want. The <code>[^\1]*</code> is not working as expected. Or is the problem elsewhere?</p>

<pre><code>import re

s = "'Well, I've tried to say \"How Doth the Little Busy Bee,\" but it all came different!' Alice replied in a very melancholy voice. She continued, 'I'll try again.'"

for i, m in enumerate(re.finditer(r'([\'"])(?!(?:ve|m|re|s|t|d|ll))(?=([^\1]*)\1)', s)):
    print("\nGroup {:d}: ".format(i+1))
    for g in m.groups():
        print('  '+g)
</code></pre>

<p>This is a great question for Python regex because sadly, in my opinion the <code>re</code> module is <a href="http://www.rexegg.com/regex-python.html#missing_in_re" rel="nofollow">one of the most underpowered of mainstream regex engines</a>. That's why for any serious regex work in Python, I turn to Matthew Barnett's stellar <a href="https://pypi.python.org/pypi/regex" rel="nofollow">regex</a> module, which incorporates some terrific features from Perl, PCRE and .NET.</p>

<p>The solution I'll show you can be adapted to work with <code>re</code>, but it is much more readable with <code>regex</code> because it is made modular. Also, consider it as a starting block for more complex nested matching, because <code>regex</code> lets you write <a href="http://www.rexegg.com/regex-recursion.html" rel="nofollow">recursive regular expressions</a> similar to those found in Perl and PCRE.</p>

<p>Okay, enough talk, here's the code (a mere four lines apart from the import and definitions). Please don't let the long regex scare you: it is long because it is designed to be readable. Explanations follow.</p>

<p><strong>The Code</strong></p>

<pre><code>import regex

quote = regex.compile(r'''(?x)
(?(DEFINE)
(?&lt;qmark&gt;["']) # what we'll consider a quotation mark
(?&lt;not_qmark&gt;[^'"]+) # chunk without quotes
(?&lt;a_quote&gt;(?P&lt;qopen&gt;(?&amp;qmark))(?&amp;not_qmark)(?P=qopen)) # a non-nested quote
) # End DEFINE block

# Start Match block
(?&amp;a_quote)
|
(?P&lt;open&gt;(?&amp;qmark))
  (?&amp;not_qmark)?
  (?P&lt;quote&gt;(?&amp;a_quote))
  (?&amp;not_qmark)?
(?P=open)
''')

str = """'Well, I have tried to say "How Doth the Little Busy Bee," but it all came different!' Alice replied in a very melancholy voice. She continued, 'I will try again.'"""

for match in quote.finditer(str):
    print(match.group())
    if match.group('quote'):
        print(match.group('quote'))
</code></pre>

<p><strong>The Output</strong></p>

<pre><code>'Well, I have tried to say "How Doth the Little Busy Bee," but it all came different!'
"How Doth the Little Busy Bee,"
'I will try again.'
</code></pre>

<p><strong>How it Works</strong></p>

<p>First, to simplify, note that I have taken the liberty of converting <code>I'll</code> to <code>I will</code>, reducing confusion with quotes. Addressing <code>I'll</code> would be no problem with a negative lookahead, but I wanted to make the regex readable.</p>

<p>In the <code>(?(DEFINE)...)</code> block, we define the three sub-expressions <code>qmark</code>, <code>not_qmark</code> and <code>a_quote</code>, much in the way that you define variables or subroutines to avoid repeating yourself. </p>

<p>After the definition block, we proceed to matching:</p>

<ul>
<li><code>(?&amp;a_quote)</code> matches an entire quote,</li>
<li><code>|</code> or...</li>
<li><code>(?P&lt;open&gt;(?&amp;qmark))</code> matches a quotation mark and captures it to the <code>open</code> group,</li>
<li><code>(?&amp;not_qmark)?</code> matches optional text that is not quotes,</li>
<li><code>(?P&lt;quote&gt;(?&amp;a_quote))</code> matches a full quote and captures it to the <code>quote</code> group,</li>
<li><code>(?&amp;not_qmark)?</code> matches optional text that is not quotes,</li>
<li><code>(?P=open)</code> matches the same quotation mark that was captured at the opening of the quote.</li>
</ul>

<p>The Python code then only needs to print the match and the <code>quote</code> capture group if present.</p>

<p>Can this be refined? You bet. Working with <code>(?(DEFINE)...)</code> in this way, you can build beautiful patterns that you can later re-read and understand. </p>

<p><strong>Adding Recursion</strong></p>

<p>If you want to handle more complex nesting using pure regex, you'll need to turn to recursion.</p>

<p>To add recursion, all you need to do is define a group and refer to it using the subroutine syntax. For instance, to execute the code within Group 1, use <code>(?1)</code>. To execute the code within group <code>something</code>, use <code>(?&amp;something)</code>. Remember to leave an exit for the engine by either making the recursion optional (<code>?</code>) or one side of an alternation.</p>

<p><strong>References</strong></p>

<ul>
<li><a href="http://www.rexegg.com/regex-disambiguation.html#define" rel="nofollow">Pre-defined regex subroutines</a></li>
<li><a href="http://www.rexegg.com/regex-capture.html#namedgroups" rel="nofollow">Named capture groups</a></li>
</ul>

python regex