Splitting arithmetric expressions in python with regexps
<p>i have a following snippet, generally i want to split arithmetic expression (with negative numbers) to tokens.</p>

<pre><code>import re
import collections

NUM     = r'(?P&lt;NUM&gt;-?\d+)'
PLUS    = r'(?P&lt;PLUS&gt;\+)'
MINUS   = r'(?P&lt;MINUS&gt;-)'
TIMES   = r'(?P&lt;TIMES&gt;\*)'
DIVIDE  = r'(?P&lt;DIVIDE&gt;/)'
LPAREN  = r'(?P&lt;LPAREN&gt;\()'
RPAREN  = r'(?P&lt;RPAREN&gt;\))'
WS      = r'(?P&lt;WS&gt;\s+)'

Token = collections.namedtuple('Token', ['type', 'value'])

def generate_tokens(text):

    pattern = re.compile('|'.join((NUM, PLUS, MINUS, TIMES, DIVIDE, LPAREN, RPAREN, WS)))

    scanner = pattern.scanner(text)
    for m in iter(scanner.match, None):
        token = Token(m.lastgroup, m.group())

        if token.type != 'WS':
            yield token

expr = "2-2"
out = [token for token in generate_tokens(expr)]
for token in out:
    print(token)
</code></pre>

<p>And there is problem with splitting negative numbers with this code, output is</p>

<pre><code>Token(type='NUM', value='2')
Token(type='NUM', value='-2')
</code></pre>

<p>But should be</p>

<pre><code>Token(type='NUM', value='2')
Token(type='MINUS', value='-')
Token(type='NUM', value='2')
</code></pre>

<p>How to fix this?</p>

<p>I think the easiest way is just to create expressions by seperating each part with space.</p>

<p>ex.</p>

<pre><code>expr= '2 - -2' 
Token(type='NUM', value='2')
Token(type='MINUS', value='-')
Token(type='NUM', value='-2')
</code></pre>

<p>The problem is that the computer dosen't know wheter you've forgot to add a minus sign in "2-2" expression or if you wanted to substract 2 - 2, which would be 0. I think it shouldn't be a problem to just specific the way how the input should be given.</p>

python regex parsing