Python parser ply matches wrong regex
<p>I'm trying to create a parser using Ply but I am confronted to a strange error.
Here is a MCVE where the matching error occurs :</p>

<p>Lexer</p>

<pre><code>import ply.lex as lex

tokens = (
    'IDENTIFIER',
    'NAME',
    'EQUALS'
)

def t_IDENTIFIER(t):
    r'\* *[a-zA-Z_]+'
    print("identifier")
    return t

def t_NAME(t):
    r"[a-zA-Z_]+"
    print("name")
    return t

t_EQUALS  = r"="
t_ignore = ' \t'


def t_newline(t):
    r'\n+'
    t.lexer.lineno += len(t.value)

# Error handling rule
def t_error(t):
    print("Illegal character '%s' at line' %s'" % (t.value[0] , t.lexer.lineno ))
    t.lexer.skip(1)

# Build the lexer
lexer = lex.lex()
</code></pre>

<p>Parser</p>

<pre><code>import ply.yacc as yacc
from l import tokens

def p_main(p):
    '''
    main : NAME EQUALS NAME
    '''

def p_error(p):
    if p is not None:
        print ("Line %s, illegal token %s" % (p.lineno, p.value))
    else:
        print("Syntax error in input!")

parser = yacc.yacc()

with open('simple2','r') as f:
    result = parser.parse(f.read())
</code></pre>

<p>My input file only contains this :</p>

<pre><code>A = B
</code></pre>

<p>And what happens is that the first word <code>A</code> is matched by the token <code>IDENTIFIER</code> even if it is not supposed to do since the regex requires a <code>*</code> before the letters.
After this the parser is unable to recognize the expression since the lexer does not return the right tokens.</p>

<p>What is wrong ? The regex used for the token IDENTIFIER works perfectly in Python.</p>

<p>According to the <a href="http://www.dabeaz.com/ply/ply.html#ply_nn6" rel="nofollow">PLY manual</a>: (emphasis added)</p>

<blockquote>
  <p>Internally, <code>lex.py</code> uses the <code>re</code> module to do its pattern matching. <em>Patterns are compiled using the <code>re.VERBOSE</code> flag</em> which can be used to help readability. However, be aware that unescaped whitespace is ignored and comments are allowed in this mode. If your pattern involves whitespace, make sure you use <code>\s</code>. If you need to match the <code>#</code> character, use <code>[#]</code>.</p>
</blockquote>

<p>So the space character in your regular expression <code>\* *[a-zA-Z_]+</code> is ignored, making the regular expression, effectively, <code>\**[a-zA-Z_]+</code>; i.e., zero or more stars. If you really want it to be a star followed by one or more spaces, you would want something like: <code>\*\ [a-zA-Z_]+</code>. </p>

python regex parsing lexer ply