Do AND, OR strings have special meaning in PLY?
<p>When using PLY (<a href="http://www.dabeaz.com/ply/" rel="nofollow">http://www.dabeaz.com/ply/</a>) I've noticed what seems to be a very strange problem: when I'm using tokens like <code>&amp;</code> for conjunction, the program below works, but when I use <code>AND</code> in the same place, PLY claims syntax error.</p>

<p>Program:</p>

<pre><code>#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import os

from ply import lex

import ply.yacc as yacc

parser = None
lexer = None


def flatten_list(lst):
    flat = []
    for x in lst:
        if isinstance(x, list):
            flat.extend(flatten_list(x))
        else:
            flat.append(x)
    return flat


############## Tokenization ##############

tokens = (
    'number',
    'lparen',
    'rparen',
    'textw',
    'titlew',
    'qword',
    'AND'
)

t_lparen = r'\('
t_rparen = r'\)'
t_textw = r'TEXTW:'
t_titlew = r'TITLEW:'
t_qword = r'\w+'
t_AND = r'AND'


def t_number(t):
    r'\d+'
    t.value = int(t.value)
    return t


t_ignore = ' \t'


def t_error(t):
    raise ValueError(
        'Illegal character "{}" at position {}, query text: {}'.format(t.value[0], t.lexpos, t.lexer.lexdata))


lexer = lex.lex()



################# Parsing #################


def p_querylist_boolop(p):
    """querylist : subquery AND subquery"""
    print &gt;&gt; sys.stderr, 'p_querylist', list(p)
    p[0] = []
    p[0].append(p[1])
    p[0].append(p[3])


def p_subquery(p):
    """subquery : lparen querykw qwordseq rparen"""
    print &gt;&gt; sys.stderr, 'p_subquery', list(p)
    p[0] = flatten_list(p[3])


def p_querykw(p):
    """querykw : textw
                | titlew"""
    print &gt;&gt; sys.stderr, 'p_querykw', list(p)
    p[0] = p[1]


def p_qwordseq(p):
    """qwordseq : qwordseq qword
                  | qwordseq number
                  | qword
                  | number"""
    print &gt;&gt; sys.stderr, 'p_qwordseq', list(p)
    if p[0]:
        p[0].extend(p[1:])
    else:
        p[0] = p[1:]



def p_error(p):
    global parser
    if p:
        tok = parser.token()
        if tok:
            msg = 'Syntax error in input, token "{}" at position {}, query text: {}'.format(tok.value, tok.lexpos,
                                                                                            lexer.lexdata)
            raise ValueError(msg)
    msg = 'Syntax error at the end of input, query text: {}'.format(lexer.lexdata)
    raise ValueError(msg)


parser = yacc.yacc()
# parser = yacc.yacc(debug=0, write_tables=0)


def parse_query(q):
    return parser.parse(q)


if __name__ == '__main__':
    query_texts = ["""(TEXTW: one article) AND (TEXTW: two books)"""]
    for qt in query_texts:
        res = parse_query(qt)
        print '***', res
</code></pre>

<p>This produces:</p>

<pre><code>ValueError: Syntax error in input, token "(" at position 19, query text: ( TEXTW: abc ) AND ( TEXTW: aaa )
</code></pre>

<p>However, when I change the following to:</p>

<pre><code>t_AND = r'&amp;'
query_texts = ["""(TEXTW: one article) &amp; (TEXTW: two books)"""]
</code></pre>

<p>..it works just fine:</p>

<pre><code>*** [['one', 'article'], ['two', 'books']]
</code></pre>

<p>Ply has a slightly eccentric approach to ordering token regular expressions, in part because it depends on the underlying python regular expression library. Tokens defined with functions, such as your <code>number</code> token, are recognuzed in the order they appear, and unlike many lexical scanner generators, Ply makes no attempt to perform a longest match. Tokens defined by assignment -- all your other token types -- have lower priority than functions, and are placed in order by decreasing length (of the regular expression).</p>

<p>The Ply manual (section 4.3) strongly suggests not using variable assignment style for keyword tokens such as <code>AND</code>, because the pattern <code>r'AND'</code>, for example, will recognize the first three characters of, for example, <code>ANDROGYNOUS</code>, which you would probably expect to be a variable.
Instead, it recommends using a function with a simple pattern to first recognize all keywords and variables as simple words, and then use a dictionary to recognize the specific keywords. Sample code and a less telegraphic explanation are in the Ply manual (in the section I cited above).</p>

python parsing ply