Read and parse a file of tokens?
<p><strong>EDIT</strong>: <em>I have edited the question to correct a major mistake (which unfortunately invalidates all the answers provided so far): the command lines can contain spaces between the words, so no solution based on using spaces as delimiters between the tokens and their parameters will work! I deeply apologize for this omission in my original post.</em></p>

<p>I have a text file containing commands in a simple (hypothetical) command language, as follows:</p>

<pre><code>$BOOLEAN_COMMAND

$NUMERIC COMMAND ALPHA 1 3 6 9 10

$NUMERIC COMMAND BETA
2 7 9 10 15
25 40 900 2000
$NUMERIC COMMAND GAMMA 6 9 11
</code></pre>

<p>1) Each "COMMAND" starts with a special character ('$') and may be followed by a sequence of digits (the "command parameters").</p>

<p>2) Commands without parameters are considered "boolean commands" and assume by default a value of True.</p>

<p>3) There can be many commands with parameters (I call them here "Alpha", "Beta", etc.), but no matter their names, all are followed by one of more lines containing parameters.</p>

<p>4) There may or may not be blank lines between lines contaning commands.</p>

<p>I wrote a function which reads a file containing said commands and parameters and returns only the parameters of a specific command (passed as a function parameter). Here it is:</p>

<pre><code>def get_params(fname, command):
    fspecs = open(fname,"r")

    params = []
    for cline in fspecs:
        cline = cline.strip()
        if not cline:
            continue     # Blank line
        if cline.startswith('$'):
            if command in cline:
                params = cline.partition(command)[-1].split()
        #else:    # Continuation of a command.
        #    params.append(cline)
    fspecs.close()

    if len(params) == 0: # Boolean command, defaults to True
        ret_val = True
    else:
        ret_val = ' '.join(params) # Numeric command, gets parameters
    return ret_val

p = get_params('command_file', '$BOOLEAN COMMAND')
print p # returns True
p = get_params('command_file', '$NUMERIC COMMAND ALPHA')
print p # returns 1 3 6 9 10
p = get_params('command_file', '$NUMERIC COMMAND BETA')
print p # should return 2 7 9 10 15, but returns True
</code></pre>

<p>The above code works when the parameters of a given command are in a single line (immediately after the command token), but fails when the parameters are in subsequent lines (in that case, it just returns 'True' because no parameters are found after the command token). If the 'else' clause is not commented out, it just takes all lines containing parameters of whatever tokens there are up to the end of the file. Actually running the above code will better demonstrate the problem.</p>

<p>What I want is being able to read one specific token (passed to the function) and get only its parameters, no matter if they extend into several lines or how many other tokens there may be in the command file.</p>

<p>Here is another solution. This one uses regular expression and it does not squeeze multiple spaces within a command:</p>

<pre><code>import re

def parse_commands_file(filename):
    command_pattern = r"""
        (\$[A-Z _]+)*  # The command, optional
        ([0-9 \n]+)*   # The parameter which might span multiple lines, optional
    """
    command_pattern = re.compile(command_pattern, flags=re.VERBOSE)

    with open(filename) as f:
        tokens = re.findall(command_pattern, f.read())
        return {cmd.strip(): [int(n) for n in params.split()] for cmd, params in tokens}

if __name__ == '__main__':
    command = parse_commands_file('commands.txt')
    print '$BOOLEAN_COMMAND:', command.get('$BOOLEAN_COMMAND')
    print '$NUMERIC COMMAND ALPHA:', command.get('$NUMERIC COMMAND ALPHA')
    print '$NUMERIC COMMAND BETA:', command.get('$NUMERIC COMMAND BETA')
</code></pre>

<h1>Discussion</h1>

<p>Basically, the command pattern says each line might contain two parts the command name and the numerical parameters, both are optional. </p>

<p>Note that a command might contain trailing space, that is why we strip them off using the expression <code>cmd.strip()</code>.</p>

<p>Also, the parameters part returned by <code>re.findall()</code> needs to be parsed by splitting them off by white spaces, then convert to <code>int</code> with the expression <code>[int(n) for n in params.split()]</code></p>

python python-2.7