Read and parse a file of tokens?
<p><strong>EDIT</strong>: <em>I have edited the question to correct a major mistake (which unfortunately invalidates all the answers provided so far): the command lines can contain spaces between the words, so no solution based on using spaces as delimiters between the tokens and their parameters will work! I deeply apologize for this omission in my original post.</em></p>

<p>I have a text file containing commands in a simple (hypothetical) command language, as follows:</p>

<pre><code>$BOOLEAN_COMMAND

$NUMERIC COMMAND ALPHA 1 3 6 9 10

$NUMERIC COMMAND BETA
2 7 9 10 15
25 40 900 2000
$NUMERIC COMMAND GAMMA 6 9 11
</code></pre>

<p>1) Each "COMMAND" starts with a special character ('$') and may be followed by a sequence of digits (the "command parameters").</p>

<p>2) Commands without parameters are considered "boolean commands" and assume by default a value of True.</p>

<p>3) There can be many commands with parameters (I call them here "Alpha", "Beta", etc.), but no matter their names, all are followed by one of more lines containing parameters.</p>

<p>4) There may or may not be blank lines between lines contaning commands.</p>

<p>I wrote a function which reads a file containing said commands and parameters and returns only the parameters of a specific command (passed as a function parameter). Here it is:</p>

<pre><code>def get_params(fname, command):
    fspecs = open(fname,"r")

    params = []
    for cline in fspecs:
        cline = cline.strip()
        if not cline:
            continue     # Blank line
        if cline.startswith('$'):
            if command in cline:
                params = cline.partition(command)[-1].split()
        #else:    # Continuation of a command.
        #    params.append(cline)
    fspecs.close()

    if len(params) == 0: # Boolean command, defaults to True
        ret_val = True
    else:
        ret_val = ' '.join(params) # Numeric command, gets parameters
    return ret_val

p = get_params('command_file', '$BOOLEAN COMMAND')
print p # returns True
p = get_params('command_file', '$NUMERIC COMMAND ALPHA')
print p # returns 1 3 6 9 10
p = get_params('command_file', '$NUMERIC COMMAND BETA')
print p # should return 2 7 9 10 15, but returns True
</code></pre>

<p>The above code works when the parameters of a given command are in a single line (immediately after the command token), but fails when the parameters are in subsequent lines (in that case, it just returns 'True' because no parameters are found after the command token). If the 'else' clause is not commented out, it just takes all lines containing parameters of whatever tokens there are up to the end of the file. Actually running the above code will better demonstrate the problem.</p>

<p>What I want is being able to read one specific token (passed to the function) and get only its parameters, no matter if they extend into several lines or how many other tokens there may be in the command file.</p>

<p>Here is my approach: split everything based on white spaces (spaces, tabs and new lines). Then construct a dictionary with command names as keys and parameters as values. From this dictionary, you can look up parameters for any command. This approach opens and reads the file only once:</p>

<pre><code>from collections import deque

def parse_commands_file(filename):
    with open(filename) as f:
        tokens = deque(f.read().split())

    command2parameters = dict()
    while tokens:
        command_name = tokens.popleft()
        # Added
        while tokens and tokens[0].isalpha() and not tokens[0].startswith('$'):
            command_name = command_name + ' ' + tokens.popleft()
        # end added

        parameters = []
        while tokens and not tokens[0].startswith('$'):
            parameters.append(int(tokens.popleft()))
        command2parameters[command_name] = parameters or True

    return command2parameters

if __name__ == '__main__':
    command = parse_commands_file('commands.txt')
    print '$BOOLEAN_COMMAND:', command.get('$BOOLEAN_COMMAND')
    print '$NUMERIC_COMMAND_ALPHA:', command.get('$NUMERIC_COMMAND_ALPHA')
    print '$NUMERIC_COMMAND_BETA:', command.get('$NUMERIC_COMMAND_BETA')
</code></pre>

<p>Output:</p>

<pre><code>$BOOLEAN_COMMAND: True
$NUMERIC_COMMAND_ALPHA: [1, 3, 6, 9, 10]
$NUMERIC_COMMAND_BETA: [2, 7, 9, 10, 15, 25, 40, 900, 2000]
</code></pre>

<h1>Discussion</h1>

<ul>
<li>I use the <code>deque</code> data structure, which stands for <em>double-end queue</em>. This structure behaves like a list, but more efficient in term of insert and pop from both ends</li>
<li>When parsing the parameters, I converted them to <code>int</code>, you can convert them to float or leave them be</li>
<li>The expression <code>parameters or True</code> basically says: if parameters is empty, use <code>True</code>, otherwise, leave it be</li>
</ul>

<h1>Update</h1>

<p>I have added a patch to handle commands with spaces in their names. However, this solution is just a patch, it does not work if you have multiple spaces such as:</p>

<pre><code>$MY      COMMAND HERE
</code></pre>

<p>In this case, multiple spaces got squeezed into one.</p>

python python-2.7