Text Parser in Python
<p>I have to write a code to read data in text file. This text file has a specific format. It is like comma-separated values (CSV) file that stores tabular data. And, I must be able to perform calculations on the data of that file.</p>

<p>Here's the format instruction of that file:</p>

<p>A dataset has to start with a declaration of its name:</p>

<p>@relation name</p>

<p>followed by a list of all the attributes in the dataset</p>

<p>@attribute attribute_name specification</p>

<p>If an attribute is nominal, specification contains a list of the possible attribute values in curly brackets:</p>

<p>@attribute nominal_attribute {first_value, second_value, third_value}</p>

<p>If an attribute is numeric, specification is replaced by the keyword </p>

<p>@attribute numeric_attribute numeric</p>

<p>After the attribute declarations, the actual data is introduced by a </p>

<p>@data</p>

<p>tag, which is followed by a list of all the instances. The instances  are listed in comma-separated format, with a question mark representing a missing value. </p>

<p>Comments are lines starting with % and are ignored.</p>

<p>I must be able to make calculations on this data separated by comma, and must know which data is associated to which attribute.</p>

<p>Example dataset file:
1: <a href="https://drive.google.com/open?id=0By6GDPYLwp2cSkd5M0J0ZjczVW8" rel="nofollow">https://drive.google.com/open?id=0By6GDPYLwp2cSkd5M0J0ZjczVW8</a>
2: <a href="https://drive.google.com/open?id=0By6GDPYLwp2cejB5SVlhTFdubnM" rel="nofollow">https://drive.google.com/open?id=0By6GDPYLwp2cejB5SVlhTFdubnM</a></p>

<p>I have no experience with parsing and very little experience with Python. So, I felt to ask the experts for the easy way to do it.</p>

<p>Thanks</p>

<p>Here is a simple solution that I came up with:</p>

<p>The idea is to read the file line by line and apply rules depending on the type of line encountered.</p>

<p>As you see in the sample input, there could be broadly 5 types of input you may encounter.</p>

<ol>
<li><p>A comment which could start with '%' -> no action is needed here.</p></li>
<li><p>A blank line i.e. '\n' -> no action needed here.</p></li>
<li><p>A line that starts with @, which indicates it could be an attribute or name of the relation.</p></li>
<li><p>If not any of these, then it is the data itself.</p></li>
</ol>

<p>The code follows a simple if-else logic taking actions at every step. based on the above 4 rules.</p>

<pre><code>with open("../Downloads/Reading_Data_Files.txt","r") as dataFl:
    lines = [line for line in dataFl]

attribute = []
data = []
for line in lines:
    if line.startswith("%") or 'data' in line or line=='\n': # this is a comment or the data line
        pass
    elif line.startswith("@"):
        if "relation" in line:
            relationName = line.split(" ")[1]
        elif "attribute" in line:
            attribute.append(line.split(" ")[1])
    else:
        data.append(list(map(lambda x : x.strip(),line.split(","))))

print("Relation Name is : %s" %relationName)
print("Attributes are " + ','.join(attribute))
print(data)
</code></pre>

<p>If you want to see which attribute is what here is a solution, which is essentially the same solution as above but with a minor tweak. The only issue with solution above is that the output is a list of lists and to tell which attribute is which is an issue. Hence, a rather better solution would be annotate each data element with the corresponding attribute name. The output will be of the form:
<code>{'distance': '45', 'temperature': '75', 'BusArrival': 'on_time', 'Students': '25'}</code></p>

<pre><code>with open("/Users/sreejithmenon/Downloads/Reading_Data_Files.txt","r") as dataFl:
    lines = [line for line in dataFl]

attribute = []
data = []
for line in lines:
    if line.startswith("%") or 'data' in line or line=='\n': # this is a comment or the data line
        pass
    elif line.startswith("@"):
        if "relation" in line:
            relationName = line.split(" ")[1]
        elif "attribute" in line:
            attribute.append(line.split(" ")[1])
    else:
        dataLine = list(map(lambda x : x.strip(),line.split(",")))
        dataDict = {attribute[i] : dataLine[i] for i in range(len(attribute))} # each line of data is now a dictionary.
        data.append(dataDict)

print("Relation Name is : %s" %relationName)
print("Attributes are " + ','.join(attribute))
print(data)
</code></pre>

<p>You could use pandas Data frames to do more analysis, slicing, querying etc. Here is a link that should help you get started with <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html" rel="nofollow">http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html</a></p>

<p>Edit: Explanation to comments
Meaning of the line: <code>dataLine = list(map(lambda x : x.strip(),line.split(",")))</code>
<code>split(&lt;delimiter&gt;)</code> function will split a string into pieces wherever there is a delimiter and returns a list(iterator). </p>

<p>For instance, 
<code>"hello, world".split(",")</code> will return <code>['hello',' world']</code> <em>Notice the space in front of "world".</em></p>

<p><code>map</code> is a function that can apply a function(first argument) to each element in a <code>iterator</code>(second argument). It is generally used as a short-hand to apply transformations to each element of the iterator. <code>strip()</code> removes any leading or trailing whitespace. A <code>lambda expression</code> is a function and here it simply applies the strip function. <code>map()</code> extracts each element from the iterator and passes it to the lambda function and appends the returned answer to the final solution. Please read more about <code>map function</code> online. Pre-req: <code>lambda expressions</code>.</p>

<p>Part II in the comment: <em>And when I am typing 'print(data[0])' all the data along with their attribute is printed. What if I want to print only no. of students of 5th row? What is I want to multiple all no. of students with corresponding temperature and store it in a new column with corresponding index?</em></p>

<p>When you <code>print(data[0])</code> it should give you the first row as is, with the related attributes and should look something like this.</p>

<pre><code>data[0]
Out[63]: 
{'BusArrival': 'on_time',
 'Students': '25',
 'distance': '45',
 'temperature': '75'} 
</code></pre>

<p>I suggest you use pandas dataframe for quick manipulations of the data. </p>

<pre><code>import pandas as pd
df = pd.DataFrame(data)
df
Out[69]: 
  BusArrival Students distance temperature
0     on_time       25       45          75
1      before       12       40          70
2       after       49       50          80
3     on_time       24       44          74
4      before       15       38          75
    # and so on
</code></pre>

<p>Now you want to extract the 5th row only,</p>

<pre><code>df.iloc[5]
Out[73]: 
BusArrival     after
Students          45
distance          49
temperature       85
Name: 5, dtype: object
</code></pre>

<p>Product of students and temperature is now simply,</p>

<pre><code>df['Students'] = df['Students'].astype('int') # making sure they are not strings
df['temperature'] = df['temperature'].astype('int') 
df['studentTempProd'] = df['Students'] * df['temperature']

df
Out[82]: 
   BusArrival  Students distance  temperature  studentTempProd
0     on_time        25       45           75             1875
1      before        12       40           70              840
2       after        49       50           80             3920
3     on_time        24       44           74             1776
4      before        15       38           75             1125
</code></pre>

<p>There is a lot more you can do with pandas. Like only extracting the 'on_time' bus arrivals etc. </p>

python parsing text