Parsing Messy Data
<p>I'm relatively new to python and was wondering if I could get some assistance in parsing data so that it is easier to analyze.</p>

<hr>

<p>My data is in the following form (each is an entire line):</p>

<pre><code>20160930-07:06:54.481737|I|MTP_4|CL:BF K7-M7-N7 Restrict for maxAggressive: -4.237195
20160930-07:06:54.481738|I|MTP_4|CL:BF K7-M7-N7 BidPrice: -5.0 mktBestBid: -5.0 bidTheo: -4.096774 bidSeedEdge: 0.195028 bidUnseedEdge: CL:BF K7-M7-N7 = 0.14042 Min Bid: -6.0 Max Ticks Offset: 1 Max Aggressive Ticks: 1
</code></pre>

<p>This is my code so far</p>

<pre><code># Output file
output_filename = os.path.normpath("Mypath/testList.log")  
# Overwrites the file
with open(output_filename, "w") as out_file:  
    out_file.write("")

# Open output file
with open(output_filename, "a") as out_file:  
    # Open input file in 'read' mode
    with open("mypath/tradedata.log", "r") as in_file:
        # Loop over each log line, Grabs lines with necessary data
        for line in islice(in_file, 177004, 8349710):
                  out_file.write(line)
</code></pre>

<p>Would it be easiest to just go through and do it by keywords like; bidUnseedEdge, mktBesdBid, etc. ?</p>

<pre><code>infilename = "path/data.log"
outfilename = "path/OutputData.csv"

with open(infilename, 'r') as infile,\
     open(outfilename, "w") as outfile:
    lineCounter = 0
    for line in infile:
        lineCounter += 1
        if lineCounter % 1000000 == 0:
            print lineCounter
        data = line.split("|")
        if len(data) &lt; 4:
            continue
        bidsplit = data[3].split("bidTheo:")
        namebid = data[3].split("BidPrice:")
        if len(bidsplit) == 2:
            bid = float(bidsplit[1].strip().split()[0])
            bidname = namebid[0].strip().split(",")[0]
            #print "bidTheo," + data[0] + "," + str(bid)
            outfile.write("bidTheo," + data[0] + "," + bidname + "," + str(bid) + "\n")
        offersplit = data[3].split("offerTheo:")
        nameoffer = data[3].split("AskPrice:")
        if len(offersplit) == 2:
            offer = float(offersplit[1].strip().split()[0])
            offername = nameoffer[0].strip().split(",")[0]
            #print "offerTheo," + data[0] + "," + str(offer)
            outfile.write("offerTheo," + data[0] + "," + offername + "," +  str(offer) + "\n")

print "Done"
</code></pre>

python