Python script for searching variable strings between two constant strings
<pre><code>import re

infile = open('document.txt','r')
outfile= open('output.txt','w')
copy = False
for line in infile:

    if line.strip() == "--operation():":
        bucket = []
        copy = True

    elif line.strip() == "StartOperation":
        for strings in bucket:
            outfile.write( strings + ',')
        for strings in bucket:
            outfile.write('\n')
        copy = False

    elif copy:
        bucket.append(line.strip()
</code></pre>

<p>CSV format is like this:</p>

<pre><code>id,          name,                poid,         error
5896, AutoAuthOSUserSubmit,     900105270,      0x4002
</code></pre>

<p>My log file has several sections starting with <code>==== START ====</code> and ending with <code>==== END   ====</code>.  I want to extract the string between <code>--operation():</code> and <code>StartOperation</code>. For example, <code>AutoAuthOSUserSubmit.</code> I also want to extract the <code>poid</code> value from line <code>poid: 900105270, poidLen: 9</code>. Finally, I want to extract the return value, e.g <code>0x4002</code> if <code>Roll back all updates</code> is found after it. </p>

<p>I am not even able to extract point the original text if <code>Start</code> and <code>End</code> are not on the same line. How do I go about doing that?</p>

<p>This is a sample LOG extract with two paragraphs:</p>

<pre><code>-- 08/24 02:07:56 [mds.ecas(5896) ECAS_CP1] **==== START ====**
open file /ecas/public/onsite-be/config/timer.conf failed
INFO 08/24/16 02:07:56  salt1be-d1-ap(**5896**/0)  main.c(780*****):--operation(): AutoAuthOSUserSubmit. StartOperation*****
INFO 08/24/16 02:07:56  salt1be-d1-ap(5896/0)  main.c(784):--Client Information: Request from host 'malt-d1-wb' process id 12382.
DEBUG 08/24/16 02:07:56  salt1be-d1-ap(5896/0)  TOci.cc(571):FetchServiceObjects: ServiceCert.sql
DEBUG 08/22/16 23:15:53  pepper1be-d1-ap(2680/0)  vsserviceagent.cpp(517):Generate Certificate 2: c1cd00d5c3de082360a08730fef9cd1d
DEBUG 08/22/16 23:15:53  pepper1be-d1-ap(2680/0)  junk.c(1373):GenerateWebPin : poid: **900105270**, poidLen: 9
DEBUG 08/22/16 23:15:53  pepper1be-d1-ap(2680/0)  junk.c(1408):GenerateWebPin : pinStr 
DEBUG 08/24/16 02:07:56  salt1be-d1-ap(5896/0)  uaadapter_vasco_totp.c(275):UAVascoTOTPImpl.close() -- Releasing Adapter Context
DEBUG 08/22/16 23:15:53  pepper1be-d1-ap(2680/0)  vsenterprise.cpp(288):VSEnterprise::Engage returns 0x4002 - Unknown error code **(0x4002)**
ERROR 08/22/16 23:15:53  pepper1be-d1-ap(2680/0)  vsautoauth.cpp(696):OSAAEndUserEnroll: error occurred. **Roll back** all updates!
INFO 08/24/16 02:07:56  salt1be-d1-ap(5896/0)  uaotptokenstoreqmimpl.cpp(199):Close token store
INFO 08/24/16 02:07:56  salt1be-d1-ap(5896/0)  main.c(990):-- EndOperation
-- 08/24 02:07:56 [mds.ecas(5896) ECAS_CP1] **==== END   ====**
    OPERATION = AutoAuthOSUserSubmit, rc = 0x0 (0)
    SYSINFO Elapse = 0.687, Heap = 1334K, Stack = 64K
</code></pre>

<p>This looks like a job for Regular Expressions!  Several in fact.  Thankfully, they are not very complicated to use in this case.</p>

<p>There are 2 main observations that would make me choose regexes over something else:</p>

<ol>
<li>Need to extract one bit of variable text from between two known constant values</li>
<li>Need to follow this same pattern several times for different strings</li>
</ol>

<p>You can try something like this:</p>

<pre><code>import re

def capture(text, pattern_string, flags=0):
    pattern = re.compile(pattern_string, flags)
    match = pattern.search(text)
    if match:
        output = match.group(1)
        print '{}\n'.format(output)
        return output
    return ''

if __name__ == '__main__':
    file = read_my_file()

    log_pattern = "\*\*==== START ====\*\*(.+)\*\*==== END   ====\*\*"
    log_text = capture(file, log_pattern, flags=re.MULTILINE|re.DOTALL)

    op_pattern = "--operation\(\): (.+). StartOperation\*\*\*\*\*"
    op_name = capture(log_text, op_pattern)

    poid_pattern = "poid: \*\*([\d]+)\*\*, poidLen: "
    op_name = capture(log_text, poid_pattern)

    retcode_pattern = "Unknown error code \*\*\((.+)\)\*\*.+\*\*Roll back\*\* all updates!"
    retcode = capture(log_text, retcode_pattern, flags=re.MULTILINE|re.DOTALL)
</code></pre>

<p>This approach essentially divides up the problem into several largely independent steps.  I'm using capturing groups in each regex - the parens like <code>(.+)</code> and <code>([\d]+)</code> - in between long strings of constant characters.     The multiline and dotall flags allow you to easily deal with line breaks in the text and treat them just like any other part of the string.</p>

<p><strong>I'm also making a big assumption here and that is your logs are not huge files, maybe a few hundred megabytes tops.</strong> Note the call to <code>read_my_file()</code> - rather than try to solve this problem a line at a time, I chose to read the entire file and work in memory.  If the files get really big though, or you're building an app that will get a lot of traffic, this may be a bad idea. </p>

<p>Hope this helps!</p>

python