Regex for log lines that may be slightly different from each other- in spark
<p>I could write a regex for matching one log line. But my file contains some log lines which differ from the other log lines in that there are just a few extra fields or in a few, the key value pairs are mixed up. </p>

<p>Eg. one log line :
 <code>case_id=1 event_id=35654423 date=30-12-2010 time=11.02 activity=registerrequest name=Pete costs=50 rerr="-"</code></p>

<p>another one : <code>case_id=1 event_id=35654424 date=31-12-2010 time=11.07 costs=400 rerr="-" activity=examinethoroughly name=Sue rloc="-"</code></p>

<p>My log pattern = <code>"^([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)"</code>
This is my code for grouping in spark: </p>

<pre><code>match = re.search(LOG_PATTERN,line)
Case_ID = match.group(1)
Event_ID = match.group(2)
Date = match.group(3)
Time = match.group(4)
Activity = match.group(5)
Resource = match.group(6)
Costs = match.group(7)
Rerr = match.group(8)
</code></pre>

<p>This code would match only the 1st log line.
How do i write a regex so that i can group them in spark without ignoring any of the log lines?</p>

<p>I suggest you use a simple regex to grab keys and values:</p>

<pre><code>rx = "(\\S+)=(\\S+)"
</code></pre>

<p>or</p>

<pre><code>rx = "(\\w+)=(\\S+)"
</code></pre>

<p>See a <a href="https://regex101.com/r/jA0zB2/1" rel="nofollow">regex demo</a>.</p>

<p>Then, create a dictionary with <code>re.findall</code>:</p>

<pre><code>d = dict([(x,y) for x,y in re.findall(rx,line)])
</code></pre>

<p>Then, you have access to each key-value.</p>

python regex apache-spark pyspark