Apache Spark Count State Change
<p>I am new to Apache Spark (Pyspark) and would be glad to get some help resolving this problem. I am currently using Pyspark 1.6 (Had to ditch 2.0 since there is not MQTT support).</p>

<p>So, I have a data frame that has the following information,</p>

<pre><code>+----------+-----------+
|      time|door_status|
+----------+-----------+
|1473678846|          2|
|1473678852|          1|
|1473679029|          3|
|1473679039|          3|
|1473679045|          2|
|1473679055|          1|
</code></pre>

<p>This is basically the time and status of the door. I need to calculate the number of times the door opened &amp; closed. So I need to identify the state change and keep independent counters for each status.</p>

<p>Since I am new to this I am finding it extremely hard to conceive any idea of how to implement this. If somebody can suggest an idea &amp; point me in the right direction it will be great. </p>

<p>Thanks in advance!!</p>

<p>In this case, using accumulator should solve the issue.
Basically, you create three different accumulators for the three statuses.</p>

<pre><code>status_1 = sc.accumulator(0)   
status_2 = sc.accumulator(0)  
status_3 = sc.accumulator(0)   
</code></pre>

<p>then you can do something like the following   </p>

<pre><code>if (status == 1):  
    status_1 += 1
</code></pre>

python apache-spark pyspark