Do not write None or empty lines in Spark (Python)
<p>I am new in Spark, but I have had some experience in Hadoop. I've trying to adapt a python code I use in Hadoop streaming that filters out some tweets in JSON format.</p>

<p>Normally, my function has a condition that prints to stdout the tweet if the condition is true and prints nothing otherwise.</p>

<pre><code>def filter(tweet):
   if criteria(tweet) is True:
      print json.dumps(tweet)
</code></pre>

<p>This way, the final output file will contain only the tweets I want.</p>

<p>However, when trying to use Spark, I had to change the <code>print</code> statement with a <code>return</code> so I return the tweet if the condition is True, and <code>None</code> otherwise. </p>

<pre><code>def filter(tweet):
       if criteria(tweet) is True:
          return json.dumps(tweet)
</code></pre>

<p>The problem appears when trying to save the results to disk. Using <code>saveAsTextFile</code> method of Pyspark, it saves not only the tweets I want but also the <code>None</code> I return when condition is False.</p>

<p>How can I avoid writing <code>None</code> to the file so I only save the desired tweets?</p>

<p>Many thanks in advance.</p>

<p>Jorge</p>

<p>If you are using your function in a map, it will not reduce the number of elements you have. To filter elements, you must to use the <code>filter</code> method to test if an element is <code>None</code> after you <code>map</code>.</p>

python hadoop apache-spark pyspark