Unable to use filter on an iterable RDD in pyspark
<p>I'm trying to apply a function that filters out certain values in a dataset based on ranges of data in another dataset. I've performed a few groupBys and joins, so the format of the parameter I'm passing into the function has two Iterables, and goes as follows:</p>

<pre><code>g1 = g0.map(lambda x: timefilter(x[0]))
</code></pre>

<p>where x[0] is <code>&lt;pyspark.resultiterable.ResultIterable object at 0x23b6610&gt;, &lt;pyspark.resultiterable.ResultIterable object at 0x23b6310&gt;)</code></p>

<p>When I enter the function <code>timefilter</code> I need to now be able to filter out the values in x[1] based on values in x[0]. But when I try the following (on both <code>twoList</code> and <code>twoRDD</code>, although I just show twoList here):</p>

<pre><code>def timefilter(RDDList):
    oneList = list(RDDList[0])
    twoList = list(RDDList[1])
    twoRDD = RDDList[1]
    test = twoList.filter(lambda x: x[4]=='helloworld')
    return test
</code></pre>

<p>It gives me the following errors: <code>AttributeError: 'ResultIterable' object has no attribute 'filter'</code>
and then a bunch of errors after. </p>

<p>It seems as though I can't use filter on any format of the iterables, but feel like I'm missing something very simple. Is there a transform I'm missing in the function?</p>

<p>Turns out that filtering on an iterable RDD isn't possible, so I just used the python in-built filter function. Something along the lines of the following: <code>filter(lambda x: x[1] in oneList, twoList)</code>.</p>

python list filter pyspark iterable