Skipping the unmatched Index lines pyspark
<p>I want to skip the lines which are out of list index, i.e keep only the lines which are matching to given index. </p>

<p>Following is my data,</p>

<pre><code>12,34,5,6,7,8,.......

23,45,657,78,34,.......

0,2,34

15,78,65,78,9,...
</code></pre>

<p><strong>I want to extract the fields x[0],x[1],x[2],x[3]</strong> also in my dataset some lines have less fields and it will throw list index out of range, SO I want to skipp the lines having index &lt;3. How could I achieve this in spark python</p>

<p>This i tried,</p>

<pre><code>def takeOnly3fields(data):
    for row in data:
        if not len(row) &lt;=3:
            return ",".join(row) 

ff = file1.map(takeOnly3fields)
print(ff.collect()) will return NULL,NULL,NULL
</code></pre>

<p>If the data is single dimensional, following is the code you need:</p>

<pre><code>def takeOnly3fields(data):
    return ",".join(data) if len(data)&gt;3 else None
</code></pre>

<p>map will iterate the entire data for you and you just need a transformation function on the data set.</p>

python list apache-spark pyspark