pyspark redueByKey modify single results
<p>I have a dataset that looks like this in pyspark:</p>

<pre><code>samp = sc.parallelize([(1,'TAGA'), (1, 'TGGA'), (1, 'ATGA'), (1, 'GTGT'), (2, 'GTAT'), (2, 'ATGT'), (3, 'TAAT'), (4, 'TAGC')])
</code></pre>

<p>I have a function that I'm using to combine the strings:</p>

<pre><code>   def combine_strings(x,y):
        if (isinstance(x,list) and isinstance(y, list)):
            z = x + y
            return z
        if (isinstance(x, list) and isinstance(y, str)):
            x.append(y)
            return x
        if (isinstance(x, str) and isinstance(y, list)):
            y.append(x)
            return y
        return [x,y]
</code></pre>

<p>The result I get is:</p>

<pre><code>samp.reduceByKey(lambda x,y : combine_strings(x,y)).collect()
[(1, ['TAGA', 'TGGA', 'ATGA', 'GTGT']), (2, ['GTAT', 'ATGT']), (3, 'TAAT'), (4, 'TAGC')]
</code></pre>

<p>What I want is:</p>

<p>[(1, ['TAGA', 'TGGA', 'ATGA', 'GTGT']), (2, ['GTAT', 'ATGT']), (3, ['TAAT']), (4, ['TAGC'])]</p>

<p>Where everything is an array.  I can't tell if pyspark is calling combine_strings on a result where there's 1 entry or if I can tell reduceByKey to do something with singleton results?  How do I modify the reduceByKey() or the combine_strings function to produce what I'd like? </p>

<p>You could first map the values into lists and then only combine those lists:</p>

<pre><code>samp.mapValues(lambda x : [x]).reduceByKey(lambda x,y : x + y).collect()
</code></pre>

<p>The problem here is that those singletons are not affected by <code>reduceByKey</code>. Here is another example:</p>

<pre><code>samp = sc.parallelize([(1,1),(2,2),(2,2),(3,3)])
&gt;&gt;&gt; samp.reduceByKey(lambda x, y : x + y + 1).collect()
[(3, 3), (1, 1), (2, 5)]
</code></pre>

python mapreduce pyspark