How to split an RDD into two RDDs and save the result as RDDs with PySpark?
<p>I'm looking for a way to split an RDD into two or more RDDs, and save the results obtained as two separated RDDs. Given for exemple :</p>

<pre><code>rdd_test = sc.parallelize(range(50), 1)
</code></pre>

<p>My code :</p>

<pre><code>def split_population_into_parts(rdd_test):

    N = 2
    repartionned_rdd = rdd_test.repartition(N).distinct()
    rdds_for_testab_populations = repartionned_rdd.glom()

    return rdds_for_testab_populations

rdds_for_testab_populations = split_population_into_parts(rdd_test)
</code></pre>

<p>Which gives :</p>

<p><strong>[[0,
  2,
  4,
  6,
  8,
  10,
  12,
  14,
  16,
  18,
  20,
  22,
  24,
  26,
  28,
  30,
  32,
  34,
  36,
  38,
  40,
  42,
  44,
  46,
  48],
 [1,
  3,
  5,
  7,
  9,
  11,
  13,
  15,
  17,
  19,
  21,
  23,
  25,
  27,
  29,
  31,
  33,
  35,
  37,
  39,
  41,
  43,
  45,
  47,
  49]]</strong></p>

<p>Now I want to associate avery list here to a new RDD. RDD1 and RDD2 for exemple. What to do ? Thx !</p>

<p>I got the solutions.</p>

<pre><code>def get_testab_populations_tables(rdds_for_testab_populations):
i = 0
while i &lt; len(rdds_for_testab_populations.collect()):
    for testab_table in rdds_for_testab_populations.toLocalIterator():
        namespace = globals()
        namespace['tAB_%d' % i] = sc.parallelize(testab_table)
        i += 1

return;
</code></pre>

<p>Then you can do :</p>

<pre><code>print tAB_0.collect()
print tAB_1.collect()
etc.
</code></pre>

python list pyspark rdd pyspark-sql