Reshaping pyspark dataframe to 4-dimensional numpy array for Keras/Theano
<p>I'm trying to get a spark dataframe, traindf into a 4-d numpy array. I've tried this:</p>

<pre><code>traindf = sqlContext.createDataFrame([
    (1, 1, 2, 3),
    (1, 2, 2, 3),
    (1, 3, 2, 3),
    (1, 4, 2, 3),
    (2, 4, 5, 6),
    (2, 4, 5, 6),
    (3, 7, 8, 9),
    (2, 4, 5, 6),
    (3, 7, 8, 9),
    (3, 7, 8, 9)
], ("id", "image", "s", "t"))

values = (traindf.rdd.map(lambda l: [map(lambda r: float(r), l)]).collect())
x = np.array(values)
x = np.array_split(x, x.shape[0]/2)
x = np.asarray(x)
x.shape
</code></pre>

<p>This yields (5, 2, 1, 4), but it appears keras needs (5, 1, 2, 4). I've tried a couple ways, but am not seeing a good way to get the correct format.</p>

<p>Any suggestions?</p>

<p>Just figured it out, tack this onto the end</p>

<pre><code>x = np.reshape(x, (5, 1, 2, 4))
</code></pre>

python numpy apache-spark pyspark keras