How to merge two dataframes in Spark Hadoop without common key?
<p>I am trying to join two dataframes.</p>

<p>data: DataFrame[_1: bigint, _2: vector]</p>

<p>cluster: DataFrame[cluster: bigint]</p>

<pre><code>result = data.join(broadcast(cluster))
</code></pre>

<p>The strange thing is, that all the executors are failing on the joining step.</p>

<p>I have no idea what I could do.</p>

<p>The data file is 2.8 gb on HDFS and the cluster data only 5 mb.
The files are read using Parquet.</p>

<p>What does work is this:</p>

<pre><code>data = sqlContext.read.parquet(data_path)
data = data.withColumn("id", monotonicallyIncreasingId())

cluster = sqlContext.read.parquet(cluster_path)  
cluster = cluster.withColumn("id", monotonicallyIncreasingId())

result = data.join(cluster, on="id")
</code></pre>

<p>Adding the cluster DataFrame directly to the data DataFrame with:</p>

<pre><code>data.withColumn("cluster", cluster.cluster)
</code></pre>

<p>Does not work.</p>

<pre><code>data.join(cluster)
</code></pre>

<p>Also does not work, executors are failing whilst having enough memory.</p>

<p>No idea why it wasn't working...</p>

python hadoop apache-spark spark-dataframe parquet