Adding data to an existing apache spark dataframe from a csv file
<p>i have a spark dataframe that has two columns: name, age as follows:</p>

<pre><code>[Row(name=u'Alice', age=2), Row(name=u'Bob', age=5)]
</code></pre>

<p>The dataframe was created using </p>

<pre><code>sqlContext.createDataFrame()
</code></pre>

<p>What i need to do next is to add a third column 'UserId' from an external 'csv' file. The external file has several columns but i need to include the first column only, which is the 'UserId':</p>

<p><a href="http://i.stack.imgur.com/udp5W.png" rel="nofollow"><img src="http://i.stack.imgur.com/udp5W.png" alt="enter image description here"></a></p>

<p>The number of records in both data sources is the same. I am using a standalone pyspark version on windows os. The final result should be a new dataframe with three columns: UserId, Name, Age. </p>

<p>Any suggestion? </p>

<p>You can do this by join of two data frame but for that you need to have in booth tables either ids or other keys. I would suggest to just copy it to an excel file if the location of lines is the same other wise you do not have enough information to merge them.</p>

python apache-spark pyspark spark-dataframe