Creating a Pandas DataFrame with HDFS file in .csv format
<p>Im trying to create a Spark workflow from fetching .csv data from a hadoop cluster and putting it into Pandas DataFrame. I'm able to pull the Data from HDFS and put it in an RDD, but unable to process it into the Pandas Dataframe. 
The following is my code:</p>

<pre><code>import pandas as pd
import numpy as nm

A=sc.textFile("hdfs://localhost:9000/sales_ord_univ.csv") # this creates the RDD
B=pd.DataFrame(A) # this gives me the following error:pandas.core.common.PandasError: DataFrame constructor not properly called!
</code></pre>

<p>I'm pretty sure this error is as such due to the RDD being a big single <strong>list</strong>, 
Hence I tried splitting the data by ';'( i.e each new row is a different string)
But that didn't seem to help either. </p>

<p>My overall goal is to use Pandas to change CSV into JSON and output into MongoDB. I have done this project using DictReader, PysparkSQL, but wanted to check if it is possible using Pandas.</p>

<p>Any help would be appreciated
Thanks!</p>

<p>I would recommend to load the csv into a Spark DataFrame and convert it to a Pandas DataFrame.</p>

<pre><code>csvDf = sqlContext.read.format("csv").option("header", "true").option("inferschema", "true").option("mode", "DROPMALFORMED").load("hdfs://localhost:9000/sales_ord_univ.csv") 
B = csvDf.toPandas()
</code></pre>

<p>If you are still using a Spark version &lt; 2.0, you have to use <code>read.format("com.databricks.spark.csv")</code> and include the com.databricks.spark.csv package (e.g. with the <code>--packages</code> parameter when using the pyspark shell).</p>

python csv hadoop pandas pyspark