pyspark type error on reading a pandas dataframe
<p>I read some CSV file into pandas, nicely preprocessed it and set dtypes to desired values of float, int, category. However, when trying to import it into spark I get the following error:</p>

<pre><code>Can not merge type &lt;class 'pyspark.sql.types.DoubleType'&gt; and &lt;class 'pyspark.sql.types.StringType'&gt;
</code></pre>

<p>After trying to trace it for a while I some source for my troubles -> see the CSV file:</p>

<pre><code>"myColumns"
""
"A"
</code></pre>

<p>Red into pandas like: <code>small = pd.read_csv(os.path.expanduser('myCsv.csv'))</code></p>

<p>And failing to import it to spark with:</p>

<pre><code>sparkDF = spark.createDataFrame(small)
</code></pre>

<p>Currently I use Spark 2.0.0</p>

<h3>Possibly multiple columns are affected. How can I deal with this problem?</h3>

<p><a href="http://i.stack.imgur.com/55CXw.jpg" rel="nofollow"><img src="http://i.stack.imgur.com/55CXw.jpg" alt="enter image description here"></a></p>

<p>You'll need to define the spark <code>DataFrame</code> schema explicitly and pass it to the <code>createDataFrame</code> function :</p>

 

<pre><code>from pyspark.sql.types import *
import pandas as pd

small = pdf.read_csv("data.csv")
small.head()
#  myColumns
# 0       NaN
# 1         A
sch = StructType([StructField("myColumns", StringType(), True)])

df = spark.createDataFrame(small, sch)
df.show()
# +---------+
# |myColumns|
# +---------+
# |      NaN|
# |        A|
# +---------+

df.printSchema()
# root
# |-- myColumns: string (nullable = true)
</code></pre>

python csv pandas apache-spark pyspark