pyspark: StructField(..., ..., False) always returns `nullable=true` instead of `nullable=false`
<p>I'm new to pyspark and am facing a strange problem. I'm trying to set some column to non-nullable while loading a CSV dataset. I can reproduce my case with a very small dataset (<code>test.csv</code>):</p>

<pre><code>col1,col2,col3
11,12,13
21,22,23
31,32,33
41,42,43
51,,53
</code></pre>

<p>There is a null value at row 5, column 2 and I don't want to get that row inside my DF. I set all fields as non-nullable (<code>nullable=false</code>) but I get a schema with all the three columns having <code>nullable=true</code>. This happens even if I set all the three columns as non-nullable! I'm running the latest available version of Spark, 2.0.1.</p>

<p>Here's the code:</p>

<pre><code>from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

struct = StructType([   StructField("col1", StringType(), False), \
                        StructField("col2", StringType(), False), \
                        StructField("col3", StringType(), False) \
                    ])

df = spark.read.load("test.csv", schema=struct, format="csv", header="true")
</code></pre>

<p><code>df.printSchema()</code> returns:</p>

<pre><code>root
 |-- col1: string (nullable = true)
 |-- col2: string (nullable = true)
 |-- col3: string (nullable = true)
</code></pre>

<p>and <code>df.show()</code> returns:</p>

<pre><code>+----+----+----+
|col1|col2|col3|
+----+----+----+
|  11|  12|  13|
|  21|  22|  23|
|  31|  32|  33|
|  41|  42|  43|
|  51|null|  53|
+----+----+----+
</code></pre>

<p>while I expect this:</p>

<pre><code>root
 |-- col1: string (nullable = false)
 |-- col2: string (nullable = false)
 |-- col3: string (nullable = false)

+----+----+----+
|col1|col2|col3|
+----+----+----+
|  11|  12|  13|
|  21|  22|  23|
|  31|  32|  33|
|  41|  42|  43|
+----+----+----+
</code></pre>

<p>While Spark behavior (switch from <code>False</code> to <code>True</code> here is confusing there is nothing fundamentally wrong going on here. <code>nullable</code> argument is not a constraint but a reflection of the source and type semantics which enables certain types of optimization</p>

<p>You state that you want to avoid null values in your data. For this you should use <code>na.drop</code> method.</p>

<pre><code>df.na.drop()
</code></pre>

<p>For other ways of handling nulls please take a look at the <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameNaFunctions" rel="nofollow"><code>DataFrameNaFunctions</code></a> (exposed using <code>DataFrame.na</code> property) documentation. </p>

<p>CSV format doesn't provide any tools which allow you to specify data constraints so by definition reader cannot assume that input is not null and your data indeed contains nulls.</p>

python apache-spark pyspark spark-dataframe