What is causing 'unicode' object has no attribute 'toordinal' in pyspark?
<p>I got this error but I don't what causes it. My python code ran in pyspark. The stacktrace is long and i just show some of them. All the stacktrace doesn't show my code in it so I don't know where to look for. What is possible the cause for this error? </p>

<pre><code>/usr/hdp/2.4.2.0-258/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    306                 raise Py4JJavaError(
    307                     "An error occurred while calling {0}{1}{2}.\n".
--&gt; 308                     format(target_id, ".", name), value)
    309             else:
    310                 raise Py4JError(

Py4JJavaError: An error occurred while calling o107.parquet.

...
File "/usr/hdp/2.4.2.0-258/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 435, in toInternal
    return self.dataType.toInternal(obj)
  File "/usr/hdp/2.4.2.0-258/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 172, in toInternal
    return d.toordinal() - self.EPOCH_ORDINAL
AttributeError: 'unicode' object has no attribute 'toordinal'
</code></pre>

<p>Thanks,</p>

<p>The specific exception is caused by trying to store a <code>unicode</code> value in a <em>date</em> datatype that is part of a struct. The conversion of the Python type to Spark internal representation expected to be able to call <a href="https://docs.python.org/3/library/datetime.html#datetime.date.toordinal" rel="nofollow"><code>date.toordinal()</code></a> method.</p>

<p>Presumably you have a dataframe schema somewhere that consists of a struct type with a date field, and something tried to stuff a string into that.</p>

<p>You can trace this based on the traceback you <em>do</em> have. The <a href="https://github.com/apache/spark" rel="nofollow">Apache Spark source code</a> is hosted on GitHub, and your traceback points to the <a href="https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py" rel="nofollow"><code>pyspark/sql/types.py</code> file</a>. The lines point to the <a href="https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L435-L436" rel="nofollow"><code>StructField.toInternal()</code> method</a>, which delegates to the <code>self.dataType.toInternal()</code> method:</p>

<pre><code>class StructField(DataType):
    # ...
    def toInternal(self, obj):
        return self.dataType.toInternal(obj)
</code></pre>

<p>which in your traceback ends up at the <a href="https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L170-L172" rel="nofollow"><code>DateType.toInternal()</code> method</a>:</p>

<pre><code>class DateType(AtomicType):
    # ...
    def toInternal(self, d):
        if d is not None:
            return d.toordinal() - self.EPOCH_ORDINAL
</code></pre>

<p>So we know this is about a date field in a struct. The <code>DateType.fromInternal()</code> shows you what Python type is produced in the opposite direction:</p>

<pre><code>def fromInternal(self, v):
    if v is not None:
        return datetime.date.fromordinal(v + self.EPOCH_ORDINAL)
</code></pre>

<p>It is safe to assume that <code>toInternal()</code> expects the same type when converting in the other direction.</p>

python pyspark