Apache Spark 2.0.0 PySpark manual dataframe creation head scratcher
<p>When I assemble a one row data frame as follows my method successfully brings back the expected data frame.</p>

<pre><code>def build_job_finish_data_frame(sql_context, job_load_id, is_success):
    job_complete_record_schema = StructType(
        [
            StructField("job_load_id", IntegerType(), False),
            StructField("terminate_datetime", TimestampType(), False),
            StructField("was_success", BooleanType(), False)
        ]
    )
    data = [
        Row(
            job_load_id=job_load_id,
            terminate_datetime=datetime.now(),
            was_success=is_success
        )
    ]

    return sql_context.createDataFrame(data, job_complete_record_schema)
</code></pre>

<p>If I change the "terminate_datetime" to "end_datetime" or "finish_datetime" as shown below it throws an error.</p>

<pre><code>def build_job_finish_data_frame(sql_context, job_load_id, is_success):
    job_complete_record_schema = StructType(
        [
            StructField("job_load_id", IntegerType(), False),
            StructField("end_datetime", TimestampType(), False),
            StructField("was_success", BooleanType(), False)
        ]
    )
    data = [
        Row(
            job_load_id=job_load_id,
            end_datetime=datetime.now(),
            was_success=is_success
        )
    ]

    return sql_context.createDataFrame(data, job_complete_record_schema)
</code></pre>

<p>The error I receive is </p>

<pre><code>TypeError: IntegerType can not accept object datetime.datetime(2016, 10, 5, 11, 19, 31, 915745) in type &lt;class 'datetime.datetime'&gt;
</code></pre>

<p>I can change "terminate_datetime" to "start_datetime" and have experimented with other words.</p>

<p>I can see no reason for field name changes breaking this code as it is doing nothing more than building a manual data frame.</p>

<p>This is worrying as I am using data frames to load up a data warehouse where I have no control of the field names.</p>

<p>I am running PySpark on Python 3.3.2 on Fedora 20.</p>

<p>Why the name changes things? The problem is that <code>Row</code> is a <code>tuple</code> <strong>sorted</strong> by <code>__fields__</code>. So the first case creates </p>

<pre><code>from pyspark.sql import Row
from datetime import datetime

x = Row(job_load_id=1, terminate_datetime=datetime.now(), was_success=True)
x.__fields__
## ['job_load_id', 'terminate_datetime', 'was_success']
</code></pre>

<p>while the second one creates:</p>

<pre><code>y = Row(job_load_id=1, end_datetime=datetime.now(), was_success=True)
y.__fields__
## ['end_datetime', 'job_load_id', 'was_success']
</code></pre>

<p>This no longer matches the schema you defined which expects <code>(IntegerType, TimestampType, Boolean)</code>.</p>

<p>Because <code>Row</code> is useful mostly for schema inference and you provide schema directly you can address that by using standard <code>tuple</code>:</p>

<pre><code>def build_job_finish_data_frame(sql_context, job_load_id, is_success):
    job_complete_record_schema = StructType(
        [
            StructField("job_load_id", IntegerType(), False),
            StructField("end_datetime", TimestampType(), False),
            StructField("was_success", BooleanType(), False)
        ]
    )
    data = [tuple(job_load_id, datetime.now(), is_success)]

    return sql_context.createDataFrame(data, job_complete_record_schema)
</code></pre>

<p>although creating a single element <code>DataFrame</code> looks strange if not pointless.</p>

python apache-spark dataframe