Spark job aborted
<p>I would like to add a prediction column to my dataframe given the logistic regression model. The function is below:</p>

<pre><code>def add_probability(df, model):

coefficients_broadcast = sc.broadcast(model.coefficients)
intercept = model.intercept

    def get_p(features):

    # Compute the raw value
        raw_prediction = coefficients_broadcast.value.dot(features)

    # Bound the raw value between 20 and -20
        if raw_prediction&gt;20: raw_prediction=20
        if raw_prediction&lt;-20: raw_prediction=-20
        print raw_prediction

    # Return the probability
        return (1+exp(-raw_prediction))^(-1)

    get_p_udf = udf(get_p, DoubleType())
    return df.withColumn('p', get_p_udf('features'))
</code></pre>

<p>In the nested function get_p, it calculates the probability for an observation given a list of features.
So, after I define the function, I apply it to my training dataframe. </p>

<pre><code>add_probability_model_basic = lambda df: add_probability(df, lr_model_basic)
training_predictions = add_probability_model_basic(ohe_train_df).cache()

print training_predictions.first()
</code></pre>

<p>However, when I tried to see the first row, the following error comes:</p>

<p><code>org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 98.0 failed 1 times, most recent failure: Lost task 0.0 in stage 98.0 (TID 270, localhost): org.apache.spark.api.python.PythonException: Traceback (most recent call last)</code></p>

<p>And if I comment out the last print command, it seems that my code generates the <code>training_predictions</code> dataframe successfully.I am very frustrated why it cannot print out the first row?</p>

<p>You will absolutely love the reason why it fails:</p>

<pre><code>return (1+exp(-raw_prediction))^(-1) 
</code></pre>

<p>should be</p>

<pre><code>return (1+exp(-raw_prediction))**(-1)
</code></pre>

<p>Glad I could help</p>

python apache-spark