can't define a udf inside pyspark project
<p>I have a python project that uses pyspark and i am trying to define a udf function inside the spark project (not in my python project) specifically in spark\python\pyspark\ml\tuning.py but i get pickling problems. it can't load the udf. 
The code:</p>

<pre><code>from pyspark.sql.functions import udf, log
test_udf = udf(lambda x : -x[1], returnType=FloatType())
d = data.withColumn("new_col", test_udf(data["x"]))
d.show()
</code></pre>

<p>when i try d.show() i am getting exception of unknown attribute test_udf</p>

<p>In my python project i defined many udf and it worked fine. </p>

<p>Found it there was 2 problems</p>

<p>1) for some reason it didn't like the returnType=FloatType() i needed to convert it to just FloatType() though this was the signature</p>

<p>2) The data in column x was a vector and for some reason i had to cast it to float</p>

<p>The working code:</p>

<pre><code>from pyspark.sql.functions import udf, log
test_udf = udf(lambda x : -float(x[1]), FloatType())
d = data.withColumn("new_col", test_udf(data["x"]))
d.show()
</code></pre>

python pyspark udf apache-spark-ml