can't define a udf inside pyspark project
<p>I have a python project that uses pyspark and i am trying to define a udf function inside the spark project (not in my python project) specifically in spark\python\pyspark\ml\tuning.py but i get pickling problems. it can't load the udf. 
The code:</p>

<pre><code>from pyspark.sql.functions import udf, log
test_udf = udf(lambda x : -x[1], returnType=FloatType())
d = data.withColumn("new_col", test_udf(data["x"]))
d.show()
</code></pre>

<p>when i try d.show() i am getting exception of unknown attribute test_udf</p>

<p>In my python project i defined many udf and it worked fine. </p>

<p>add the following to your code. It isn't recognizing the datatype.</p>

<pre><code>from pyspark.sql.types import *
</code></pre>

<p>Let me know if this helps. Thanks.</p>

python pyspark udf apache-spark-ml