Column transform in Spark MLlib
<p>I have read <a href="https://spark.apache.org/docs/latest/ml-features.html#binarizer" rel="nofollow">Spark MLlib doc</a> for feature transform, but I am still confused about two simple cases:</p>

<p>1.How to deal with single column flexible?
For example, I have one column named "date", it's "YYYY-MM-DD" format, I want to generate one new column called "week" based on "date". If use pandas.Dataframe, it could be done with Series.apply, my quesion is how to do that in Spark MLlib?</p>

<p>2.How to generate new column based on multi-columns?
For example, I want to calculate roi based on spend and income, it's simple in pandas.DataFrame:</p>

<pre><code>df['roi'] = (df['income'] - df['spend'])/df['spend']
</code></pre>

<p>For Spark.MLlib, I find SQLTransformer may be used for the same work, but I am not sure</p>

<p>Could any guy tell me how to deal with that in Spark.MLlib? Thanks a lot</p>

<p>A clean option is to define your own function, and apply to your <code>DataFrame</code> using <code>withColumn()</code>. Note that this has nothing to do with <code>MLlib</code> as that refers to the machine learning module of <code>Spark</code>.</p>

<pre><code>from pyspark.sql.types import FloatType
from pyspark.sql.functions import udf

def roiCalc(income, spend): # Define function
  return((income - spend)/spend)

roiCalculator = udf(roiCalc, FloatType()) # Convert to udf
df.withColumn("roi", roiCalculator(df["income"],df["spend"])) # Apply to df
</code></pre>

python apache-spark spark-dataframe apache-spark-mllib