Pyspark - Get all parameters of models created with ParamGridBuilder
<p>Im using pySpark 2.0 for a kaggle competition. I'd like to know the behavior of a model (randomForest) depending on different parameters. ParamGridBuilder() allows to specify different values for a single parameters, and then perform (i guess) a cartesian product of the entire set of parameters. Assuming my dataframe is already defined:</p>

<pre><code>rdc = RandomForestClassifier()
pipeline = Pipeline(stages=STAGES + [rdc])
paramGrid = ParamGridBuilder().addGrid(rdc.maxDepth, [3, 10, 20])
                              .addGrid(rdc.minInfoGain, [0.01, 0.001])
                              .addGrid(rdc.numTrees, [5, 10, 20, 30])
                              .build()
evaluator = MulticlassClassificationEvaluator()
valid = TrainValidationSplit(estimator=pipeline,
                             estimatorParamMaps=paramGrid,
                             evaluator=evaluator,
                             trainRatio=0.50)
model = valid.fit(df)
result = model.bestModel.transform(df)
</code></pre>

<p>Ok so now I'm able to retrieves simple information with a handmade function:</p>

<pre><code>def evaluate(result):
    predictionAndLabels = result.select("prediction", "label")
    metrics = ["f1","weightedPrecision","weightedRecall","accuracy"]
    for m in metrics:
        evaluator = MulticlassClassificationEvaluator(metricName=m)
        print(str(m) + ": " + str(evaluator.evaluate(predictionAndLabels)))
</code></pre>

<p>Now I want several things:</p>

<p>*What are the parameters of the best model? This post partially answers the question: <a href="http://stackoverflow.com/questions/36697304/how-to-extract-model-hyper-parameters-from-spark-ml-in-pyspark">How to extract model hyper-parameters from spark.ml in PySpark?</a></p>

<p>*What are the parameters of all models ?
  *What are the results (aka recall, accuracy, etc...) of each model ? I only found <code>print(model.validationMetrics)</code> that displays (it seems) a list containing the accuracy of each model, but I can't get to know which model to refers</p>

<p>If I can retrieve all those informations, I should be able to display graphs, bar charts, and work as I do with panda and sklearn.</p>

<p>Long story short you simply cannot get parameters for all models because, <a href="http://stackoverflow.com/a/38874828/1560062">similarly to <code>CrossValidator</code></a>, <code>TrainValidationSplitModel</code> retains only the best model. These classes are designed for semi-automated model selection not exploration or experiments.</p>

<blockquote>
  <p>What are the parameters of all models?</p>
</blockquote>

<p>While you cannot retrieve actual models <code>validationMetrics</code> correspond to input <code>Params</code> so you should be able to simply <code>zip</code> both:</p>

<pre><code>from typing import Dict, Tuple, List, Any
from pyspark.ml.param import Param
from pyspark.ml.tuning import TrainValidationSplitModel

EvalParam = List[Tuple[float, Dict[Param, Any]]]

def get_metrics_and_params(model: TrainValidationSplitModel) -&gt; EvalParam:
    return list(zip(model.validationMetrics, model.getEstimatorParamMaps()))
</code></pre>

<p>to get some about relationship between metrics and parameters.</p>

<p>If you need more information you should use <a href="http://stackoverflow.com/q/35253990/1560062">Pipeline <code>Params</code></a>. It will preserve all model which can be used for further processing:</p>

<pre><code>models = pipeline.fit(df, params=paramGrid)
</code></pre>

<p>It will generate a list of the <code>PipelineModels</code> corresponding to the <code>params</code> argument:</p>

<pre><code>zip(models, params)
</code></pre>

python machine-learning pyspark hyperparameters