Feature Importance for Random Forest Regressor in Python
<p>I'm trying to find out which features have the most importance for my predictive model.</p>

<p>Currently I'm using sklearn's inbuilt attribute as such</p>

<pre><code>Model = Model.fit(Train_Features, Labels_Train)
print(Model.feature_importances_)
</code></pre>

<p>It's just that its more of a black box type method, I'm not understanding what method it uses to weight the importance towards the features. Is there a better approach for doing this?</p>

<p>Feature importance is not a black-box when it comes to decision trees. From the documentation for a <a href="http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html#sklearn.tree.DecisionTreeRegressor.feature_importances_" rel="nofollow">DecisionTreeRegressor</a>:</p>

<blockquote>
  <p>The importance of a feature is computed as the (normalized) total
  reduction of the criterion brought by that feature. It is also known
  as the Gini importance.</p>
</blockquote>

<p>For a forest, it just averages across the different trees in your forest. Check out the <a href="https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/ensemble/forest.py#L322" rel="nofollow">source code</a>:</p>

<pre><code>def feature_importances_(self):
    """Return the feature importances (the higher, the more important the
       feature).
    Returns
    -------
    feature_importances_ : array, shape = [n_features]
    """
    if self.estimators_ is None or len(self.estimators_) == 0:
        raise NotFittedError("Estimator not fitted, "
                             "call `fit` before `feature_importances_`.")

    all_importances = Parallel(n_jobs=self.n_jobs,
                               backend="threading")(
        delayed(getattr)(tree, 'feature_importances_')
        for tree in self.estimators_)

    return sum(all_importances) / len(self.estimators_)
</code></pre>

python regression random-forest feature-selection