How to get predictive attributes of each target in `Random Forest`?
<p>I've been messing around with <code>Random Forest</code> models lately and they are really useful w/ the <code>feature_importance_</code> attribute!  </p>

<p><strong>It would be useful to know which variables are more predictive of particular targets.</strong> </p>

<p>For example, what if the <code>1st and 2nd attributes</code> were more predictive of distringuishing <code>target 0</code> but the <code>3rd and 4th attributes</code> were more predictive of <code>target 1</code>?  </p>

<p><strong>Is there a way to get the <code>feature_importance_</code> array for each target separately?</strong> With <code>sklearn</code>, <code>scipy</code>, <code>pandas</code>, or <code>numpy</code> preferably. </p>

<pre><code># Iris dataset
DF_iris = pd.DataFrame(load_iris().data,
                       index = ["iris_%d" % i for i in range(load_iris().data.shape[0])],
                       columns = load_iris().feature_names)

Se_iris = pd.Series(load_iris().target,
                       index = ["iris_%d" % i for i in range(load_iris().data.shape[0])],
                       name = "Species")
# Import modules
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import train_test_split

# Split Data
X_tr, X_te, y_tr, y_te = train_test_split(DF_iris, Se_iris, test_size=0.3, random_state=0)

# Create model
Mod_rf = RandomForestClassifier(random_state=0)
Mod_rf.fit(X_tr,y_tr)

# Variable Importance
Mod_rf.feature_importances_
# array([ 0.14334485,  0.0264803 ,  0.40058315,  0.42959169])

# Target groups
Se_iris.unique()
# array([0, 1, 2])
</code></pre>

<p>This is not really how RF works. Since there is no simple "feature voting" (which takes place in linear models) it is really hard to answer the question what "feature X is more predictive for target Y" even means. What feature_importance of RF captures is "how probable is, in general, to use this feature in the decision process". The problem with addressing your question is that if you ask "how probable is, in general, to use this feature in decision process <strong>leading to label Y</strong>" you would have to pretty much run the same procedure but remove all subtrees which do not contain label Y in a leaf - this way you remove parts of the decision process which do not address the problem "is it Y or not Y" but rather try to answer which "not Y" it is. However, in practice, due to very stochastic nature of RF, cutting its depth etc. this might barely reduce anything. The bad news is also, that I never seen it implemented in any standard RF library, you could do this on your own, just the way I said:</p>

<pre><code>for i = 1 to K (K is number of distinct labels)
  tmp_RF = deepcopy(RF)
  for tree in tmp_RF:
    tree = remove_all_subtrees_that_do_not_contain_given_label(tree, i)
    for x in X (X is your dataset)
      features_importance[i] += how_many_times_each_feature_is_used(tree, x) / |X| 
  features_importance[i] /= |tmp_RF|
return features_importance 
</code></pre>

<p>in particular you could use existing feature_importance codes, simply by doing</p>

<pre><code>for i = 1 to K (K is number of distinct labels)
  tmp_RF = deepcopy(RF)
  for tree in tmp_RF:
    tree = remove_all_subtrees_that_do_not_contain_given_label(tree, i)
  features_importance[i] = run_regular_feature_importance(tmp_RF)
return features_importance 
</code></pre>

python machine-learning classification feature-extraction