scikit-learn - train_test_split and ShuffleSplit yielding very different results
<p>I'm trying to do run a simple RandomForestClassifier() with a large-ish dataset. I typically first do the cross-validation using <code>train_test_split</code>, and then start using <code>cross_val_score</code>.</p>

<p>In this case though, I get very different results from these two approaches, and I can't figure out why. My understanding these is that these two snippets should do <em>exactly the same thing</em>:</p>

<pre><code>cfc = RandomForestClassifier(n_estimators=50)

scores = cross_val_score(cfc, X, y, 
                         cv = ShuffleSplit(len(X), 1, 0.25), 
                         scoring = 'roc_auc')
print(scores)
&gt;&gt;&gt; [ 0.88482262]
</code></pre>

<p>and this:</p>

<pre><code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)

cfc = RandomForestClassifier(n_estimators=50)
cfc.fit(X_train, y_train)

roc_auc_score(y_test, cfc.predict(X_test))
&gt;&gt;&gt; 0.57733474562203269
</code></pre>

<p>And yet the scores are widely different. (The scores are very representative, I observed the same behavior across many many runs).</p>

<p>Any ideas why this can be? I am tempted to trust the <code>cross_val_score</code> result, but I want to be sure that I am not messing up somewhere..</p>

<p>** Update **</p>

<p>I noticed that when I reverse the order of the arguments to <code>roc_auc_score</code>, I get a similar result:</p>

<pre><code>roc_auc_score(cfc.predict(X_test), y_test)
</code></pre>

<p>But the documentation explicitly states that the first element should be the real values, and the second one the target.</p>

<p>I'm not sure what's the issue but here are two things you could try:</p>

<ol>
<li><p>ROC AUC needs prediction probabilities for proper evaluation, not hard scores (i.e. 0 or 1). Therefore change the <code>cross_val_score</code> to work with probabilities. You can check the first answer on <a href="http://stackoverflow.com/questions/28787500/predict-proba-for-a-cross-validated-model">this link</a> for more details.</p>

<p>Compare this with <code>roc_auc_score(y_test, cfc.predict_proba(X_test)[:,1])</code></p></li>
<li><p>As xysmas said, try setting a random_state to both <code>cross_val_score</code> and <code>roc_auc_score</code></p></li>
</ol>

python pandas scikit-learn