using confusion matrix as scoring metric in cross validation in scikit learn
<p>I am creating a pipeline in scikit learn, </p>

<pre><code>pipeline = Pipeline([
    ('bow', CountVectorizer()),  
    ('classifier', BernoulliNB()), 
])
</code></pre>

<p>and  computing the accuracy using cross validation</p>

<pre><code>scores = cross_val_score(pipeline,  # steps to convert raw messages      into models
                     train_set,  # training data
                     label_train,  # training labels
                     cv=5,  # split data randomly into 10 parts: 9 for training, 1 for scoring
                     scoring='accuracy',  # which scoring metric?
                     n_jobs=-1,  # -1 = use all cores = faster
                     )
</code></pre>

<p>How can I report confusion matrix instead of 'accuracy'?</p>

<p>Short answer is "you cannot".</p>

<p>You need to understand difference between <code>cross_val_score</code> and cross validation as model selection method. <code>cross_val_score</code> as name suggests, works only on <strong>scores</strong>. Confusion matrix is not a score, it is a kind of summary of what happened during evaluation. A major distinction is that a score is supposed to return <strong>an orderable object</strong>, in particular in scikit-learn - a <strong>float</strong>. So, based on score you can tell whether method b is better from a by simply comparing if b has bigger score. You cannot do this with confusion matrix which, again as name suggests, is a matrix. </p>

<p>If you want to obtain confusion matrices for multiple evaluation runs (such as cross validation) you have to do this by hand, which is not that bad in scikit-learn - it is actually a few lines of code.</p>

<pre><code>kf = cross_validation.KFold(len(y), n_folds=5)
for train_index, test_index in kf:

   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

   model.fit(X_train, y_train)
   print confusion_matrix(y_test, model.predict(X_test))
</code></pre>

python machine-learning scikit-learn