precision and recall on biased data set
<p>Suppose a two class classification problem. One class has more than 95% of labelled data, and the other class has 5% of labelled data. The two class are very biased.</p>

<p>I am doing class validation to evaluate different classifiers, I found if a classifier intentionally to predict to the class which has majority (95%) label, even if the prediction result on other class is not accurate, from precision/recall, it is hard to distinguish since the other class has only 5% labelled data.</p>

<p>Here are the methods/metrics (using precision/recall) I am using. I am wondering if any other better metrics or method to evaluate considering the minor 5% class? I am assign a weight to the minor 5% class, but I ask here for a more systematic method to measure biased data set.</p>

<p>Using scikit learn + python 2.7.</p>

<pre><code>scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='recall_weighted')
print("Recall: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='precision_weighted')
print("Precision: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
</code></pre>

<p>This is a common problem in statistics, thus you will find plenty of resources in the internet. Check, e.g., <a href="http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/" rel="nofollow">8 Tactics To CombatÂ ImbalancedÂ Training Data</a>.</p>

<p>The probably easiest method is to resample your data. The simplest way would be to duplicate the minority class until both classes are equally represented.
A statistically more sound approach would be to first learn a probability distribution for reach of your classes, and then draw <em>n</em> samples for every class. Thus, you then have a balanced dataset.</p>

<p>This, of course, depends on your data - or simply learn only on an unbiased subset of your data.
See the article for more options.</p>

python python-2.7 machine-learning scikit-learn precision-recall