Getting good accuracy but very low precision for classification task on an unbalanced dataset
<p>I have an unbalanced dataset , where the positive class is about 10,000 entries and the negative class is some 8,00,000 entries. I am trying a simple scikit's LogisticRegression model as a base line model, with class_weight='balanced' (hopefully unbalanced problem should be solved?).</p>

<p>However, I am getting an accuracy score of 0.83, but a precision score of 0.03. What could be the issue? Do I need to handle the unbalanced part separately?</p>

<p>This is my current code:</p>

<pre><code>&gt;&gt;&gt; train = []
&gt;&gt;&gt; target = []
&gt;&gt;&gt; len(posList)
... 10214
&gt;&gt;&gt; len(negList)
... 831134
&gt;&gt;&gt; for entry in posList:
...     train.append(entry)
...     target.append(1)
...
&gt;&gt;&gt; for entry in negList:
...     train.append(entry)
...     target.append(-1)
...
&gt;&gt;&gt; train = np.array(train)
&gt;&gt;&gt; target = np.array(target)
&gt;&gt;&gt; 
&gt;&gt;&gt; X_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=42)
&gt;&gt;&gt; 
&gt;&gt;&gt; model = LogisticRegression(class_weight='balanced')
&gt;&gt;&gt; model.fit(X_train, y_train)
LogisticRegression(C=1.0, class_weight='balanced', dual=False,
          fit_intercept=True, intercept_scaling=1, max_iter=100,
          multi_class='ovr', n_jobs=1, penalty='l2', random_state=None,
          solver='liblinear', tol=0.0001, verbose=0, warm_start=False)
&gt;&gt;&gt; 
&gt;&gt;&gt; predicted = model.predict(X_test)
&gt;&gt;&gt; 
&gt;&gt;&gt; metrics.accuracy_score(y_test, predicted)
0.835596671213
&gt;&gt;&gt; 
&gt;&gt;&gt; metrics.precision_score(y_test, predicted, average='weighted')
/usr/local/lib/python2.7/dist-packages/sklearn/metrics/classification.py:976: DeprecationWarning: From version 0.18, binary input will not be handled specially when using averaged precision/recall/F-score. Please use average='binary' to report only the positive class performance.
  'positive class performance.', DeprecationWarning)
0.033512518766
</code></pre>

<p>I think I understand what is going on.</p>

<p>Consider a dummy classifier that would return the majority class for each sample of your data set. For an unbalanced set like yours it seems pretty fair (let's call your positive class <em>class 1</em> and the negative one <em>class 0</em>). The classifier would have an accuracy of : <code>831134/(831134+10214.0) = 0.987859958067292</code>. Yes, <strong>an accuracy of 99%</strong>, but it doesn't give a good representation of your classifier. Instead, we'd better look at its <strong>precision</strong>. So since your data is really unbalanced (ratio <code>1:80</code>) the Logistic Regression performs poorly, but as for the dummy classifier, it has a high accuracy.</p>

<p><strong>Precision</strong> is defined as the ratio of <em>true positives</em> and the sum of <em>true positives</em> and <em>false positives</em>. In other words, it is the proportion of <em>elements truly belonging to class 1</em> amongst <em>all the elements detected as belonging to class 1</em>.</p>

<p>The accuracy of your LinearRegression classifier is <code>acc = 0.835596671213</code>. Thus, their is a difference of accuracy between the dummy classifier and the logistic regression of : <code>diff = 0.987859958067292 - 0.835596671213 = 0.15226328685429202</code>. So <strong>15%</strong> of the data has been <strong>misclassified</strong> by the dummy classifier, which corresponds to almost <code>n_misclass = 0.15*(831134+10214.0)=126202.2</code> samples. Thus, the Logistic regression classifies <strong>126202</strong> samples as being from <em>class 1</em> while they are only  <strong>10214</strong>.</p>

<p>The <strong>precision that Logistic Regression</strong> might then be something like : <code>prec = 10214/126202.0 = 0.081</code>.</p>

<p>In your case it seems that it performs less good in terms of accuracy than what we've seen above. But that was roughly to give you a clue about what might be going on.</p>

python python-2.7 scikit-learn