Why is sklearn's Perceptron predicting with accuracy, precision etc. of 1?
<p>I am using sklearn.linear_model.Perceptron on a synthetic dataset I created. The data consists of 2 classes each of which is a multivariate Gaussian with a common non-diagonal covariance matrix. The centroids of the classes are close enough that there is significant overlap. </p>

<pre><code>mean1 = np.ones((20,))
mean2 = 2 * np.ones((20,))

A = 0.1 * np.random.randn(20,20)
cov = np.dot(A, A.T)

class1 = np.random.multivariate_normal(mean1, cov, 2000)
class2 = np.random.multivariate_normal(mean2, cov, 2000)

class1 = np.concatenate((class1, np.ones((len(class1), 1))), axis=1)
class2 = np.concatenate((class2, 2*np.ones((len(class2), 1))), axis=1)

class1_train, class1_test = train_test_split(class1, test_size=0.3)
class2_train, class2_test = train_test_split(class2, test_size=0.3)
train = np.concatenate((class1_train, class2_train), axis=0)
test = np.concatenate((class1_test, class2_test), axis=0)

np.random.shuffle(train)
np.random.shuffle(test)
y_train = train[:,20]
x_train = train[:,0:20]
y_test = test[:,20]
x_test = test[:,0:20]
</code></pre>

<p>After saving this data, I just used :</p>

<pre><code>classifier = sklearn.linear_model.Perceptron()
classifier.fit(x_train, y_train)
predicted_test = classifier.predict(x_test)
accuracy = sklearn.metrics.accuracy_score(y_test, predicted_test)
precision = sklearn.metrics.precision_score(y_test, predicted_test)
recall = sklearn.metrics.recall_score(y_test, predicted_test)
f_measure = sklearn.metrics.f1_score(y_test, predicted_test)
print(accuracy, precision, recall, f_measure)  
</code></pre>

<p>The data is overlapping by design. But yet the linear classifier is able to predict perfectly somehow with accuracy, precision etc. all being 1.</p>

<p>The correct way of using <code>cross_validation.train_test_split</code> is to give it the complete dataset, and letting it partition the data to <code>x_train, x_test, y_train, y_test</code>.</p>

<p>The following code works better:</p>

<pre><code>class1 = np.random.multivariate_normal(mean1, cov, 2000)
class2 = np.random.multivariate_normal(mean2, cov, 2000)

class1 = np.concatenate((class1, np.ones((len(class1), 1))), axis=1)
class2 = np.concatenate((class2, 2*np.ones((len(class2), 1))), axis=1)

dataset = np.concatenate((class1, class2), axis=0)

np.random.shuffle(dataset)

x_train, x_test, y_train, y_test = \
    cross_validation.train_test_split(dataset[:,:20], dataset[:,20], test_size=0.3)
</code></pre>

<p>Notice that the Perceptron can actually acheive 100% accuracy with your data. Try adding some noise to it, in order to get a feeling of it.</p>

<p>For instance:</p>

<pre><code>noise = np.random.normal(0,1,(4000, 20))

dataset[:, 0:20] = dataset[:, 0:20] + noise

x_train, x_test, y_train, y_test = \
    cross_validation.train_test_split(dataset[:,:20], dataset[:,20], test_size=0.3)
</code></pre>

python scikit-learn