Text Classification for multiple label
<p>I doing Text Classification by Convolution Neural Network. I used health documents (ICD-9-CM code) for my project and I used the same model as <a href="http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/" rel="nofollow">dennybritz</a> used but my data has 36 labels. I used one_hot encoding to encode my label.</p>

<p>Here is my problem, when I run data which has one label for each document my code the accuracy is perfect from 0.8 to 1. If I run data which has more than one labels, the accuracy is significantly reduced. </p>

<p>For example: a document has single label as <code>"782.0"</code>: <code>[0 0 1 0 ... 0]</code>,<br>
a document has multiple label as <code>"782.0 V13.09 593.5"</code>: <code>[1 0 1 0 ... 1]</code>.</p>

<p>Could anyone suggest why this happen and how to improve it?</p>

<p>The label encoding seems correct. If you have multiple correct labels, <code>[1 0 1 0 ... 1]</code> looks totally fine. The loss function used in Denny's <a href="http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/" rel="nofollow">post</a> is <code>tf.nn.softmax_cross_entropy_with_logits</code>, which is the loss function you want to use for a multi-label problem. </p>

<blockquote>
  <p>Computes softmax cross entropy between logits and labels.</p>
  
  <p>Measures the probability error in discrete classification tasks in
  which the classes are <strong>mutually exclusive</strong> (each entry is in exactly one class).</p>
</blockquote>

<h3>Multiple predictions</h3>

<p>First you might want to have multiple predictions (for example top k). Then you should consider replace </p>

<pre><code>self.predictions = tf.argmax(self.scores, 1, name="predictions")
</code></pre>

<p>with <code>tf.nn.top_k()</code>, for example top 2 labels.</p>

<pre><code>values, self.predictions = tf.nn.top_k(self.scores, 2)
</code></pre>

<p>Further, you can remove predictions if their <code>values</code> are less than a threshold.</p>

<h3>Fix the accuracy measure</h3>

<p>In order to measure the accuracy correctly for a multi-label problem, the code below needs to be changed.</p>

<pre><code># Calculate Accuracy
with tf.name_scope("accuracy"):
    correct_predictions = tf.equal(self.predictions, tf.argmax(self.input_y, 1))
    self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, "float"), name="accuracy")
</code></pre>

<p>The logic of <code>correct_predictions</code> above is incorrect when you could have multiple correct labels. For example, say <code>num_classes=4</code>, and label 0 and 2 are correct. Thus your <code>input_y=[1, 0, 1, 0].</code> The <code>correct_predictions</code> would need to break tie between index 0 and index 2. I am not sure how <code>tf.argmax</code> breaks tie but if it breaks the tie by choosing the smaller index, a prediction of label 2 is always considered wrong, which definitely hurt your accuracy measure. </p>

<p>Actually in a multi-label problem, <a href="https://en.wikipedia.org/wiki/Precision_and_recall" rel="nofollow">precision and recall</a> are better metrics than accuracy. Also you can consider using precision@k (<code>tf.nn.in_top_k</code>) to report classifier performance.</p>

python machine-learning tensorflow text-classification