Vocabulary Processor function
<p>I am researching about embedding input for Convolution Neural Network and I understand Word2vec. However, in <a href="https://github.com/dennybritz/cnn-text-classification-tf/blob/master/train.py" rel="nofollow">CNN text classification</a>. dennybritz used function <code>learn.preprocessing.VocabularyProcessor</code>. In the <a href="http://tflearn.org/data_utils/" rel="nofollow">document</a>. They said it Maps documents to sequences of word ids. I am not quite sure how this function work. Does it creates a list of Ids then maps the Ids with Words or It has an dictionary of words and their Ids, when run function it only give the ids ? </p>

<p>Lets say that you have just two documents <code>I like pizza</code> and <code>I like Pasta</code>. Your whole vocabulary consists of these words <code>(I, like, pizza, pasta)</code> For every word in the vocabulary, there is an index associated like so (1, 2, 3, 4). Now given a document like <code>I like pasta</code> it can be converted into a vector [1, 2, 4]. This is what the <code>learn.preprocessing.VocabularyProcessor</code> does. The parameter <code>max_document_length</code> makes sure that all the documents are represented by a vector of length <code>max_document_length</code> either by padding numbers if their length is shorter than <code>max_document_length</code> and clipping them if their length is greater than <code>max_document_length</code> Hope this helps you </p>

python tensorflow text-classification