doc2vec - Input Format for doc2vec training and infer_vector() in python
<p>In gensim, when I give a string as input for training doc2vec model,  I get this error : </p>

<blockquote>
  <p>TypeError('don\'t know how to handle uri %s' % repr(uri))</p>
</blockquote>

<p>I referred to this question <a href="https://stackoverflow.com/questions/36780138/doc2vec-taggedlinedocument">Doc2vec : TaggedLineDocument()</a>
but still have a doubt about the input format. </p>

<p><code>documents = TaggedLineDocument('myfile.txt')</code></p>

<p>Should the myFile.txt have tokens as list of lists or separate list in each line for each document or a string? </p>

<p>For eg - I have 2 documents.</p>

<p>Doc 1 : Machine learning is a subfield of computer science that evolved from the study of pattern recognition.</p>

<p>Doc 2 :  Arthur Samuel defined machine learning as a "Field of study that gives computers the ability to learn".</p>

<p>So, what should the myFile.txt look like?</p>

<p>Case 1 : simple text of each document in each line</p>

<p>Machine learning is a subfield of computer science that evolved from the study of pattern recognition</p>

<p>Arthur Samuel defined machine learning as a Field of study that gives computers the ability to learn</p>

<p>Case 2 : a list of lists having tokens of each document</p>

<p>[ ["Machine", "learning", "is", "a", "subfield", "of", "computer", "science", "that", "evolved", "from", "the", "study", "of", "pattern", "recognition"],</p>

<p>["Arthur", "Samuel", "defined", "machine", "learning", "as", "a", "Field", "of", "study", "that", "gives", "computers" ,"the", "ability", "to", "learn"] ]</p>

<p>Case 3 : list of tokens of each document in a separate line</p>

<p>["Machine", "learning", "is", "a", "subfield", "of", "computer", "science", "that", "evolved", "from", "the", "study", "of", "pattern", "recognition"]</p>

<p>["Arthur", "Samuel", "defined", "machine", "learning", "as", "a", "Field", "of", "study", "that", "gives", "computers" ,"the", "ability", "to", "learn"]</p>

<p>And when I am running it on the test data, what should be the format of the sentence which i want to predict the doc vector for? Should it be like case 1 or case 2 below or something else?</p>

<p><code>model.infer_vector(testSentence, alpha=start_alpha, steps=infer_epoch)</code></p>

<p>Should the testSentence be :</p>

<p>Case 1 : string</p>

<p>testSentence = "Machine learning is an evolving field"</p>

<p>Case 2 : list of tokens</p>

<p>testSentence = ["Machine", "learning", "is", "an", "evolving", "field"]</p>

<p><code>TaggedLineDocument</code> is a convenience class that expects its source file (or file-like object) to be space-delimited tokens, one per line. (That is, what you refer to as 'Case 1' in your 1st question.)</p>

<p>But you can write your own iterable object to feed to gensim <code>Doc2Vec</code> as the <code>documents</code> corpus, as long as this corpus (1) iterably-returns <code>next()</code> objects that, like TaggedDocument, have <code>words</code> and <code>tags</code> lists; and (2) can be iterated over multiple times, for the multiple passes <code>Doc2Vec</code> requires for both the initial vocabulary-survey and then <code>iter</code> training passes. </p>

<p>The <code>infer_vector()</code> method takes lists-of-tokens, similar to the <code>words</code> attribute of individual <code>TaggedDocument</code>-like objects. (That is, what you refer to as 'Case 2' in your 2nd question.)</p>

python gensim word2vec doc2vec