tfidfvectorizer Predict in saved classifier
<p>I trained my model by using TfIdfVectorizer and MultinomialNB and I saved it into a pickle file.</p>

<p>Now that I am trying to use the classifier from another file to predict in unseen data, I cannot do it because it is telling my that the number of features of the classifier is not the same than the number of features of my current corpus.</p>

<p>This is the code where I am trying to predict. The function do_vectorize is exactly the same used in training.</p>

<pre><code>def do_vectorize(data, stop_words=[], tokenizer_fn=tokenize):
    vectorizer = TfidfVectorizer(stop_words=stop_words, tokenizer=tokenizer_fn)
    X = vectorizer.fit_transform(data)
    return X, vectorizer

# Vectorizing the unseen documents 
matrix, vectorizer = do_vectorize(corpus, stop_words=stop_words)

# Predicting on the trained model
clf = pickle.load(open('../data/classifier_0.5_function.pkl', 'rb'))
predictions = clf.predict(matrix)
</code></pre>

<p>However I receive the error that the number of features are different</p>

<pre><code>ValueError: Expected input with 65264 features, got 472546 instead
</code></pre>

<p>This means I also have to save my vocabulary from training in order to test? What will happen if there are terms that did not exist on training?</p>

<p>I tried to used pipelines from scikit-learn with the same vectorizer and classifier, and the same parameters for both. However, it turned too slow from 1 hour to more than 6 hours, so I prefer to do it manually.</p>

<blockquote>
  <p>This means I also have to save my vocabulary from training in order to test? </p>
</blockquote>

<p>Yes, you have to save <strong>whole tfidf vectorizer</strong>, which in particular means saving vocabulary.</p>

<blockquote>
  <p>What will happen if there are terms that did not exist on training?</p>
</blockquote>

<p>They will be ignored, which makes perfect sense since you have <strong>no training data</strong> about this, thus there is nothing to take into consideration (there are more complex methods which could still use it, but they do not use such simple approaches as tfidf).</p>

<blockquote>
  <p>I tried to used pipelines from scikit-learn with the same vectorizer and classifier, and the same parameters for both. However, it turned too slow from 1 hour to more than 6 hours, so I prefer to do it manually.</p>
</blockquote>

<p>There should be little to no overhead when using pipelines, however doing things manually is fine as long as you remember to store vectorizer as well.</p>

python machine-learning scikit-learn classification prediction