How to handle NA values when tokenizing the contents of a data frame?
<p>I have a pandas dataframe and I am trying to tokenize the contents of each row. </p>

<pre><code>import pandas as pd
import nltk as nk
from nltk import word_tokenize

TextData = pd.read_csv('TextData.csv')
TextData['tokenized_summary'] = TextData.apply(lambda row: nk.word_tokenize(row['Summary']), axis=1)
</code></pre>

<p>When I run it, I get an error at line 67, </p>

<pre><code>TypeError: ('expected string or buffer', u'occurred at index 67') 
</code></pre>

<p>Which I think I am getting because the value for 'Summary' at iloc[67] is an NA value. </p>

<pre><code>TextData.Summary.iloc[67]

Out[45]: nan
</code></pre>

<p>Assuming it is the na value which is causing this, is there a way to tell word_tokenize or pandas to ignore the NA values whenever it comes across them?  </p>

<p>Else, what else might be causing this? </p>

<p>You can use <code>fillna()</code> to replace NaN with a specified value:</p>

<pre><code>import pandas as pd
import nltk as nk
from nltk import word_tokenize

TextData = pd.read_csv('TextData.csv')
TextData.fillna('some value') # or just: TextData['Summary'].fillna('some value')
TextData['tokenized_summary'] = TextData.apply(lambda row: nk.word_tokenize(row['Summary']), axis=1)
</code></pre>

<hr>

<h3>Previous Answer</h3>

<p>You can simply "eliminate" the records where that value is null:</p>

<pre><code>TextData = TextData[TextData['tokenized_summary'].notnull()]
</code></pre>

<p>Making the final product look like:</p>

<pre><code>import pandas as pd
import nltk as nk
from nltk import word_tokenize

TextData = pd.read_csv('TextData.csv')
TextData = TextData[TextData['tokenized_summary'].notnull()]
TextData['tokenized_summary'] = TextData.apply(lambda row: nk.word_tokenize(row['Summary']), axis=1)
</code></pre>

python pandas nltk