Pandas: Filter in rows that have a Null/None/NaN value in any of several specific columns
<p>I have a csv file which has a lot of strings called <code>"NULL"</code> in it, in several columns.</p>

<p>I would like to select (filter in) rows that have a <code>"NULL"</code> value in <em>any</em> of <em>several specific</em> columns.</p>

<p>Example:</p>

<blockquote>
<pre><code>["Firstname"]   ["Lastname"]    ["Profession"]
"Jeff"          "Goldblum"      "NULL"
"NULL"          "Coltrane"      "Musician"
"Richard"       "NULL"          "Physicist"
</code></pre>
</blockquote>

<p>Here, I would like to filter in (select) rows in <code>df</code> that have the value <code>"NULL"</code> in the column <code>"Firstname"</code> <em>or</em> <code>"Lastname"</code> â but not if the value is <code>"NULL"</code> in <code>"Profession"</code>.</p>

<p>This manages to filter in strings (not <code>None</code>) in one column:</p>

<pre><code>df = df[df["Firstname"].str.contains("NULL", case=False)]
</code></pre>

<p><br>
I have however attempted to convert the <code>"NULL"</code> strings to <code>None</code> via:</p>

<pre><code>df = df.where((pd.notnull(df)), None)
df.columns = df.columns.str.lower()
</code></pre>

<p>Given the above <code>str.contains</code> filtering, perhaps it's easier to filter in <code>"NULL"</code> strings before converting to <code>None</code>?</p>

<p>I think you need first <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.replace.html" rel="nofollow"><code>replace</code></a> <code>NULL</code> string to <code>NaN</code>. Then check all <code>NaN</code> values in selected columns by <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.isnull.html" rel="nofollow"><code>isnull</code></a> and select all rows where is <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.any.html" rel="nofollow"><code>any</code></a> <code>True</code> by <a href="http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing" rel="nofollow"><code>boolean indexing</code></a>:</p>

<pre><code>df = df.replace("NULL", np.nan)

print (df[['Firstname','Lastname']].isnull())
  Firstname Lastname
0     False    False
1      True    False
2     False     True

print (df[df[['Firstname','Lastname']].isnull().any(1)])
  Firstname  Lastname Profession
1       NaN  Coltrane   Musician
2   Richard       NaN  Physicist
</code></pre>

python pandas filtering