error using where with pandas and categorical columns
<p>Problem: using the where clause with a dataframe with categorical columns produces <strong>ValueError: Wrong number of dimensions</strong></p>

<p>I Just can't figure out what am I doing wrong.</p>

<pre><code>df=pd.read_csv("F:/python/projects/mail/Inbox_20160911-1646/rows.csv",header=0,sep=",",quotechar="'",quoting=1)
df.where(df &gt; 100)  # WORKS !!!!

for c in [x for x in df.columns[2:] if df[x].dtype == "object" ]:
    cl="c"+c
    df[cl]=df[c].astype("category")

df.where(df &gt; 100) # ---&gt; ValueError: Wrong number of dimensions

    df.where(df &gt; 100)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-278-7469c620cf83&gt; in &lt;module&gt;()
----&gt; 1 df.where(df &gt; 100)

F:\python\anaconda3\lib\site-packages\pandas\core\ops.py in f(self, other)
   1182             # straight boolean comparisions we want to allow all columns
   1183             # (regardless of dtype to pass thru) See #4537 for discussion.
-&gt; 1184             res = self._combine_const(other, func, raise_on_error=False)
   1185             return res.fillna(True).astype(bool)
   1186 

F:\python\anaconda3\lib\site-packages\pandas\core\frame.py in _combine_const(self, other, func, raise_on_error)
   3553 
   3554         new_data = self._data.eval(func=func, other=other,
-&gt; 3555                                    raise_on_error=raise_on_error)
   3556         return self._constructor(new_data)
   3557 

F:\python\anaconda3\lib\site-packages\pandas\core\internals.py in eval(self, **kwargs)
   2909 
   2910     def eval(self, **kwargs):
-&gt; 2911         return self.apply('eval', **kwargs)
   2912 
   2913     def quantile(self, **kwargs):

F:\python\anaconda3\lib\site-packages\pandas\core\internals.py in apply(self, f, axes, filter, do_integrity_check, consolidate, raw, **kwargs)
   2888 
   2889             kwargs['mgr'] = self
-&gt; 2890             applied = getattr(b, f)(**kwargs)
   2891             result_blocks = _extend_blocks(applied, result_blocks)
   2892 

F:\python\anaconda3\lib\site-packages\pandas\core\internals.py in eval(self, func, other, raise_on_error, try_cast, mgr)
   1160             result = self._try_cast_result(result)
   1161 
-&gt; 1162         return [self.make_block(result, fastpath=True, )]
   1163 
   1164     def where(self, other, cond, align=True, raise_on_error=True,

F:\python\anaconda3\lib\site-packages\pandas\core\internals.py in make_block(self, values, placement, ndim, **kwargs)
    179             ndim = self.ndim
    180 
--&gt; 181         return make_block(values, placement=placement, ndim=ndim, **kwargs)
    182 
    183     def make_block_same_class(self, values, placement=None, fastpath=True,

F:\python\anaconda3\lib\site-packages\pandas\core\internals.py in make_block(values, placement, klass, ndim, dtype, fastpath)
   2516                      placement=placement, dtype=dtype)
   2517 
-&gt; 2518     return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)
   2519 
   2520 # TODO: flexible with index=None and/or items=None

F:\python\anaconda3\lib\site-packages\pandas\core\internals.py in __init__(self, values, ndim, fastpath, placement, **kwargs)
   1661 
   1662         super(ObjectBlock, self).__init__(values, ndim=ndim, fastpath=fastpath,
-&gt; 1663                                           placement=placement, **kwargs)
   1664 
   1665     @property

F:\python\anaconda3\lib\site-packages\pandas\core\internals.py in __init__(self, values, placement, ndim, fastpath)
     79             ndim = values.ndim
     80         elif values.ndim != ndim:
---&gt; 81             raise ValueError('Wrong number of dimensions')
     82         self.ndim = ndim
     83 
</code></pre>

<p>ValueError: Wrong number of dimensions</p>

<p>Here is a small demo, that reproduces your error:</p>

<pre><code>In [11]: df = pd.DataFrame(np.random.randint(0, 10, (5,3)), columns=list('abc'))

In [12]: df
Out[12]:
   a  b  c
0  9  9  8
1  5  6  1
2  2  9  8
3  8  1  3
4  1  5  1
</code></pre>

<p>this works:</p>

<pre><code>In [13]: df &gt; 1
Out[13]:
       a      b      c
0   True   True   True
1   True   True  False
2   True   True   True
3   True  False   True
4  False   True  False

In [14]: df['cat'] = df.c.astype('category')

In [15]: df
Out[15]:
   a  b  c cat
0  9  9  8   8
1  5  6  1   1
2  2  9  8   8
3  8  1  3   3
4  1  5  1   1
</code></pre>

<p>this throws an <code>Wrong number of dimensions</code> exception:</p>

<pre><code>In [16]: df &gt; 1
...skipped...
ValueError: Wrong number of dimensions
</code></pre>

<p>and this is a real reason for the previous error:</p>

<pre><code>In [19]: df.cat &gt; 1
...skipped...
TypeError: Unordered Categoricals can only compare equality or not
</code></pre>

<p><strong>Solution:</strong></p>

<pre><code>In [22]: df.select_dtypes(include=['number']) &gt; 1
Out[22]:
       a      b      c
0   True   True   True
1   True   True  False
2   True   True   True
3   True  False   True
4  False   True  False

In [23]: np.where(df.select_dtypes(exclude=['category']) &gt; 1)
Out[23]:
(array([0, 0, 0, 1, 1, 2, 2, 2, 3, 3, 4], dtype=int64),
 array([0, 1, 2, 0, 1, 0, 1, 2, 0, 2, 1], dtype=int64))
</code></pre>

python pandas categorical-data