Build successive condition to when
<p>I'm trying to build a sql request of consecutive <code>when</code>.</p>

<pre><code>def build_modify_function(df, ids_colname, modified_colname, modification_list):
if len(modification_list) == 0:
    pass
# Small optimization
id_col      = df[ids_colname]
modif_col   = df[modified_colname]
# There is no "identity element" so :
ret = None
for (row_ids, new_value) in modification_list:
    if type(row_ids) != type(list()):
        row_ids = list(row_ids)
    if ret == None:
        ret = when(id_col.isin(row_ids), new_value) # .isin(row_ids)
    else:
        ret = ret.when(id_col.isin(row_ids), new_value)
return modif_col if ret == None else ret.otherwise(modif_col)
</code></pre>

<p><code>df</code> is a dataframe,</p>

<p><code>ids_colname</code> is the column name of my ids,</p>

<p><code>modified_colname</code> is the column name I want to modify,</p>

<p><code>modification_list</code> is a list of tuple <code>[(list_of_ids, new_value)]</code></p>

<p>I am having this error:</p>

<pre><code>ValueError: Cannot convert column into bool: 
please use '&amp;' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.
</code></pre>

<p>I don't understand why. When I build a very simple function returning :</p>

<pre><code> tmp = when(id_col.isin(row_ids), new_value)
 return tmp\
     .when(id_col.isin(row_ids), new_value)\
     .otherwise(modif_col)
</code></pre>

<p>It perfectly works. Any suggestions?</p>

<p>I believe that the problem is here:</p>

<pre><code>if ret == None:
   ...
</code></pre>

<p>In general you should never use equality operators to compare to singleton objects in Python and always use <code>is</code> or <code>is not</code>:</p>

<pre><code>if ret is None:
    ...
</code></pre>

<p>In this particular case equality operator for column returns a <code>Column</code> and <code>Column</code> object explicitly forbids conversion to bool.</p>

python apache-spark apache-spark-sql