How to drop columns which have same values in all rows via pandas or spark dataframe?
<p>Suppose I've data similar to following:</p>

<pre><code>  index id   name  value  value2  value3  data1  val5
    0  345  name1    1      99      23     3      66
    1   12  name2    1      99      23     2      66
    5    2  name6    1      99      23     7      66
</code></pre>

<p>How can we drop all those columns like (<code>value</code>, <code>value2</code>, <code>value3</code>) where all rows have same values, in one command or couple of commands using <strong>python</strong> ? </p>

<p>Consider we have many columns similar to <code>value</code>,<code>value2</code>,<code>value3</code>...<code>value200</code>.</p>

<p>Output:</p>

<pre><code>   index id      name   data1
        0  345  name1    3
        1   12  name2    2
        5    2  name6    7
</code></pre>

<p>Another solution is <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html" rel="nofollow"><code>set_index</code></a> from column which are not compared and then compare first row selected by <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iloc.html" rel="nofollow"><code>iloc</code></a> by <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.eq.html" rel="nofollow"><code>eq</code></a> with all <code>DataFrame</code> and last use <a href="http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing" rel="nofollow"><code>boolean indexing</code></a>:</p>

<pre><code>df1 = df.set_index(['index','id','name',])
print (~df1.eq(df1.iloc[0]).all())
value     False
value2    False
value3    False
data1      True
val5      False
dtype: bool

print (df1.ix[:, (~df1.eq(df1.iloc[0]).all())].reset_index())
   index   id   name  data1
0      0  345  name1      3
1      1   12  name2      2
2      5    2  name6      7
</code></pre>

python pandas duplicates multiple-columns spark-dataframe