spark Dataframe/RDD equivalent to pandas command given in description?
<p>How to perform same functionality as this pandas command via Pyspark dataframe or RDD ?</p>

<pre><code>df.drop(df.std()[(df.std() == 0)].index, axis=1)
</code></pre>

<p>For details on what this command does, refer:
<a href="http://stackoverflow.com/questions/39658574/how-to-drop-columns-which-have-same-values-in-all-rows-via-pandas-or-spark-dataf">How to drop columns which have same values in all rows via pandas or spark dataframe?</a></p>

<p>NOTE: file is too big to use df.toPandas().</p>

<p>In general you can use <code>countDistinct</code>:</p>

<pre><code>from pyspark.sql.functions import countDistinct 

cnts = (df
    .select([countDistinct(c).alias(c) for c in df.columns])
    .first()
    .asDict())

df.select(*[k for (k, v) in cnts.items() if v &gt; 1])

## +---+-----+-----+-----+
## | id|index| name|data1|
## +---+-----+-----+-----+
## |345|    0|name1|    3|
## | 12|    1|name2|    2|
## |  2|    5|name6|    7|
## +---+-----+-----+-----+
</code></pre>

<p>This won't work on data with cardinality but can handle non-numeric columns.</p>

<p>You can use the same approach to filter with standard deviations:</p>

<pre><code>from pyspark.sql.functions import stddev

stddevs = df.select(*[stddev(c).alias(c) for c in df.columns]).first().asDict()

df.select(*[k for (k, v) in stddevs.items() if v is None or v != 0.0])

## +---+-----+-----+-----+
## | id|index| name|data1|
## +---+-----+-----+-----+
## |345|    0|name1|    3|
## | 12|    1|name2|    2|
## |  2|    5|name6|    7|
## +---+-----+-----+-----+
</code></pre>

python pandas pyspark rdd spark-dataframe