Filter a large number of IDs from a dataframe Spark
<p>I have a large dataframe with a format similar to</p>

<pre><code>+-----+------+------+
|ID   |Cat   |date  |
+-----+------+------+
|12   | A    |201602|
|14   | B    |201601|
|19   | A    |201608|
|12   | F    |201605|
|11   | G    |201603|
+-----+------+------+
</code></pre>

<p>and I need to filter rows based on a list with around 5000 thousand IDs. The straighforward way would be to filter with <code>isin</code> but that has really bad performance. How can this filter be done?</p>

<p>If you're committed to using Spark SQL and <code>isin</code> doesn't scale anymore then inner equi-join should be a decent fit. </p>

<p>First convert id list to as single column <code>DataFrame</code>. If this is a local collection</p>

<pre><code>ids_df = sc.parallelize(id_list).map(lambda x: (x, )).toDF(["id"])
</code></pre>

<p>and <code>join</code>:</p>

<pre><code>df.join(ids_df, ["ID"], "inner")
</code></pre>

python apache-spark pyspark