Spark - How many Executors and Cores are allocated to my spark job
<p>Spark architecture is entirely revolves around the concept of executors and cores. I would like to see practically how many executors and cores running for my spark application running in a cluster. </p>

<p>I was trying to use below snippet in my application but no luck.</p>

<pre><code>val conf = new SparkConf().setAppName("ExecutorTestJob")
val sc = new SparkContext(conf)
conf.get("spark.executor.instances")
conf.get("spark.executor.cores")
</code></pre>

<p>Is there any way to get those values using <code>SparkContext</code> Object or <code>SparkConf</code> object etc..</p>

<h1>Scala (Programmatic way) :</h1>

<p><code>getExecutorStorageStatus</code> and <code>getExecutorMemoryStatus</code> both return the number of executors including driver.
like below example snippet.</p>

<pre><code>/** Method that just returns the current active/registered executors
        * excluding the driver.
        * @param sc The spark context to retrieve registered executors.
        * @return a list of executors each in the form of host:port.
        */
       def currentActiveExecutors(sc: SparkContext): Seq[String] = {
         val allExecutors = sc.getExecutorMemoryStatus.map(_._1)
         val driverHost: String = sc.getConf.get("spark.driver.host")
         allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList
       }

sc.getConf.getInt("spark.executor.instances", 1)
</code></pre>

<p>similarly get all properties and print like below you may get cores information as well..</p>

<pre><code>sc.getConf.getAll.mkString("\n")
</code></pre>

<p>OR </p>

<pre><code>sc.getConf.toDebugString
</code></pre>

<p>Mostly <code>spark.executor.cores</code> for executors <code>spark.driver.cores</code> driver should have this value.</p>

<h1>Python :</h1>

<p><a href="https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.SparkContext" rel="nofollow">Above methods getExecutorStorageStatus and getExecutorMemoryStatus, In python api were not implemented</a></p>

python scala hadoop apache-spark executors