Spark HiveContext: Tables with multiple files on HDFS
<p>I have a Hive table X which has multiple files on HDFS.
Table X location on HDFS is /data/hive/X.
Files:</p>

<pre><code>/data/hive/X/f1
/data/hive/X/f2
/data/hive/X/f3 ...
</code></pre>

<p>Now, I run the below commands:</p>

<pre><code>df=hiveContext.sql("SELECT count(*) from X")
df.show()
</code></pre>

<p>What happens internally? Does each file be considered as a separate partition and is processed by a separate node and then results are collated?</p>

<p>If yes, is there a way to instruct Spark to load all the files into 1 partition and then process the data?</p>

<p>Thanks in advance.</p>

<p>Spark will contact Hive metastore to find out (a) Location of data (b) How to read the data. At low level, Spark will get Input Splits based on Input Formats used in hive to store the data. Once Splits are decided, Spark will read data 1 split/partition. 
In Spark, one physical node can run one or more executors. Each executor will have one or more partitions. 
Once data is read into memory, spark will run a count, which will be (a) local counts on map (b) global count after a shuffle. then it is returned to driver as a result. </p>

python apache-spark dataframe hdfs