Get a list of subdirectories
<p>I know I can do this:</p>

<pre><code>data = sc.textFile('/hadoop_foo/a')
data.count()
240
data = sc.textFile('/hadoop_foo/*')
data.count()
168129
</code></pre>

<p>However, I would like to count the size of the data of every subdirectory of "/hadoop_foo/". Can I do that?</p>

<p>In other words, what I want is something like this:</p>

<pre><code>subdirectories = magicFunction()
for subdir in subdirectories:
  data sc.textFile(subdir)
  data.count()
</code></pre>

<hr>

<p>I tried with:</p>

<pre><code>In [9]: [x[0] for x in os.walk("/hadoop_foo/")]
Out[9]: []
</code></pre>

<p>but I think that fails, because it searches at the local directory of the driver (the gateway in that case), while "/hadoop_foo/" lies in the <a href="/questions/tagged/hdfs" class="post-tag" title="show questions tagged &#39;hdfs&#39;" rel="tag">hdfs</a>. Same for "hdfs:///hadoop_foo/".</p>

<hr>

<p>After reading <a href="http://stackoverflow.com/questions/31056680/how-can-i-list-subdirectories-recursively-for-hdfs">How can I list subdirectories recursively for HDFS?</a>, I am wondering if there is a way to execute:</p>

<pre><code>hadoop dfs -lsr /hadoop_foo/
</code></pre>

<p>in code..</p>

<hr>

<p>From <a href="http://stackoverflow.com/questions/39303218/correct-way-of-writing-two-floats-into-a-regular-txt">Correct way of writing two floats into a regular txt</a>:</p>

<pre><code>In [28]: os.getcwd()
Out[28]: '/homes/gsamaras'  &lt;-- which is my local directory
</code></pre>

<p>With python use <a href="https://pypi.python.org/pypi/hdfs/" rel="nofollow">hdfs</a> module; <a href="https://hdfscli.readthedocs.io/en/latest/api.html#hdfs.client.Client.walk" rel="nofollow">walk()</a> method can get you list of files.</p>

<p>The code sould look something like this:</p>

<pre><code>from hdfs import InsecureClient

client = InsecureClient('http://host:port', user='user')
for stuff in client.walk(dir, 0, True):
...
</code></pre>

<p>With Scala you can get the filesystem (<code>val fs = FileSystem.get(new Configuration())</code>) and run <a href="https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path" rel="nofollow">https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path</a>, boolean)</p>

<p>You can also execute a shell command from your script with <a href="https://docs.python.org/2/library/subprocess.html" rel="nofollow">os.subprocess</a> but this is never a recommended approach since you depend on text output of a shell utility here.</p>

<hr>

<p>Eventually, what worked for the OP was using <a href="/hadoop_foo/">subprocess.check_output()</a>:</p>

<pre><code>subdirectories = subprocess.check_output(["hadoop","fs","-ls", "/hadoop_foo/"])
</code></pre>

python hadoop apache-spark hdfs bigdata