parse xml files in subdirectories using beautifulsoup in python
<p>I have more than 5000 XML files in multiple sub directories named f1, f2, f3, f4,...
Each folder contains more than 200 files. At the moment I want to extract all the files using BeautifulSoup only as I have already tried lxml, elemetTree and minidom but am struggling to get it done through BeautifulSoup. </p>

<p>I can extract single file in sub directory but not able to get all the files through BeautifulSoup.</p>

<p>I have checked the below posts:</p>

<p><a href="http://stackoverflow.com/questions/7785831/xml-parsing-in-python-using-beautifulsoup">XML parsing in Python using BeautifulSoup</a> (Extract Single File)</p>

<p><a href="http://stackoverflow.com/questions/38211588/parsing-all-xml-files-in-directory-and-all-subdirectories">Parsing all XML files in directory and all subdirectories</a> (This is minidom)</p>

<p><a href="http://stackoverflow.com/questions/30267755/reading-1000s-of-xml-documents-with-beautifulsoup">Reading 1000s of XML documents with BeautifulSoup</a> (Unable to get the files through this post)</p>

<p>Here is the code which I have written to extract a single file:</p>

<pre><code>from bs4 import BeautifulSoup

file = BeautifulSoup(open('./Folder/SubFolder1/file1.XML'),'lxml-xml') 

print(file.prettify())
</code></pre>

<p>When I try to get all files in all folders I am using the below code:</p>

<pre><code>from bs4 import BeautifulSoup

file = BeautifulSoup('//Folder/*/*.XML','lxml-xml') 

print(file.prettify())
</code></pre>

<p>Then I am only getting XML Version and nothing else. I know that I have to use a for loop and am not sure how to use it in order to parse all the files through the loop. </p>

<p>I know that it will be very very slow but for the sake of learning I want to use beautifulsoup to parse all the files, or if for loop is not recommended then will be grateful if I can get a better solution but only in beautifulsoup only.</p>

<p>Regards,</p>

<p>Use <code>glob.glob</code> to find the XML documents:</p>

<pre><code>import glob

from bs4 import BeautifulSoup

for filename in glob.glob('//Folder/*/*.XML'):
    content = BeautifulSoup(filename, 'lxml-xml')
    print(content.prettify())
</code></pre>

<p><em>note</em>: don't shadow the builtin function/class <code>file</code>. </p>

<p>Read the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" rel="nofollow">BeautifulSoup Quick Start</a></p>

python xml-parsing beautifulsoup