parse xml files in subdirectories using beautifulsoup in python
<p>I have more than 5000 XML files in multiple sub directories named f1, f2, f3, f4,...
Each folder contains more than 200 files. At the moment I want to extract all the files using BeautifulSoup only as I have already tried lxml, elemetTree and minidom but am struggling to get it done through BeautifulSoup. </p>

<p>I can extract single file in sub directory but not able to get all the files through BeautifulSoup.</p>

<p>I have checked the below posts:</p>

<p><a href="http://stackoverflow.com/questions/7785831/xml-parsing-in-python-using-beautifulsoup">XML parsing in Python using BeautifulSoup</a> (Extract Single File)</p>

<p><a href="http://stackoverflow.com/questions/38211588/parsing-all-xml-files-in-directory-and-all-subdirectories">Parsing all XML files in directory and all subdirectories</a> (This is minidom)</p>

<p><a href="http://stackoverflow.com/questions/30267755/reading-1000s-of-xml-documents-with-beautifulsoup">Reading 1000s of XML documents with BeautifulSoup</a> (Unable to get the files through this post)</p>

<p>Here is the code which I have written to extract a single file:</p>

<pre><code>from bs4 import BeautifulSoup

file = BeautifulSoup(open('./Folder/SubFolder1/file1.XML'),'lxml-xml') 

print(file.prettify())
</code></pre>

<p>When I try to get all files in all folders I am using the below code:</p>

<pre><code>from bs4 import BeautifulSoup

file = BeautifulSoup('//Folder/*/*.XML','lxml-xml') 

print(file.prettify())
</code></pre>

<p>Then I am only getting XML Version and nothing else. I know that I have to use a for loop and am not sure how to use it in order to parse all the files through the loop. </p>

<p>I know that it will be very very slow but for the sake of learning I want to use beautifulsoup to parse all the files, or if for loop is not recommended then will be grateful if I can get a better solution but only in beautifulsoup only.</p>

<p>Regards,</p>

<p>If I understood you correctly, then you do need to loop through the files, as you had already thought:</p>

<pre><code>from bs4 import BeautifulSoup
from pathlib import Path

for filepath in Path('./Folder').glob('*/*.XML'):
    with filepath.open() as f:
        soup = BeautifulSoup(f,'lxml-xml')
    print(soup.prettify())
</code></pre>

<p><a href="https://docs.python.org/3/library/pathlib.html" rel="nofollow"><code>pathlib</code></a> is just one approach to handling paths, on a higher level using objects. You could achieve the same with <a href="https://docs.python.org/3/library/glob.html" rel="nofollow"><code>glob</code></a> and string paths.</p>

python xml-parsing beautifulsoup