Cleaning up broken XML in Python
<p>A server I don't control sends broken XML with characters such as '>', '&amp;', '&lt;' etc. in attributes and text.</p>

<p>A small sample:</p>

<pre><code>&lt;StockFormula Description="" Name="F_ÃâTURN" RankType="Higher" Scope="Universe" Weight="10.86%"&gt;
    &lt;Formula&gt;AstTurnTTM&gt;AstTurnPTM&lt;/Formula&gt;
&lt;/StockFormula&gt;
&lt;Composite Name="Piotroski &amp; Trends - &lt;11@4w600k 70b" Weight="0%" RankType="Higher"&gt;
&lt;/Composite&gt;
</code></pre>

<p>I settled on using the lxml module because it is case sensitive, is very fast and gets the job done. </p>

<p>How would I go about fixing up this type of XML? Basically, I am trying to replace all occurrences of the invalid characters with the proper escape sequences.</p>

<pre><code>import re

broken = '&lt;StockFormula Description="" Name="F_ÃâTURN" RankType="Higher" Scope="Universe" Weight="10.86%"&gt;\n&lt;Formula&gt;AstTurnTTM&gt;AstTurnPTM&lt;/Formula&gt;\n&lt;Composite Name="Piotroski &amp; Trends - &lt;11@4w600k 70b" Weight="0%" RankType="Higher"&gt;\n&lt;/Composite&gt;'
print re.sub(r'(.*Name=".*)&amp;(")', r'\g&lt;1&gt;&amp;gt;\g&lt;2&gt;', broken)
</code></pre>

<p>Output:</p>

<pre><code>&lt;StockFormula Description="" Name="F_ÃÅ½Ã¢â¬TURN" RankType="Higher" Scope="Universe" Weight="10.86%"&gt;
    &lt;Formula&gt;AstTurnTTM&gt;AstTurnPTM&lt;/Formula&gt;
&lt;/StockFormula&gt;
&lt;Composite Name="Piotroski &amp; Trends - &lt;11@4w600k 70b" Weight="0%" RankType="Higher"&gt;
&lt;/Composite&gt;
</code></pre>

<p>First, realize no XML parser can help you with "broken XML."  XML parsers only operate over XML, which by definition must be <strong><a href="http://stackoverflow.com/a/25830482/290085">well-formed</a></strong>.</p>

<p>Second, it is not possible to repair "broken XML" in the general case.  There are no rules governing "broken XML."  Without a clear definition of "broken XML," you cannot be guaranteed to be able to process it and convert it to real XML.</p>

<p>That said, <a href="http://www.html-tidy.org/" rel="nofollow">HTML Tidy</a> does a decent job of repairing (X)HTML, and it has limited capabilities for repairing XML as well.  It's your best bet for automated repair of "broken XML."  There is a Python package, <a href="http://countergram.com/open-source/pytidylib/docs/index.html" rel="nofollow">PyTidyLib</a>, which wraps the HTML Tidy library.  </p>

python xml lxml