Python: null character breaking xml format
<p>I have some python code that processes input files and dumps certain fields from the input to XML files.  This code broke when passing a null character from input -- throwing an invalid token error:</p>

<pre><code>def pretty_print_xml(elem):

    rough_string = ET.tostring(elem, 'utf-8')
    reparsed = minidom.parseString(rough_string)
    return reparsed.toprettyxml(indent='    ')
</code></pre>

<p>This surprised me and I would like to know why it broke and what else might need to be sanitized from the input.  I thought only an XML meta character could throw this error and these are already being handled by minidom.</p>

<p>NUL literals are not allowed in XML. See <a href="https://www.w3.org/TR/xml11/#charsets" rel="nofollow">the XML standard, version 1.1</a>:</p>

<blockquote>
  <h2>2.2 Characters</h2>
  
  <p>[Definition: A parsed entity contains text, a sequence of <a href="https://www.w3.org/TR/xml11/#dt-character" rel="nofollow">characters</a>, which may represent markup or character data.] [Definition: A character is an atomic unit of text as specified by ISO/IEC 10646 <A HREF="https://www.w3.org/TR/xml11/#ISO10646" rel="nofollow">[ISO/IEC 10646]</A>. Legal characters are tab, carriage return, line feed, and the legal characters of Unicode and ISO/IEC 10646. The versions of these standards cited in <A HREF="https://www.w3.org/TR/xml11/#sec-existing-stds" rel="nofollow">A.1 Normative References</A> were current at the time this document was prepared. New characters may be added to these standards by amendments or new editions. Consequently, XML processors must accept any character in the range specified for <A HREF="https://www.w3.org/TR/xml11/#NT-Char" rel="nofollow">Char</A>.]</p>

<pre><code>[2]       Char       ::=      [#x1-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
[2a]      RestrictedChar     ::=      [#x1-#x8] | [#xB-#xC] | [#xE-#x1F] | [#x7F-#x84] | [#x86-#x9F]
</code></pre>
</blockquote>

<p>Note that <code>Char</code> is defined to allow (among other ranges) <code>\x01</code> through <code>\xD7FF</code> -- but <strong>not</strong> <code>\x00</code>.</p>

<hr>

<p>By the way -- if your goal is pretty-printing, I'd suggest using <code>lxml.etree</code>. If <A HREF="http://lxml.de/tutorial.html#serialisation" rel="nofollow">the <code>pretty_print=True</code> argument</A> on serialization calls doesn't work out-of-the-box, see <A HREF="http://lxml.de/FAQ.html#why-doesn-t-the-pretty-print-option-reformat-my-xml-output" rel="nofollow">the relevant FAQ entry</A>.</p>

python xml