xml.etree.ElementTree.
-parse() function, which can take a filename or a file-like object [FIXME xref]. This function parses the entire document at once. If memory is tight, there are ways to parse an XML document incrementally instead.
+parse() function, which can take a filename or a file-like object [FIXME xref]. This function parses the entire document at once. If memory is tight, there are ways to parse an XML document incrementally instead. [FIXME href]
parse() function returns an object which represents the entire document. This is not the root element. To get a reference to the root element, call the getroot() method.
feed element in the http://www.w3.org/2005/Atom namespace. The string representation of this object reinforces an important point: an XML element is a combination of its namespace and its tag name (also called the local name). Every element in this document is in the Atom namespace, so the root element is represented as {http://www.w3.org/2005/Atom}feed.
author elements that have an Atom uri element as a child. This only returns two author elements, the ones in the first and second entry. The author in the last entry contains only a name, not a uri.
-Overall, ElementTree’s findall() method is a very powerful feature, but the query language can be a bit surprising. It is officially described as “limited support for XPath expressions.” XPath is a W3C standard for querying XML documents. ElementTree’s query language is similar enough to XPath to do basic searching, but dissimilar enough that it may annoy you if you already know XPath. Now let’s look at a third-party XML library that extends the ElementTree API with full XPath support.
-
-
lxml FIXME +
What’s that? You say you want the power of the findall() method, but you want to work with an iterator instead of building a complete list? ElementTree can do that too.
->>> from lxml import etree -. -. FIXME (show how it's a drop-in replacement for everything we've done so far) -. -- -
FIXME: from here on out, we use lxml.etree explicitly because these functions are specific to lxml - -
->>> import lxml.etree
->>> tree = lxml.etree.parse("examples/feed.xml")
->>> it = tree.iterfind("//{http://www.w3.org/2005/Atom}link")
->>> next(it)
+# continuing from the previous example
+>>> it = tree.getiterator("{http://www.w3.org/2005/Atom}link") ①
+>>> next(it) ②
<Element {http://www.w3.org/2005/Atom}link at 122f1b0>
>>> next(it)
<Element {http://www.w3.org/2005/Atom}link at 122f1e0>
@@ -438,32 +424,59 @@ mark{display:inline}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
+getiterator() method can zero or one arguments. If called with no arguments, it returns an iterator that spits out every element and child element in the entire document. Or, as shown here, you can call it with an element name in standard ElementTree format. This returns an iterator that spits out only elements of that name.
+Overall, ElementTree’s findall() method is a very powerful feature, but the query language can be a bit surprising. It is officially described as “limited support for XPath expressions.” XPath is a W3C standard for querying XML documents. ElementTree’s query language is similar enough to XPath to do basic searching, but dissimilar enough that it may annoy you if you already know XPath. Now let’s look at a third-party XML library that extends the ElementTree API with full XPath support.
+
+
lxml is an open source third-party library that builds on the popular libxml2 parser [FIXME href]. It provides a 100% compatible ElementTree API, then extends it with full XPath support and a few other niceties. There are installers available for Windows and Mac OS X (FIXME really?); Linux users can probably use distribution-specific tools like yum or apt-get to install precompiled binaries from their repositories.
->>> NSMAP = {"atom": "http://www.w3.org/2005/Atom"}
->>> entries = tree.xpath("//atom:category[@term='accessibility']/..", namespaces=NSMAP)
->>> entries
+>>> from lxml import etree ①
+>>> tree = etree.parse("examples/feed.xml") ②
+>>> root = tree.getroot() ③
+>>> root.findall("{http://www.w3.org/2005/Atom}entry") ④
+[<Element {http://www.w3.org/2005/Atom}entry at e2b4e0>,
+ <Element {http://www.w3.org/2005/Atom}entry at e2b510>,
+ <Element {http://www.w3.org/2005/Atom}entry at e2b540>]
+parse() function: same as ElementTree.
+getroot() method: also the same.
+findall() method: exactly the same.
+For large XML documents, lxml is significantly faster than the built-in ElementTree libary. If you’re only using the ElementTree API and want to use the fastest available implementation, you can try to import lxml and fall back to the built-in ElementTree. + +
try:
+ from lxml import etree
+except ImportError:
+ import xml.etree.ElementTree as etreeBut lxml is more than just a faster ElementTree. It also integrates support for arbitrary XPath expressions. I’m not going to go into depth about XPath syntax (it can get quite complicated). [FIXME href] is a good beginner’s guide to XPath. + +
+>>> import lxml.etree ① +>>> tree = lxml.etree.parse("examples/feed.xml") +>>> NSMAP = {"atom": "http://www.w3.org/2005/Atom"} ② +>>> entries = tree.xpath("//atom:category[@term='accessibility']/..", ③ +... namespaces=NSMAP) +>>> entries ④ [<Element {http://www.w3.org/2005/Atom}entry at e2b630>] >>> entry = entries[0] ->>> entry.xpath("./atom:title/text()", namespaces=nsmap) +>>> entry.xpath("./atom:title/text()", namespaces=nsmap) ⑤ ['Accessibility is a harsh mistress']- -
FIXME - -
->>> import lxml.etree
->>> parser = lxml.etree.XMLParser(no_network=True, ns_clean=True, recover=True, remove_blank_text=True, remove_comments=True)
->>> tree = lxml.etree.parse("examples/feed.xml", parser)
-.
-.
-.
-
-
-FIXME +
import lxml.etree (instead of, say, from lxml import etree), to emphasize that these features are specific to lxml.
+category elements (in the Atom namespace) that contain a term attribute with the value accessibility. But that’s not actually the query result. Look at the very end of the query string; did you notice the /.. bit? That means “and then return the parent element of the category element you just found.” So this single XPath query will find all entries with a child element of <category term="accessibility">.
+xpath() function returns a list of ElementTree objects. In this document, there is only one entry with a category whose term is accessibility.
+text()) of the title element (atom:title) that is a child of the current element (./).
+The XML specification mandates that all conforming XML parsers employ “draconian error handling.” That is, they must halt and catch fire as soon as they detect any sort of wellformedness error in the XML document. Wellformedness errors include mismatched start and end tags, undefined entities, illegal Unicode characters, and a number of other esoteric rules. This is in stark contrast to other common formats like HTML — your browser doesn’t stop rendering a web page if you forget to close an HTML tag or escape an ampersand in an attribute value. (It is a common misconception that HTML has no defined error handling. HTML error handling is actually quite well-defined [FIXME href], but it’s significantly more complicated than “halt and catch fire on first error.”) + +
Some people (myself included) believe that it was a mistake for the inventors of XML to mandate draconian error handling. Don’t get me wrong; I can certainly see the allure of simplifying the error handling rules. But in practice, the concept of “wellformedness” is trickier than it sounds, especially for XML documents (like Atom feeds) that are published on the web and served over HTTP. Despite the maturity of XML, which standardized on draconian error handling in 1997, surveys continually show a significant fraction of Atom feeds on the web are plagued with wellformedness errors.
+
+
So, I have both theoretical and practical reasons to parse XML documents “at any cost,” that is, not to halt and catch fire at the first wellformedness error. If you find yourself wanting to do this too, lxml can help.
+
+
Here is a fragment of a broken XML document. I’ve highlighted the wellformedness error. + +
<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
+ <title>dive into …</title>
+...
+</feed>That’s an error, because the … entity is not defined in XML. (It is defined in HTML.) If you try to parse this broken feed with the default settings, lxml will choke on the undefined entity.
+
+
+>>> import lxml.etree
+>>> tree = lxml.etree.parse("examples/feed-broken.xml")
+Traceback (most recent call last):
+ File "<stdin>", line 1, in <module>
+ File "lxml.etree.pyx", line 2693, in lxml.etree.parse (src/lxml/lxml.etree.c:52591)
+ File "parser.pxi", line 1478, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:75665)
+ File "parser.pxi", line 1507, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:75993)
+ File "parser.pxi", line 1407, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:75002)
+ File "parser.pxi", line 965, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:72023)
+ File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:67830)
+ File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:68877)
+ File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:68125)
+lxml.etree.XMLSyntaxError: Entity 'hellip' not defined, line 3, column 28
+
+To parse this broken XML document, despite its wellformedness error, you need to create a custom XML parser. + +
+>>> parser = lxml.etree.XMLParser(recover=True) ① +>>> tree = lxml.etree.parse("examples/feed-broken.xml", parser) ② +>>> parser.error_log ③ +examples/feed-broken.xml:3:28:FATAL:PARSER:ERR_UNDECLARED_ENTITY: Entity 'hellip' not defined +>>> tree.findall("{http://www.w3.org/2005/Atom}title") +[<Element {http://www.w3.org/2005/Atom}title at ead510>] +>>> title = tree.findall("{http://www.w3.org/2005/Atom}title")[0] +>>> title.text ④ +'dive into ' +>>> print(lxml.etree.tounicode(tree.getroot())) ⑤ +<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"> + <title>dive into </title> +. +. [rest of serialization snipped for brevity] +.+
lxml.etree.XMLParser class. It can take a number of different named arguments [FIXME href]. The one we’re interested in here is the recover argument. When set to True, the XML parser will try its best to “recover” from wellformedness errors.
+XML document with your custom parser, pass the parser object as the second argument to the parse() function. Note that lxml does not raise an exception about the undefined … entity.
+… entity, the parser just silently dropped it. The text content of the title element becomes "dive into ".
+… entity didn’t get moved; it was simply dropped.
+It is important to reiterate that there is no guarantee of interoperability with “recovering” XML parsers. A different parser might decide that it recognized the … entity from HTML, and replace it with &hellip; instead. Is that “better”? Maybe. Is it “more correct”? No, they are both equally incorrect. The correct behavior (according to the XML specification) is to halt and catch fire. If you’ve decided not to do that, you’re on your own.
+