From 2c114c103585580e81cb28bf07364178dd1fb4d1 Mon Sep 17 00:00:00 2001 From: Mark Pilgrim Date: Tue, 26 May 2009 11:48:44 -0700 Subject: [PATCH] finished XML chapter, modulo a few hrefs to fill in later --- examples/feed-ns0.xml | 63 +++++++++++++++++ xml.html | 154 +++++++++++++++++++++++++++++++----------- 2 files changed, 177 insertions(+), 40 deletions(-) create mode 100644 examples/feed-ns0.xml diff --git a/examples/feed-ns0.xml b/examples/feed-ns0.xml new file mode 100644 index 0000000..9590116 --- /dev/null +++ b/examples/feed-ns0.xml @@ -0,0 +1,63 @@ + + + dive into mark + currently between addictions + tag:diveintomark.org,2001-07-29:/ + 2009-03-27T21:56:07Z + + + + Mark + http://diveintomark.org/ + + Dive into history, 2009 edition + + tag:diveintomark.org,2009-03-27:/archives/20090327172042 + 2009-03-27T21:56:07Z + 2009-03-27T17:20:42Z + + + + Putting an entire chapter on one page sounds + bloated, but consider this &mdash; my longest chapter so far + would be 75 printed pages, and it loads in under 5 seconds&hellip; + On dialup. + + + + Mark + http://diveintomark.org/ + + Accessibility is a harsh mistress + + tag:diveintomark.org,2009-03-21:/archives/20090321200928 + 2009-03-22T01:05:37Z + 2009-03-21T20:09:28Z + + The accessibility orthodoxy does not permit people to + question the value of features that are rarely useful and rarely used. + + + + Mark + + A gentle introduction to video encoding, part 1: container formats + + tag:diveintomark.org,2008-12-18:/archives/20081218155422 + 2009-01-11T19:39:22Z + 2008-12-18T15:54:22Z + + + + + + + + + These notes will eventually become part of a + tech talk on video encoding. + + diff --git a/xml.html b/xml.html index 3c222ca..ef6443c 100644 --- a/xml.html +++ b/xml.html @@ -253,7 +253,7 @@ mark{display:inline} <Element {http://www.w3.org/2005/Atom}feed at cd1eb0>
  1. The ElementTree library is part of the Python standard library, in xml.etree.ElementTree. -
  2. The primary entry point for the ElementTree library is the parse() function, which can take a filename or a file-like object [FIXME xref]. This function parses the entire document at once. If memory is tight, there are ways to parse an XML document incrementally instead. +
  3. The primary entry point for the ElementTree library is the parse() function, which can take a filename or a file-like object [FIXME xref]. This function parses the entire document at once. If memory is tight, there are ways to parse an XML document incrementally instead. [FIXME href]
  4. The parse() function returns an object which represents the entire document. This is not the root element. To get a reference to the root element, call the getroot() method.
  5. As expected, the root element is the feed element in the http://www.w3.org/2005/Atom namespace. The string representation of this object reinforces an important point: an XML element is a combination of its namespace and its tag name (also called the local name). Every element in this document is in the Atom namespace, so the root element is represented as {http://www.w3.org/2005/Atom}feed.
@@ -407,26 +407,12 @@ mark{display:inline}
  • After doing some quick string formatting (because otherwise these compound queries get ridiculously long), this query searches for Atom author elements that have an Atom uri element as a child. This only returns two author elements, the ones in the first and second entry. The author in the last entry contains only a name, not a uri. -

    Overall, ElementTree’s findall() method is a very powerful feature, but the query language can be a bit surprising. It is officially described as “limited support for XPath expressions.” XPath is a W3C standard for querying XML documents. ElementTree’s query language is similar enough to XPath to do basic searching, but dissimilar enough that it may annoy you if you already know XPath. Now let’s look at a third-party XML library that extends the ElementTree API with full XPath support. - -

    Going Further With lxml

    - -

    lxml FIXME +

    What’s that? You say you want the power of the findall() method, but you want to work with an iterator instead of building a complete list? ElementTree can do that too. 

    ->>> from lxml import etree
-.
-.  FIXME (show how it's a drop-in replacement for everything we've done so far)
-.
-
    - -

    FIXME: from here on out, we use lxml.etree explicitly because these functions are specific to lxml - -

    ->>> import lxml.etree
->>> tree = lxml.etree.parse("examples/feed.xml")
->>> it = tree.iterfind("//{http://www.w3.org/2005/Atom}link")
->>> next(it)
+# continuing from the previous example
+>>> it = tree.getiterator("{http://www.w3.org/2005/Atom}link")  ①
+>>> next(it)                                                    ②
 <Element {http://www.w3.org/2005/Atom}link at 122f1b0>
 >>> next(it)
 <Element {http://www.w3.org/2005/Atom}link at 122f1e0>
@@ -438,32 +424,59 @@ mark{display:inline}
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 StopIteration
    +
      +
    1. The getiterator() method can zero or one arguments. If called with no arguments, it returns an iterator that spits out every element and child element in the entire document. Or, as shown here, you can call it with an element name in standard ElementTree format. This returns an iterator that spits out only elements of that name. +
    2. +
    + +

    Overall, ElementTree’s findall() method is a very powerful feature, but the query language can be a bit surprising. It is officially described as “limited support for XPath expressions.” XPath is a W3C standard for querying XML documents. ElementTree’s query language is similar enough to XPath to do basic searching, but dissimilar enough that it may annoy you if you already know XPath. Now let’s look at a third-party XML library that extends the ElementTree API with full XPath support. + +

    Going Further With lxml

    + +

    lxml is an open source third-party library that builds on the popular libxml2 parser [FIXME href]. It provides a 100% compatible ElementTree API, then extends it with full XPath support and a few other niceties. There are installers available for Windows and Mac OS X (FIXME really?); Linux users can probably use distribution-specific tools like yum or apt-get to install precompiled binaries from their repositories. 

    ->>> NSMAP = {"atom": "http://www.w3.org/2005/Atom"}
->>> entries = tree.xpath("//atom:category[@term='accessibility']/..", namespaces=NSMAP)
->>> entries
+>>> from lxml import etree                   ①
+>>> tree = etree.parse("examples/feed.xml")  ②
+>>> root = tree.getroot()                    ③
+>>> root.findall("{http://www.w3.org/2005/Atom}entry")  ④
+[<Element {http://www.w3.org/2005/Atom}entry at e2b4e0>,
+ <Element {http://www.w3.org/2005/Atom}entry at e2b510>,
+ <Element {http://www.w3.org/2005/Atom}entry at e2b540>]
    +
      +
    1. Once imported, lxml provides the same API as the built-in ElementTree libary. +
    2. parse() function: same as ElementTree. +
    3. getroot() method: also the same. +
    4. findall() method: exactly the same. +
    + +

    For large XML documents, lxml is significantly faster than the built-in ElementTree libary. If you’re only using the ElementTree API and want to use the fastest available implementation, you can try to import lxml and fall back to the built-in ElementTree. + +

    try:
+    from lxml import etree
+except ImportError:
+    import xml.etree.ElementTree as etree
    + +

    But lxml is more than just a faster ElementTree. It also integrates support for arbitrary XPath expressions. I’m not going to go into depth about XPath syntax (it can get quite complicated). [FIXME href] is a good beginner’s guide to XPath. + +

    +>>> import lxml.etree                                                  ① 
+>>> tree = lxml.etree.parse("examples/feed.xml")
+>>> NSMAP = {"atom": "http://www.w3.org/2005/Atom"}                    ②
+>>> entries = tree.xpath("//atom:category[@term='accessibility']/..",  ③
+...     namespaces=NSMAP)
+>>> entries                                                            ④
 [<Element {http://www.w3.org/2005/Atom}entry at e2b630>]
 >>> entry = entries[0]
->>> entry.xpath("./atom:title/text()", namespaces=nsmap)
+>>> entry.xpath("./atom:title/text()", namespaces=nsmap)               ⑤
 ['Accessibility is a harsh mistress']
    - -

    Customizing Your XML Parser

    - -

    FIXME - -

    ->>> import lxml.etree
->>> parser = lxml.etree.XMLParser(no_network=True, ns_clean=True, recover=True, remove_blank_text=True, remove_comments=True)
->>> tree = lxml.etree.parse("examples/feed.xml", parser)
-.
-.
-.
-
    - -

    Incremental Parsing

    - -

    FIXME +

      +
    1. In this example, I’m going to import lxml.etree (instead of, say, from lxml import etree), to emphasize that these features are specific to lxml. +
    2. To perform XPath queries on namespaced elements, you need to define a namespace prefix mapping. This is just a Python dictionary. +
    3. Here is an XPath query. The XPath expression searches for category elements (in the Atom namespace) that contain a term attribute with the value accessibility. But that’s not actually the query result. Look at the very end of the query string; did you notice the /.. bit? That means “and then return the parent element of the category element you just found.” So this single XPath query will find all entries with a child element of <category term="accessibility">. +
    4. The xpath() function returns a list of ElementTree objects. In this document, there is only one entry with a category whose term is accessibility. +
    5. XPath expressions don’t always return a list of elements. Technically, the DOM of a parsed XML document doesn’t contain elements; it contains nodes. Depending on their type, nodes can be elements, attributes, or even text content. The result of an XPath query is a list of nodes. This query returns a list of text nodes: the text content (text()) of the title element (atom:title) that is a child of the current element (./). +

    Generating XML

    @@ -534,6 +547,67 @@ StopIteration
  • You can also apply “pretty printing” to the serialization, which inserts line breaks after end tags, and after start tags of elements that contain child elements but no text content. In technical terms, lxml adds “insignificant whitespace” to make the output more readable. +

    Customizing Your XML Parser

    + +

    The XML specification mandates that all conforming XML parsers employ “draconian error handling.” That is, they must halt and catch fire as soon as they detect any sort of wellformedness error in the XML document. Wellformedness errors include mismatched start and end tags, undefined entities, illegal Unicode characters, and a number of other esoteric rules. This is in stark contrast to other common formats like HTML — your browser doesn’t stop rendering a web page if you forget to close an HTML tag or escape an ampersand in an attribute value. (It is a common misconception that HTML has no defined error handling. HTML error handling is actually quite well-defined [FIXME href], but it’s significantly more complicated than “halt and catch fire on first error.”) + +

    Some people (myself included) believe that it was a mistake for the inventors of XML to mandate draconian error handling. Don’t get me wrong; I can certainly see the allure of simplifying the error handling rules. But in practice, the concept of “wellformedness” is trickier than it sounds, especially for XML documents (like Atom feeds) that are published on the web and served over HTTP. Despite the maturity of XML, which standardized on draconian error handling in 1997, surveys continually show a significant fraction of Atom feeds on the web are plagued with wellformedness errors. + +

    So, I have both theoretical and practical reasons to parse XML documents “at any cost,” that is, not to halt and catch fire at the first wellformedness error. If you find yourself wanting to do this too, lxml can help. + +

    Here is a fragment of a broken XML document. I’ve highlighted the wellformedness error. + +

    <?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
+  <title>dive into …</title>
+...
+</feed>
    + +

    That’s an error, because the &hellip; entity is not defined in XML. (It is defined in HTML.) If you try to parse this broken feed with the default settings, lxml will choke on the undefined entity. + +

    +>>> import lxml.etree
+>>> tree = lxml.etree.parse("examples/feed-broken.xml")
+Traceback (most recent call last):
+  File "<stdin>", line 1, in <module>
+  File "lxml.etree.pyx", line 2693, in lxml.etree.parse (src/lxml/lxml.etree.c:52591)
+  File "parser.pxi", line 1478, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:75665)
+  File "parser.pxi", line 1507, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:75993)
+  File "parser.pxi", line 1407, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:75002)
+  File "parser.pxi", line 965, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:72023)
+  File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:67830)
+  File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:68877)
+  File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:68125)
+lxml.etree.XMLSyntaxError: Entity 'hellip' not defined, line 3, column 28
    + +

    To parse this broken XML document, despite its wellformedness error, you need to create a custom XML parser. + +

    +>>> parser = lxml.etree.XMLParser(recover=True)                  ①
+>>> tree = lxml.etree.parse("examples/feed-broken.xml", parser)  ②
+>>> parser.error_log                                             ③
+examples/feed-broken.xml:3:28:FATAL:PARSER:ERR_UNDECLARED_ENTITY: Entity 'hellip' not defined
+>>> tree.findall("{http://www.w3.org/2005/Atom}title")
+[<Element {http://www.w3.org/2005/Atom}title at ead510>]
+>>> title = tree.findall("{http://www.w3.org/2005/Atom}title")[0]
+>>> title.text                                                   ④
+'dive into '
+>>> print(lxml.etree.tounicode(tree.getroot()))                  ⑤
+<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
+  <title>dive into </title>
+.
+. [rest of serialization snipped for brevity]
+.
    +
      +
    1. To create a custom parser, instantiate the lxml.etree.XMLParser class. It can take a number of different named arguments [FIXME href]. The one we’re interested in here is the recover argument. When set to True, the XML parser will try its best to “recover” from wellformedness errors. +
    2. To parse an XML document with your custom parser, pass the parser object as the second argument to the parse() function. Note that lxml does not raise an exception about the undefined &hellip; entity. +
    3. The parser keeps a log of the wellformedness errors that it has encountered. (This is actually true regardless of whether it is set to recover from those errors or not.) +
    4. Since it didn’t know what to do with the undefined &hellip; entity, the parser just silently dropped it. The text content of the title element becomes "dive into ". +
    5. As you can see from the serialization, the &hellip; entity didn’t get moved; it was simply dropped. +
    + +

    It is important to reiterate that there is no guarantee of interoperability with “recovering” XML parsers. A different parser might decide that it recognized the &hellip; entity from HTML, and replace it with &amp;hellip; instead. Is that “better”? Maybe. Is it “more correct”? No, they are both equally incorrect. The correct behavior (according to the XML specification) is to halt and catch fire. If you’ve decided not to do that, you’re on your own. +

    Further Reading