diff --git a/xml.html b/xml.html index c90aa8d..45defe8 100644 --- a/xml.html +++ b/xml.html @@ -396,27 +396,6 @@ mark{display:inline}
  • The other three results are each entry-level alternate links. Each entry has a single link child element, and because of the double slash at the beginning of the query, this query finds all of them. -

    The findall() method has a few other tricks up its sleeve. - -

    -# continuing from the previous example
->>> tree.findall("//{http://www.w3.org/2005/Atom}*[@href]")                             ①
-[<Element {http://www.w3.org/2005/Atom}link at eeb8a0>,
- <Element {http://www.w3.org/2005/Atom}link at eeb990>,
- <Element {http://www.w3.org/2005/Atom}link at eeb960>,
- <Element {http://www.w3.org/2005/Atom}link at eeb9c0>]
->>> tree.findall("//{http://www.w3.org/2005/Atom}*[@href='http://diveintomark.org/']")  ②
-[<Element {http://www.w3.org/2005/Atom}link at eeb930>]
->>> NS = "{http://www.w3.org/2005/Atom}"
->>> tree.findall("//{NS}author[{NS}uri]".format(NS=NS))                                 ③
-[<Element {http://www.w3.org/2005/Atom}author at eeba80>,
- <Element {http://www.w3.org/2005/Atom}author at eebba0>]
    -
      -
    1. This query finds all elements in the Atom namespace, anywhere in the document, that have an href attribute. The // at the beginning of the query means “elements anywhere (not just as children of the root element).” {http://www.w3.org/2005/Atom} means “only elements in the Atom namespace.” * means “elements with any local name.” And [@href] means “has an href attribute.” -
    2. The query finds all Atom elements with an href whose value is http://diveintomark.org/. -
    3. After doing some quick string formatting (because otherwise these compound queries get ridiculously long), this query searches for Atom author elements that have an Atom uri element as a child. This only returns two author elements, the ones in the first and second entry. The author in the last entry contains only a name, not a uri. -
    -

    What’s that? You say you want the power of the findall() method, but you want to work with an iterator instead of building a complete list? ElementTree can do that too. 

    @@ -445,7 +424,7 @@ StopIteration

    Going Further With lxml

    -

    lxml is an open source third-party library that builds on the popular libxml2 parser. It provides a 100% compatible ElementTree API, then extends it with full XPath support and a few other niceties. There are installers available for Windows; Linux users should always try to use distribution-specific tools like yum or apt-get to install precompiled binaries from their repositories. Otherwise you’ll need to install lxml manually. +

    lxml is an open source third-party library that builds on the popular libxml2 parser. It provides a 100% compatible ElementTree API, then extends it with full XPath support and a few other niceties. There are installers available for Windows; Linux users should always try to use distribution-specific tools like yum or apt-get to install precompiled binaries from their repositories. Otherwise you’ll need to install lxml manually. 

     >>> from lxml import etree                   ①
@@ -456,34 +435,56 @@ StopIteration
    <Element {http://www.w3.org/2005/Atom}entry at e2b510>, <Element {http://www.w3.org/2005/Atom}entry at e2b540>]
      -
    1. Once imported, lxml provides the same API as the built-in ElementTree libary. +
    2. Once imported, lxml provides the same API as the built-in ElementTree libary.
    3. parse() function: same as ElementTree.
    4. getroot() method: also the same.
    5. findall() method: exactly the same.
    -

    For large XML documents, lxml is significantly faster than the built-in ElementTree libary. If you’re only using the ElementTree API and want to use the fastest available implementation, you can try to import lxml and fall back to the built-in ElementTree. +

    For large XML documents, lxml is significantly faster than the built-in ElementTree libary. If you’re only using the ElementTree API and want to use the fastest available implementation, you can try to import lxml and fall back to the built-in ElementTree. 

    try:
     from lxml import etree
 except ImportError:
     import xml.etree.ElementTree as etree
    -

    But lxml is more than just a faster ElementTree. It also integrates support for arbitrary XPath expressions. I’m not going to go into depth about XPath syntax. (That could be a whole book unto itself!) But I will show you how it integrates into lxml. +

    But lxml is more than just a faster ElementTree. Its findall() method includes support for more complicated expressions. 

    ->>> import lxml.etree                                                  ① 
+>>> import lxml.etree                                                                   ①
 >>> tree = lxml.etree.parse("examples/feed.xml")
->>> NSMAP = {"atom": "http://www.w3.org/2005/Atom"}                    ②
->>> entries = tree.xpath("//atom:category[@term='accessibility']/..",  ③
+>>> tree.findall("//{http://www.w3.org/2005/Atom}*[@href]")                             ②
+[<Element {http://www.w3.org/2005/Atom}link at eeb8a0>,
+ <Element {http://www.w3.org/2005/Atom}link at eeb990>,
+ <Element {http://www.w3.org/2005/Atom}link at eeb960>,
+ <Element {http://www.w3.org/2005/Atom}link at eeb9c0>]
+>>> tree.findall("//{http://www.w3.org/2005/Atom}*[@href='http://diveintomark.org/']")  ③
+[<Element {http://www.w3.org/2005/Atom}link at eeb930>]
+>>> NS = "{http://www.w3.org/2005/Atom}"
+>>> tree.findall("//{NS}author[{NS}uri]".format(NS=NS))                                 ④
+[<Element {http://www.w3.org/2005/Atom}author at eeba80>,
+ <Element {http://www.w3.org/2005/Atom}author at eebba0>]
    +
      +
    1. In this example, I’m going to import lxml.etree (instead of, say, from lxml import etree), to emphasize that these features are specific to lxml. +
    2. This query finds all elements in the Atom namespace, anywhere in the document, that have an href attribute. The // at the beginning of the query means “elements anywhere (not just as children of the root element).” {http://www.w3.org/2005/Atom} means “only elements in the Atom namespace.” * means “elements with any local name.” And [@href] means “has an href attribute.” +
    3. The query finds all Atom elements with an href whose value is http://diveintomark.org/. +
    4. After doing some quick string formatting (because otherwise these compound queries get ridiculously long), this query searches for Atom author elements that have an Atom uri element as a child. This only returns two author elements, the ones in the first and second entry. The author in the last entry contains only a name, not a uri. +
    + +

    Not enough for you? lxml also integrates support for arbitrary XPath expressions. I’m not going to go into depth about XPath syntax; that could be a whole book unto itself! But I will show you how it integrates into lxml. + +

    +>>> import lxml.etree
+>>> tree = lxml.etree.parse("examples/feed.xml")
+>>> NSMAP = {"atom": "http://www.w3.org/2005/Atom"}                    ①
+>>> entries = tree.xpath("//atom:category[@term='accessibility']/..",  ②
 ...     namespaces=NSMAP)
->>> entries                                                            ④
+>>> entries                                                            ③
 [<Element {http://www.w3.org/2005/Atom}entry at e2b630>]
 >>> entry = entries[0]
->>> entry.xpath("./atom:title/text()", namespaces=nsmap)               ⑤
+>>> entry.xpath("./atom:title/text()", namespaces=nsmap)               ④
 ['Accessibility is a harsh mistress']
      -
    1. In this example, I’m going to import lxml.etree (instead of, say, from lxml import etree), to emphasize that these features are specific to lxml.
    2. To perform XPath queries on namespaced elements, you need to define a namespace prefix mapping. This is just a Python dictionary.
    3. Here is an XPath query. The XPath expression searches for category elements (in the Atom namespace) that contain a term attribute with the value accessibility. But that’s not actually the query result. Look at the very end of the query string; did you notice the /.. bit? That means “and then return the parent element of the category element you just found.” So this single XPath query will find all entries with a child element of <category term="accessibility">.
    4. The xpath() function returns a list of ElementTree objects. In this document, there is only one entry with a category whose term is accessibility. @@ -520,7 +521,7 @@ except ImportError:

      The only practical difference is that the second serialization is several characters shorter. If we were to recast our entire sample feed with a ns0: prefix in every start and end tag, it would add 4 characters per start tag × 79 tags + 4 characters for the namespace declaration itself, for a total of 316 characters. Assuming UTF-8 encoding, that’s 316 extra bytes. (After gzipping, the difference drops to 21 bytes, but still, 21 bytes is 21 bytes.) Maybe that doesn’t matter to you, but for something like an Atom feed, which may be downloaded several thousand times whenever it changes, saving a few bytes per request can quickly add up. -

      The built-in ElementTree library does not offer this fine-grained control over serializing namespaced elements, but lxml does. +

      The built-in ElementTree library does not offer this fine-grained control over serializing namespaced elements, but lxml does. 

       >>> import lxml.etree
@@ -533,9 +534,9 @@ except ImportError:
 <feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"/>
      1. To start, define a namespace mapping as a dictionary. Dictionary values are namespaces; dictionary keys are the desired prefix. Using None as a prefix effectively declares a default namespace. -
      2. Now you can pass the lxml-specific nsmap argument when you create an element, and lxml will respect the namespace prefixes you’ve defined. +
      3. Now you can pass the lxml-specific nsmap argument when you create an element, and lxml will respect the namespace prefixes you’ve defined.
      4. As expected, this serialization defines the Atom namespace as the default namespace and declares the feed element without a namespace prefix. -
      5. Oops, we forgot to add the xml:lang attribute. You can always add attributes to any element with the set() method. It takes two arguments: the attribute name in standard ElementTree format, then the attribute value. (This method is not lxml-specific. The only lxml-specific part of this example was the nsmap argument to control the namespace prefixes in the serialized output.) +
      6. Oops, we forgot to add the xml:lang attribute. You can always add attributes to any element with the set() method. It takes two arguments: the attribute name in standard ElementTree format, then the attribute value. (This method is not lxml-specific. The only lxml-specific part of this example was the nsmap argument to control the namespace prefixes in the serialized output.)

      Are XML documents limited to one element per document? No, of course not. You can easily create child elements, too. @@ -555,21 +556,21 @@ except ImportError:

      1. To create a child element of an existing element, instantiate the SubElement class. The only required arguments are the parent element (new_feed in this case) and the new element’s name. Since this child element will inherit the namespace mapping of its parent, there is no need to redeclare the namespace or prefix here.
      2. You can also pass in an attribute dictionary. Keys are attribute names; values are attribute values. -
      3. As expected, the new title element was created in the Atom namespace, and it was inserted as a child of the feed element. Since the title element has no text content and no children of its own, lxml serializes it as an empty element (with the /> shortcut). +
      4. As expected, the new title element was created in the Atom namespace, and it was inserted as a child of the feed element. Since the title element has no text content and no children of its own, lxml serializes it as an empty element (with the /> shortcut).
      5. To set the text content of an element, simply set its .text property. -
      6. Now the title element is serialized with its text content. Any text content that contains less-than signs or ampersands needs to be escaped when serialized. lxml handles this escaping automatically. -
      7. You can also apply “pretty printing” to the serialization, which inserts line breaks after end tags, and after start tags of elements that contain child elements but no text content. In technical terms, lxml adds “insignificant whitespace” to make the output more readable. +
      8. Now the title element is serialized with its text content. Any text content that contains less-than signs or ampersands needs to be escaped when serialized. lxml handles this escaping automatically. +
      9. You can also apply “pretty printing” to the serialization, which inserts line breaks after end tags, and after start tags of elements that contain child elements but no text content. In technical terms, lxml adds “insignificant whitespace” to make the output more readable.

      ⁂ -

      Customizing Your XML Parser

      +

      Parsing Broken XML

      The XML specification mandates that all conforming XML parsers employ “draconian error handling.” That is, they must halt and catch fire as soon as they detect any sort of wellformedness error in the XML document. Wellformedness errors include mismatched start and end tags, undefined entities, illegal Unicode characters, and a number of other esoteric rules. This is in stark contrast to other common formats like HTML — your browser doesn’t stop rendering a web page if you forget to close an HTML tag or escape an ampersand in an attribute value. (It is a common misconception that HTML has no defined error handling. HTML error handling is actually quite well-defined, but it’s significantly more complicated than “halt and catch fire on first error.”)

      Some people (myself included) believe that it was a mistake for the inventors of XML to mandate draconian error handling. Don’t get me wrong; I can certainly see the allure of simplifying the error handling rules. But in practice, the concept of “wellformedness” is trickier than it sounds, especially for XML documents (like Atom feeds) that are published on the web and served over HTTP. Despite the maturity of XML, which standardized on draconian error handling in 1997, surveys continually show a significant fraction of Atom feeds on the web are plagued with wellformedness errors. -

      So, I have both theoretical and practical reasons to parse XML documents “at any cost,” that is, not to halt and catch fire at the first wellformedness error. If you find yourself wanting to do this too, lxml can help. +

      So, I have both theoretical and practical reasons to parse XML documents “at any cost,” that is, not to halt and catch fire at the first wellformedness error. If you find yourself wanting to do this too, lxml can help.

      Here is a fragment of a broken XML document. I’ve highlighted the wellformedness error. @@ -579,7 +580,7 @@ except ImportError: ... </feed> -

      That’s an error, because the &hellip; entity is not defined in XML. (It is defined in HTML.) If you try to parse this broken feed with the default settings, lxml will choke on the undefined entity. +

      That’s an error, because the &hellip; entity is not defined in XML. (It is defined in HTML.) If you try to parse this broken feed with the default settings, lxml will choke on the undefined entity. 

       >>> import lxml.etree
@@ -616,7 +617,7 @@ lxml.etree.XMLSyntaxError: Entity 'hellip' not defined, line 3, column 28
 .
      1. To create a custom parser, instantiate the lxml.etree.XMLParser class. It can take a number of different named arguments. The one we’re interested in here is the recover argument. When set to True, the XML parser will try its best to “recover” from wellformedness errors. -
      2. To parse an XML document with your custom parser, pass the parser object as the second argument to the parse() function. Note that lxml does not raise an exception about the undefined &hellip; entity. +
      3. To parse an XML document with your custom parser, pass the parser object as the second argument to the parse() function. Note that lxml does not raise an exception about the undefined &hellip; entity.
      4. The parser keeps a log of the wellformedness errors that it has encountered. (This is actually true regardless of whether it is set to recover from those errors or not.)
      5. Since it didn’t know what to do with the undefined &hellip; entity, the parser just silently dropped it. The text content of the title element becomes "dive into ".
      6. As you can see from the serialization, the &hellip; entity didn’t get moved; it was simply dropped. @@ -634,9 +635,9 @@ lxml.etree.XMLSyntaxError: Entity 'hellip' not defined, line 3, column 28
      7. Elements and Element Trees
      8. XPath Support in ElementTree
      9. The ElementTree iterparse Function -
      10. lxml -
      11. Parsing XML and HTML with lxml -
      12. XPath and XSLT with lxml +
      13. lxml +
      14. Parsing XML and HTML with lxml +
      15. XPath and XSLT with lxml

        © 2001–9 Mark Pilgrim