# Dive Into Python

20 May 2004

Copyright © 2000, 2001, 2002, 2003, 2004 Mark Pilgrim

This book lives at http://diveintopython3.org/. If you're reading it somewhere else, you may not have the latest version.

Table of Contents

- 1. Installing Python
  - 1.1. Which Python is right for you?
  - 1.2. Python on Windows
  - 1.3. Python on Mac OS X
  - 1.4. Python on Mac OS 9
  - 1.5. Python on RedHat Linux
  - 1.6. Python on Debian GNU/Linux
  - 1.7. Python Installation from Source
  - 1.8. The Interactive Shell
  - 1.9. Summary
- 3.3. Introducing Tuples
- 3.4. Declaring variables
  - 3.4.1. Referencing Variables
  - 3.4.2. Assigning Multiple Values at Once
- 4. The Power Of Introspection
  - 4.1. Diving In
  - 4.2. Using Optional and Named Arguments
  - 4.3. Using type, str, dir, and Other Built-In Functions
    - 4.3.1. The type Function
    - 4.3.2. The str Function
    - 4.3.3. Built-In Functions
  - 4.4. Getting Object References With getattr
    - 4.4.1. getattr with Modules
    - 4.4.2. getattr As a Dispatcher
  - 4.5. Filtering Lists
  - 4.6. The Peculiar Nature of and and or
    - 4.6.1. Using the and-or Trick
  - 4.8. Putting It All Together
  - 4.9. Summary
- 5. Objects and Object-Orientation
  - 5.1. Diving In
  - 5.2. Importing Modules Using from module import
  - 5.3. Defining Classes
    - 5.3.1. Initializing and Coding Classes
    - 5.3.2. Knowing When to Use self and __init__
  - 5.4. Instantiating Classes
    - 5.4.1. Garbage Collection
  - 5.5. Exploring UserDict: A Wrapper Class
  - 5.6. Special Class Methods
    - 5.6.1. Getting and Setting Items
  - 5.7. Advanced Special Class Methods
  - 5.8. Introducing Class Attributes
  - 5.9. Private Functions
  - 5.10. Summary
- 6. Exceptions and File Handling
  - 6.1. Handling Exceptions
    - 6.1.1. Using Exceptions For Other Purposes
  - 6.2. Working with File Objects
    - 6.2.1. Reading Files
    - 6.2.2. Closing Files
    - 6.2.3. Handling I/O Errors
    - 6.2.4. Writing to Files
  - 6.3. Iterating with for Loops
  - 6.4. Using sys.modules
  - 6.5. Working with Directories
  - 6.6. Putting It All Together
  - 6.7. Summary
- 8. HTML Processing
  - 8.1. Diving in
  - 8.2. Introducing sgmllib.py
  - 8.3. Extracting data from HTML documents
  - 8.4. Introducing BaseHTMLProcessor.py
  - 8.5. locals and globals
  - 8.6. Dictionary-based string formatting
  - 8.7. Quoting attribute values
  - 8.8. Introducing dialect.py
  - 8.9. Putting it all together
  - 8.10. Summary
- 9. XML Processing
  - 9.1. Diving in
  - 9.2. Packages
  - 9.3. Parsing XML
  - 9.5. Searching for elements
  - 9.6. Accessing element attributes
  - 9.7. Segue
- 10. Scripts and Streams
  - 10.1. Abstracting input sources
  - 10.2. Standard input, output, and error
  - 10.3. Caching node lookups
  - 10.4. Finding direct children of a node
  - 10.5. Creating separate handlers by node type
  - 10.6. Handling command-line arguments
  - 10.7. Putting it all together
  - 10.8. Summary
- 11. HTTP Web Services
  - 11.1. Diving in
  - 11.2. How not to fetch data over HTTP
  - 11.3. Features of HTTP
    - 11.3.1. User-Agent
    - 11.3.2. Redirects
    - 11.3.3. Last-Modified/If-Modified-Since
    - 11.3.4. ETag/If-None-Match
    - 11.3.5. Compression
  - 11.4. Debugging HTTP web services
  - 11.5. Setting the User-Agent
  - 11.6. Handling Last-Modified and ETag
  - 11.7. Handling redirects
  - 11.8. Handling compressed data
  - 11.9. Putting it all together
  - 11.10. Summary
- 13. Unit Testing
  - 13.1. Introduction to Roman numerals
  - 13.2. Diving in
  - 13.3. Introducing romantest.py
  - 13.4. Testing for success
  - 13.5. Testing for failure
  - 13.6. Testing for sanity
- 14. Test-First Programming
  - 14.1. roman.py, stage 1
  - 14.2. roman.py, stage 2
  - 14.3. roman.py, stage 3
  - 14.4. roman.py, stage 4
  - 14.5. roman.py, stage 5
- 15. Refactoring
  - 15.1. Handling bugs
  - 15.2. Handling changing requirements
  - 15.3. Refactoring
  - 15.4. Postscript
  - 15.5. Summary
- 16. Functional Programming
  - 16.1. Diving in
  - 16.2. Finding the path
  - 16.3. Filtering lists revisited
  - 16.4. Mapping lists revisited
  - 16.5. Data-centric programming
  - 16.6. Dynamically importing modules
  - 16.7. Putting it all together
  - 16.8. Summary
- 17. Dynamic functions
  - 17.1. Diving in
  - 17.2. plural.py, stage 1
  - 17.3. plural.py, stage 2
  - 17.4. plural.py, stage 3
  - 17.5. plural.py, stage 4
  - 17.6. plural.py, stage 5
  - 17.7. plural.py, stage 6
  - 17.8. Summary
- 18. Performance Tuning
  - 18.1. Diving in
  - 18.2. Using the timeit Module
  - 18.3. Optimizing Regular Expressions
  - 18.4. Optimizing Dictionary Lookups
  - 18.5. Optimizing List Operations
  - 18.6. Optimizing String Manipulation
  - 18.7. Summary

## Chapter 1. Installing Python

Welcome to Python. Let's dive in. In this chapter, you'll install the version of Python that's right for you.

## 1.1. Which Python is right for you?

The first thing you need to do with Python is install it. Or do you?

If you're using an account on a hosted server, your ISP may have already installed Python. Most popular Linux distributions come with Python in the default installation. Mac OS X 10.2 and later includes a command-line version of Python, although you'll probably want to install a version that includes a more Mac-like graphical interface.

Windows does not come with any version of Python, but don't despair! There are several ways to point-and-click your way to Python on Windows.

As you can see already, Python runs on a great many operating systems. The full list includes Windows, Mac OS, Mac OS X, and all varieties of free UNIX-compatible systems like Linux. There are also versions that run on Sun Solaris, AS/400, Amiga, OS/2, BeOS, and a plethora of other platforms you've probably never even heard of.

What's more, Python programs written on one platform can, with a little care, run on any supported platform. For instance, I regularly develop Python programs on Windows and later deploy them on Linux.

So back to the question that started this section, “Which Python is right for you?” The answer is whichever one runs on the computer you already have.

## 1.2. Python on Windows

On Windows, you have a couple choices for installing Python.

ActiveState makes a Windows installer for Python called ActivePython, which includes a complete version of Python, an IDE with a Python-aware code editor, plus some Windows extensions for Python that allow complete access to Windows-specific services, APIs, and the Windows Registry.

ActivePython is freely downloadable, although it is not open source. It is the IDE I used to learn Python, and I recommend you try it unless you have a specific reason not to. One such reason might be that ActiveState is generally several months behind in updating their ActivePython installer when new version of Python are released. If you absolutely need the latest version of Python and ActivePython is still a version behind as you read this, you'll want to use the second option for installing Python on Windows.

The second option is the “official” Python installer, distributed by the people who develop Python itself. It is freely downloadable and open source, and it is always current with the latest version of Python.

### Procedure 1.1. Option 1: Installing ActivePython

Here is the procedure for installing ActivePython:

1. Download ActivePython from http://www.activestate.com/Products/ActivePython/.

2. If you are using Windows 95, Windows 98, or Windows ME, you will also need to download and install Windows Installer 2.0 before installing ActivePython.

3. Double-click the installer, `ActivePython-2.2.2-224-win32-ix86.msi`.

4. Step through the installer program.

5. If space is tight, you can do a custom installation and deselect the documentation, but I don't recommend this unless you absolutely can't spare the 14MB.

6. After the installation is complete, close the installer and choose Start->Programs->ActiveState ActivePython 2.2->PythonWin IDE. You'll see something like the following:

    PythonWin 2.2.2 (#37, Nov 26 2002, 10:24:37) [MSC 32 bit (Intel)] on win32.
    Portions Copyright 1994-2001 Mark Hammond (mhammond@skippinet.com.au) -
    see 'Help/About PythonWin' for further copyright information.
    >>>

### Procedure 1.2. Option 2: Installing Python from Python.org

1. Download the latest Python Windows installer by going to http://www.python.org/ftp/python/ and selecting the highest version number listed, then downloading the `.exe` installer.

2. Double-click the installer, `Python-2.xxx.yyy.exe`. The name will depend on the version of Python available when you read this.

3. Step through the installer program.

4. If disk space is tight, you can deselect the HTMLHelp file, the utility scripts (`Tools/`), and/or the test suite (`Lib/test/`).

5. If you do not have administrative rights on your machine, you can select Advanced Options, then choose Non-Admin Install. This just affects where Registry entries and Start menu shortcuts are created.

6. After the installation is complete, close the installer and select Start->Programs->Python 2.3->IDLE (Python GUI). You'll see something like the following:

    Python 2.3.2 (#49, Oct  2 2003, 20:02:00) [MSC v.1200 32 bit (Intel)] on win32
    Type "copyright", "credits" or "license()" for more information.

        ****************************************************************
        Personal firewall software may warn about the connection IDLE
        makes to its subprocess using this computer's internal loopback
        interface.  This connection is not visible on any external
        interface and no data is sent to or received from the Internet.
        ****************************************************************

    IDLE 1.0
    >>>

## 1.3. Python on Mac OS X

On Mac OS X, you have two choices for installing Python: install it, or don't install it. You probably want to install it.

Mac OS X 10.2 and later comes with a command-line version of Python preinstalled. If you are comfortable with the command line, you can use this version for the first third of the book. However, the preinstalled version does not come with an XML parser, so when you get to the XML chapter, you'll need to install the full version.

Rather than using the preinstalled version, you'll probably want to install the latest version, which also comes with a graphical interactive shell.

### Procedure 1.3. Running the Preinstalled Version of Python on Mac OS X

To use the preinstalled version of Python, follow these steps:

1. Open the `/Applications` folder.

2. Open the `Utilities` folder.

3. Double-click `Terminal` to open a terminal window and get to a command line.

4. Type `python` at the command prompt.

Try it out:

    Welcome to Darwin!
    [localhost:~] you% python
    Python 2.2 (#1, 07/14/02, 23:25:09)
    [GCC Apple cpp-precomp 6.14] on darwin
    Type "help", "copyright", "credits", or "license" for more information.
    >>> [press Ctrl+D to get back to the command prompt]
    [localhost:~] you%

### Procedure 1.4. Installing the Latest Version of Python on Mac OS X

Follow these steps to download and install the latest version of Python:

1. Download the `MacPython-OSX` disk image from http://homepages.cwi.nl/~jack/macpython/download.html.

2. If your browser has not already done so, double-click `MacPython-OSX-2.3-1.dmg` to mount the disk image on your desktop.

3. Double-click the installer, `MacPython-OSX.pkg`.

4. The installer will prompt you for your administrative username and password.

5. Step through the installer program.

6. After installation is complete, close the installer and open the `/Applications` folder.

7. Open the `MacPython-2.3` folder

8. Double-click `PythonIDE` to launch Python.

The MacPython IDE should display a splash screen, then take you to the interactive shell. If the interactive shell does not appear, select Window->Python Interactive (`Cmd-0`). The opening window will look something like this:

    Python 2.3 (#2, Jul 30 2003, 11:45:28)
    [GCC 3.1 20020420 (prerelease)]
    Type "copyright", "credits" or "license" for more information.
    MacPython IDE 1.0.1
    >>>

Note that once you install the latest version, the pre-installed version is still present. If you are running scripts from the command line, you need to be aware which version of Python you are using.

### Example 1.1. Two versions of Python

    [localhost:~] you% python
    Python 2.2 (#1, 07/14/02, 23:25:09)
    [GCC Apple cpp-precomp 6.14] on darwin
    Type "help", "copyright", "credits", or "license" for more information.
    >>> [press Ctrl+D to get back to the command prompt]
    [localhost:~] you% /usr/local/bin/python
    Python 2.3 (#2, Jul 30 2003, 11:45:28)
    [GCC 3.1 20020420 (prerelease)] on darwin
    Type "help", "copyright", "credits", or "license" for more information.
    >>> [press Ctrl+D to get back to the command prompt]
    [localhost:~] you%

## 1.4. Python on Mac OS 9

Mac OS 9 does not come with any version of Python, but installation is very simple, and there is only one choice.

Follow these steps to install Python on Mac OS 9:

1. Download the `MacPython23full.bin` file from http://homepages.cwi.nl/~jack/macpython/download.html.

2. If your browser does not decompress the file automatically, double-click `MacPython23full.bin` to decompress the file with Stuffit Expander.

3. Double-click the installer, `MacPython23full`.

4. Step through the installer program.

5. AFter installation is complete, close the installer and open the `/Applications` folder.

6. Open the `MacPython-OS9 2.3` folder.

7. Double-click `Python IDE` to launch Python.

The MacPython IDE should display a splash screen, and then take you to the interactive shell. If the interactive shell does not appear, select Window->Python Interactive (`Cmd-0`). You'll see a screen like this:

    Python 2.3 (#2, Jul 30 2003, 11:45:28)
    [GCC 3.1 20020420 (prerelease)]
    Type "copyright", "credits" or "license" for more information.
    MacPython IDE 1.0.1
    >>>

## 1.5. Python on RedHat Linux

Installing under UNIX-compatible operating systems such as Linux is easy if you're willing to install a binary package. Pre-built binary packages are available for most popular Linux distributions. Or you can always compile from source.

Download the latest Python RPM by going to http://www.python.org/ftp/python/ and selecting the highest version number listed, then selecting the `rpms/` directory within that. Then download the RPM with the highest version number. You can install it with the `rpm` command, as shown here:

### Example 1.2. Installing on RedHat Linux 9

    localhost:~$ su -
    Password: [enter your root password]
    [root@localhost root]# wget http://python.org/ftp/python/2.3/rpms/redhat-9/python2.3-2.3-5pydotorg.i386.rpm
    Resolving python.org... done.
    Connecting to python.org[194.109.137.226]:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 7,495,111 [application/octet-stream]
    ...
    [root@localhost root]# rpm -Uvh python2.3-2.3-5pydotorg.i386.rpm
    Preparing...                ########################################### [100%]
       1:python2.3              ########################################### [100%]
    [root@localhost root]# python          ①
    Python 2.2.2 (#1, Feb 24 2003, 19:13:11)
    [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2
    Type "help", "copyright", "credits", or "license" for more information.
    >>> [press Ctrl+D to exit]
    [root@localhost root]# python2.3       ②
    Python 2.3 (#1, Sep 12 2003, 10:53:56)
    [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
    Type "help", "copyright", "credits", or "license" for more information.
    >>> [press Ctrl+D to exit]
    [root@localhost root]# which python2.3 ③
    /usr/bin/python2.3

1. Whoops! Just typing `python` gives you the older version of Python -- the one that was installed by default. That's not the one you want.
2. At the time of this writing, the newest version is called `python2.3`. You'll probably want to change the path on the first line of the sample scripts to point to the newer version.
3. This is the complete path of the newer version of Python that you just installed. Use this on the `#!` line (the first line of each script) to ensure that scripts are running under the latest version of Python, and be sure to type `python2.3` to get into the interactive shell.

   ## 1.6. Python on Debian GNU/Linux

   If you are lucky enough to be running Debian GNU/Linux, you install Python through the `apt` command.

   ### Example 1.3. Installing on Debian GNU/Linux

       localhost:~$ su -
       Password: [enter your root password]
       localhost:~# apt-get install python
       Reading Package Lists... Done
       Building Dependency Tree... Done
       The following extra packages will be installed:
         python2.3
       Suggested packages:
         python-tk python2.3-doc
       The following NEW packages will be installed:
         python python2.3
       0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.
       Need to get 0B/2880kB of archives.
       After unpacking 9351kB of additional disk space will be used.
       Do you want to continue? [Y/n] Y
       Selecting previously deselected package python2.3.
       (Reading database ... 22848 files and directories currently installed.)
       Unpacking python2.3 (from .../python2.3_2.3.1-1_i386.deb) ...
       Selecting previously deselected package python.
       Unpacking python (from .../python_2.3.1-1_all.deb) ...
       Setting up python (2.3.1-1) ...
       Setting up python2.3 (2.3.1-1) ...
       Compiling python modules in /usr/lib/python2.3 ...
       Compiling optimized python modules in /usr/lib/python2.3 ...
       localhost:~# exit
       logout
       localhost:~$ python
       Python 2.3.1 (#2, Sep 24 2003, 11:39:14)
       [GCC 3.3.2 20030908 (Debian prerelease)] on linux2
       Type "help", "copyright", "credits" or "license" for more information.
       >>> [press Ctrl+D to exit]

   ## 1.7. Python Installation from Source

   If you prefer to build from source, you can download the Python source code from http://www.python.org/ftp/python/. Select the highest version number listed, download the `.tgz` file), and then do the usual `configure`, `make`, `make install` dance.

   ### Example 1.4. Installing from source

       localhost:~$ su -
       Password: [enter your root password]
       localhost:~# wget http://www.python.org/ftp/python/2.3/Python-2.3.tgz
       Resolving www.python.org... done.
       Connecting to www.python.org[194.109.137.226]:80... connected.
       HTTP request sent, awaiting response... 200 OK
       Length: 8,436,880 [application/x-tar]
       ...
       localhost:~# tar xfz Python-2.3.tgz
       localhost:~# cd Python-2.3
       localhost:~/Python-2.3# ./configure
       checking MACHDEP... linux2
       checking EXTRAPLATDIR...
       checking for --without-gcc... no
       ...
       localhost:~/Python-2.3# make
       gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes
       -I. -I./Include  -DPy_BUILD_CORE -o Modules/python.o Modules/python.c
       gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes
       -I. -I./Include  -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
       gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes
       -I. -I./Include  -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c
       ...
       localhost:~/Python-2.3# make install
       /usr/bin/install -c python /usr/local/bin/python2.3
       ...
       localhost:~/Python-2.3# exit
       logout
       localhost:~$ which python
       /usr/local/bin/python
       localhost:~$ python
       Python 2.3.1 (#2, Sep 24 2003, 11:39:14)
       [GCC 3.3.2 20030908 (Debian prerelease)] on linux2
       Type "help", "copyright", "credits" or "license" for more information.
       >>> [press Ctrl+D to get back to the command prompt]
       localhost:~$

   ## 1.8. The Interactive Shell

   Now that you have Python installed, what's this interactive shell thing you're running?

   It's like this: Python leads a double life. It's an interpreter for scripts that you can run from the command line or run like applications, by double-clicking the scripts. But it's also an interactive shell that can evaluate arbitrary statements and expressions. This is extremely useful for debugging, quick hacking, and testing. I even know some people who use the Python interactive shell in lieu of a calculator!

   Launch the Python interactive shell in whatever way works on your platform, and let's dive in with the steps shown here:

   ### Example 1.5. First Steps in the Interactive Shell

       >>> 1 + 1               ①
       2
       >>> print 'hello world' ②
       hello world
       >>> x = 1               ③
       >>> y = 2
       >>> x + y
       3

   1. The Python interactive shell can evaluate arbitrary Python expressions, including any basic arithmetic expression.
   2. The interactive shell can execute arbitrary Python statements, including the `print` statement.
   3. You can also assign values to variables, and the values will be remembered as long as the shell is open (but not any longer than that).

      ## 1.9. Summary

      You should now have a version of Python installed that works for you.

      Depending on your platform, you may have more than one version of Python intsalled. If so, you need to be aware of your paths. If simply typing `python` on the command line doesn't run the version of Python that you want to use, you may need to enter the full pathname of your preferred version.

      Congratulations, and welcome to Python.

      ## 2.3. Documenting Functions

      You can document a Python function by giving it a `docstring`.

      ### Example 2.2. Defining the `buildConnectionString` Function's `docstring`

          def buildConnectionString(params):
              """Build a connection string from a dictionary of parameters.

              Returns string."""

      Triple quotes signify a multi-line string. Everything between the start and end quotes is part of a single string, including carriage returns and other quote characters. You can use them anywhere, but you'll see them most often used when defining a `docstring`.

      Triple quotes are also an easy way to define a string with both single and double quotes, like `qq/.../` in Perl.

      Everything between the triple quotes is the function's `docstring`, which documents what the function does. A `docstring`, if it exists, must be the first thing defined in a function (that is, the first thing after the colon). You don't technically need to give your function a `docstring`, but you always should. I know you've heard this in every programming class you've ever taken, but Python gives you an added incentive: the `docstring` is available at runtime as an attribute of the function.

      Many Python IDEs use the `docstring` to provide context-sensitive documentation, so that when you type a function name, its `docstring` appears as a tooltip. This can be incredibly helpful, but it's only as good as the `docstring`s you write.

      ## 2.4. Everything Is an Object

      ## 2.6. Testing Modules

      Python modules are objects and have several useful attributes. You can use this to easily test your modules as you write them. Here's an example that uses the `if` `__name__` trick.

          if __name__ == "__main__":

      Some quick observations before you get to the good stuff. First, parentheses are not required around the `if` expression. Second, the `if` statement ends with a colon, and is followed by indented code.

      Like C, Python uses `==` for comparison and `=` for assignment. Unlike C, Python does not support in-line assignment, so there's no chance of accidentally assigning the value you thought you were comparing.

      So why is this particular `if` statement a trick? Modules are objects, and all modules have a built-in attribute `__name__`. A module's `__name__` depends on how you're using the module. If you `import` the module, then `__name__` is the module's filename, without a directory path or file extension. But you can also run the module directly as a standalone program, in which case `__name__` will be a special default value, `__main__`.

          >>> import odbchelper
          >>> odbchelper.__name__
          'odbchelper'

      Knowing this, you can design a test suite for your module within the module itself by putting it in this `if` statement. When you run the module directly, `__name__` is `__main__`, so the test suite executes. When you import the module, `__name__` is something else, so the test suite is ignored. This makes it easier to develop and debug new modules before integrating them into a larger program.

      On MacPython, there is an additional step to make the `if` `__name__` trick work. Pop up the module's options menu by clicking the black triangle in the upper-right corner of the window, and make sure Run as __main__ is checked.

      ### Further Reading on Importing Modules

      - Python Reference Manual discusses the low-level details of importing modules.

      ## 3.4. Declaring variables

      Now that you know something about dictionaries, tuples, and lists (oh my!), let's get back to the sample program from Chapter 2, `odbchelper.py`.

      Python has local and global variables like most other languages, but it has no explicit variable declarations. Variables spring into existence by being assigned a value, and they are automatically destroyed when they go out of scope.

      ### Example 3.17. Defining the myParams Variable

          if __name__ == "__main__":
              myParams = {"server":"mpilgrim", \
                          "database":"master", \
                          "uid":"sa", \
                          "pwd":"secret" \
                          }

      Notice the indentation. An `if` statement is a code block and needs to be indented just like a function.

      Also notice that the variable assignment is one command split over several lines, with a backslash (“`\`”) serving as a line-continuation marker.

      When a command is split among several lines with the line-continuation marker (“`\`”), the continued lines can be indented in any manner; Python's normally stringent indentation rules do not apply. If your Python IDE auto-indents the continued line, you should probably accept its default unless you have a burning reason not to.

      Strictly speaking, expressions in parentheses, straight brackets, or curly braces (like defining a dictionary) can be split into multiple lines with or without the line continuation character (“`\`”). I like to include the backslash even when it's not required because I think it makes the code easier to read, but that's a matter of style.

      Third, you never declared the variable myParams, you just assigned a value to it. This is like VBScript without the `option explicit` option. Luckily, unlike VBScript, Python will not allow you to reference a variable that has never been assigned a value; trying to do so will raise an exception.

      ### 3.4.1. Referencing Variables

      ### Example 3.18. Referencing an Unbound Variable

          >>> x
          Traceback (innermost last):
            File "<interactive input>", line 1, in ?
          NameError: There is no variable named 'x'
          >>> x = 1
          >>> x
          1

      You will thank Python for this one day.

      ### 3.4.2. Assigning Multiple Values at Once

      One of the cooler programming shortcuts in Python is using sequences to assign multiple values at once.

      ### Example 3.19. Assigning multiple values at once

          >>> v = ('a', 'b', 'e')
          >>> (x, y, z) = v     ①
          >>> x
          'a'
          >>> y
          'b'
          >>> z
          'e'

      1. v is a tuple of three elements, and `(x, y, z)` is a tuple of three variables. Assigning one to the other assigns each of the values of v to each of the variables, in order.

         This has all sorts of uses. I often want to assign names to a range of values. In C, you would use `enum` and manually list each constant and its associated value, which seems especially tedious when the values are consecutive. In Python, you can use the built-in `range` function with multi-variable assignment to quickly assign consecutive values.

         ### Example 3.20. Assigning Consecutive Values

             >>> range(7)              ①
             [0, 1, 2, 3, 4, 5, 6]
             >>> (MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY) = range(7) ②
             >>> MONDAY                ③
             0
             >>> TUESDAY
             1
             >>> SUNDAY
             6

         1. The built-in `range` function returns a list of integers. In its simplest form, it takes an upper limit and returns a zero-based list counting up to but not including the upper limit. (If you like, you can pass other parameters to specify a base other than `0` and a step other than `1`. You can `print range.__doc__` for details.)
         2. MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, and SUNDAY are the variables you're defining. (This example came from the `calendar` module, a fun little module that prints calendars, like the UNIX program `cal`. The `calendar` module defines integer constants for days of the week.)
         3. Now each variable has its value: MONDAY is `0`, TUESDAY is `1`, and so forth.

            You can also use multi-variable assignment to build functions that return multiple values, simply by returning a tuple of all the values. The caller can treat it as a tuple, or assign the values to individual variables. Many standard Python libraries do this, including the `os` module, which you'll discuss in Chapter 6.

            ### Further Reading on Variables

            - Python Reference Manual shows examples of when you can skip the line continuation character and when you need to use it.
            - How to Think Like a Computer Scientist shows how to use multi-variable assignment to swap the values of two variables.

            ## 3.6. Mapping Lists

            One of the most powerful features of Python is the list comprehension, which provides a compact way of mapping a list into another list by applying a function to each of the elements of the list.

            ### Example 3.24. Introducing List Comprehensions

                >>> li = [1, 9, 8, 4]
                >>> [elem*2 for elem in li]      ①
                [2, 18, 16, 8]
                >>> li         ②
                [1, 9, 8, 4]
                >>> li = [elem*2 for elem in li] ③
                >>> li
                [2, 18, 16, 8]

            1. To make sense of this, look at it from right to left. li is the list you're mapping. Python loops through li one element at a time, temporarily assigning the value of each element to the variable elem. Python then applies the function `elem*2` and appends that result to the returned list.
            2. Note that list comprehensions do not change the original list.
            3. It is safe to assign the result of a list comprehension to the variable that you're mapping. Python constructs the new list in memory, and when the list comprehension is complete, it assigns the result to the variable.

               Here are the list comprehensions in the `buildConnectionString` function that you declared in Chapter 2:

                   ["%s=%s" % (k, v) for k, v in params.items()]

               First, notice that you're calling the `items` function of the params dictionary. This function returns a list of tuples of all the data in the dictionary.

               ### Example 3.25. The `keys`, `values`, and `items` Functions

                   >>> params = {"server":"mpilgrim", "database":"master", "uid":"sa", "pwd":"secret"}
                   >>> params.keys()   ①
                   ['server', 'uid', 'database', 'pwd']
                   >>> params.values() ②
                   ['mpilgrim', 'sa', 'master', 'secret']
                   >>> params.items()  ③
                   [('server', 'mpilgrim'), ('uid', 'sa'), ('database', 'master'), ('pwd', 'secret')]

               1. The `keys` method of a dictionary returns a list of all the keys. The list is not in the order in which the dictionary was defined (remember that elements in a dictionary are unordered), but it is a list.
               2. The `values` method returns a list of all the values. The list is in the same order as the list returned by `keys`, so `params.values()[n] == params[params.keys()[n]]` for all values of n.
               3. The `items` method returns a list of tuples of the form `(key, value)`. The list contains all the data in the dictionary.

                  Now let's see what `buildConnectionString` does. It takes a list, `params.`items`()`, and maps it to a new list by applying string formatting to each element. The new list will have the same number of elements as `params.`items`()`, but each element in the new list will be a string that contains both a key and its associated value from the params dictionary.

                  ### Example 3.26. List Comprehensions in `buildConnectionString`, Step by Step

                      >>> params = {"server":"mpilgrim", "database":"master", "uid":"sa", "pwd":"secret"}
                      >>> params.items()
                      [('server', 'mpilgrim'), ('uid', 'sa'), ('database', 'master'), ('pwd', 'secret')]
                      >>> [k for k, v in params.items()]                ①
                      ['server', 'uid', 'database', 'pwd']
                      >>> [v for k, v in params.items()]                ②
                      ['mpilgrim', 'sa', 'master', 'secret']
                      >>> ["%s=%s" % (k, v) for k, v in params.items()] ③
                      ['server=mpilgrim', 'uid=sa', 'database=master', 'pwd=secret']

                  1. Note that you're using two variables to iterate through the `params.items()` list. This is another use of multi-variable assignment. The first element of `params.items()` is `('server', 'mpilgrim')`, so in the first iteration of the list comprehension, k will get `'server'` and v will get `'mpilgrim'`. In this case, you're ignoring the value of v and only including the value of k in the returned list, so this list comprehension ends up being equivalent to `params.`keys`()`.
                  2. Here you're doing the same thing, but ignoring the value of k, so this list comprehension ends up being equivalent to `params.`values`()`.
                  3. Combining the previous two examples with some simple string formatting, you get a list of strings that include both the key and value of each element of the dictionary. This looks suspiciously like the output of the program. All that remains is to join the elements in this list into a single string.

                     ### Further Reading on List Comprehensions

                     - Python Tutorial discusses another way to map lists using the built-in `map` function.
                     - Python Tutorial shows how to do nested list comprehensions.
                     (String splitting stuff was here)

                     Before diving into the next chapter, make sure you're comfortable doing all of these things:

                     - Using the Python IDE to test expressions interactively
                     - Writing Python programs and running them from within your IDE, or from the command line
                     - Importing modules and calling their functions
                     - Declaring functions and using `docstring`s, local variables, and proper indentation
                     - Defining dictionaries, tuples, and lists
                     - Accessing attributes and methods of any object, including strings, lists, dictionaries, functions, and modules
                     - Concatenating values through string formatting
                     - Mapping lists into other lists using list comprehensions
                     - Splitting strings into lists and joining lists into strings

                     ## Chapter 4. The Power Of Introspection

                     This chapter covers one of Python's strengths: introspection. As you know, everything in Python is an object, and introspection is code looking at other modules and functions in memory as objects, getting information about them, and manipulating them. Along the way, you'll define functions with no name, call functions with arguments out of order, and reference functions whose names you don't even know ahead of time.

                     ## 4.1. Diving In

                     Here is a complete, working Python program. You should understand a good deal about it just by looking at it. The numbered lines illustrate concepts covered in Chapter 2, Your First Python Program. Don't worry if the rest of the code looks intimidating; you'll learn all about it throughout this chapter.

                     ### Example 4.1. `apihelper.py`

                     If you have not already done so, you can download this and other examples used in this book.

                         def info(object, spacing=10, collapse=1): ① ② ③
                             """Print methods and docstrings.

                             Takes module, class, list, dictionary, or string."""
                             methodList = [method for method in dir(object) if callable(getattr(object, method))]
                             processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)
                             print "\n".join(["%s %s" %
                             (method.ljust(spacing),
                              processFunc(str(getattr(object, method).__doc__)))
                            for method in methodList])

                         if __name__ == "__main__":                ④ ⑤
                             print info.__doc__

                     1. This module has one function, `info`. According to its function declaration, it takes three parameters: object, spacing, and collapse. The last two are actually optional parameters, as you'll see shortly.
                     2. The `info` function has a multi-line `docstring` that succinctly describes the function's purpose. Note that no return value is mentioned; this function will be used solely for its effects, rather than its value.
                     3. Code within the function is indented.
                     4. The `if __name__` trick allows this program do something useful when run by itself, without interfering with its use as a module for other programs. In this case, the program simply prints out the `docstring` of the `info` function.
                     5. `if` statements use `==` for comparison, and parentheses are not required.

                        The `info` function is designed to be used by you, the programmer, while working in the Python IDE. It takes any object that has functions or methods (like a module, which has functions, or a list, which has methods) and prints out the functions and their `docstring`s.

                        ### Example 4.2. Sample Usage of `apihelper.py`

                            >>> from apihelper import info
                            >>> li = []
                            >>> info(li)
                            append     L.append(object) -- append object to end
                            count      L.count(value) -> integer -- return number of occurrences of value
                            extend     L.extend(list) -- extend list by appending list elements
                            index      L.index(value) -> integer -- return index of first occurrence of value
                            insert     L.insert(index, object) -- insert object before index
                            pop        L.pop([index]) -> item -- remove and return item at index (default last)
                            remove     L.remove(value) -- remove first occurrence of value
                            reverse    L.reverse() -- reverse *IN PLACE*
                            sort       L.sort([cmpfunc]) -- sort *IN PLACE*; if given, cmpfunc(x, y) -> -1, 0, 1

                        By default the output is formatted to be easy to read. Multi-line `docstring`s are collapsed into a single long line, but this option can be changed by specifying `0` for the `collapse` argument. If the function names are longer than 10 characters, you can specify a larger value for the `spacing` argument to make the output easier to read.

                        ### Example 4.3. Advanced Usage of `apihelper.py`

                        ###

                        ###

                        4. ###

                           ###

                           4. ###

                              3. ###

                                 5. ###

                                    ###

                                    ###

                                    ##

                                    ###

                                    5. ###

                                       ###

                                       5. ###

                                          ###

                                          3. ###

                                             1. ##

                                                ###

                                                3. ###

                                                   ##

                                                   ###

                                                   3. ###

                                                      4. ###

                                                         ###

                                                         2. ###

                                                            1. ###

                                                               1. ###

                                                                  ##

                                                                  ###

                                                                  2. ###

                                                                     ###

                                                                     3. ###

                                                                        ##

                                                                        ###

                                                                        4. ###

                                                                           3. ###

                                                                              2. ###

                                                                                 1. ##

                                                                                    ##

                                                                                    ##

                                                                                    ###

                                                                                    1. ##

                                                                                       ###

                                                                                       4. ###

                                                                                          ###

                                                                                          1. ###

                                                                                             ###

                                                                                             2. ###

                                                                                                ##

                                                                                                ###

                                                                                                5. ###

                                                                                                   5. ###

                                                                                                      2. ###

                                                                                                         ##

                                                                                                         ###

                                                                                                         ###

                                                                                                         2. ###

                                                                                                            2. ###

                                                                                                               4. ###

                                                                                                                  3. ##

                                                                                                                     ###

                                                                                                                     4. ###

                                                                                                                        ##

                                                                                                                        ###

                                                                                                                        3. ###

                                                                                                                           5. ##

                                                                                                                              ###

                                                                                                                              1. ###

                                                                                                                                 ##

                                                                                                                                 ##

                                                                                                                                 ##

                                                                                                                                 ###

                                                                                                                                 4. ###

                                                                                                                                    ###

                                                                                                                                    5. ###

                                                                                                                                       ##

                                                                                                                                       ###

                                                                                                                                       4. ###

                                                                                                                                          ###

                                                                                                                                          5. ###

                                                                                                                                             ###

                                                                                                                                             5. ###

                                                                                                                                                ###

                                                                                                                                                6. ###

                                                                                                                                                   ###

                                                                                                                                                   5. ###

                                                                                                                                                      ##

                                                                                                                                                      ###

                                                                                                                                                      3. ###

                                                                                                                                                         2. ###

                                                                                                                                                            3. ###

                                                                                                                                                               3. ##

                                                                                                                                                                  ###

                            >>> import odbchelper
                            >>> info(odbchelper)
                            buildConnectionString Build a connection string from a dictionary Returns string.
                            >>> info(odbchelper, 30)
                            buildConnectionString          Build a connection string from a dictionary Returns string.
                            >>> info(odbchelper, 30, 0)
                            buildConnectionString          Build a connection string from a dictionary

                                Returns string.

                            (optional and named arguments stuff was here)

                            4.3.1. The type Function
                            The type function returns the datatype of any arbitrary object. The possible types are listed in the types module. This is useful for helper functions that can handle several types of data.
                            Example 4.5. Introducing type>>> type(1)           ①
                            <type 'int'>
                            >>> li = []
                            >>> type(li)          ②
                            <type 'list'>
                            >>> import odbchelper
                            >>> type(odbchelper)  ③
                            <type 'module'>
                            >>> import types      ④
                            >>> type(odbchelper) == types.ModuleType
                            True

                            type takes anything -- and I mean anything -- and returns its datatype. Integers, strings, lists, dictionaries, tuples, functions,
                                           classes, modules, even types are acceptable.
                            type can take a variable and return its datatype.
                            type also works on modules.
                            You can use the constants in the types module to compare types of objects. This is what the info function does, as you'll see shortly.
                            4.3.2. The str Function
                            The str coerces data into a string. Every datatype can be coerced into a string.
                            Example 4.6. Introducing str
                            >>> str(1)          ①
                            '1'
                            >>> horsemen = ['war', 'pestilence', 'famine']
                            >>> horsemen
                            ['war', 'pestilence', 'famine']
                            >>> horsemen.append('Powerbuilder')
                            >>> str(horsemen)   ②
                            "['war', 'pestilence', 'famine', 'Powerbuilder']"
                            >>> str(odbchelper) ③
                            "<module 'odbchelper' from 'c:\\docbook\\dip\\py\\odbchelper.py'>"
                            >>> str(None)       ④
                            'None'

                            For simple datatypes like integers, you would expect str to work, because almost every language has a function to convert an integer to a string.
                            However, str works on any object of any type. Here it works on a list which you've constructed in bits and pieces.
                            str also works on modules. Note that the string representation of the module includes the pathname of the module on disk, so
                                           yours will be different.
                            A subtle but important behavior of str is that it works on None, the Python null value. It returns the string 'None'. You'll use this to your advantage in the info function, as you'll see shortly.
                            At the heart of the info function is the powerful dir function. dir returns a list of the attributes and methods of any object: modules, functions, strings, lists, dictionaries... pretty much
                               anything.
                            Example 4.7. Introducing dir>>> li = []
                            >>> dir(li)           ①
                            ['append', 'count', 'extend', 'index', 'insert',
                            'pop', 'remove', 'reverse', 'sort']
                            >>> d = {}
                            >>> dir(d)            ②
                            ['clear', 'copy', 'get', 'has_key', 'items', 'keys', 'setdefault', 'update', 'values']
                            >>> import odbchelper
                            >>> dir(odbchelper)   ③
                            ['__builtins__', '__doc__', '__file__', '__name__', 'buildConnectionString']

                            li is a list, so dir(li) returns a list of all the methods of a list. Note that the returned list contains the names of the methods as strings, not
                                           the methods themselves.
                            d is a dictionary, so dir(d) returns a list of the names of dictionary methods. At least one of these, keys, should look familiar.
                            This is where it really gets interesting. odbchelper is a module, so dir(odbchelper) returns a list of all kinds of stuff defined in the module, including built-in attributes, like __name__, __doc__, and whatever other attributes and methods you define. In this case, odbchelper has only one user-defined method, the buildConnectionString function described in Chapter 2.
                            Finally, the callable function takes any object and returns True if the object can be called, or False otherwise. Callable objects include functions, class methods, even classes themselves. (More on classes in the next chapter.)
                            Example 4.8. Introducing callable
                            >>> import string
                            >>> string.punctuation           ①
                            '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
                            >>> string.join②
                            <function join at 00C55A7C>
                            >>> callable(string.punctuation) ③
                            False
                            >>> callable(string.join)        ④
                            True
                            >>> print string.join.__doc__    ⑤
                            join(list [,sep]) -> string

                                Return a string composed of the words in list, with
                                intervening occurrences of sep. The default separator is a
                                single space.

                                (joinfields and join are synonymous)

                            The functions in the string module are deprecated (although many people still use the join function), but the module contains a lot of useful constants like this string.punctuation, which contains all the standard punctuation characters.
                            string.join is a function that joins a list of strings.
                            string.punctuation is not callable; it is a string. (A string does have callable methods, but the string itself is not callable.)
                            string.join is callable; it's a function that takes two arguments.
                            Any callable object may have a docstring. By using the callable function on each of an object's attributes, you can determine which attributes you care about (methods, functions, classes)
                                           and which you want to ignore (constants and so on) without knowing anything about the object ahead of time.
                            4.3.3. Built-In Functions
                            type, str, dir, and all the rest of Python's built-in functions are grouped into a special module called __builtin__. (That's two underscores before and after.)  If it helps, you can think of Python automatically executing from __builtin__ import * on startup, which imports all the “built-in” functions into the namespace so you can use them directly.
                            The advantage of thinking like this is that you can access all the built-in functions and attributes as a group by getting
                               information about the __builtin__ module. And guess what, Python has a function called info. Try it yourself and skim through the list now. We'll dive into some of the more important functions later. (Some of the
                               built-in error classes, like AttributeError, should already look familiar.)
                            Example 4.9. Built-in Attributes and Functions>>> from apihelper import info
                            >>> import __builtin__
                            >>> info(__builtin__, 20)
                            ArithmeticError      Base class for arithmetic errors.
                            AssertionError       Assertion failed.
                            AttributeError       Attribute not found.
                            EOFError             Read beyond end of file.
                            EnvironmentError     Base class for I/O related errors.
                            Exception            Common base class for all exceptions.
                            FloatingPointError   Floating point operation failed.
                            IOError              I/O operation failed.

                            [...snip...]

                            Python comes with excellent reference manuals, which you should peruse thoroughly to learn all the modules Python has to offer. But unlike most languages, where you would find yourself referring back to the manuals or man pages to remind
                                     yourself how to use these modules, Python is largely self-documenting.

                            Further Reading on Built-In Functions

                            Python Library Reference documents all the built-in functions and all the built-in exceptions.

                            4.4. Getting Object References With getattr
                            You already know that Python functions are objects. What you don't know is that you can get a reference to a function without knowing its name until run-time, by using the
                            getattr function.
                            Example 4.10. Introducing getattr>>> li = ["Larry", "Curly"]
                            >>> li.pop     ①
                            <built-in method pop of list object at 010DF884>
                            >>> getattr(li, "pop")           ②
                            <built-in method pop of list object at 010DF884>
                            >>> getattr(li, "append")("Moe") ③
                            >>> li
                            ["Larry", "Curly", "Moe"]
                            >>> getattr({}, "clear")         ④
                            <built-in method clear of dictionary object at 00F113D4>
                            >>> getattr((), "pop")           ⑤
                            Traceback (innermost last):
                              File "<interactive input>", line 1, in ?
                            AttributeError: 'tuple' object has no attribute 'pop'

                            This gets a reference to the pop method of the list. Note that this is not calling the pop method; that would be li.pop(). This is the method itself.
                            This also returns a reference to the pop method, but this time, the method name is specified as a string argument to the getattr function. getattr is an incredibly useful built-in function that returns any attribute of any object. In this case, the object is a list,
                                        and the attribute is the pop method.
                            In case it hasn't sunk in just how incredibly useful this is, try this: the return value of getattr is the method, which you can then call just as if you had said li.append("Moe") directly. But you didn't call the function directly; you specified the function name as a string instead.
                            getattr also works on dictionaries.
                            In theory, getattr would work on tuples, except that tuples have no methods, so getattr will raise an exception no matter what attribute name you give.
                            4.4.1. getattr with Modules
                            getattr isn't just for built-in datatypes. It also works on modules.
                            Example 4.11. The getattr Function in apihelper.py>>> import odbchelper
                            >>> odbchelper.buildConnectionString             ①
                            <function buildConnectionString at 00D18DD4>
                            >>> getattr(odbchelper, "buildConnectionString") ②
                            <function buildConnectionString at 00D18DD4>
                            >>> object = odbchelper
                            >>> method = "buildConnectionString"
                            >>> getattr(object, method)    ③
                            <function buildConnectionString at 00D18DD4>
                            >>> type(getattr(object, method))                ④
                            <type 'function'>
                            >>> import types
                            >>> type(getattr(object, method)) == types.FunctionType
                            True
                            >>> callable(getattr(object, method))            ⑤
                            True

                            This returns a reference to the buildConnectionString function in the odbchelper module, which you studied in Chapter 2, Your First Python Program. (The hex address you see is specific to my machine; your output will be different.)
                            Using getattr, you can get the same reference to the same function. In general, getattr(object, "attribute") is equivalent to object.attribute. If object is a module, then attribute can be anything defined in the module: a function, class, or global variable.
                            And this is what you actually use in the info function. object is passed into the function as an argument; method is a string which is the name of a method or function.
                            In this case, method is the name of a function, which you can prove by getting its type.
                            Since method is a function, it is callable.
                            4.4.2. getattr As a Dispatcher
                            A common usage pattern of getattr is as a dispatcher. For example, if you had a program that could output data in a variety of different formats, you could
                               define separate functions for each output format and use a single dispatch function to call the right one.
                            For example, let's imagine a program that prints site statistics in HTML, XML, and plain text formats. The choice of output format could be specified on the command line, or stored in a configuration
                               file. A statsout module defines three functions, output_html, output_xml, and output_text. Then the main program defines a single output function, like this:
                            Example 4.12. Creating a Dispatcher with getattr
                            import statsout

                            def output(data, format="text"):            ①
                                output_function = getattr(statsout, "output_%s" % format) ②
                                return output_function(data)            ③

                            The output function takes one required argument, data, and one optional argument, format. If format is not specified, it defaults to text, and you will end up calling the plain text output function.
                            You concatenate the format argument with "output_" to produce a function name, and then go get that function from the statsout module. This allows you to easily extend the program later to support other output formats, without changing this dispatch
                                        function. Just add another function to statsout named, for instance, output_pdf, and pass "pdf" as the format into the output function.
                            Now you can simply call the output function in the same way as any other function. The output_function variable is a reference to the appropriate function from the statsout module.
                            Did you see the bug in the previous example?  This is a very loose coupling of strings and functions, and there is no error
                               checking. What happens if the user passes in a format that doesn't have a corresponding function defined in statsout?  Well, getattr will return None, which will be assigned to output_function instead of a valid function, and the next line that attempts to call that function will crash and raise an exception. That's
                               bad.
                            Luckily, getattr takes an optional third argument, a default value.
                            Example 4.13. getattr Default Values
                            import statsout

                            def output(data, format="text"):
                                output_function = getattr(statsout, "output_%s" % format, statsout.output_text)
                                return output_function(data) ①

                            This function call is guaranteed to work, because you added a third argument to the call to getattr. The third argument is a default value that is returned if the attribute or method specified by the second argument wasn't
                                           found.
                            As you can see, getattr is quite powerful. It is the heart of introspection, and you'll see even more powerful examples of it in later chapters.
                            4.5. Filtering Lists
                            As you know, Python has powerful capabilities for mapping lists into other lists, via list comprehensions (Section 3.6, “Mapping Lists”). This can be combined with a filtering mechanism, where some elements in the list are mapped while others are skipped entirely.

                            Here is the list filtering syntax:
                            [mapping-expression for element in source-list if filter-expression]This is an extension of the list comprehensions that you know and love. The first two thirds are the same; the last part, starting with the if, is the filter expression. A filter expression can be any expression that evaluates true or false (which in Python can be almost anything). Any element for which the filter expression evaluates true will be included in the mapping. All other elements are ignored,
                            so they are never put through the mapping expression and are not included in the output list.
                            Example 4.14. Introducing List Filtering>>> li = ["a", "mpilgrim", "foo", "b", "c", "b", "d", "d"]
                            >>> [elem for elem in li if len(elem) > 1]       ①
                            ['mpilgrim', 'foo']
                            >>> [elem for elem in li if elem != "b"]         ②
                            ['a', 'mpilgrim', 'foo', 'c', 'd', 'd']
                            >>> [elem for elem in li if li.count(elem) == 1] ③
                            ['a', 'mpilgrim', 'foo', 'c']

                            The mapping expression here is simple (it just returns the value of each element), so concentrate on the filter expression.
                                         As Python loops through the list, it runs each element through the filter expression. If the filter expression is true, the element
                                        is mapped and the result of the mapping expression is included in the returned list. Here, you are filtering out all the
                                        one-character strings, so you're left with a list of all the longer strings.
                            Here, you are filtering out a specific value, b. Note that this filters all occurrences of b, since each time it comes up, the filter expression will be false.
                            count is a list method that returns the number of times a value occurs in a list. You might think that this filter would eliminate
                                        duplicates from a list, returning a list containing only one copy of each value in the original list. But it doesn't, because
                                        values that appear twice in the original list (in this case, b and d) are excluded completely. There are ways of eliminating duplicates from a list, but filtering is not the solution.
                            Let's id="apihelper.filter.care" get back to this line from apihelper.py:
                                methodList = [method for method in dir(object) if callable(getattr(object, method))]This looks complicated, and it is complicated, but the basic structure is the same. The whole filter expression returns a
                            list, which is assigned to the methodList variable. The first half of the expression is the list mapping part. The mapping expression is an identity expression,
                            which it returns the value of each element. dir(object) returns a list of object's attributes and methods -- that's the list you're mapping. So the only new part is the filter expression after the if.
                            The filter expression looks scary, but it's not. You already know about callable, getattr, and in. As you saw in the previous section, the expression getattr(object, method) returns a function object if object is a module and method is the name of a function in that module.
                            So this expression takes an object (named object). Then it gets a list of the names of the object's attributes, methods, functions, and a few other things. Then it filters
                            that list to weed out all the stuff that you don't care about. You do the weeding out by taking the name of each attribute/method/function
                            and getting a reference to the real thing, via the getattr function. Then you check to see if that object is callable, which will be any methods and functions, both built-in (like
                            the pop method of a list) and user-defined (like the buildConnectionString function of the odbchelper module). You don't care about other attributes, like the __name__ attribute that's built in to every module.

                            Further Reading on Filtering Lists

                            Python Tutorial discusses another way to filter lists using the built-in filter function.

                            4.6. The Peculiar Nature of and and or
                            In Python, and and or perform boolean logic as you would expect, but they do not return boolean values; instead, they return one of the actual
                               values they are comparing.
                            Example 4.15. Introducing and>>> 'a' and 'b'         ①
                            'b'
                            >>> '' and 'b'          ②
                            ''
                            >>> 'a' and 'b' and 'c' ③
                            'c'

                            When using and, values are evaluated in a boolean context from left to right. 0, '', [], (), {}, and None are false in a boolean context; everything else is true. Well, almost everything. By default, instances of classes are
                                        true in a boolean context, but you can define special methods in your class to make an instance evaluate to false. You'll
                                        learn all about classes and special methods in Chapter 5. If all values are true in a boolean context, and returns the last value. In this case, and evaluates 'a', which is true, then 'b', which is true, and returns 'b'.
                            If any value is false in a boolean context, and returns the first false value. In this case, '' is the first false value.
                            All values are true, so and returns the last value, 'c'.
                            Example 4.16. Introducing or>>> 'a' or 'b'          ①
                            'a'
                            >>> '' or 'b'           ②
                            'b'
                            >>> '' or [] or {}      ③
                            {}
                            >>> def sidefx():
                            ...    print "in sidefx()"
                            ...    return 1
                            >>> 'a' or sidefx()     ④
                            'a'

                            When using or, values are evaluated in a boolean context from left to right, just like and. If any value is true, or returns that value immediately. In this case, 'a' is the first true value.
                            or evaluates '', which is false, then 'b', which is true, and returns 'b'.
                            If all values are false, or returns the last value. or evaluates '', which is false, then [], which is false, then {}, which is false, and returns {}.
                            Note that or evaluates values only until it finds one that is true in a boolean context, and then it ignores the rest. This distinction
                                        is important if some values can have side effects. Here, the function sidefx is never called, because or evaluates 'a', which is true, and returns 'a' immediately.
                            If you're a C hacker, you are certainly familiar with the bool ? a : b expression, which evaluates to a if bool is true, and b otherwise. Because of the way and and or work in Python, you can accomplish the same thing.
                            4.6.1. Using the and-or Trick
                            Example 4.17. Introducing the and-or Trick>>> a = "first"
                            >>> b = "second"
                            >>> 1 and a or b ①
                            'first'
                            >>> 0 and a or b ②
                            'second'

                            This syntax looks similar to the bool ? a : b expression in C. The entire expression is evaluated from left to right, so the and is evaluated first. 1 and 'first' evalutes to 'first', then 'first' or 'second' evalutes to 'first'.
                            0 and 'first' evalutes to False, and then 0 or 'second' evaluates to 'second'.
                            However, since this Python expression is simply boolean logic, and not a special construct of the language, there is one extremely important difference
                               between this and-or trick in Python and the bool ? a : b syntax in C. If the value of a is false, the expression will not work as you would expect it to. (Can you tell I was bitten by this?  More than once?)
                            Example 4.18. When the and-or Trick Fails>>> a = ""
                            >>> b = "second"
                            >>> 1 and a or b         ①
                            'second'

                            Since a is an empty string, which Python considers false in a boolean context, 1 and '' evalutes to '', and then '' or 'second' evalutes to 'second'. Oops!  That's not what you wanted.
                            The and-or trick, bool and a or b, will not work like the C expression bool ? a : b when a is false in a boolean context.
                            The real trick behind the and-or trick, then, is to make sure that the value of a is never false. One common way of doing this is to turn a into [a] and b into [b], then taking the first element of the returned list, which will be either a or b.
                            Example 4.19. Using the and-or Trick Safely>>> a = ""
                            >>> b = "second"
                            >>> (1 and [a] or [b])[0] ①
                            ''

                            Since [a] is a non-empty list, it is never false. Even if a is 0 or '' or some other false value, the list [a] is true because it has one element.
                            By now, this trick may seem like more trouble than it's worth. You could, after all, accomplish the same thing with an if statement, so why go through all this fuss?  Well, in many cases, you are choosing between two constant values, so you can
                               use the simpler syntax and not worry, because you know that the a value will always be true. And even if you need to use the more complicated safe form, there are good reasons to do so.
                               For example, there are some cases in Python where if statements are not allowed, such as in lambda functions.

                            Further Reading on the and-or Trick

                            Python Cookbook discusses alternatives to the and-or trick.

                            4.7. Using lambda Functions
                            Python supports an interesting syntax that lets you define one-line mini-functions on the fly. Borrowed from Lisp, these so-called lambda functions can be used anywhere a function is required.
                            Example 4.20. Introducing lambda Functions>>> def f(x):
                            ...    return x*2
                            ...
                            >>> f(3)
                            6
                            >>> g = lambda x: x*2  ①
                            >>> g(3)
                            6
                            >>> (lambda x: x*2)(3) ②
                            6

                            This is a lambda function that accomplishes the same thing as the normal function above it. Note the abbreviated syntax here: there are no
                                        parentheses around the argument list, and the return keyword is missing (it is implied, since the entire function can only be one expression). Also, the function has no name,
                                        but it can be called through the variable it is assigned to.
                            You can use a lambda function without even assigning it to a variable. This may not be the most useful thing in the world, but it just goes to
                                        show that a lambda is just an in-line function.
                            To generalize, a lambda function is a function that takes any number of arguments (including optional arguments) and returns the value of a single expression. lambda functions can not contain commands, and they can not contain more than one expression. Don't try to squeeze too much into
                            a lambda function; if you need something more complex, define a normal function instead and make it as long as you want.

                            lambda functions are a matter of style. Using them is never required; anywhere you could use them, you could define a separate
                                  normal function and use that instead. I use them in places where I want to encapsulate specific, non-reusable code without
                                  littering my code with a lot of little one-line functions.
                            4.7.1. Real-World lambda Functions

                            Here are the lambda functions in apihelper.py:
                                processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)Notice that this uses the simple form of the and-or trick, which is okay, because a lambda function is always true in a boolean context. (That doesn't mean that a lambda function can't return a false value. The function is always true; its return value could be anything.)
                            Also notice that you're using the split function with no arguments. You've already seen it used with one or two arguments, but without any arguments it splits on whitespace.
                            Example 4.21. split With No Arguments>>> s = "this   is\na\ttest"  ①
                            >>> print s
                            this   is
                            a	test
                            >>> print s.split()           ②
                            ['this', 'is', 'a', 'test']
                            >>> print " ".join(s.split()) ③
                            'this is a test'

                            This is a multiline string, defined by escape characters instead of triple quotes. \n is a carriage return, and \t is a tab character.
                            split without any arguments splits on whitespace. So three spaces, a carriage return, and a tab character are all the same.
                            You can normalize whitespace by splitting a string with split and then rejoining it with join, using a single space as a delimiter. This is what the info function does to collapse multi-line docstrings into a single line.
                            So what is the info function actually doing with these lambda functions, splits, and and-or tricks?

                                processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)processFunc is now a function, but which function it is depends on the value of the collapse variable. If collapse is true, processFunc(string) will collapse whitespace; otherwise, processFunc(string) will return its argument unchanged.
                            To do this in a less robust language, like Visual Basic, you would probably create a function that took a string and a collapse argument and used an if statement to decide whether to collapse the whitespace or not, then returned the appropriate value. This would be inefficient,
                               because the function would need to handle every possible case. Every time you called it, it would need to decide whether
                               to collapse whitespace before it could give you what you wanted. In Python, you can take that decision logic out of the function and define a lambda function that is custom-tailored to give you exactly (and only) what you want. This is more efficient, more elegant, and
                               less prone to those nasty oh-I-thought-those-arguments-were-reversed kinds of errors.

                            Further Reading on lambda Functions

                            Python Knowledge Base discusses using lambda to call functions indirectly.

                            Python Tutorial shows how to access outside variables from inside a lambda function. (PEP 227 explains how this will change in future versions of Python.)

                            The Whole Python FAQ has examples of obfuscated one-liners using lambda.

                            4.8. Putting It All Together
                            The last line of code, the only one you haven't deconstructed yet, is the one that does all the work. But by now the work
                               is easy, because everything you need is already set up just the way you need it. All the dominoes are in place; it's time
                               to knock them down.

                            This is the meat of apihelper.py:
                                print "\n".join(["%s %s" %
                                (method.ljust(spacing),
                                 processFunc(str(getattr(object, method).__doc__)))
                               for method in methodList])Note that this is one command, split over multiple lines, but it doesn't use the line continuation character (\). Remember when I said that some expressions can be split into multiple lines without using a backslash?  A list comprehension is one of those expressions, since the entire expression is contained in
                            square brackets.
                            Now, let's take it from the end and work backwards. The
                            for method in methodListshows that this is a list comprehension. As you know, methodList is a list of all the methods you care about in object. So you're looping through that list with method.
                            Example 4.22. Getting a docstring Dynamically>>> import odbchelper
                            >>> object = odbchelper ①
                            >>> method = 'buildConnectionString'      ②
                            >>> getattr(object, method)               ③
                            <function buildConnectionString at 010D6D74>
                            >>> print getattr(object, method).__doc__ ④
                            Build a connection string from a dictionary of parameters.

                                Returns string.

                            In the info function, object is the object you're getting help on, passed in as an argument.
                            As you're looping through methodList, method is the name of the current method.
                            Using the getattr function, you're getting a reference to the method function in the object module.
                            Now, printing the actual docstring of the method is easy.
                            The next piece of the puzzle is the use of str around the docstring. As you may recall, str is a built-in function that coerces data into a string. But a docstring is always a string, so why bother with the str function?  The answer is that not every function has a docstring, and if it doesn't, its __doc__ attribute is None.
                            Example 4.23. Why Use str on a docstring?>>> >>> def foo(): print 2
                            >>> >>> foo()
                            2
                            >>> >>> foo.__doc__     ①
                            >>> foo.__doc__ == None ②
                            True
                            >>> str(foo.__doc__)    ③
                            'None'

                            You can easily define a function that has no docstring, so its __doc__ attribute is None. Confusingly, if you evaluate the __doc__ attribute directly, the Python IDE prints nothing at all, which makes sense if you think about it, but is still unhelpful.
                            You can verify that the value of the __doc__ attribute is actually None by comparing it directly.
                            The str function takes the null value and returns a string representation of it, 'None'.

                            In SQL, you must use IS NULL instead of = NULL to compare a null value. In Python, you can use either == None or is None, but is None is faster.
                            Now that you are guaranteed to have a string, you can pass the string to processFunc, which you have already defined as a function that either does or doesn't collapse whitespace. Now you see why it was important to use str to convert a None value into a string representation. processFunc is assuming a string argument and calling its split method, which would crash if you passed it None because None doesn't have a split method.
                            Stepping back even further, you see that you're using string formatting again to concatenate the return value of processFunc with the return value of method's ljust method. This is a new string method that you haven't seen before.
                            Example 4.24. Introducing ljust>>> s = 'buildConnectionString'
                            >>> s.ljust(30) ①
                            'buildConnectionString         '
                            >>> s.ljust(20) ②
                            'buildConnectionString'

                            ljust pads the string with spaces to the given length. This is what the info function uses to make two columns of output and line up all the docstrings in the second column.
                            If the given length is smaller than the length of the string, ljust will simply return the string unchanged. It never truncates the string.
                            You're almost finished. Given the padded method name from the ljust method and the (possibly collapsed) docstring from the call to processFunc, you concatenate the two and get a single string. Since you're mapping methodList, you end up with a list of strings. Using the join method of the string "\n", you join this list into a single string, with each element of the list on a separate line, and print the result.
                            Example 4.25. Printing a List>>> li = ['a', 'b', 'c']
                            >>> print "\n".join(li) ①
                            a
                            b
                            c

                            This is also a useful debugging trick when you're working with lists. And in Python, you're always working with lists.
                            That's the last piece of the puzzle. You should now understand this code.

                                print "\n".join(["%s %s" %
                                (method.ljust(spacing),
                                 processFunc(str(getattr(object, method).__doc__)))
                               for method in methodList])4.9. Summary
                            The apihelper.py program and its output should now make perfect sense.

                            def info(object, spacing=10, collapse=1):
                                """Print methods and docstrings.

                                Takes module, class, list, dictionary, or string."""
                                methodList = [method for method in dir(object) if callable(getattr(object, method))]
                                processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)
                                print "\n".join(["%s %s" %
                                (method.ljust(spacing),
                                 processFunc(str(getattr(object, method).__doc__)))
                               for method in methodList])

                            if __name__ == "__main__":
                                print info.__doc__
                            Here is the output of apihelper.py:>>> from apihelper import info
                            >>> li = []
                            >>> info(li)
                            append     L.append(object) -- append object to end
                            count      L.count(value) -> integer -- return number of occurrences of value
                            extend     L.extend(list) -- extend list by appending list elements
                            index      L.index(value) -> integer -- return index of first occurrence of value
                            insert     L.insert(index, object) -- insert object before index
                            pop        L.pop([index]) -> item -- remove and return item at index (default last)
                            remove     L.remove(value) -- remove first occurrence of value
                            reverse    L.reverse() -- reverse *IN PLACE*
                            sort       L.sort([cmpfunc]) -- sort *IN PLACE*; if given, cmpfunc(x, y) -> -1, 0, 1
                            Before diving into the next chapter, make sure you're comfortable doing all of these things:

                            Defining and calling functions with optional and named arguments
                            Using str to coerce any arbitrary value into a string representation

                            Using getattr to get references to functions and other attributes dynamically

                            Extending the list comprehension syntax to do list filtering
                            Recognizing the and-or trick and using it safely

                            Defining lambda functions
                            Assigning functions to variables and calling the function by referencing the variable. I can't emphasize this enough, because this mode of thought is vital
                                     to advancing your understanding of Python. You'll see more complex applications of this concept throughout this book.

                            Chapter 5. Objects and Object-Orientation
                            This chapter, and pretty much every chapter after this, deals with object-oriented Python programming.
                            5.1. Diving In
                            Here is a complete, working Python program. Read the docstrings of the module, the classes, and the functions to get an overview of what this program does and how it works. As usual, don't
                               worry about the stuff you don't understand; that's what the rest of the chapter is for.
                            Example 5.1. fileinfo.py
                            If you have not already done so, you can download this and other examples used in this book.

                            """Framework for getting filetype-specific metadata.

                            Instantiate appropriate class with filename. Returned object acts like a
                            dictionary, with key-value pairs for each piece of metadata.
                                import fileinfo
                                info = fileinfo.MP3FileInfo("/music/ap/mahadeva.mp3")
                                print "\\n".join(["%s=%s" % (k, v) for k, v in info.items()])

                            Or use listDirectory function to get info on all files in a directory.
                                for info in fileinfo.listDirectory("/music/ap/", [".mp3"]):
                                    ...

                            Framework can be extended by adding classes for particular file types, e.g.
                            HTMLFileInfo, MPGFileInfo, DOCFileInfo. Each class is completely responsible for
                            parsing its files appropriately; see MP3FileInfo for example.
                            """
                            import os
                            import sys
                            from UserDict import UserDict

                            def stripnulls(data):
                                "strip whitespace and nulls"
                                return data.replace("\00", "").strip()

                            class FileInfo(UserDict):
                                "store file metadata"
                                def __init__(self, filename=None):
                                    UserDict.__init__(self)
                                    self["name"] = filename

                            class MP3FileInfo(FileInfo):
                                "store ID3v1.0 MP3 tags"
                                tagDataMap = {"title"   : (  3,  33, stripnulls),
                            "artist"  : ( 33,  63, stripnulls),
                            "album"   : ( 63,  93, stripnulls),
                            "year"    : ( 93,  97, stripnulls),
                            "comment" : ( 97, 126, stripnulls),
                            "genre"   : (127, 128, ord)}

                                def __parse(self, filename):
                                    "parse ID3v1.0 tags from MP3 file"
                                    self.clear()
                                    try:
                                        fsock = open(filename, "rb", 0)
                                        try:
                                            fsock.seek(-128, 2)
                                            tagdata = fsock.read(128)
                                        finally:
                                            fsock.close()
                                        if tagdata[:3] == "TAG":
                                            for tag, (start, end, parseFunc) in self.tagDataMap.items():
                              self[tag] = parseFunc(tagdata[start:end])
                                    except IOError:
                                        pass

                                def __setitem__(self, key, item):
                                    if key == "name" and item:
                                        self.__parse(item)
                                    FileInfo.__setitem__(self, key, item)

                            def listDirectory(directory, fileExtList):
                                "get list of file info objects for files of particular extensions"
                                fileList = [os.path.normcase(f)
                                            for f in os.listdir(directory)]
                                fileList = [os.path.join(directory, f)
                                           for f in fileList
                                            if os.path.splitext(f)[1] in fileExtList]
                                def getFileInfoClass(filename, module=sys.modules[FileInfo.__module__]):
                                    "get file info class from filename extension"
                                    subclass = "%sFileInfo" % os.path.splitext(filename)[1].upper()[1:]
                                    return hasattr(module, subclass) and getattr(module, subclass) or FileInfo
                                return [getFileInfoClass(f)(f) for f in fileList]

                            if __name__ == "__main__":
                                for info in listDirectory("/music/_singles/", [".mp3"]): ①
                                    print "\n".join(["%s=%s" % (k, v) for k, v in info.items()])
                                    print

                            This program's output depends on the files on your hard drive. To get meaningful output, you'll need to change the directory
                                        path to point to a directory of MP3 files on your own machine.

                            This is the output I got on my machine. Your output will be different, unless, by some startling coincidence, you share my
                               exact taste in music.
                            album=
                            artist=Ghost in the Machine
                            title=A Time Long Forgotten (Concept
                            genre=31
                            name=/music/_singles/a_time_long_forgotten_con.mp3
                            year=1999
                            comment=http://mp3.com/ghostmachine

                            album=Rave Mix
                            artist=***DJ MARY-JANE***
                            title=HELLRAISER****Trance from Hell
                            genre=31
                            name=/music/_singles/hellraiser.mp3
                            year=2000
                            comment=http://mp3.com/DJMARYJANE

                            album=Rave Mix
                            artist=***DJ MARY-JANE***
                            title=KAIRO****THE BEST GOA
                            genre=31
                            name=/music/_singles/kairo.mp3
                            year=2000
                            comment=http://mp3.com/DJMARYJANE

                            album=Journeys
                            artist=Masters of Balance
                            title=Long Way Home
                            genre=31
                            name=/music/_singles/long_way_home1.mp3
                            year=2000
                            comment=http://mp3.com/MastersofBalan

                            album=
                            artist=The Cynic Project
                            title=Sidewinder
                            genre=18
                            name=/music/_singles/sidewinder.mp3
                            year=2000
                            comment=http://mp3.com/cynicproject

                            album=Digitosis@128k
                            artist=VXpanded
                            title=Spinning
                            genre=255
                            name=/music/_singles/spinning.mp3
                            year=2000
                            comment=http://mp3.com/artists/95/vxp5.2. Importing Modules Using from module import
                            Python has two ways of importing modules. Both are useful, and you should know when to use each. One way, import module, you've already seen in Section 2.4, “Everything Is an Object”. The other way accomplishes the same thing, but it has subtle and important differences.

                            Here is the basic from module import syntax:
                            from UserDict import UserDict
                            This is similar to the import module syntax that you know and love, but with an important difference: the attributes and methods of the imported module types are imported directly into the local namespace, so they are available directly, without qualification by module name. You
                            can import individual items or use from module import * to import everything.

                            from module import * in Python is like use module in Perl; import module in Python is like require module in Perl.

                            from module import * in Python is like import module.* in Java; import module in Python is like import module in Java.
                            Example 5.2. import module vs. from module import>>> import types
                            >>> types.FunctionType             ①
                            <type 'function'>
                            >>> FunctionType ②
                            Traceback (innermost last):
                              File "<interactive input>", line 1, in ?
                            NameError: There is no variable named 'FunctionType'
                            >>> from types import FunctionType ③
                            >>> FunctionType ④
                            <type 'function'>

                            The types module contains no methods; it just has attributes for each Python object type. Note that the attribute, FunctionType, must be qualified by the module name, types.
                            FunctionType by itself has not been defined in this namespace; it exists only in the context of types.
                            This syntax imports the attribute FunctionType from the types module directly into the local namespace.
                            Now FunctionType can be accessed directly, without reference to types.
                            When should you use from module import?

                            If you will be accessing attributes and methods often and don't want to type the module name over and over, use from module import.

                            If you want to selectively import some attributes and methods but not others, use from module import.

                            If the module contains attributes or functions with the same name as ones in your module, you must use import module to avoid name conflicts.

                            Other than that, it's just a matter of style, and you will see Python code written both ways.

                            Use from module import * sparingly, because it makes it difficult to determine where a particular function or attribute came from, and that makes
                                  debugging and refactoring more difficult.

                            Further Reading on Module Importing Techniques

                            eff-bot has more to say on import module vs. from module import.

                            Python Tutorial discusses advanced import techniques, including from module import *.

                            [classes stuff was here]

                            Example 5.4. Defining the FileInfo Class
                            from UserDict import UserDict

                            class FileInfo(UserDict): ①

                            In Python, the ancestor of a class is simply listed in parentheses immediately after the class name. So the FileInfo class is inherited from the UserDict class (which was imported from the UserDict module). UserDict is a class that acts like a dictionary, allowing you to essentially subclass the dictionary datatype and add your own behavior.
                                         (There are similar classes UserList and UserString which allow you to subclass lists and strings.)  There is a bit of black magic behind this, which you will demystify later
                                        in this chapter when you explore the UserDict class in more depth.

                            In Python, the ancestor of a class is simply listed in parentheses immediately after the class name. There is no special keyword like
                            extends in Java.
                            Python supports multiple inheritance. In the parentheses following the class name, you can list as many ancestor classes as you
                            like, separated by commas.
                            5.3.1. Initializing and Coding Classes

                            If creating new instances is easy, destroying them is even easier. In general, there is no need to explicitly free instances,
                               because they are freed automatically when the variables assigned to them go out of scope. Memory leaks are rare in Python.
                            Example 5.8. Trying to Implement a Memory Leak>>> def leakmem():
                            ...    f = fileinfo.FileInfo('/music/_singles/kairo.mp3') ①
                            ...
                            >>> for i in range(100):
                            ...    leakmem()      ②

                            Every time the leakmem function is called, you are creating an instance of FileInfo and assigning it to the variable f, which is a local variable within the function. Then the function ends without ever freeing f, so you would expect a memory leak, but you would be wrong. When the function ends, the local variable f goes out of scope. At this point, there are no longer any references to the newly created instance of FileInfo (since you never assigned it to anything other than f), so Python destroys the instance for us.
                            No matter how many times you call the leakmem function, it will never leak memory, because every time, Python will destroy the newly created FileInfo class before returning from leakmem.
                            The technical term for this form of garbage collection is “reference counting”. Python keeps a list of references to every instance created. In the above example, there was only one reference to the FileInfo instance: the local variable f. When the function ends, the variable f goes out of scope, so the reference count drops to 0, and Python destroys the instance automatically.
                            In previous versions of Python, there were situations where reference counting failed, and Python couldn't clean up after you. If you created two instances that referenced each other (for instance, a doubly-linked list,
                               where each node has a pointer to the previous and next node in the list), neither instance would ever be destroyed automatically
                               because Python (correctly) believed that there is always a reference to each instance. Python 2.0 has an additional form of garbage collection called “mark-and-sweep” which is smart enough to notice this virtual gridlock and clean up circular references correctly.
                            As a former philosophy major, it disturbs me to think that things disappear when no one is looking at them, but that's exactly
                               what happens in Python. In general, you can simply forget about memory management and let Python clean up after you.

                            Further Reading on Garbage Collection

                            Python Library Reference summarizes built-in attributes like __class__.

                            Python Library Reference documents the gc module, which gives you low-level control over Python's garbage collection.

                            5.5. Exploring UserDict: A Wrapper Class
                            As you've seen, FileInfo is a class that acts like a dictionary. To explore this further, let's look at the UserDict class in the UserDict module, which is the ancestor of the FileInfo class. This is nothing special; the class is written in Python and stored in a .py file, just like any other Python code. In particular, it's stored in the lib directory in your Python installation.

                            In the ActivePython IDE on Windows, you can quickly open any module in your library path by selecting
                            File->Locate... (Ctrl-L).
                            Example 5.9. Defining the UserDict Class
                            class UserDict:              ①
                                def __init__(self, dict=None):             ②
                                    self.data = {}       ③
                                    if dict is not None: self.update(dict) ④ ⑤

                            Note that UserDict is a base class, not inherited from any other class.
                            This is the __init__ method that you overrode in the FileInfo class. Note that the argument list in this ancestor class is different than the descendant. That's okay; each subclass can have
                                        its own set of arguments, as long as it calls the ancestor with the correct arguments. Here the ancestor class has a way
                                        to define initial values (by passing a dictionary in the dict argument) which the FileInfo does not use.
                            Python supports data attributes (called “instance variables” in Java and Powerbuilder, and “member variables” in C++). Data attributes are pieces of data held by a specific instance of a class. In this case, each instance of UserDict will have a data attribute data. To reference this attribute from code outside the class, you qualify it with the instance name, instance.data, in the same way that you qualify a function with its module name. To reference a data attribute from within the class,
                                        you use self as the qualifier. By convention, all data attributes are initialized to reasonable values in the __init__ method. However, this is not required, since data attributes, like local variables, spring into existence when they are first assigned a value.
                            The update method is a dictionary duplicator: it copies all the keys and values from one dictionary to another. This does not clear the target dictionary first; if the target dictionary already has some keys, the ones from the source dictionary will
                                        be overwritten, but others will be left untouched. Think of update as a merge function, not a copy function.
                            This is a syntax you may not have seen before (I haven't used it in the examples in this book). It's an if statement, but instead of having an indented block starting on the next line, there is just a single statement on the same
                                        line, after the colon. This is perfectly legal syntax, which is just a shortcut you can use when you have only one statement
                                        in a block. (It's like specifying a single statement without braces in C++.)  You can use this syntax, or you can have indented code on subsequent lines, but you can't do both for the same block.

                            Java and Powerbuilder support function overloading by argument list, i.e. one class can have multiple methods with the same name but a different number of arguments, or arguments of different types.
                                   Other languages (most notably PL/SQL) even support function overloading by argument name; i.e. one class can have multiple methods with the same name and the same number of arguments of the same type but different argument
                                  names. Python supports neither of these; it has no form of function overloading whatsoever. Methods are defined solely by their name,
                                  and there can be only one method per class with a given name. So if a descendant class has an __init__ method, it always overrides the ancestor __init__ method, even if the descendant defines it with a different argument list. And the same rule applies to any other method.

                            Guido, the original author of Python, explains method overriding this way: "Derived classes may override methods of their base classes. Because methods have no
                                  special privileges when calling other methods of the same object, a method of a base class that calls another method defined
                                  in the same base class, may in fact end up calling a method of a derived class that overrides it. (For C++ programmers: all methods in Python are effectively virtual.)"  If that doesn't make sense to you (it confuses the hell out of me), feel free to ignore it.
                                  I just thought I'd pass it along.

                            Always assign an initial value to all of an instance's data attributes in the __init__ method. It will save you hours of debugging later, tracking down AttributeError exceptions because you're referencing uninitialized (and therefore non-existent) attributes.
                            Example 5.10. UserDict Normal Methods
                                def clear(self): self.data.clear()          ①
                                def copy(self):           ②
                                    if self.__class__ is UserDict:          ③
                                        return UserDict(self.data)
                                    import copy           ④
                                    return copy.copy(self)
                                def keys(self): return self.data.keys()     ⑤
                                def items(self): return self.data.items()
                                def values(self): return self.data.values()

                            clear is a normal class method; it is publicly available to be called by anyone at any time. Notice that clear, like all class methods, has self as its first argument. (Remember that you don't include self when you call the method; it's something that Python adds for you.)  Also note the basic technique of this wrapper class: store a real dictionary (data) as a data attribute, define all the methods that a real dictionary has, and have each class method redirect to the corresponding
                                        method on the real dictionary. (In case you'd forgotten, a dictionary's clear method deletes all of its keys and their associated values.)
                            The copy method of a real dictionary returns a new dictionary that is an exact duplicate of the original (all the same key-value pairs).
                                         But UserDict can't simply redirect to self.data.copy, because that method returns a real dictionary, and what you want is to return a new instance that is the same class as self.
                            You use the __class__ attribute to see if self is a UserDict; if so, you're golden, because you know how to copy a UserDict: just create a new UserDict and give it the real dictionary that you've squirreled away in self.data. Then you immediately return the new UserDict you don't even get to the import copy on the next line.
                            If self.__class__ is not UserDict, then self must be some subclass of UserDict (like maybe FileInfo), in which case life gets trickier. UserDict doesn't know how to make an exact copy of one of its descendants; there could, for instance, be other data attributes defined
                                        in the subclass, so you would need to iterate through them and make sure to copy all of them. Luckily, Python comes with a module to do exactly this, and it's called copy. I won't go into the details here (though it's a wicked cool module, if you're ever inclined to dive into it on your own).
                                         Suffice it to say that copy can copy arbitrary Python objects, and that's how you're using it here.
                            The rest of the methods are straightforward, redirecting the calls to the built-in methods on self.data.

                            In versions of Python prior to 2.2, you could not directly subclass built-in datatypes like strings, lists, and dictionaries. To compensate for
                                  this, Python comes with wrapper classes that mimic the behavior of these built-in datatypes: UserString, UserList, and UserDict. Using a combination of normal and special methods, the UserDict class does an excellent imitation of a dictionary. In Python 2.2 and later, you can inherit classes directly from built-in datatypes like dict. An example of this is given in the examples that come with this book, in fileinfo_fromdict.py.
                            In Python, you can inherit directly from the dict built-in datatype, as shown in this example. There are three differences here compared to the UserDict version.
                            Example 5.11. Inheriting Directly from Built-In Datatype dict
                            class FileInfo(dict):①
                                "store file metadata"
                                def __init__(self, filename=None): ②
                                    self["name"] = filename

                            The first difference is that you don't need to import the UserDict module, since dict is a built-in datatype and is always available. The second is that you are inheriting from dict directly, instead of from UserDict.UserDict.
                            The third difference is subtle but important. Because of the way UserDict works internally, it requires you to manually call its __init__ method to properly initialize its internal data structures. dict does not work like this; it is not a wrapper, and it requires no explicit initialization.

                            Further Reading on UserDict

                            Python Library Reference documents the UserDict module and the copy module.

                            5.6. Special Class Methods
                            In addition to normal class methods, there are a number of special methods that Python classes can define. Instead of being called directly by your code (like normal methods), special methods are called for
                               you by Python in particular circumstances or when specific syntax is used.
                            As you saw in the previous section, normal methods go a long way towards wrapping a dictionary in a class. But normal methods alone are not enough, because
                            there are a lot of things you can do with dictionaries besides call methods on them. For starters, you can get and set items with a syntax that doesn't include explicitly invoking methods. This is where special class methods come in: they
                            provide a way to map non-method-calling syntax into method calls.
                            5.6.1. Getting and Setting Items
                            Example 5.12. The __getitem__ Special Method
                                def __getitem__(self, key): return self.data[key]>>> f = fileinfo.FileInfo("/music/_singles/kairo.mp3")
                            >>> f
                            {'name':'/music/_singles/kairo.mp3'}
                            >>> f.__getitem__("name") ①
                            '/music/_singles/kairo.mp3'
                            >>> f["name"]             ②
                            '/music/_singles/kairo.mp3'

                            The __getitem__ special method looks simple enough. Like the normal methods clear, keys, and values, it just redirects to the dictionary to return its value. But how does it get called?  Well, you can call __getitem__ directly, but in practice you wouldn't actually do that; I'm just doing it here to show you how it works. The right way
                                           to use __getitem__ is to get Python to call it for you.
                            This looks just like the syntax you would use to get a dictionary value, and in fact it returns the value you would expect. But here's the missing link: under the covers, Python has converted this syntax to the method call f.__getitem__("name"). That's why __getitem__ is a special class method; not only can you call it yourself, you can get Python to call it for you by using the right syntax.
                            Of course, Python has a __setitem__ special method to go along with __getitem__, as shown in the next example.
                            Example 5.13. The __setitem__ Special Method
                                def __setitem__(self, key, item): self.data[key] = item>>> f
                            {'name':'/music/_singles/kairo.mp3'}
                            >>> f.__setitem__("genre", 31) ①
                            >>> f
                            {'name':'/music/_singles/kairo.mp3', 'genre':31}
                            >>> f["genre"] = 32            ②
                            >>> f
                            {'name':'/music/_singles/kairo.mp3', 'genre':32}

                            Like the __getitem__ method, __setitem__ simply redirects to the real dictionary self.data to do its work. And like __getitem__, you wouldn't ordinarily call it directly like this; Python calls __setitem__ for you when you use the right syntax.
                            This looks like regular dictionary syntax, except of course that f is really a class that's trying very hard to masquerade as a dictionary, and __setitem__ is an essential part of that masquerade. This line of code actually calls f.__setitem__("genre", 32) under the covers.
                            __setitem__ is a special class method because it gets called for you, but it's still a class method. Just as easily as the __setitem__ method was defined in UserDict, you can redefine it in the descendant class to override the ancestor method. This allows you to define classes that act
                               like dictionaries in some ways but define their own behavior above and beyond the built-in dictionary.
                            This concept is the basis of the entire framework you're studying in this chapter. Each file type can have a handler class
                               that knows how to get metadata from a particular type of file. Once some attributes (like the file's name and location) are
                               known, the handler class knows how to derive other attributes automatically. This is done by overriding the __setitem__ method, checking for particular keys, and adding additional processing when they are found.
                            For example, MP3FileInfo is a descendant of FileInfo. When an MP3FileInfo's name is set, it doesn't just set the name key (like the ancestor FileInfo does); it also looks in the file itself for MP3 tags and populates a whole set of keys. The next example shows how this works.
                            Example 5.14. Overriding __setitem__ in MP3FileInfo
                                def __setitem__(self, key, item):         ①
                                    if key == "name" and item:            ②
                                        self.__parse(item)                ③
                                    FileInfo.__setitem__(self, key, item) ④

                            Notice that this __setitem__ method is defined exactly the same way as the ancestor method. This is important, since Python will be calling the method for you, and it expects it to be defined with a certain number of arguments. (Technically speaking,
                                           the names of the arguments don't matter; only the number of arguments is important.)
                            Here's the crux of the entire MP3FileInfo class: if you're assigning a value to the name key, you want to do something extra.
                            The extra processing you do for names is encapsulated in the __parse method. This is another class method defined in MP3FileInfo, and when you call it, you qualify it with self. Just calling __parse would look for a normal function defined outside the class, which is not what you want. Calling self.__parse will look for a class method defined within the class. This isn't anything new; you reference data attributes the same way.
                            After doing this extra processing, you want to call the ancestor method. Remember that this is never done for you in Python; you must do it manually. Note that you're calling the immediate ancestor, FileInfo, even though it doesn't have a __setitem__ method. That's okay, because Python will walk up the ancestor tree until it finds a class with the method you're calling, so this line of code will eventually
                                           find and call the __setitem__ defined in UserDict.

                            When accessing data attributes within a class, you need to qualify the attribute name: self.attribute. When calling other methods within a class, you need to qualify the method name: self.method.
                            Example 5.15. Setting an MP3FileInfo's name>>> import fileinfo
                            >>> mp3file = fileinfo.MP3FileInfo() ①
                            >>> mp3file
                            {'name':None}
                            >>> mp3file["name"] = "/music/_singles/kairo.mp3"      ②
                            >>> mp3file
                            {'album': 'Rave Mix', 'artist': '***DJ MARY-JANE***', 'genre': 31,
                            'title': 'KAIRO****THE BEST GOA', 'name': '/music/_singles/kairo.mp3',
                            'year': '2000', 'comment': 'http://mp3.com/DJMARYJANE'}
                            >>> mp3file["name"] = "/music/_singles/sidewinder.mp3" ③
                            >>> mp3file
                            {'album': '', 'artist': 'The Cynic Project', 'genre': 18, 'title': 'Sidewinder',
                            'name': '/music/_singles/sidewinder.mp3', 'year': '2000',
                            'comment': 'http://mp3.com/cynicproject'}

                            First, you create an instance of MP3FileInfo, without passing it a filename. (You can get away with this because the filename argument of the __init__ method is optional.)  Since MP3FileInfo has no __init__ method of its own, Python walks up the ancestor tree and finds the __init__ method of FileInfo. This __init__ method manually calls the __init__ method of UserDict and then sets the name key to filename, which is None, since you didn't pass a filename. Thus, mp3file initially looks like a dictionary with one key, name, whose value is None.

                            Now the real fun begins. Setting the name key of mp3file triggers the __setitem__ method on MP3FileInfo (not UserDict), which notices that you're setting the name key with a real value and calls self.__parse. Although you haven't traced through the __parse method yet, you can see from the output that it sets several other keys: album, artist, genre, title, year, and comment.

                            Modifying the name key will go through the same process again: Python calls __setitem__, which calls self.__parse, which sets all the other keys.

                            5.7. Advanced Special Class Methods
                            Python has more special methods than just __getitem__ and __setitem__. Some of them let you emulate functionality that you may not even know about.
                            This example shows some of the other special methods in UserDict.
                            Example 5.16. More Special Methods in UserDict
                                def __repr__(self): return repr(self.data)     ①
                                def __cmp__(self, dict):     ②
                                    if isinstance(dict, UserDict):
                                        return cmp(self.data, dict.data)
                                    else:
                                        return cmp(self.data, dict)
                                def __len__(self): return len(self.data)       ③
                                def __delitem__(self, key): del self.data[key] ④

                            __repr__ is a special method that is called when you call repr(instance). The repr function is a built-in function that returns a string representation of an object. It works on any object, not just class
                                        instances. You're already intimately familiar with repr and you don't even know it. In the interactive window, when you type just a variable name and press the ENTER key, Python uses repr to display the variable's value. Go create a dictionary d with some data and then print repr(d) to see for yourself.
                            __cmp__ is called when you compare class instances. In general, you can compare any two Python objects, not just class instances, by using ==. There are rules that define when built-in datatypes are considered equal; for instance, dictionaries are equal when they
                                        have all the same keys and values, and strings are equal when they are the same length and contain the same sequence of characters.
                                         For class instances, you can define the __cmp__ method and code the comparison logic yourself, and then you can use == to compare instances of your class and Python will call your __cmp__ special method for you.
                            __len__ is called when you call len(instance). The len function is a built-in function that returns the length of an object. It works on any object that could reasonably be thought
                                        of as having a length. The len of a string is its number of characters; the len of a dictionary is its number of keys; the len of a list or tuple is its number of elements. For class instances, define the __len__ method and code the length calculation yourself, and then call len(instance) and Python will call your __len__ special method for you.
                            __delitem__ is called when you call del instance[key], which you may remember as the way to delete individual items from a dictionary. When you use del on a class instance, Python calls the __delitem__ special method for you.

                            In Java, you determine whether two string variables reference the same physical memory location by using str1 == str2. This is called object identity, and it is written in Python as str1 is str2. To compare string values in Java, you would use str1.equals(str2); in Python, you would use str1 == str2. Java programmers who have been taught to believe that the world is a better place because == in Java compares by identity instead of by value may have a difficult time adjusting to Python's lack of such “gotchas”.
                            At this point, you may be thinking, “All this work just to do something in a class that I can do with a built-in datatype.”  And it's true that life would be easier (and the entire UserDict class would be unnecessary) if you could inherit from built-in datatypes like a dictionary. But even if you could, special
                            methods would still be useful, because they can be used in any class, not just wrapper classes like UserDict.
                            Special methods mean that any class can store key/value pairs like a dictionary, just by defining the __setitem__ method. Any class can act like a sequence, just by defining the __getitem__ method. Any class that defines the __cmp__ method can be compared with ==. And if your class represents something that has a length, don't define a GetLength method; define the __len__ method and use len(instance).

                            While other object-oriented languages only let you define the physical model of an object (“this object has a GetLength method”), Python's special class methods like __len__ allow you to define the logical model of an object (“this object has a length”).
                            Python has a lot of other special methods. There's a whole set of them that let classes act like numbers, allowing you to add,
                            subtract, and do other arithmetic operations on class instances. (The canonical example of this is a class that represents
                            complex numbers, numbers with both real and imaginary components.)  The __call__ method lets a class act like a function, allowing you to call a class instance directly. And there are other special methods
                            that allow classes to have read-only and write-only data attributes; you'll talk more about those in later chapters.

                            Further Reading on Special Class Methods

                            Python Reference Manual documents all the special class methods.

                            5.8. Introducing Class Attributes
                            You already know about data attributes, which are variables owned by a specific instance of a class. Python also supports class attributes, which are variables owned by the class itself.
                            Example 5.17. Introducing Class Attributes
                            class MP3FileInfo(FileInfo):
                                "store ID3v1.0 MP3 tags"
                                tagDataMap = {"title"   : (  3,  33, stripnulls),
                            "artist"  : ( 33,  63, stripnulls),
                            "album"   : ( 63,  93, stripnulls),
                            "year"    : ( 93,  97, stripnulls),
                            "comment" : ( 97, 126, stripnulls),
                            "genre"   : (127, 128, ord)}>>> import fileinfo
                            >>> fileinfo.MP3FileInfo            ①
                            <class fileinfo.MP3FileInfo at 01257FDC>
                            >>> fileinfo.MP3FileInfo.tagDataMap ②
                            {'title': (3, 33, <function stripnulls at 0260C8D4>),
                            'genre': (127, 128, <built-in function ord>),
                            'artist': (33, 63, <function stripnulls at 0260C8D4>),
                            'year': (93, 97, <function stripnulls at 0260C8D4>),
                            'comment': (97, 126, <function stripnulls at 0260C8D4>),
                            'album': (63, 93, <function stripnulls at 0260C8D4>)}
                            >>> m = fileinfo.MP3FileInfo()      ③
                            >>> m.tagDataMap
                            {'title': (3, 33, <function stripnulls at 0260C8D4>),
                            'genre': (127, 128, <built-in function ord>),
                            'artist': (33, 63, <function stripnulls at 0260C8D4>),
                            'year': (93, 97, <function stripnulls at 0260C8D4>),
                            'comment': (97, 126, <function stripnulls at 0260C8D4>),
                            'album': (63, 93, <function stripnulls at 0260C8D4>)}

                            MP3FileInfo is the class itself, not any particular instance of the class.
                            tagDataMap is a class attribute: literally, an attribute of the class. It is available before creating any instances of the class.
                            Class attributes are available both through direct reference to the class and through any instance of the class.

                            In Java, both static variables (called class attributes in Python) and instance variables (called data attributes in Python) are defined immediately after the class definition (one with the static keyword, one without). In Python, only class attributes can be defined here; data attributes are defined in the __init__ method.
                            Class attributes can be used as class-level constants (which is how you use them in MP3FileInfo), but they are not really constants. You can also change them.

                            There are no constants in Python. Everything can be changed if you try hard enough. This fits with one of the core principles of Python: bad behavior should be discouraged but not banned. If you really want to change the value of None, you can do it, but don't come running to me when your code is impossible to debug.
                            Example 5.18. Modifying Class Attributes>>> class counter:
                            ...    count = 0   ①
                            ...    def __init__(self):
                            ...        self.__class__.count += 1 ②
                            ...
                            >>> counter
                            <class __main__.counter at 010EAECC>
                            >>> counter.count   ③
                            0
                            >>> c = counter()
                            >>> c.count         ④
                            1
                            >>> counter.count
                            1
                            >>> d = counter()   ⑤
                            >>> d.count
                            2
                            >>> c.count
                            2
                            >>> counter.count
                            2

                            count is a class attribute of the counter class.
                            __class__ is a built-in attribute of every class instance (of every class). It is a reference to the class that self is an instance of (in this case, the counter class).
                            Because count is a class attribute, it is available through direct reference to the class, before you have created any instances of the
                                        class.
                            Creating an instance of the class calls the __init__ method, which increments the class attribute count by 1. This affects the class itself, not just the newly created instance.
                            Creating a second instance will increment the class attribute count again. Notice how the class attribute is shared by the class and all instances of the class.
                            5.9. Private Functions
                            Like most languages, Python has the concept of private elements:

                            Private functions, which can't be called from outside their module
                            Private class methods, which can't be called from outside their class
                            Private attributes, which can't be accessed from outside their class.

                            Unlike in most languages, whether a Python function, method, or attribute is private or public is determined entirely by its name.
                            If the name of a Python function, class method, or attribute starts with (but doesn't end with) two underscores, it's private; everything else is
                            public. Python has no concept of protected class methods (accessible only in their own class and descendant classes). Class methods are either private (accessible
                            only in their own class) or public (accessible from anywhere).
                            In MP3FileInfo, there are two methods: __parse and __setitem__. As you have already discussed, __setitem__ is a special method; normally, you would call it indirectly by using the dictionary syntax on a class instance, but it is public, and you could
                            call it directly (even from outside the fileinfo module) if you had a really good reason. However, __parse is private, because it has two underscores at the beginning of its name.

                            In Python, all special methods (like __setitem__) and built-in attributes (like __doc__) follow a standard naming convention: they both start with and end with two underscores. Don't name your own methods and
                                  attributes this way, because it will only confuse you (and others) later.
                            Example 5.19. Trying to Call a Private Method>>> import fileinfo
                            >>> m = fileinfo.MP3FileInfo()
                            >>> m.__parse("/music/_singles/kairo.mp3") ①
                            Traceback (innermost last):
                              File "<interactive input>", line 1, in ?
                            AttributeError: 'MP3FileInfo' instance has no attribute '__parse'

                            If you try to call a private method, Python will raise a slightly misleading exception, saying that the method does not exist. Of course it does exist, but it's private,
                                        so it's not accessible outside the class.Strictly speaking, private methods are accessible outside their class, just not easily accessible. Nothing in Python is truly private; internally, the names of private methods and attributes are mangled and unmangled on the fly to make them
                                        seem inaccessible by their given names. You can access the __parse method of the MP3FileInfo class by the name _MP3FileInfo__parse. Acknowledge that this is interesting, but promise to never, ever do it in real code. Private methods are private for a
                                        reason, but like many other things in Python, their privateness is ultimately a matter of convention, not force.

                            Further Reading on Private Functions

                            Python Tutorial discusses the inner workings of private variables.

                            5.10. Summary
                            That's it for the hard-core object trickery. You'll see a real-world application of special class methods in Chapter 12, which uses getattr to create a proxy to a remote web service.
                            The next chapter will continue using this code sample to explore other Python concepts, such as exceptions, file objects, and for loops.
                            Before diving into the next chapter, make sure you're comfortable doing all of these things:

                            Importing modules using either import module or from module import
                            Defining and instantiating classes

                            Defining __init__ methods and other special class methods, and understanding when they are called

                            Subclassing UserDict to define classes that act like dictionaries

                            Defining data attributes and class attributes, and understanding the differences between them

                            Defining private attributes and methods

                            Chapter 6. Exceptions and File Handling
                            In this chapter, you will dive into exceptions, file objects, for loops, and the os and sys modules. If you've used exceptions in another programming language, you can skim the first section to get a sense of Python's syntax. Be sure to tune in again for file handling.
                            6.1. Handling Exceptions
                            Like many other programming languages, Python has exception handling via try...except blocks.

                            Python uses try...except to handle exceptions and raise to generate them. Java and C++ use try...catch to handle exceptions, and throw to generate them.
                            Exceptions are everywhere in Python. Virtually every module in the standard Python library uses them, and Python itself will raise them in a lot of different circumstances. You've already seen them repeatedly throughout this book.

                            Accessing a non-existent dictionary key will raise a KeyError exception.

                            Searching a list for a non-existent value will raise a ValueError exception.

                            Calling a non-existent method will raise an AttributeError exception.

                            Referencing a non-existent variable will raise a NameError exception.

                            Mixing datatypes without coercion will raise a TypeError exception.

                            In each of these cases, you were simply playing around in the Python IDE: an error occurred, the exception was printed (depending on your IDE, perhaps in an intentionally jarring shade of red), and that was that. This is called an unhandled exception. When the exception was raised, there was no code to explicitly notice it and deal with it, so it bubbled its
                            way back to the default behavior built in to Python, which is to spit out some debugging information and give up. In the IDE, that's no big deal, but if that happened while your actual Python program was running, the entire program would come to a screeching halt.
                            An exception doesn't need result in a complete program crash, though. Exceptions, when raised, can be handled. Sometimes an exception is really because you have a bug in your code (like accessing a variable that doesn't exist), but
                            many times, an exception is something you can anticipate. If you're opening a file, it might not exist. If you're connecting
                            to a database, it might be unavailable, or you might not have the correct security credentials to access it. If you know
                            a line of code may raise an exception, you should handle the exception using a try...except block.
                            Example 6.1. Opening a Non-Existent File>>> fsock = open("/notthere", "r")      ①
                            Traceback (innermost last):
                              File "<interactive input>", line 1, in ?
                            IOError: [Errno 2] No such file or directory: '/notthere'
                            >>> try:
                            ...    fsock = open("/notthere")       ②
                            ... except IOError:   ③
                            ...    print "The file does not exist, exiting gracefully"
                            ... print "This line will always print" ④
                            The file does not exist, exiting gracefully
                            This line will always print

                            Using the built-in open function, you can try to open a file for reading (more on open in the next section). But the file doesn't exist, so this raises the IOError exception. Since you haven't provided any explicit check for an IOError exception, Python just prints out some debugging information about what happened and then gives up.
                            You're trying to open the same non-existent file, but this time you're doing it within a try...except block.
                            When the open method raises an IOError exception, you're ready for it. The except IOError: line catches the exception and executes your own block of code, which in this case just prints a more pleasant error message.
                            Once an exception has been handled, processing continues normally on the first line after the try...except block. Note that this line will always print, whether or not an exception occurs. If you really did have a file called
                            notthere in your root directory, the call to open would succeed, the except clause would be ignored, and this line would still be executed.
                            Exceptions may seem unfriendly (after all, if you don't catch the exception, your entire program will crash), but consider
                            the alternative. Would you rather get back an unusable file object to a non-existent file?  You'd need to check its validity
                            somehow anyway, and if you forgot, somewhere down the line, your program would give you strange errors somewhere down the
                            line that you would need to trace back to the source. I'm sure you've experienced this, and you know it's not fun. With
                            exceptions, errors occur immediately, and you can handle them in a standard way at the source of the problem.
                            6.1.1. Using Exceptions For Other Purposes
                            There are a lot of other uses for exceptions besides handling actual error conditions. A common use in the standard Python library is to try to import a module, and then check whether it worked. Importing a module that does not exist will raise
                               an ImportError exception. You can use this to define multiple levels of functionality based on which modules are available at run-time,
                               or to support multiple platforms (where platform-specific code is separated into different modules).
                            You can also define your own exceptions by creating a class that inherits from the built-in Exception class, and then raise your exceptions with the raise command. See the further reading section if you're interested in doing this.
                            The next example demonstrates how to use an exception to support platform-specific functionality. This code comes from the
                            getpass module, a wrapper module for getting a password from the user. Getting a password is accomplished differently on UNIX, Windows, and Mac OS platforms, but this code encapsulates all of those differences.
                            Example 6.2. Supporting Platform-Specific Functionality
                              # Bind the name getpass to the appropriate function
                              try:
                                  import termios, TERMIOS   ①
                              except ImportError:
                                  try:
                                      import msvcrt         ②
                                  except ImportError:
                                      try:
                                          from EasyDialogs import AskPassword ③
                                      except ImportError:
                                          getpass = default_getpass           ④
                                      else:                 ⑤
                                          getpass = AskPassword
                                  else:
                                      getpass = win_getpass
                              else:
                                  getpass = unix_getpass

                            termios is a UNIX-specific module that provides low-level control over the input terminal. If this module is not available (because it's not
                                           on your system, or your system doesn't support it), the import fails and Python raises an ImportError, which you catch.
                            OK, you didn't have termios, so let's try msvcrt, which is a Windows-specific module that provides an API to many useful functions in the Microsoft Visual C++ runtime services. If this import fails, Python will raise an ImportError, which you catch.
                            If the first two didn't work, you try to import a function from EasyDialogs, which is a Mac OS-specific module that provides functions to pop up dialog boxes of various types. Once again, if this import fails, Python will raise an ImportError, which you catch.
                            None of these platform-specific modules is available (which is possible, since Python has been ported to a lot of different platforms), so you need to fall back on a default password input function (which is
                                           defined elsewhere in the getpass module). Notice what you're doing here: assigning the function default_getpass to the variable getpass. If you read the official getpass documentation, it tells you that the getpass module defines a getpass function. It does this by binding getpass to the correct function for your platform. Then when you call the getpass function, you're really calling a platform-specific function that this code has set up for you. You don't need to know or
                                           care which platform your code is running on -- just call getpass, and it will always do the right thing.
                            A try...except block can have an else clause, like an if statement. If no exception is raised during the try block, the else clause is executed afterwards. In this case, that means that the from EasyDialogs import AskPassword import worked, so you should bind getpass to the AskPassword function. Each of the other try...except blocks has similar else clauses to bind getpass to the appropriate function when you find an import that works.

                            Further Reading on Exception Handling

                            Python Tutorial discusses defining and raising your own exceptions, and handling multiple exceptions at once.

                            Python Library Reference summarizes all the built-in exceptions.

                            Python Library Reference documents the getpass module.

                            Python Library Reference documents the traceback module, which provides low-level access to exception attributes after an exception is raised.

                            Python Reference Manual discusses the inner workings of the try...except block.

                            6.2. Working with File Objects
                            Python has a built-in function, open, for opening a file on disk. open returns a file object, which has methods and attributes for getting information about and manipulating the opened file.
                            Example 6.3. Opening a File>>> f = open("/music/_singles/kairo.mp3", "rb") ①
                            >>> f       ②
                            <open file '/music/_singles/kairo.mp3', mode 'rb' at 010E3988>
                            >>> f.mode  ③
                            'rb'
                            >>> f.name  ④
                            '/music/_singles/kairo.mp3'

                            The open method can take up to three parameters: a filename, a mode, and a buffering parameter. Only the first one, the filename,
                                        is required; the other two are optional. If not specified, the file is opened for reading in text mode. Here you are opening the file for reading in binary mode.
                                         (print open.__doc__ displays a great explanation of all the possible modes.)
                            The open function returns an object (by now, this should not surprise you). A file object has several useful attributes.
                            The mode attribute of a file object tells you in which mode the file was opened.
                            The name attribute of a file object tells you the name of the file that the file object has open.
                            6.2.1. Reading Files
                            After you open a file, the first thing you'll want to do is read from it, as shown in the next example.
                            Example 6.4. Reading a File
                            >>> f
                            <open file '/music/_singles/kairo.mp3', mode 'rb' at 010E3988>
                            >>> f.tell()              ①
                            0
                            >>> f.seek(-128, 2)       ②
                            >>> f.tell()              ③
                            7542909
                            >>> tagData = f.read(128) ④
                            >>> tagData
                            'TAGKAIRO****THE BEST GOA         ***DJ MARY-JANE***
                            Rave Mix    2000http://mp3.com/DJMARYJANE     \037'
                            >>> f.tell()              ⑤
                            7543037

                            A file object maintains state about the file it has open. The tell method of a file object tells you your current position in the open file. Since you haven't done anything with this file
                                           yet, the current position is 0, which is the beginning of the file.
                            The seek method of a file object moves to another position in the open file. The second parameter specifies what the first one means;
                            0 means move to an absolute position (counting from the start of the file), 1 means move to a relative position (counting from the current position), and 2 means move to a position relative to the end of the file. Since the MP3 tags you're looking for are stored at the end of the file, you use 2 and tell the file object to move to a position 128 bytes from the end of the file.
                            The tell method confirms that the current file position has moved.
                            The read method reads a specified number of bytes from the open file and returns a string with the data that was read. The optional
                                           parameter specifies the maximum number of bytes to read. If no parameter is specified, read will read until the end of the file. (You could have simply said read() here, since you know exactly where you are in the file and you are, in fact, reading the last 128 bytes.)  The read data
                                           is assigned to the tagData variable, and the current position is updated based on how many bytes were read.
                            The tell method confirms that the current position has moved. If you do the math, you'll see that after reading 128 bytes, the position
                                           has been incremented by 128.
                            6.2.2. Closing Files
                            Open files consume system resources, and depending on the file mode, other programs may not be able to access them. It's
                               important to close files as soon as you're finished with them.
                            Example 6.5. Closing a File
                            >>> f
                            <open file '/music/_singles/kairo.mp3', mode 'rb' at 010E3988>
                            >>> f.closed       ①
                            False
                            >>> f.close()      ②
                            >>> f
                            <closed file '/music/_singles/kairo.mp3', mode 'rb' at 010E3988>
                            >>> f.closed       ③
                            True
                            >>> f.seek(0)      ④
                            Traceback (innermost last):
                              File "<interactive input>", line 1, in ?
                            ValueError: I/O operation on closed file
                            >>> f.tell()
                            Traceback (innermost last):
                              File "<interactive input>", line 1, in ?
                            ValueError: I/O operation on closed file
                            >>> f.read()
                            Traceback (innermost last):
                              File "<interactive input>", line 1, in ?
                            ValueError: I/O operation on closed file
                            >>> f.close()      ⑤

                            The closed attribute of a file object indicates whether the object has a file open or not. In this case, the file is still open (closed is False).
                            To close a file, call the close method of the file object. This frees the lock (if any) that you were holding on the file, flushes buffered writes (if any)
                                           that the system hadn't gotten around to actually writing yet, and releases the system resources.
                            The closed attribute confirms that the file is closed.
                            Just because a file is closed doesn't mean that the file object ceases to exist. The variable f will continue to exist until it goes out of scope or gets manually deleted. However, none of the methods that manipulate an open file will work once the file has been closed;
                                           they all raise an exception.
                            Calling close on a file object whose file is already closed does not raise an exception; it fails silently.
                            6.2.3. Handling I/O Errors
                            Now you've seen enough to understand the file handling code in the fileinfo.py sample code from teh previous chapter. This example shows how to safely open and read from a file and gracefully handle
                               errors.
                            Example 6.6. File Objects in MP3FileInfo
                                    try:              ①
                                        fsock = open(filename, "rb", 0) ②
                                        try:
                                            fsock.seek(-128, 2)         ③
                                            tagdata = fsock.read(128)   ④
                                        finally:      ⑤
                                            fsock.close()
                                        .
                                        .
                                        .
                                    except IOError:   ⑥
                                        pass

                            Because opening and reading files is risky and may raise an exception, all of this code is wrapped in a try...except block. (Hey, isn't standardized indentation great?  This is where you start to appreciate it.)
                            The open function may raise an IOError. (Maybe the file doesn't exist.)
                            The seek method may raise an IOError. (Maybe the file is smaller than 128 bytes.)
                            The read method may raise an IOError. (Maybe the disk has a bad sector, or it's on a network drive and the network just went down.)
                            This is new: a try...finally block. Once the file has been opened successfully by the open function, you want to make absolutely sure that you close it, even if an exception is raised by the seek or read methods. That's what a try...finally block is for: code in the finally block will always be executed, even if something in the try block raises an exception. Think of it as code that gets executed on the way out, regardless of what happened before.
                            At last, you handle your IOError exception. This could be the IOError exception raised by the call to open, seek, or read. Here, you really don't care, because all you're going to do is ignore it silently and continue. (Remember, pass is a Python statement that does nothing.)  That's perfectly legal; “handling” an exception can mean explicitly doing nothing. It still counts as handled, and processing will continue normally on the
                                           next line of code after the try...except block.
                            6.2.4. Writing to Files
                            As you would expect, you can also write to files in much the same way that you read from them. There are two basic file modes:

                            "Append" mode will add data to the end of the file.
                            "write" mode will overwrite the file.

                            Either mode will create the file automatically if it doesn't already exist, so there's never a need for any sort of fiddly
                               "if the log file doesn't exist yet, create a new empty file just so you can open it for the first time" logic. Just open
                               it and start writing.
                            Example 6.7. Writing to Files
                            >>> logfile = open('test.log', 'w') ①
                            >>> logfile.write('test succeeded') ②
                            >>> logfile.close()
                            >>> print file('test.log').read()   ③
                            test succeeded
                            >>> logfile = open('test.log', 'a') ④
                            >>> logfile.write('line 2')
                            >>> logfile.close()
                            >>> print file('test.log').read()   ⑤
                            test succeededline 2

                            You start boldly by creating either the new file test.log or overwrites the existing file, and opening the file for writing. (The second parameter "w" means open the file for writing.)  Yes, that's all as dangerous as it sounds. I hope you didn't care about the previous
                                           contents of that file, because it's gone now.
                            You can add data to the newly opened file with the write method of the file object returned by open.
                            file is a synonym for open. This one-liner opens the file, reads its contents, and prints them.
                            You happen to know that test.log exists (since you just finished writing to it), so you can open it and append to it. (The "a" parameter means open the file for appending.)  Actually you could do this even if the file didn't exist, because opening
                                           the file for appending will create the file if necessary. But appending will never harm the existing contents of the file.
                            As you can see, both the original line you wrote and the second line you appended are now in test.log. Also note that carriage returns are not included. Since you didn't write them explicitly to the file either time, the
                                           file doesn't include them. You can write a carriage return with the "\n" character. Since you didn't do this, everything you wrote to the file ended up smooshed together on the same line.

                            Further Reading on File Handling

                            Python Tutorial discusses reading and writing files, including how to read a file one line at a time into a list.

                            eff-bot discusses efficiency and performance of various ways of reading a file.

                            Python Knowledge Base answers common questions about files.

                            Python Library Reference summarizes all the file object methods.

                            6.3. Iterating with for Loops
                            Like most other languages, Python has for loops. The only reason you haven't seen them until now is that Python is good at so many other things that you don't need them as often.
                            Most other languages don't have a powerful list datatype like Python, so you end up doing a lot of manual work, specifying a start, end, and step to define a range of integers or characters
                            or other iteratable entities. But in Python, a for loop simply iterates over a list, the same way list comprehensions work.
                            Example 6.8. Introducing the for Loop>>> li = ['a', 'b', 'e']
                            >>> for s in li:         ①
                            ...    print s          ②
                            a
                            b
                            e
                            >>> print "\n".join(li)  ③
                            a
                            b
                            e

                            The syntax for a for loop is similar to list comprehensions. li is a list, and s will take the value of each element in turn, starting from the first element.
                            Like an if statement or any other indented block, a for loop can have any number of lines of code in it.
                            This is the reason you haven't seen the for loop yet: you haven't needed it yet. It's amazing how often you use for loops in other languages when all you really want is a join or a list comprehension.
                            Doing a “normal” (by Visual Basic standards) counter for loop is also simple.
                            Example 6.9. Simple Counters
                            >>> for i in range(5):             ①
                            ...    print i
                            0
                            1
                            2
                            3
                            4
                            >>> li = ['a', 'b', 'c', 'd', 'e']
                            >>> for i in range(len(li)):       ②
                            ...    print li[i]
                            a
                            b
                            c
                            d
                            e

                            As you saw in Example 3.20, “Assigning Consecutive Values”, range produces a list of integers, which you then loop through. I know it looks a bit odd, but it is occasionally (and I stress
                            occasionally) useful to have a counter loop.
                            Don't ever do this. This is Visual Basic-style thinking. Break out of it. Just iterate through the list, as shown in the previous example.
                            for loops are not just for simple counters. They can iterate through all kinds of things. Here is an example of using a for loop to iterate through a dictionary.
                            Example 6.10. Iterating Through a Dictionary
                            >>> import os
                            >>> for k, v in os.environ.items():      ① ②
                            ...    print "%s=%s" % (k, v)
                            USERPROFILE=C:\Documents and Settings\mpilgrim
                            OS=Windows_NT
                            COMPUTERNAME=MPILGRIM
                            USERNAME=mpilgrim

                            [...snip...]
                            >>> print "\n".join(["%s=%s" % (k, v)
                            ...    for k, v in os.environ.items()]) ③
                            USERPROFILE=C:\Documents and Settings\mpilgrim
                            OS=Windows_NT
                            COMPUTERNAME=MPILGRIM
                            USERNAME=mpilgrim

                            [...snip...]

                            os.environ is a dictionary of the environment variables defined on your system. In Windows, these are your user and system variables
                                        accessible from MS-DOS. In UNIX, they are the variables exported in your shell's startup scripts. In Mac OS, there is no concept of environment variables, so this dictionary is empty.
                            os.environ.items() returns a list of tuples: [(key1, value1), (key2, value2), ...]. The for loop iterates through this list. The first round, it assigns key1 to k and value1 to v, so k = USERPROFILE and v = C:\Documents and Settings\mpilgrim. In the second round, k gets the second key, OS, and v gets the corresponding value, Windows_NT.
                            With multi-variable assignment and list comprehensions, you can replace the entire for loop with a single statement. Whether you actually do this in real code is a matter of personal coding style. I like it
                                        because it makes it clear that what I'm doing is mapping a dictionary into a list, then joining the list into a single string.
                                         Other programmers prefer to write this out as a for loop. The output is the same in either case, although this version is slightly faster, because there is only one print statement instead of many.
                            Now we can look at the for loop in MP3FileInfo, from the sample fileinfo.py program introduced in Chapter 5.
                            Example 6.11. for Loop in MP3FileInfo
                                tagDataMap = {"title"   : (  3,  33, stripnulls),
                            "artist"  : ( 33,  63, stripnulls),
                            "album"   : ( 63,  93, stripnulls),
                            "year"    : ( 93,  97, stripnulls),
                            "comment" : ( 97, 126, stripnulls),
                            "genre"   : (127, 128, ord)}             ①
                                .
                                .
                                .
                                        if tagdata[:3] == "TAG":
                                            for tag, (start, end, parseFunc) in self.tagDataMap.items(): ②
                              self[tag] = parseFunc(tagdata[start:end])                ③

                            tagDataMap is a class attribute that defines the tags you're looking for in an MP3 file. Tags are stored in fixed-length fields. Once you read the last 128 bytes of the file, bytes 3 through 32 of those
                                        are always the song title, 33 through 62 are always the artist name, 63 through 92 are the album name, and so forth. Note
                                        that tagDataMap is a dictionary of tuples, and each tuple contains two integers and a function reference.
                            This looks complicated, but it's not. The structure of the for variables matches the structure of the elements of the list returned by items. Remember that items returns a list of tuples of the form (key, value). The first element of that list is ("title", (3, 33, <function stripnulls>)), so the first time around the loop, tag gets "title", start gets 3, end gets 33, and parseFunc gets the function stripnulls.
                            Now that you've extracted all the parameters for a single MP3 tag, saving the tag data is easy. You slice tagdata from start to end to get the actual data for this tag, call parseFunc to post-process the data, and assign this as the value for the key tag in the pseudo-dictionary self. After iterating through all the elements in tagDataMap, self has the values for all the tags, and you know what that looks like.
                            6.4. Using sys.modules
                            Modules, like everything else in Python, are objects. Once imported, you can always get a reference to a module through the global dictionary sys.modules.
                            Example 6.12. Introducing sys.modules>>> import sys        ①
                            >>> print '\n'.join(sys.modules.keys()) ②
                            win32api
                            os.path
                            os
                            exceptions
                            __main__
                            ntpath
                            nt
                            sys
                            __builtin__
                            site
                            signal
                            UserDict
                            stat

                        1. The `sys` module contains system-level information, such as the version of Python you're running (``sys`.version` or ``sys`.version_info`), and system-level options such as the maximum allowed recursion depth (``sys`.getrecursionlimit()` and ``sys`.setrecursionlimit()`).
                        2. ``sys`.modules` is a dictionary containing all the modules that have ever been imported since Python was started; the key is the module name, the value is the module object. Note that this is more than just the modules your program has imported. Python preloads some modules on startup, and if you're using a Python IDE, ``sys`.modules` contains all the modules imported by all the programs you've run within the IDE.

                           This example demonstrates how to use ``sys`.modules`.

                           ### Example 6.13. Using ``sys`.modules`

                               >>> import fileinfo         ①
                               >>> print '\n'.join(sys.modules.keys())
                               win32api
                               os.path
                               os
                               fileinfo
                               exceptions
                               __main__
                               ntpath
                               nt
                               sys
                               __builtin__
                               site
                               signal
                               UserDict
                               stat
                               >>> fileinfo
                               <module 'fileinfo' from 'fileinfo.pyc'>
                               >>> sys.modules["fileinfo"] ②
                               <module 'fileinfo' from 'fileinfo.pyc'>

                           1. As new modules are imported, they are added to ``sys`.modules`. This explains why importing the same module twice is very fast: Python has already loaded and cached the module in ``sys`.modules`, so importing the second time is simply a dictionary lookup.
                           2. Given the name (as a string) of any previously-imported module, you can get a reference to the module itself through the ``sys`.modules` dictionary.

                              The next example shows how to use the `__module__` class attribute with the ``sys`.modules` dictionary to get a reference to the module in which a class is defined.

                              ### Example 6.14. The `__module__` Class Attribute

                                  >>> from fileinfo import MP3FileInfo
                                  >>> MP3FileInfo.__module__              ①
                                  'fileinfo'
                                  >>> sys.modules[MP3FileInfo.__module__] ②
                                  <module 'fileinfo' from 'fileinfo.pyc'>

                              1. Every Python class has a built-in class attribute `__module__`, which is the name of the module in which the class is defined.
                              2. Combining this with the ``sys`.modules` dictionary, you can get a reference to the module in which a class is defined.

                                 Now you're ready to see how ``sys`.modules` is used in `fileinfo.py`, the sample program introduced in Chapter 5. This example shows that portion of the code.

                                 ### Example 6.15. ``sys`.modules` in `fileinfo.py`

                                         def getFileInfoClass(filename, module=sys.modules[FileInfo.__module__]):       ①
                                             "get file info class from filename extension"
                                             subclass = "%sFileInfo" % os.path.splitext(filename)[1].upper()[1:]        ②
                                             return hasattr(module, subclass) and getattr(module, subclass) or FileInfo ③

                                 1. This is a function with two arguments; filename is required, but module is optional and defaults to the module that contains the `FileInfo` class. This looks inefficient, because you might expect Python to evaluate the ``sys`.modules` expression every time the function is called. In fact, Python evaluates default expressions only once, the first time the module is imported. As you'll see later, you never call this function with a module argument, so module serves as a function-level constant.
                                 2. You'll plow through this line later, after you dive into the `os` module. For now, take it on faith that subclass ends up as the name of a class, like `MP3FileInfo`.
                                 3. You already know about `getattr`, which gets a reference to an object by name. `hasattr` is a complementary function that checks whether an object has a particular attribute; in this case, whether a module has a particular class (although it works for any object and any attribute, just like `getattr`). In English, this line of code says, “If this module has the class named by subclass then return it, otherwise return the base class `FileInfo`.”

                                    ### Further Reading on Modules

                                    - Python Tutorial discusses exactly when and how default arguments are evaluated.
                                    - Python Library Reference documents the `sys` module.

                                    ## 6.5. Working with Directories

                                    The `os.path` module has several functions for manipulating files and directories. Here, we're looking at handling pathnames and listing the contents of a directory.

                                    ### Example 6.16. Constructing Pathnames

                                        >>> import os
                                        >>> os.path.join("c:\\music\\ap\\", "mahadeva.mp3") ① ②
                                        'c:\\music\\ap\\mahadeva.mp3'
                                        >>> os.path.join("c:\\music\\ap", "mahadeva.mp3")   ③
                                        'c:\\music\\ap\\mahadeva.mp3'
                                        >>> os.path.expanduser("~")       ④
                                        'c:\\Documents and Settings\\mpilgrim\\My Documents'
                                        >>> os.path.join(os.path.expanduser("~"), "Python") ⑤
                                        'c:\\Documents and Settings\\mpilgrim\\My Documents\\Python'

                                    1. `os.path` is a reference to a module -- which module depends on your platform. Just as `getpass` encapsulates differences between platforms by setting getpass to a platform-specific function, `os` encapsulates differences between platforms by setting path to a platform-specific module.
                                    2. The `join` function of `os.path` constructs a pathname out of one or more partial pathnames. In this case, it simply concatenates strings. (Note that dealing with pathnames on Windows is annoying because the backslash character must be escaped.)
                                    3. In this slightly less trivial case, `join` will add an extra backslash to the pathname before joining it to the filename. I was overjoyed when I discovered this, since `addSlashIfNecessary` is one of the stupid little functions I always need to write when building up my toolbox in a new language. Do not write this stupid little function in Python; smart people have already taken care of it for you.
                                    4. `expanduser` will expand a pathname that uses `~` to represent the current user's home directory. This works on any platform where users have a home directory, like Windows, UNIX, and Mac OS X; it has no effect on Mac OS.
                                    5. Combining these techniques, you can easily construct pathnames for directories and files under the user's home directory.

                                       ### Example 6.17. Splitting Pathnames

                                           >>> os.path.split("c:\\music\\ap\\mahadeva.mp3")      ①
                                           ('c:\\music\\ap', 'mahadeva.mp3')
                                           >>> (filepath, filename) = os.path.split("c:\\music\\ap\\mahadeva.mp3") ②
                                           >>> filepath      ③
                                           'c:\\music\\ap'
                                           >>> filename      ④
                                           'mahadeva.mp3'
                                           >>> (shortname, extension) = os.path.splitext(filename)                 ⑤
                                           >>> shortname
                                           'mahadeva'
                                           >>> extension
                                           '.mp3'

                                       1. The `split` function splits a full pathname and returns a tuple containing the path and filename. Remember when I said you could use multi-variable assignment to return multiple values from a function? Well, `split` is such a function.
                                       2. You assign the return value of the `split` function into a tuple of two variables. Each variable receives the value of the corresponding element of the returned tuple.
                                       3. The first variable, filepath, receives the value of the first element of the tuple returned from `split`, the file path.
                                       4. The second variable, filename, receives the value of the second element of the tuple returned from `split`, the filename.
                                       5. `os.path` also contains a function `splitext`, which splits a filename and returns a tuple containing the filename and the file extension. You use the same technique to assign each of them to separate variables.

                                          ### Example 6.18. Listing Directories

                                              >>> os.listdir("c:\\music\\_singles\\")              ①
                                              ['a_time_long_forgotten_con.mp3', 'hellraiser.mp3',
                                              'kairo.mp3', 'long_way_home1.mp3', 'sidewinder.mp3',
                                              'spinning.mp3']
                                              >>> dirname = "c:\\"
                                              >>> os.listdir(dirname)            ②
                                              ['AUTOEXEC.BAT', 'boot.ini', 'CONFIG.SYS', 'cygwin',
                                              'docbook', 'Documents and Settings', 'Incoming', 'Inetpub', 'IO.SYS',
                                              'MSDOS.SYS', 'Music', 'NTDETECT.COM', 'ntldr', 'pagefile.sys',
                                              'Program Files', 'Python20', 'RECYCLER',
                                              'System Volume Information', 'TEMP', 'WINNT']
                                              >>> [f for f in os.listdir(dirname)
                                              ...    if os.path.isfile(os.path.join(dirname, f))] ③
                                              ['AUTOEXEC.BAT', 'boot.ini', 'CONFIG.SYS', 'IO.SYS', 'MSDOS.SYS',
                                              'NTDETECT.COM', 'ntldr', 'pagefile.sys']
                                              >>> [f for f in os.listdir(dirname)
                                              ...    if os.path.isdir(os.path.join(dirname, f))]  ④
                                              ['cygwin', 'docbook', 'Documents and Settings', 'Incoming',
                                              'Inetpub', 'Music', 'Program Files', 'Python20', 'RECYCLER',
                                              'System Volume Information', 'TEMP', 'WINNT']

                                          1. The `listdir` function takes a pathname and returns a list of the contents of the directory.
                                          2. `listdir` returns both files and folders, with no indication of which is which.
                                          3. You can use list filtering and the `isfile` function of the `os.path` module to separate the files from the folders. `isfile` takes a pathname and returns 1 if the path represents a file, and 0 otherwise. Here you're using ``os.path`.`join`` to ensure a full pathname, but `isfile` also works with a partial path, relative to the current working directory. You can use `os.getcwd()` to get the current working directory.
                                          4. `os.path` also has a `isdir` function which returns 1 if the path represents a directory, and 0 otherwise. You can use this to get a list of the subdirectories within a directory.

                                             ### Example 6.19. Listing Directories in `fileinfo.py`

                                                 def listDirectory(directory, fileExtList):
                                                     "get list of file info objects for files of particular extensions"
                                                     fileList = [os.path.normcase(f)
                                                                 for f in os.listdir(directory)]            ① ②
                                                     fileList = [os.path.join(directory, f)
                                                                for f in fileList
                                                                 if os.path.splitext(f)[1] in fileExtList]  ③ ④ ⑤

                                             1. `os.listdir(directory)` returns a list of all the files and folders in directory.
                                             2. Iterating through the list with f, you use `os.path.normcase(f)` to normalize the case according to operating system defaults. `normcase` is a useful little function that compensates for case-insensitive operating systems that think that `mahadeva.mp3` and `mahadeva.MP3` are the same file. For instance, on Windows and Mac OS, `normcase` will convert the entire filename to lowercase; on UNIX-compatible systems, it will return the filename unchanged.
                                             3. Iterating through the normalized list with f again, you use `os.path.splitext(f)` to split each filename into name and extension.
                                             4. For each file, you see if the extension is in the list of file extensions you care about (fileExtList, which was passed to the `listDirectory` function).
                                             5. For each file you care about, you use `os.path.join(directory, f)` to construct the full pathname of the file, and return a list of the full pathnames.

                                                Whenever possible, you should use the functions in `os` and `os.path` for file, directory, and path manipulations. These modules are wrappers for platform-specific modules, so functions like `os.path.split` work on UNIX, Windows, Mac OS, and any other platform supported by Python.

                                                There is one other way to get the contents of a directory. It's very powerful, and it uses the sort of wildcards that you may already be familiar with from working on the command line.

                                                ### Example 6.20. Listing Directories with `glob`

                                                    >>> os.listdir("c:\\music\\_singles\\")               ①
                                                    ['a_time_long_forgotten_con.mp3', 'hellraiser.mp3',
                                                    'kairo.mp3', 'long_way_home1.mp3', 'sidewinder.mp3',
                                                    'spinning.mp3']
                                                    >>> import glob
                                                    >>> glob.glob('c:\\music\\_singles\\*.mp3')           ②
                                                    ['c:\\music\\_singles\\a_time_long_forgotten_con.mp3',
                                                    'c:\\music\\_singles\\hellraiser.mp3',
                                                    'c:\\music\\_singles\\kairo.mp3',
                                                    'c:\\music\\_singles\\long_way_home1.mp3',
                                                    'c:\\music\\_singles\\sidewinder.mp3',
                                                    'c:\\music\\_singles\\spinning.mp3']
                                                    >>> glob.glob('c:\\music\\_singles\\s*.mp3')          ③
                                                    ['c:\\music\\_singles\\sidewinder.mp3',
                                                    'c:\\music\\_singles\\spinning.mp3']
                                                    >>> glob.glob('c:\\music\\*\\*.mp3')④

                                                1. As you saw earlier, `os.listdir` simply takes a directory path and lists all files and directories in that directory.
                                                2. The `glob` module, on the other hand, takes a wildcard and returns the full path of all files and directories matching the wildcard. Here the wildcard is a directory path plus "*.mp3", which will match all `.mp3` files. Note that each element of the returned list already includes the full path of the file.
                                                3. If you want to find all the files in a specific directory that start with "s" and end with ".mp3", you can do that too.
                                                4. Now consider this scenario: you have a `music` directory, with several subdirectories within it, with `.mp3` files within each subdirectory. You can get a list of all of those with a single call to `glob`, by using two wildcards at once. One wildcard is the `"*.mp3"` (to match `.mp3` files), and one wildcard is within the directory path itself, to match any subdirectory within `c:\music`. That's a crazy amount of power packed into one deceptively simple-looking function!

                                                   ### Further Reading on the `os` Module

                                                   - Python Knowledge Base answers questions about the `os` module.
                                                   - Python Library Reference documents the `os` module and the `os.path` module.

                                                   ## 6.6. Putting It All Together

                                                   Once again, all the dominoes are in place. You've seen how each line of code works. Now let's step back and see how it all fits together.

                                                   ### Example 6.21. `listDirectory`

                                                       def listDirectory(directory, fileExtList):     ①
                                                           "get list of file info objects for files of particular extensions"
                                                           fileList = [os.path.normcase(f)
                                                                       for f in os.listdir(directory)]
                                                           fileList = [os.path.join(directory, f)
                                                                      for f in fileList
                                                                       if os.path.splitext(f)[1] in fileExtList]        ②
                                                           def getFileInfoClass(filename, module=sys.modules[FileInfo.__module__]):       ③
                                                               "get file info class from filename extension"
                                                               subclass = "%sFileInfo" % os.path.splitext(filename)[1].upper()[1:]        ④
                                                               return hasattr(module, subclass) and getattr(module, subclass) or FileInfo ⑤
                                                           return [getFileInfoClass(f)(f) for f in fileList]            ⑥

                                                   1. `listDirectory` is the main attraction of this entire module. It takes a directory (like `c:\music\_singles\` in my case) and a list of interesting file extensions (like `['.mp3']`), and it returns a list of class instances that act like dictionaries that contain metadata about each interesting file in that directory. And it does it in just a few straightforward lines of code.
                                                   2. As you saw in the previous section, this line of code gets a list of the full pathnames of all the files in directory that have an interesting file extension (as specified by fileExtList).
                                                   3. Old-school Pascal programmers may be familiar with them, but most people give me a blank stare when I tell them that Python supports nested functions -- literally, a function within a function. The nested function `getFileInfoClass` can be called only from the function in which it is defined, `listDirectory`. As with any other function, you don't need an interface declaration or anything fancy; just define the function and code it.
                                                   4. Now that you've seen the `os` module, this line should make more sense. It gets the extension of the file (`os.path.splitext(filename)[1]`), forces it to uppercase (`.upper()`), slices off the dot (`[1:]`), and constructs a class name out of it with string formatting. So `c:\music\ap\mahadeva.mp3` becomes `.mp3` becomes `.MP3` becomes `MP3` becomes `MP3FileInfo`.
                                                   5. Having constructed the name of the handler class that would handle this file, you check to see if that handler class actually exists in this module. If it does, you return the class, otherwise you return the base class `FileInfo`. This is a very important point: this function returns a class. Not an instance of a class, but the class itself.
                                                   6. For each file in the “interesting files” list (fileList), you call `getFileInfoClass` with the filename (f). Calling `getFileInfoClass(f)` returns a class; you don't know exactly which class, but you don't care. You then create an instance of this class (whatever it is) and pass the filename (f again), to the `__init__` method. As you saw earlier in this chapter, the `__init__` method of `FileInfo` sets `self["name"]`, which triggers `__setitem__`, which is overridden in the descendant (`MP3FileInfo`) to parse the file appropriately to pull out the file's metadata. You do all that for each interesting file and return a list of the resulting instances.

                                                      Note that `listDirectory` is completely generic. It doesn't know ahead of time which types of files it will be getting, or which classes are defined that could potentially handle those files. It inspects the directory for the files to process, and then introspects its own module to see what special handler classes (like `MP3FileInfo`) are defined. You can extend this program to handle other types of files simply by defining an appropriately-named class: `HTMLFileInfo` for HTML files, `DOCFileInfo` for Word `.doc` files, and so forth. `listDirectory` will handle them all, without modification, by handing off the real work to the appropriate classes and collating the results.

                                                      ## 6.7. Summary

                                                      The `fileinfo.py` program introduced in Chapter 5 should now make perfect sense.

                                                          """Framework for getting filetype-specific metadata.

                                                          Instantiate appropriate class with filename. Returned object acts like a
                                                          dictionary, with key-value pairs for each piece of metadata.
                                                              import fileinfo
                                                              info = fileinfo.MP3FileInfo("/music/ap/mahadeva.mp3")
                                                              print "\\n".join(["%s=%s" % (k, v) for k, v in info.items()])

                                                          Or use listDirectory function to get info on all files in a directory.
                                                              for info in fileinfo.listDirectory("/music/ap/", [".mp3"]):
                                                                  ...

                                                          Framework can be extended by adding classes for particular file types, e.g.
                                                          HTMLFileInfo, MPGFileInfo, DOCFileInfo. Each class is completely responsible for
                                                          parsing its files appropriately; see MP3FileInfo for example.
                                                          """
                                                          import os
                                                          import sys
                                                          from UserDict import UserDict

                                                          def stripnulls(data):
                                                              "strip whitespace and nulls"
                                                              return data.replace("\00", "").strip()

                                                          class FileInfo(UserDict):
                                                              "store file metadata"
                                                              def __init__(self, filename=None):
                                                                  UserDict.__init__(self)
                                                                  self["name"] = filename

                                                          class MP3FileInfo(FileInfo):
                                                              "store ID3v1.0 MP3 tags"
                                                              tagDataMap = {"title"   : (  3,  33, stripnulls),
                                                          "artist"  : ( 33,  63, stripnulls),
                                                          "album"   : ( 63,  93, stripnulls),
                                                          "year"    : ( 93,  97, stripnulls),
                                                          "comment" : ( 97, 126, stripnulls),
                                                          "genre"   : (127, 128, ord)}

                                                              def __parse(self, filename):
                                                                  "parse ID3v1.0 tags from MP3 file"
                                                                  self.clear()
                                                                  try:
                                                                      fsock = open(filename, "rb", 0)
                                                                      try:
                                                                          fsock.seek(-128, 2)
                                                                          tagdata = fsock.read(128)
                                                                      finally:
                                                                          fsock.close()
                                                                      if tagdata[:3] == "TAG":
                                                                          for tag, (start, end, parseFunc) in self.tagDataMap.items():
                                                            self[tag] = parseFunc(tagdata[start:end])
                                                                  except IOError:
                                                                      pass

                                                              def __setitem__(self, key, item):
                                                                  if key == "name" and item:
                                                                      self.__parse(item)
                                                                  FileInfo.__setitem__(self, key, item)

                                                          def listDirectory(directory, fileExtList):
                                                              "get list of file info objects for files of particular extensions"
                                                              fileList = [os.path.normcase(f)
                                                                          for f in os.listdir(directory)]
                                                              fileList = [os.path.join(directory, f)
                                                                         for f in fileList
                                                                          if os.path.splitext(f)[1] in fileExtList]
                                                              def getFileInfoClass(filename, module=sys.modules[FileInfo.__module__]):
                                                                  "get file info class from filename extension"
                                                                  subclass = "%sFileInfo" % os.path.splitext(filename)[1].upper()[1:]
                                                                  return hasattr(module, subclass) and getattr(module, subclass) or FileInfo
                                                              return [getFileInfoClass(f)(f) for f in fileList]

                                                          if __name__ == "__main__":
                                                              for info in listDirectory("/music/_singles/", [".mp3"]):
                                                                  print "\n".join(["%s=%s" % (k, v) for k, v in info.items()])
                                                                  print

                                                      Before diving into the next chapter, make sure you're comfortable doing the following things:

                                                      - Catching exceptions with `try...except`
                                                      - Protecting external resources with `try...finally`
                                                      - Reading from files
                                                      - Assigning multiple values at once in a `for` loop
                                                      - Using the `os` module for all your cross-platform file manipulation needs
                                                      - Dynamically instantiating classes of unknown type by treating classes as objects and passing them around

                                                      ## Chapter 8. HTML Processing

                                                      ## 8.1. Diving in

                                                      I often see questions on comp.lang.python like “How can I list all the [headers|images|links] in my HTML document?” “How do I parse/translate/munge the text of my HTML document but leave the tags alone?” “How can I add/remove/quote attributes of all my HTML tags at once?” This chapter will answer all of these questions.

                                                      Here is a complete, working Python program in two parts. The first part, `BaseHTMLProcessor.py`, is a generic tool to help you process HTML files by walking through the tags and text blocks. The second part, `dialect.py`, is an example of how to use `BaseHTMLProcessor.py` to translate the text of an HTML document but leave the tags alone. Read the `docstring`s and comments to get an overview of what's going on. Most of it will seem like black magic, because it's not obvious how any of these class methods ever get called. Don't worry, all will be revealed in due time.

                                                      ### Example 8.1. `BaseHTMLProcessor.py`

                                                      If you have not already done so, you can download this and other examples used in this book.

                                                          from sgmllib import SGMLParser
                                                          import htmlentitydefs

                                                          class BaseHTMLProcessor(SGMLParser):
                                                              def reset(self):
                                                                  # extend (called by SGMLParser.__init__)
                                                                  self.pieces = []
                                                                  SGMLParser.reset(self)

                                                              def unknown_starttag(self, tag, attrs):
                                                                  # called for each start tag
                                                                  # attrs is a list of (attr, value) tuples
                                                                  # e.g. for <pre class=screen>, tag="pre", attrs=[("class", "screen")]
                                                                  # Ideally we would like to reconstruct original tag and attributes, but
                                                                  # we may end up quoting attribute values that weren't quoted in the source
                                                                  # document, or we may change the type of quotes around the attribute value
                                                                  # (single to double quotes).
                                                                  # Note that improperly embedded non-HTML code (like client-side Javascript)
                                                                  # may be parsed incorrectly by the ancestor, causing runtime script errors.
                                                                  # All non-HTML code must be enclosed in HTML comment tags (<!-- code -->)
                                                                  # to ensure that it will pass through this parser unaltered (in handle_comment).
                                                                  strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs])
                                                                  self.pieces.append("<%(tag)s%(strattrs)s>" % locals())

                                                              def unknown_endtag(self, tag):
                                                                  # called for each end tag, e.g. for </pre>, tag will be "pre"
                                                                  # Reconstruct the original end tag.
                                                                  self.pieces.append("</%(tag)s>" % locals())

                                                              def handle_charref(self, ref):
                                                                  # called for each character reference, e.g. for "&#160;", ref will be "160"
                                                                  # Reconstruct the original character reference.
                                                                  self.pieces.append("&#%(ref)s;" % locals())

                                                              def handle_entityref(self, ref):
                                                                  # called for each entity reference, e.g. for "&copy;", ref will be "copy"
                                                                  # Reconstruct the original entity reference.
                                                                  self.pieces.append("&%(ref)s" % locals())
                                                                  # standard HTML entities are closed with a semicolon; other entities are not
                                                                  if htmlentitydefs.entitydefs.has_key(ref):
                                                                      self.pieces.append(";")

                                                              def handle_data(self, text):
                                                                  # called for each block of plain text, i.e. outside of any tag and
                                                                  # not containing any character or entity references
                                                                  # Store the original text verbatim.
                                                                  self.pieces.append(text)

                                                              def handle_comment(self, text):
                                                                  # called for each HTML comment, e.g. <!-- insert Javascript code here -->
                                                                  # Reconstruct the original comment.
                                                                  # It is especially important that the source document enclose client-side
                                                                  # code (like Javascript) within comments so it can pass through this
                                                                  # processor undisturbed; see comments in unknown_starttag for details.
                                                                  self.pieces.append("<!--%(text)s-->" % locals())

                                                              def handle_pi(self, text):
                                                                  # called for each processing instruction, e.g. <?instruction>
                                                                  # Reconstruct original processing instruction.
                                                                  self.pieces.append("<?%(text)s>" % locals())

                                                              def handle_decl(self, text):
                                                                  # called for the DOCTYPE, if present, e.g.
                                                                  # <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
                                                                  #     "http://www.w3.org/TR/html4/loose.dtd">
                                                                  # Reconstruct original DOCTYPE
                                                                  self.pieces.append("<!%(text)s>" % locals())

                                                              def output(self):
                                                                  """Return processed HTML as a single string"""
                                                                  return "".join(self.pieces)

                                                      ### Example 8.2. `dialect.py`

                                                          import re
                                                          from BaseHTMLProcessor import BaseHTMLProcessor

                                                          class Dialectizer(BaseHTMLProcessor):
                                                              subs = ()

                                                              def reset(self):
                                                                  # extend (called from __init__ in ancestor)
                                                                  # Reset all data attributes
                                                                  self.verbatim = 0
                                                                  BaseHTMLProcessor.reset(self)

                                                              def start_pre(self, attrs):
                                                                  # called for every <pre> tag in HTML source
                                                                  # Increment verbatim mode count, then handle tag like normal
                                                                  self.verbatim += 1
                                                                  self.unknown_starttag("pre", attrs)

                                                              def end_pre(self):
                                                                  # called for every </pre> tag in HTML source
                                                                  # Decrement verbatim mode count
                                                                  self.unknown_endtag("pre")
                                                                  self.verbatim -= 1

                                                              def handle_data(self, text):
                                                                  # override
                                                                  # called for every block of text in HTML source
                                                                  # If in verbatim mode, save text unaltered;
                                                                  # otherwise process the text with a series of substitutions
                                                                  self.pieces.append(self.verbatim and text or self.process(text))

                                                              def process(self, text):
                                                                  # called from handle_data
                                                                  # Process text block by performing series of regular expression
                                                                  # substitutions (actual substitions are defined in descendant)
                                                                  for fromPattern, toPattern in self.subs:
                                                                      text = re.sub(fromPattern, toPattern, text)
                                                                  return text

                                                          class ChefDialectizer(Dialectizer):
                                                              """convert HTML to Swedish Chef-speak

                                                              based on the classic chef.x, copyright (c) 1992, 1993 John Hagerman
                                                              """
                                                              subs = ((r'a([nu])', r'u\1'),
                                                                      (r'A([nu])', r'U\1'),
                                                                      (r'a\B', r'e'),
                                                                      (r'A\B', r'E'),
                                                                      (r'en\b', r'ee'),
                                                                      (r'\Bew', r'oo'),
                                                                      (r'\Be\b', r'e-a'),
                                                                      (r'\be', r'i'),
                                                                      (r'\bE', r'I'),
                                                                      (r'\Bf', r'ff'),
                                                                      (r'\Bir', r'ur'),
                                                                      (r'(\w*?)i(\w*?)$', r'\1ee\2'),
                                                                      (r'\bow', r'oo'),
                                                                      (r'\bo', r'oo'),
                                                                      (r'\bO', r'Oo'),
                                                                      (r'the', r'zee'),
                                                                      (r'The', r'Zee'),
                                                                      (r'th\b', r't'),
                                                                      (r'\Btion', r'shun'),
                                                                      (r'\Bu', r'oo'),
                                                                      (r'\BU', r'Oo'),
                                                                      (r'v', r'f'),
                                                                      (r'V', r'F'),
                                                                      (r'w', r'w'),
                                                                      (r'W', r'W'),
                                                                      (r'([a-z])[.]', r'\1. Bork Bork Bork!'))

                                                          class FuddDialectizer(Dialectizer):
                                                              """convert HTML to Elmer Fudd-speak"""
                                                              subs = ((r'[rl]', r'w'),
                                                                      (r'qu', r'qw'),
                                                                      (r'th\b', r'f'),
                                                                      (r'th', r'd'),
                                                                      (r'n[.]', r'n, uh-hah-hah-hah.'))

                                                          class OldeDialectizer(Dialectizer):
                                                              """convert HTML to mock Middle English"""
                                                              subs = ((r'i([bcdfghjklmnpqrstvwxyz])e\b', r'y\1'),
                                                                      (r'i([bcdfghjklmnpqrstvwxyz])e', r'y\1\1e'),
                                                                      (r'ick\b', r'yk'),
                                                                      (r'ia([bcdfghjklmnpqrstvwxyz])', r'e\1e'),
                                                                      (r'e[ea]([bcdfghjklmnpqrstvwxyz])', r'e\1e'),
                                                                      (r'([bcdfghjklmnpqrstvwxyz])y', r'\1ee'),
                                                                      (r'([bcdfghjklmnpqrstvwxyz])er', r'\1re'),
                                                                      (r'([aeiou])re\b', r'\1r'),
                                                                      (r'ia([bcdfghjklmnpqrstvwxyz])', r'i\1e'),
                                                                      (r'tion\b', r'cioun'),
                                                                      (r'ion\b', r'ioun'),
                                                                      (r'aid', r'ayde'),
                                                                      (r'ai', r'ey'),
                                                                      (r'ay\b', r'y'),
                                                                      (r'ay', r'ey'),
                                                                      (r'ant', r'aunt'),
                                                                      (r'ea', r'ee'),
                                                                      (r'oa', r'oo'),
                                                                      (r'ue', r'e'),
                                                                      (r'oe', r'o'),
                                                                      (r'ou', r'ow'),
                                                                      (r'ow', r'ou'),
                                                                      (r'\bhe', r'hi'),
                                                                      (r've\b', r'veth'),
                                                                      (r'se\b', r'e'),
                                                                      (r"'s\b", r'es'),
                                                                      (r'ic\b', r'ick'),
                                                                      (r'ics\b', r'icc'),
                                                                      (r'ical\b', r'ick'),
                                                                      (r'tle\b', r'til'),
                                                                      (r'll\b', r'l'),
                                                                      (r'ould\b', r'olde'),
                                                                      (r'own\b', r'oune'),
                                                                      (r'un\b', r'onne'),
                                                                      (r'rry\b', r'rye'),
                                                                      (r'est\b', r'este'),
                                                                      (r'pt\b', r'pte'),
                                                                      (r'th\b', r'the'),
                                                                      (r'ch\b', r'che'),
                                                                      (r'ss\b', r'sse'),
                                                                      (r'([wybdp])\b', r'\1e'),
                                                                      (r'([rnt])\b', r'\1\1e'),
                                                                      (r'from', r'fro'),
                                                                      (r'when', r'whan'))

                                                          def translate(url, dialectName="chef"):
                                                              """fetch URL and translate using dialect

                                                              dialect in ("chef", "fudd", "olde")"""
                                                              import urllib
                                                              sock = urllib.urlopen(url)
                                                              htmlSource = sock.read()
                                                              sock.close()
                                                              parserName = "%sDialectizer" % dialectName.capitalize()
                                                              parserClass = globals()[parserName]
                                                              parser = parserClass()
                                                              parser.feed(htmlSource)
                                                              parser.close()
                                                              return parser.output()

                                                          def test(url):
                                                              """test all dialects against URL"""
                                                              for dialect in ("chef", "fudd", "olde"):
                                                                  outfile = "%s.html" % dialect
                                                                  fsock = open(outfile, "wb")
                                                                  fsock.write(translate(url, dialect))
                                                                  fsock.close()
                                                                  import webbrowser
                                                                  webbrowser.open_new(outfile)

                                                          if __name__ == "__main__":
                                                              test("http://diveintopython3.org/odbchelper_list.html")

                                                      ### Example 8.3. Output of `dialect.py`

                                                      Running this script will translate Section 3.2, “Introducing Lists” into mock Swedish Chef-speak (from The Muppets), mock Elmer Fudd-speak (from Bugs Bunny cartoons), and mock Middle English (loosely based on Chaucer's The Canterbury Tales). If you look at the HTML source of the output pages, you'll see that all the HTML tags and attributes are untouched, but the text between the tags has been “translated” into the mock language. If you look closer, you'll see that, in fact, only the titles and paragraphs were translated; the code listings and screen examples were left untouched.

                                                          <div class=abstract>
                                                          <p>Lists awe <span class=application>Pydon</span>'s wowkhowse datatype.
                                                          If youw onwy expewience wif wists is awways in
                                                          <span class=application>Visuaw Basic</span> ow (God fowbid) de datastowe
                                                          in <span class=application>Powewbuiwdew</span>, bwace youwsewf fow
                                                          <span class=application>Pydon</span> wists.</p>
                                                          </div>

                                                      ## 8.2. Introducing `sgmllib.py`

                                                      HTML processing is broken into three steps: breaking down the HTML into its constituent pieces, fiddling with the pieces, and reconstructing the pieces into HTML again. The first step is done by `sgmllib.py`, a part of the standard Python library.

                                                      The key to understanding this chapter is to realize that HTML is not just text, it is structured text. The structure is derived from the more-or-less-hierarchical sequence of start tags and end tags. Usually you don't work with HTML this way; you work with it textually in a text editor, or visually in a web browser or web authoring tool. `sgmllib.py` presents HTML structurally.

                                                      `sgmllib.py` contains one important class: `SGMLParser`. `SGMLParser` parses HTML into useful pieces, like start tags and end tags. As soon as it succeeds in breaking down some data into a useful piece, it calls a method on itself based on what it found. In order to use the parser, you subclass the `SGMLParser` class and override these methods. This is what I meant when I said that it presents HTML structurally: the structure of the HTML determines the sequence of method calls and the arguments passed to each method.

                                                      `SGMLParser` parses HTML into 8 kinds of data, and calls a separate method for each of them:

                                                      Start tag
                                                        An HTML tag that starts a block, like `<html>`, `<head>`, `<body>`, or `<pre>`, or a standalone tag like `<br>` or `<img>`. When it finds a start tag `tagname`, `SGMLParser` will look for a method called `start_`tagname`` or `do_`tagname``. For instance, when it finds a `<pre>` tag, it will look for a `start_pre` or `do_pre` method. If found, `SGMLParser` calls this method with a list of the tag's attributes; otherwise, it calls `unknown_starttag` with the tag name and list of attributes.
                                                      End tag
                                                        An HTML tag that ends a block, like `</html>`, `</head>`, `</body>`, or `</pre>`. When it finds an end tag, `SGMLParser` will look for a method called `end_`tagname``. If found, `SGMLParser` calls this method, otherwise it calls `unknown_endtag` with the tag name.
                                                      Character reference
                                                        An escaped character referenced by its decimal or hexadecimal equivalent, like `&#160;`. When found, `SGMLParser` calls `handle_charref` with the text of the decimal or hexadecimal character equivalent.
                                                      Entity reference
                                                        An HTML entity, like `&copy;`. When found, `SGMLParser` calls `handle_entityref` with the name of the HTML entity.
                                                      Comment
                                                        An HTML comment, enclosed in `<!-- ... -->`. When found, `SGMLParser` calls `handle_comment` with the body of the comment.
                                                      Processing instruction
                                                        An HTML processing instruction, enclosed in `<? ... >`. When found, `SGMLParser` calls `handle_pi` with the body of the processing instruction.
                                                      Declaration
                                                        An HTML declaration, such as a `DOCTYPE`, enclosed in `<! ... >`. When found, `SGMLParser` calls `handle_decl` with the body of the declaration.
                                                      Text data
                                                        A block of text. Anything that doesn't fit into the other 7 categories. When found, `SGMLParser` calls `handle_data` with the text.

                                                      Python 2.0 had a bug where `SGMLParser` would not recognize declarations at all (`handle_decl` would never be called), which meant that `DOCTYPE`s were silently ignored. This is fixed in Python 2.1.

                                                      `sgmllib.py` comes with a test suite to illustrate this. You can run `sgmllib.py`, passing the name of an HTML file on the command line, and it will print out the tags and other elements as it parses them. It does this by subclassing the `SGMLParser` class and defining `unknown_starttag`, `unknown_endtag`, `handle_data` and other methods which simply print their arguments.

                                                      In the ActivePython IDE on Windows, you can specify command line arguments in the “Run script” dialog. Separate multiple arguments with spaces.

                                                      ### Example 8.4. Sample test of `sgmllib.py`

                                                      Here is a snippet from the table of contents of the HTML version of this book. Of course your paths may vary. (If you haven't downloaded the HTML version of the book, you can do so at http://diveintopython3.org/.

                                                          c:\python23\lib> type "c:\downloads\diveintopython3\html\toc\index.html"

                                                          <!DOCTYPE html
                                                            PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
                                                          <html>
                                                             <head>
                                                                <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">

                                                                <title>Dive Into Python</title>
                                                                <link rel="stylesheet" href="diveintopython3.css" type="text/css">

                                                          ... rest of file omitted for brevity ...

                                                      Running this through the test suite of `sgmllib.py` yields this output:

                                                          c:\python23\lib> python sgmllib.py "c:\downloads\diveintopython3\html\toc\index.html"
                                                          data: '\n\n'
                                                          start tag: <html >
                                                          data: '\n   '
                                                          start tag: <head>
                                                          data: '\n      '
                                                          start tag: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" >
                                                          data: '\n   \n      '
                                                          start tag: <title>
                                                          data: 'Dive Into Python'
                                                          end tag: </title>
                                                          data: '\n      '
                                                          start tag: <link rel="stylesheet" href="diveintopython3.css" type="text/css" >
                                                          data: '\n      '

                                                          ... rest of output omitted for brevity ...

                                                      Here's the roadmap for the rest of the chapter:

                                                      - Subclass `SGMLParser` to create classes that extract interesting data out of HTML documents.
                                                      - Subclass `SGMLParser` to create `BaseHTMLProcessor`, which overrides all 8 handler methods and uses them to reconstruct the original HTML from the pieces.
                                                      - Subclass `BaseHTMLProcessor` to create `Dialectizer`, which adds some methods to process specific HTML tags specially, and overrides the `handle_data` method to provide a framework for processing the text blocks between the HTML tags.
                                                      - Subclass `Dialectizer` to create classes that define text processing rules used by `Dialectizer.handle_data`.
                                                      - Write a test suite that grabs a real web page from `http://diveintopython3.org/` and processes it.

                                                      Along the way, you'll also learn about `locals`, `globals`, and dictionary-based string formatting.

                                                      ## 8.3. Extracting data from HTML documents

                                                      To extract data from HTML documents, subclass the `SGMLParser` class and define methods for each tag or entity you want to capture.

                                                      The first step to extracting data from an HTML document is getting some HTML. If you have some HTML lying around on your hard drive, you can use file functions to read it, but the real fun begins when you get HTML from live web pages.

                                                      ### Example 8.5. Introducing `urllib`

                                                          >>> import urllib   ①
                                                          >>> sock = urllib.urlopen("http://diveintopython3.org/") ②
                                                          >>> htmlSource = sock.read()          ③
                                                          >>> sock.close()    ④
                                                          >>> print htmlSource⑤
                                                          <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head>
                                                                <meta http-equiv='Content-Type' content='text/html; charset=ISO-8859-1'>
                                                             <title>Dive Into Python</title>
                                                          <link rel='stylesheet' href='diveintopython3.css' type='text/css'>
                                                          <link rev='made' href='mailto:mark@diveintopython3.org'>
                                                          <meta name='keywords' content='Python, Dive Into Python, tutorial, object-oriented, programming, documentation, book, free'>
                                                          <meta name='description' content='a free Python tutorial for experienced programmers'>
                                                          </head>
                                                          <body bgcolor='white' text='black' link='#0000FF' vlink='#840084' alink='#0000FF'>
                                                          <table cellpadding='0' cellspacing='0' border='0' width='100%'>
                                                          <tr><td class='header' width='1%' valign='top'>diveintopython3.org</td>
                                                          <td width='99%' align='right'><hr size='1' noshade></td></tr>
                                                          <tr><td class='tagline' colspan='2'>Python&nbsp;for&nbsp;experienced&nbsp;programmers</td></tr>

                                                          [...snip...]

                                                      1. The `urllib` module is part of the standard Python library. It contains functions for getting information about and actually retrieving data from Internet-based URLs (mainly web pages).
                                                      2. The simplest use of `urllib` is to retrieve the entire text of a web page using the `urlopen` function. Opening a URL is similar to opening a file. The return value of `urlopen` is a file-like object, which has some of the same methods as a file object.
                                                      3. The simplest thing to do with the file-like object returned by `urlopen` is `read`, which reads the entire HTML of the web page into a single string. The object also supports `readlines`, which reads the text line by line into a list.
                                                      4. When you're done with the object, make sure to `close` it, just like a normal file object.
                                                      5. You now have the complete HTML of the home page of `http://diveintopython3.org/` in a string, and you're ready to parse it.

                                                         ### Example 8.6. Introducing `urllister.py`

                                                         If you have not already done so, you can download this and other examples used in this book.

                                                             from sgmllib import SGMLParser

                                                             class URLLister(SGMLParser):
                                                                 def reset(self):            ①
                                                                     SGMLParser.reset(self)
                                                                     self.urls = []

                                                                 def start_a(self, attrs):   ②
                                                                     href = [v for k, v in attrs if k=='href'] ③ ④
                                                                     if href:
                                                                         self.urls.extend(href)

                                                         1. `reset` is called by the `__init__` method of `SGMLParser`, and it can also be called manually once an instance of the parser has been created. So if you need to do any initialization, do it in `reset`, not in `__init__`, so that it will be re-initialized properly when someone re-uses a parser instance.
                                                         2. `start_a` is called by `SGMLParser` whenever it finds an `<a>` tag. The tag may contain an `href` attribute, and/or other attributes, like `name` or `title`. The attrs parameter is a list of tuples, `[(attribute, value), (attribute, value), ...]`. Or it may be just an `<a>`, a valid (if useless) HTML tag, in which case attrs would be an empty list.
                                                         3. You can find out whether this `<a>` tag has an `href` attribute with a simple multi-variable list comprehension.
                                                         4. String comparisons like `k=='href'` are always case-sensitive, but that's safe in this case, because `SGMLParser` converts attribute names to lowercase while building attrs.

                                                            ### Example 8.7. Using `urllister.py`

                                                                >>> import urllib, urllister
                                                                >>> usock = urllib.urlopen("http://diveintopython3.org/")
                                                                >>> parser = urllister.URLLister()
                                                                >>> parser.feed(usock.read())         ①
                                                                >>> usock.close()   ②
                                                                >>> parser.close()  ③
                                                                >>> for url in parser.urls: print url ④
                                                                toc/index.html
                                                                #download
                                                                #languages
                                                                toc/index.html
                                                                appendix/history.html
                                                                download/diveintopython3-html-5.0.zip
                                                                download/diveintopython3-pdf-5.0.zip
                                                                download/diveintopython3-word-5.0.zip
                                                                download/diveintopython3-text-5.0.zip
                                                                download/diveintopython3-html-flat-5.0.zip
                                                                download/diveintopython3-xml-5.0.zip
                                                                download/diveintopython3-common-5.0.zip

                                                                ... rest of output omitted for brevity ...

                                                            1. Call the `feed` method, defined in `SGMLParser`, to get HTML into the parser. [1] It takes a string, which is what `usock.read()` returns.
                                                            2. Like files, you should `close` your URL objects as soon as you're done with them.
                                                            3. You should `close` your parser object, too, but for a different reason. You've read all the data and fed it to the parser, but the `feed` method isn't guaranteed to have actually processed all the HTML you give it; it may buffer it, waiting for more. Be sure to call `close` to flush the buffer and force everything to be fully parsed.
                                                            4. Once the parser is `close`d, the parsing is complete, and parser.urls contains a list of all the linked URLs in the HTML document. (Your output may look different, if the download links have been updated by the time you read this.)

                                                               ## 8.4. Introducing `BaseHTMLProcessor.py`

                                                               `SGMLParser` doesn't produce anything by itself. It parses and parses and parses, and it calls a method for each interesting thing it finds, but the methods don't do anything. `SGMLParser` is an HTML consumer: it takes HTML and breaks it down into small, structured pieces. As you saw in the previous section, you can subclass `SGMLParser` to define classes that catch specific tags and produce useful things, like a list of all the links on a web page. Now you'll take this one step further by defining a class that catches everything `SGMLParser` throws at it and reconstructs the complete HTML document. In technical terms, this class will be an HTML producer.

                                                               `BaseHTMLProcessor` subclasses `SGMLParser` and provides all 8 essential handler methods: `unknown_starttag`, `unknown_endtag`, `handle_charref`, `handle_entityref`, `handle_comment`, `handle_pi`, `handle_decl`, and `handle_data`.

                                                               ### Example 8.8. Introducing `BaseHTMLProcessor`

                                                                   class BaseHTMLProcessor(SGMLParser):
                                                                       def reset(self):      ①
                                                                           self.pieces = []
                                                                           SGMLParser.reset(self)

                                                                       def unknown_starttag(self, tag, attrs): ②
                                                                           strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs])
                                                                           self.pieces.append("<%(tag)s%(strattrs)s>" % locals())

                                                                       def unknown_endtag(self, tag):          ③
                                                                           self.pieces.append("</%(tag)s>" % locals())

                                                                       def handle_charref(self, ref):          ④
                                                                           self.pieces.append("&#%(ref)s;" % locals())

                                                                       def handle_entityref(self, ref):        ⑤
                                                                           self.pieces.append("&%(ref)s" % locals())
                                                                           if htmlentitydefs.entitydefs.has_key(ref):
                                                                               self.pieces.append(";")

                                                                       def handle_data(self, text):            ⑥
                                                                           self.pieces.append(text)

                                                                       def handle_comment(self, text):         ⑦
                                                                           self.pieces.append("<!--%(text)s-->" % locals())

                                                                       def handle_pi(self, text):              ⑧
                                                                           self.pieces.append("<?%(text)s>" % locals())

                                                                       def handle_decl(self, text):
                                                                           self.pieces.append("<!%(text)s>" % locals())

                                                               1. `reset`, called by `SGMLParser.__init__`, initializes self.pieces as an empty list before calling the ancestor method. self.pieces is a data attribute which will hold the pieces of the HTML document you're constructing. Each handler method will reconstruct the HTML that `SGMLParser` parsed, and each method will append that string to self.pieces. Note that self.pieces is a list. You might be tempted to define it as a string and just keep appending each piece to it. That would work, but Python is much more efficient at dealing with lists. [2]
                                                               2. Since `BaseHTMLProcessor` does not define any methods for specific tags (like the `start_a` method in `URLLister`), `SGMLParser` will call `unknown_starttag` for every start tag. This method takes the tag (tag) and the list of attribute name/value pairs (attrs), reconstructs the original HTML, and appends it to self.pieces. The string formatting here is a little strange; you'll untangle that (and also the odd-looking `locals` function) later in this chapter.
                                                               3. Reconstructing end tags is much simpler; just take the tag name and wrap it in the `</...>` brackets.
                                                               4. When `SGMLParser` finds a character reference, it calls `handle_charref` with the bare reference. If the HTML document contains the reference `&#160;`, ref will be `160`. Reconstructing the original complete character reference just involves wrapping ref in `&#...;` characters.
                                                               5. Entity references are similar to character references, but without the hash mark. Reconstructing the original entity reference requires wrapping ref in `&...;` characters. (Actually, as an erudite reader pointed out to me, it's slightly more complicated than this. Only certain standard HTML entites end in a semicolon; other similar-looking entities do not. Luckily for us, the set of standard HTML entities is defined in a dictionary in a Python module called `htmlentitydefs`. Hence the extra `if` statement.)
                                                               6. Blocks of text are simply appended to self.pieces unaltered.
                                                               7. HTML comments are wrapped in `<!--...-->` characters.
                                                               8. Processing instructions are wrapped in `<?...>` characters.

                                                                  The HTML specification requires that all non-HTML (like client-side JavaScript) must be enclosed in HTML comments, but not all web pages do this properly (and all modern web browsers are forgiving if they don't). `BaseHTMLProcessor` is not forgiving; if script is improperly embedded, it will be parsed as if it were HTML. For instance, if the script contains less-than and equals signs, `SGMLParser` may incorrectly think that it has found tags and attributes. `SGMLParser` always converts tags and attribute names to lowercase, which may break the script, and `BaseHTMLProcessor` always encloses attribute values in double quotes (even if the original HTML document used single quotes or no quotes), which will certainly break the script. Always protect your client-side script within HTML comments.

                                                                  ### Example 8.9. `BaseHTMLProcessor` output

                                                                          def output(self):               ①
                                                                              """Return processed HTML as a single string"""
                                                                              return "".join(self.pieces) ②

                                                                  1. This is the one method in `BaseHTMLProcessor` that is never called by the ancestor `SGMLParser`. Since the other handler methods store their reconstructed HTML in self.pieces, this function is needed to join all those pieces into one string. As noted before, Python is great at lists and mediocre at strings, so you only create the complete string when somebody explicitly asks for it.
                                                                  2. If you prefer, you could use the `join` method of the `string` module instead: `string.join(self.pieces, "")`

                                                                     ### Further reading

                                                                     - W3C discusses character and entity references.
                                                                     - Python Library Reference confirms your suspicions that the `htmlentitydefs` module is exactly what it sounds like.

                                                                     ## 8.5. `locals` and `globals`

                                                                     Let's digress from HTML processing for a minute and talk about how Python handles variables. Python has two built-in functions, `locals` and `globals`, which provide dictionary-based access to local and global variables.

                                                                     Remember `locals`? You first saw it here:

                                                                             def unknown_starttag(self, tag, attrs):
                                                                                 strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs])
                                                                                 self.pieces.append("<%(tag)s%(strattrs)s>" % locals())

                                                                     No, wait, you can't learn about `locals` yet. First, you need to learn about namespaces. This is dry stuff, but it's important, so pay attention.

                                                                     Python uses what are called namespaces to keep track of variables. A namespace is just like a dictionary where the keys are names of variables and the dictionary values are the values of those variables. In fact, you can access a namespace as a Python dictionary, as you'll see in a minute.

                                                                     At any particular point in a Python program, there are several namespaces available. Each function has its own namespace, called the local namespace, which keeps track of the function's variables, including function arguments and locally defined variables. Each module has its own namespace, called the global namespace, which keeps track of the module's variables, including functions, classes, any other imported modules, and module-level variables and constants. And there is the built-in namespace, accessible from any module, which holds built-in functions and exceptions.

                                                                     When a line of code asks for the value of a variable x, Python will search for that variable in all the available namespaces, in order:

                                                                     1. local namespace - specific to the current function or class method. If the function defines a local variable x, or has an argument x, Python will use this and stop searching.
                                                                     2. global namespace - specific to the current module. If the module has defined a variable, function, or class called x, Python will use that and stop searching.
                                                                     3. built-in namespace - global to all modules. As a last resort, Python will assume that x is the name of built-in function or variable.

                                                                     If Python doesn't find x in any of these namespaces, it gives up and raises a `NameError` with the message `There is no variable named 'x'`, which you saw back in Example 3.18, “Referencing an Unbound Variable”, but you didn't appreciate how much work Python was doing before giving you that error.

                                                                     Python 2.2 introduced a subtle but important change that affects the namespace search order: nested scopes. In versions of Python prior to 2.2, when you reference a variable within a nested function or `lambda` function, Python will search for that variable in the current (nested or `lambda`) function's namespace, then in the module's namespace. Python 2.2 will search for the variable in the current (nested or `lambda`) function's namespace, then in the parent function's namespace, then in the module's namespace. Python 2.1 can work either way; by default, it works like Python 2.0, but you can add the following line of code at the top of your module to make your module work like Python 2.2:

                                                                         from __future__ import nested_scopes

                                                                     Are you confused yet? Don't despair! This is really cool, I promise. Like many things in Python, namespaces are directly accessible at run-time. How? Well, the local namespace is accessible via the built-in `locals` function, and the global (module level) namespace is accessible via the built-in `globals` function.

                                                                     ### Example 8.10. Introducing `locals`

                                                                         >>> def foo(arg): ①
                                                                         ...    x = 1
                                                                         ...    print locals()
                                                                         ...
                                                                         >>> foo(7)        ②
                                                                         {'arg': 7, 'x': 1}
                                                                         >>> foo('bar')    ③
                                                                         {'arg': 'bar', 'x': 1}

                                                                     1. The function `foo` has two variables in its local namespace: arg, whose value is passed in to the function, and x, which is defined within the function.
                                                                     2. `locals` returns a dictionary of name/value pairs. The keys of this dictionary are the names of the variables as strings; the values of the dictionary are the actual values of the variables. So calling `foo` with `7` prints the dictionary containing the function's two local variables: arg (`7`) and x (`1`).
                                                                     3. Remember, Python has dynamic typing, so you could just as easily pass a string in for arg; the function (and the call to `locals`) would still work just as well. `locals` works with all variables of all datatypes.

                                                                        What `locals` does for the local (function) namespace, `globals` does for the global (module) namespace. `globals` is more exciting, though, because a module's namespace is more exciting. [3] Not only does the module's namespace include module-level variables and constants, it includes all the functions and classes defined in the module. Plus, it includes anything that was imported into the module.

                                                                        Remember the difference between `from module import` and `import module`? With `import module`, the module itself is imported, but it retains its own namespace, which is why you need to use the module name to access any of its functions or attributes: `module.function`. But with `from module import`, you're actually importing specific functions and attributes from another module into your own namespace, which is why you access them directly without referencing the original module they came from. With the `globals` function, you can actually see this happen.

                                                                        ### Example 8.11. Introducing `globals`

                                                                        Look at the following block of code at the bottom of `BaseHTMLProcessor.py`:

                                                                            if __name__ == "__main__":
                                                                                for k, v in globals().items():             ①
                                                                                    print k, "=", v

                                                                        1. Just so you don't get intimidated, remember that you've seen all this before. The `globals` function returns a dictionary, and you're iterating through the dictionary using the `items` method and multi-variable assignment. The only thing new here is the `globals` function.

                                                                           Now running the script from the command line gives this output (note that your output may be slightly different, depending on your platform and where you installed Python):

                                                                               c:\docbook\dip\py> python BaseHTMLProcessor.py

                                                                               SGMLParser = sgmllib.SGMLParser                ①
                                                                               htmlentitydefs = <module 'htmlentitydefs' from 'C:\Python23\lib\htmlentitydefs.py'> ②
                                                                               BaseHTMLProcessor = __main__.BaseHTMLProcessor ③
                                                                               __name__ = __main__          ④
                                                                               ... rest of output omitted for brevity...

                                                                           1. `SGMLParser` was imported from `sgmllib`, using `from module import`. That means that it was imported directly into the module's namespace, and here it is.
                                                                           2. Contrast this with `htmlentitydefs`, which was imported using `import`. That means that the `htmlentitydefs` module itself is in the namespace, but the entitydefs variable defined within `htmlentitydefs` is not.
                                                                           3. This module only defines one class, `BaseHTMLProcessor`, and here it is. Note that the value here is the class itself, not a specific instance of the class.
                                                                           4. Remember the `if __name__` trick? When running a module (as opposed to importing it from another module), the built-in `__name__` attribute is a special value, `__main__`. Since you ran this module as a script from the command line, `__name__` is `__main__`, which is why the little test code to print the `globals` got executed.

                                                                              Using the `locals` and `globals` functions, you can get the value of arbitrary variables dynamically, providing the variable name as a string. This mirrors the functionality of the `getattr` function, which allows you to access arbitrary functions dynamically by providing the function name as a string.

                                                                              There is one other important difference between the `locals` and `globals` functions, which you should learn now before it bites you. It will bite you anyway, but at least then you'll remember learning it.

                                                                              ### Example 8.12. `locals` is read-only, `globals` is not

                                                                                  def foo(arg):
                                                                                      x = 1
                                                                                      print locals()    ①
                                                                                      locals()["x"] = 2 ②
                                                                                      print "x=",x      ③

                                                                                  z = 7
                                                                                  print "z=",z
                                                                                  foo(3)
                                                                                  globals()["z"] = 8    ④
                                                                                  print "z=",z          ⑤

                                                                              1. Since `foo` is called with `3`, this will print `{'arg': 3, 'x': 1}`. This should not be a surprise.
                                                                              2. `locals` is a function that returns a dictionary, and here you are setting a value in that dictionary. You might think that this would change the value of the local variable x to `2`, but it doesn't. `locals` does not actually return the local namespace, it returns a copy. So changing it does nothing to the value of the variables in the local namespace.
                                                                              3. This prints `x= 1`, not `x= 2`.
                                                                              4. After being burned by `locals`, you might think that this wouldn't change the value of z, but it does. Due to internal differences in how Python is implemented (which I'd rather not go into, since I don't fully understand them myself), `globals` returns the actual global namespace, not a copy: the exact opposite behavior of `locals`. So any changes to the dictionary returned by `globals` directly affect your global variables.
                                                                              5. This prints `z= 8`, not `z= 7`.

                                                                                 ## 8.6. Dictionary-based string formatting

                                                                                 Why did you learn about `locals` and `globals`? So you can learn about dictionary-based string formatting. As you recall, regular string formatting provides an easy way to insert values into strings. Values are listed in a tuple and inserted in order into the string in place of each formatting marker. While this is efficient, it is not always the easiest code to read, especially when multiple values are being inserted. You can't simply scan through the string in one pass and understand what the result will be; you're constantly switching between reading the string and reading the tuple of values.

                                                                                 There is an alternative form of string formatting that uses dictionaries instead of tuples of values.

                                                                                 ### Example 8.13. Introducing dictionary-based string formatting

                                                                                     >>> params = {"server":"mpilgrim", "database":"master", "uid":"sa", "pwd":"secret"}
                                                                                     >>> "%(pwd)s" % params①
                                                                                     'secret'
                                                                                     >>> "%(pwd)s is not a good password for %(uid)s" % params ②
                                                                                     'secret is not a good password for sa'
                                                                                     >>> "%(database)s of mind, %(database)s of body" % params ③
                                                                                     'master of mind, master of body'

                                                                                 1. Instead of a tuple of explicit values, this form of string formatting uses a dictionary, params. And instead of a simple `%s` marker in the string, the marker contains a name in parentheses. This name is used as a key in the params dictionary and subsitutes the corresponding value, `secret`, in place of the `%(pwd)s` marker.
                                                                                 2. Dictionary-based string formatting works with any number of named keys. Each key must exist in the given dictionary, or the formatting will fail with a `KeyError`.
                                                                                 3. You can even specify the same key twice; each occurrence will be replaced with the same value.

                                                                                    So why would you use dictionary-based string formatting? Well, it does seem like overkill to set up a dictionary of keys and values simply to do string formatting in the next line; it's really most useful when you happen to have a dictionary of meaningful keys and values already. Like `locals`.

                                                                                    ### Example 8.14. Dictionary-based string formatting in `BaseHTMLProcessor.py`

                                                                                            def handle_comment(self, text):
                                                                                                self.pieces.append("<!--%(text)s-->" % locals()) ①

                                                                                    1. Using the built-in `locals` function is the most common use of dictionary-based string formatting. It means that you can use the names of local variables within your string (in this case, text, which was passed to the class method as an argument) and each named variable will be replaced by its value. If text is `'Begin page footer'`, the string formatting `"<!--%(text)s-->" % locals()` will resolve to the string `'<!--Begin page footer-->'`.

                                                                                       ### Example 8.15. More dictionary-based string formatting

                                                                                               def unknown_starttag(self, tag, attrs):
                                                                                                   strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs]) ①
                                                                                                   self.pieces.append("<%(tag)s%(strattrs)s>" % locals())    ②

                                                                                       1. When this method is called, attrs is a list of key/value tuples, just like the `items` of a dictionary, which means you can use multi-variable assignment to iterate through it. This should be a familiar pattern by now, but there's a lot going on here, so let's break it down:
                                                                                          1. Suppose attrs is `[('href', 'index.html'), ('title', 'Go to home page')]`.
                                                                                          2. In the first round of the list comprehension, key will get `'href'`, and value will get `'index.html'`.
                                                                                          3. The string formatting `' %s="%s"' % (key, value)` will resolve to `' href="index.html"'`. This string becomes the first element of the list comprehension's return value.
                                                                                          4. In the second round, key will get `'title'`, and value will get `'Go to home page'`.
                                                                                          5. The string formatting will resolve to `' title="Go to home page"'`.
                                                                                          6. The list comprehension returns a list of these two resolved strings, and strattrs will join both elements of this list together to form `' href="index.html" title="Go to home page"'`.
                                                                                       2. Now, using dictionary-based string formatting, you insert the value of tag and strattrs into a string. So if tag is `'a'`, the final result would be `'<a href="index.html" title="Go to home page">'`, and that is what gets appended to self.pieces.

                                                                                          Using dictionary-based string formatting with `locals` is a convenient way of making complex string formatting expressions more readable, but it comes with a price. There is a slight performance hit in making the call to `locals`, since `locals` builds a copy of the local namespace.

                                                                                          ## 8.7. Quoting attribute values

                                                                                          A common question on comp.lang.python is “I have a bunch of HTML documents with unquoted attribute values, and I want to properly quote them all. How can I do this?”[4] (This is generally precipitated by a project manager who has found the HTML-is-a-standard religion joining a large project and proclaiming that all pages must validate against an HTML validator. Unquoted attribute values are a common violation of the HTML standard.) Whatever the reason, unquoted attribute values are easy to fix by feeding HTML through `BaseHTMLProcessor`.

                                                                                          `BaseHTMLProcessor` consumes HTML (since it's descended from `SGMLParser`) and produces equivalent HTML, but the HTML output is not identical to the input. Tags and attribute names will end up in lowercase, even if they started in uppercase or mixed case, and attribute values will be enclosed in double quotes, even if they started in single quotes or with no quotes at all. It is this last side effect that you can take advantage of.

                                                                                          ### Example 8.16. Quoting attribute values

                                                                                              >>> htmlSource = """        ①
                                                                                              ...    <html>
                                                                                              ...    <head>
                                                                                              ...    <title>Test page</title>
                                                                                              ...    </head>
                                                                                              ...    <body>
                                                                                              ...    <ul>
                                                                                              ...    <li><a href=index.html>Home</a></li>
                                                                                              ...    <li><a href=toc.html>Table of contents</a></li>
                                                                                              ...    <li><a href=history.html>Revision history</a></li>
                                                                                              ...    </body>
                                                                                              ...    </html>
                                                                                              ...    """
                                                                                              >>> from BaseHTMLProcessor import BaseHTMLProcessor
                                                                                              >>> parser = BaseHTMLProcessor()
                                                                                              >>> parser.feed(htmlSource) ②
                                                                                              >>> print parser.output()   ③
                                                                                              <html>
                                                                                              <head>
                                                                                              <title>Test page</title>
                                                                                              </head>
                                                                                              <body>
                                                                                              <ul>
                                                                                              <li><a href="index.html">Home</a></li>
                                                                                              <li><a href="toc.html">Table of contents</a></li>
                                                                                              <li><a href="history.html">Revision history</a></li>
                                                                                              </body>
                                                                                              </html>

                                                                                          1. Note that the attribute values of the `href` attributes in the `<a>` tags are not properly quoted. (Also note that you're using triple quotes for something other than a `docstring`. And directly in the IDE, no less. They're very useful.)
                                                                                          2. Feed the parser.
                                                                                          3. Using the `output` function defined in `BaseHTMLProcessor`, you get the output as a single string, complete with quoted attribute values. While this may seem anti-climactic, think about how much has actually happened here: `SGMLParser` parsed the entire HTML document, breaking it down into tags, refs, data, and so forth; `BaseHTMLProcessor` used those elements to reconstruct pieces of HTML (which are still stored in parser.pieces, if you want to see them); finally, you called `parser.output`, which joined all the pieces of HTML into one string.

                                                                                             ## 8.8. Introducing `dialect.py`

                                                                                             `Dialectizer` is a simple (and silly) descendant of `BaseHTMLProcessor`. It runs blocks of text through a series of substitutions, but it makes sure that anything within a ``<pre>`...`</pre>`` block passes through unaltered.

                                                                                             To handle the `<pre>` blocks, you define two methods in `Dialectizer`: `start_pre` and `end_pre`.

                                                                                             ### Example 8.17. Handling specific tags

                                                                                                     def start_pre(self, attrs):             ①
                                                                                                         self.verbatim += 1②
                                                                                                         self.unknown_starttag("pre", attrs) ③

                                                                                                     def end_pre(self):    ④
                                                                                                         self.unknown_endtag("pre")          ⑤
                                                                                                         self.verbatim -= 1⑥

                                                                                             1. `start_pre` is called every time `SGMLParser` finds a `<pre>` tag in the HTML source. (In a minute, you'll see exactly how this happens.) The method takes a single parameter, attrs, which contains the attributes of the tag (if any). attrs is a list of key/value tuples, just like `unknown_starttag` takes.
                                                                                             2. In the `reset` method, you initialize a data attribute that serves as a counter for `<pre>` tags. Every time you hit a `<pre>` tag, you increment the counter; every time you hit a `</pre>` tag, you'll decrement the counter. (You could just use this as a flag and set it to `1` and reset it to `0`, but it's just as easy to do it this way, and this handles the odd (but possible) case of nested `<pre>` tags.) In a minute, you'll see how this counter is put to good use.
                                                                                             3. That's it, that's the only special processing you do for `<pre>` tags. Now you pass the list of attributes along to `unknown_starttag` so it can do the default processing.
                                                                                             4. `end_pre` is called every time `SGMLParser` finds a `</pre>` tag. Since end tags can not contain attributes, the method takes no parameters.
                                                                                             5. First, you want to do the default processing, just like any other end tag.
                                                                                             6. Second, you decrement your counter to signal that this `<pre>` block has been closed.

                                                                                                At this point, it's worth digging a little further into `SGMLParser`. I've claimed repeatedly (and you've taken it on faith so far) that `SGMLParser` looks for and calls specific methods for each tag, if they exist. For instance, you just saw the definition of `start_pre` and `end_pre` to handle `<pre>` and `</pre>`. But how does this happen? Well, it's not magic, it's just good Python coding.

                                                                                                ### Example 8.18. `SGMLParser`

                                                                                                        def finish_starttag(self, tag, attrs):               ①
                                                                                                            try:
                                                                                                                method = getattr(self, 'start_' + tag)       ②
                                                                                                            except AttributeError:         ③
                                                                                                                try:
                                                                                                                    method = getattr(self, 'do_' + tag)      ④
                                                                                                                except AttributeError:
                                                                                                                    self.unknown_starttag(tag, attrs)        ⑤
                                                                                                                    return -1
                                                                                                                else:
                                                                                                                    self.handle_starttag(tag, method, attrs) ⑥
                                                                                                                    return 0
                                                                                                            else:
                                                                                                                self.stack.append(tag)
                                                                                                                self.handle_starttag(tag, method, attrs)
                                                                                                                return 1 ⑦

                                                                                                        def handle_starttag(self, tag, method, attrs):
                                                                                                            method(attrs)⑧

                                                                                                1. At this point, `SGMLParser` has already found a start tag and parsed the attribute list. The only thing left to do is figure out whether there is a specific handler method for this tag, or whether you should fall back on the default method (`unknown_starttag`).
                                                                                                2. The “magic” of `SGMLParser` is nothing more than your old friend, `getattr`. What you may not have realized before is that `getattr` will find methods defined in descendants of an object as well as the object itself. Here the object is self, the current instance. So if tag is `'pre'`, this call to `getattr` will look for a `start_pre` method on the current instance, which is an instance of the `Dialectizer` class.
                                                                                                3. `getattr` raises an `AttributeError` if the method it's looking for doesn't exist in the object (or any of its descendants), but that's okay, because you wrapped the call to `getattr` inside a `try...except` block and explicitly caught the `AttributeError`.
                                                                                                4. Since you didn't find a `start_xxx` method, you'll also look for a `do_xxx` method before giving up. This alternate naming scheme is generally used for standalone tags, like `<br>`, which have no corresponding end tag. But you can use either naming scheme; as you can see, `SGMLParser` tries both for every tag. (You shouldn't define both a `start_xxx` and `do_xxx` handler method for the same tag, though; only the `start_xxx` method will get called.)
                                                                                                5. Another `AttributeError`, which means that the call to `getattr` failed with `do_xxx`. Since you found neither a `start_xxx` nor a `do_xxx` method for this tag, you catch the exception and fall back on the default method, `unknown_starttag`.
                                                                                                6. Remember, `try...except` blocks can have an `else` clause, which is called if no exception is raised during the `try...except` block. Logically, that means that you did find a `do_xxx` method for this tag, so you're going to call it.
                                                                                                7. By the way, don't worry about these different return values; in theory they mean something, but they're never actually used. Don't worry about the `self.stack.append(tag)` either; `SGMLParser` keeps track internally of whether your start tags are balanced by appropriate end tags, but it doesn't do anything with this information either. In theory, you could use this module to validate that your tags were fully balanced, but it's probably not worth it, and it's beyond the scope of this chapter. You have better things to worry about right now.
                                                                                                8. `start_xxx` and `do_xxx` methods are not called directly; the tag, method, and attributes are passed to this function, `handle_starttag`, so that descendants can override it and change the way all start tags are dispatched. You don't need that level of control, so you just let this method do its thing, which is to call the method (`start_xxx` or `do_xxx`) with the list of attributes. Remember, method is a function, returned from `getattr`, and functions are objects. (I know you're getting tired of hearing it, and I promise I'll stop saying it as soon as I run out of ways to use it to my advantage.) Here, the function object is passed into this dispatch method as an argument, and this method turns around and calls the function. At this point, you don't need to know what the function is, what it's named, or where it's defined; the only thing you need to know about the function is that it is called with one argument, attrs.

                                                                                                   Now back to our regularly scheduled program: `Dialectizer`. When you left, you were in the process of defining specific handler methods for `<pre>` and `</pre>` tags. There's only one thing left to do, and that is to process text blocks with the pre-defined substitutions. For that, you need to override the `handle_data` method.

                                                                                                   ### Example 8.19. Overriding the `handle_data` method

                                                                                                           def handle_data(self, text):     ①
                                                                                                               self.pieces.append(self.verbatim and text or self.process(text)) ②

                                                                                                   1. `handle_data` is called with only one argument, the text to process.
                                                                                                   2. In the ancestor `BaseHTMLProcessor`, the `handle_data` method simply appended the text to the output buffer, self.pieces. Here the logic is only slightly more complicated. If you're in the middle of a ``<pre>`...`</pre>`` block, self.verbatim will be some value greater than `0`, and you want to put the text in the output buffer unaltered. Otherwise, you will call a separate method to process the substitutions, then put the result of that into the output buffer. In Python, this is a one-liner, using the `and-or` trick.

                                                                                                      You're close to completely understanding `Dialectizer`. The only missing link is the nature of the text substitutions themselves. If you know any Perl, you know that when complex text substitutions are required, the only real solution is regular expressions. The classes later in `dialect.py` define a series of regular expressions that operate on the text between the HTML tags. But you just had a whole chapter on regular expressions. You don't really want to slog through regular expressions again, do you? God knows I don't. I think you've learned enough for one chapter.

                                                                                                      ## 8.9. Putting it all together

                                                                                                      It's time to put everything you've learned so far to good use. I hope you were paying attention.

                                                                                                      ### Example 8.20. The `translate` function, part 1

                                                                                                          def translate(url, dialectName="chef"): ①
                                                                                                              import urllib     ②
                                                                                                              sock = urllib.urlopen(url)          ③
                                                                                                              htmlSource = sock.read()
                                                                                                              sock.close()

                                                                                                      1. The `translate` function has an optional argument dialectName, which is a string that specifies the dialect you'll be using. You'll see how this is used in a minute.
                                                                                                      2. Hey, wait a minute, there's an `import` statement in this function! That's perfectly legal in Python. You're used to seeing `import` statements at the top of a program, which means that the imported module is available anywhere in the program. But you can also import modules within a function, which means that the imported module is only available within the function. If you have a module that is only ever used in one function, this is an easy way to make your code more modular. (When you find that your weekend hack has turned into an 800-line work of art and decide to split it up into a dozen reusable modules, you'll appreciate this.)
                                                                                                      3. Now you get the source of the given URL.

                                                                                                         ### Example 8.21. The `translate` function, part 2: curiouser and curiouser

                                                                                                                 parserName = "%sDialectizer" % dialectName.capitalize() ①
                                                                                                                 parserClass = globals()[parserName]   ②
                                                                                                                 parser = parserClass()                ③

                                                                                                         1. `capitalize` is a string method you haven't seen before; it simply capitalizes the first letter of a string and forces everything else to lowercase. Combined with some string formatting, you've taken the name of a dialect and transformed it into the name of the corresponding Dialectizer class. If dialectName is the string `'chef'`, parserName will be the string `'ChefDialectizer'`.
                                                                                                         2. You have the name of a class as a string (parserName), and you have the global namespace as a dictionary (`globals`()). Combined, you can get a reference to the class which the string names. (Remember, classes are objects, and they can be assigned to variables just like any other object.) If parserName is the string `'ChefDialectizer'`, parserClass will be the class `ChefDialectizer`.
                                                                                                         3. Finally, you have a class object (parserClass), and you want an instance of the class. Well, you already know how to do that: call the class like a function. The fact that the class is being stored in a local variable makes absolutely no difference; you just call the local variable like a function, and out pops an instance of the class. If parserClass is the class `ChefDialectizer`, parser will be an instance of the class `ChefDialectizer`.

                                                                                                            Why bother? After all, there are only 3 `Dialectizer` classes; why not just use a `case` statement? (Well, there's no `case` statement in Python, but why not just use a series of `if` statements?) One reason: extensibility. The `translate` function has absolutely no idea how many Dialectizer classes you've defined. Imagine if you defined a new `FooDialectizer` tomorrow; `translate` would work by passing `'foo'` as the dialectName.

                                                                                                            Even better, imagine putting `FooDialectizer` in a separate module, and importing it with `from module import`. You've already seen that this includes it in `globals`(), so `translate` would still work without modification, even though `FooDialectizer` was in a separate file.

                                                                                                            Now imagine that the name of the dialect is coming from somewhere outside the program, maybe from a database or from a user-inputted value on a form. You can use any number of server-side Python scripting architectures to dynamically generate web pages; this function could take a URL and a dialect name (both strings) in the query string of a web page request, and output the “translated” web page.

                                                                                                            Finally, imagine a `Dialectizer` framework with a plug-in architecture. You could put each `Dialectizer` class in a separate file, leaving only the `translate` function in `dialect.py`. Assuming a consistent naming scheme, the `translate` function could dynamic import the appropiate class from the appropriate file, given nothing but the dialect name. (You haven't seen dynamic importing yet, but I promise to cover it in a later chapter.) To add a new dialect, you would simply add an appropriately-named file in the plug-ins directory (like `foodialect.py` which contains the `FooDialectizer` class). Calling the `translate` function with the dialect name `'foo'` would find the module `foodialect.py`, import the class `FooDialectizer`, and away you go.

                                                                                                            ### Example 8.22. The `translate` function, part 3

                                                                                                                    parser.feed(htmlSource) ①
                                                                                                                    parser.close()          ②
                                                                                                                    return parser.output()  ③

                                                                                                            1. After all that imagining, this is going to seem pretty boring, but the `feed` function is what does the entire transformation. You had the entire HTML source in a single string, so you only had to call `feed` once. However, you can call `feed` as often as you want, and the parser will just keep parsing. So if you were worried about memory usage (or you knew you were going to be dealing with very large HTML pages), you could set this up in a loop, where you read a few bytes of HTML and fed it to the parser. The result would be the same.
                                                                                                            2. Because `feed` maintains an internal buffer, you should always call the parser's `close` method when you're done (even if you fed it all at once, like you did). Otherwise you may find that your output is missing the last few bytes.
                                                                                                            3. Remember, `output` is the function you defined on `BaseHTMLProcessor` that joins all the pieces of output you've buffered and returns them in a single string.

                                                                                                               And just like that, you've “translated” a web page, given nothing but a URL and the name of a dialect.

                                                                                                               ### Further reading

                                                                                                               - You thought I was kidding about the server-side scripting idea. So did I, until I found this web-based dialectizer. Unfortunately, source code does not appear to be available.

                                                                                                               ## 8.10. Summary

                                                                                                               Python provides you with a powerful tool, `sgmllib.py`, to manipulate HTML by turning its structure into an object model. You can use this tool in many different ways.

                                                                                                               - parsing the HTML looking for something specific
                                                                                                               - aggregating the results, like the URL lister
                                                                                                               - altering the structure along the way, like the attribute quoter
                                                                                                               - transforming the HTML into something else by manipulating the text while leaving the tags alone, like the `Dialectizer`

                                                                                                               Along with these examples, you should be comfortable doing all of the following things:

                                                                                                               - Using `locals`() and `globals`() to access namespaces
                                                                                                               - Formatting strings using dictionary-based substitutions

                                                                                                               [1] The technical term for a parser like `SGMLParser` is a consumer: it consumes HTML and breaks it down. Presumably, the name `feed` was chosen to fit into the whole “consumer” motif. Personally, it makes me think of an exhibit in the zoo where there's just a dark cage with no trees or plants or evidence of life of any kind, but if you stand perfectly still and look really closely you can make out two beady eyes staring back at you from the far left corner, but you convince yourself that that's just your mind playing tricks on you, and the only way you can tell that the whole thing isn't just an empty cage is a small innocuous sign on the railing that reads, “Do not feed the parser.” But maybe that's just me. In any event, it's an interesting mental image.

                                                                                                               [2] The reason Python is better at lists than strings is that lists are mutable but strings are immutable. This means that appending to a list just adds the element and updates the index. Since strings can not be changed after they are created, code like `s = s + newpiece` will create an entirely new string out of the concatenation of the original and the new piece, then throw away the original string. This involves a lot of expensive memory management, and the amount of effort involved increases as the string gets longer, so doing `s = s + newpiece` in a loop is deadly. In technical terms, appending n items to a list is `O(n)`, while appending n items to a string is `O(n2)`.

                                                                                                               [3] I don't get out much.

                                                                                                               [4] All right, it's not that common a question. It's not up there with “What editor should I use to write Python code?” (answer: Emacs) or “Is Python better or worse than Perl?” (answer: “Perl is worse than Python because people wanted it worse.” -Larry Wall, 10/14/1998) But questions about HTML processing pop up in one form or another about once a month, and among those questions, this is a popular one.

                                                                                                               ## Chapter 9. XML Processing

                                                                                                               ## 9.1. Diving in

                                                                                                               These next two chapters are about XML processing in Python. It would be helpful if you already knew what an XML document looks like, that it's made up of structured tags to form a hierarchy of elements, and so on. If this doesn't make sense to you, there are many XML tutorials that can explain the basics.

                                                                                                               If you're not particularly interested in XML, you should still read these chapters, which cover important topics like Python packages, Unicode, command line arguments, and how to use `getattr` for method dispatching.

                                                                                                               Being a philosophy major is not required, although if you have ever had the misfortune of being subjected to the writings of Immanuel Kant, you will appreciate the example program a lot more than if you majored in something useful, like computer science.

                                                                                                               There are two basic ways to work with XML. One is called SAX (“Simple API for XML”), and it works by reading the XML a little bit at a time and calling a method for each element it finds. (If you read Chapter 8, HTML Processing, this should sound familiar, because that's how the `sgmllib` module works.) The other is called DOM (“Document Object Model”), and it works by reading in the entire XML document at once and creating an internal representation of it using native Python classes linked in a tree structure. Python has standard modules for both kinds of parsing, but this chapter will only deal with using the DOM.

                                                                                                               The following is a complete Python program which generates pseudo-random output based on a context-free grammar defined in an XML format. Don't worry yet if you don't understand what that means; you'll examine both the program's input and its output in more depth throughout these next two chapters.

                                                                                                               ### Example 9.1. `kgp.py`

                                                                                                               If you have not already done so, you can download this and other examples used in this book.

                                                                                                                   """Kant Generator for Python

                                                                                                                   Generates mock philosophy based on a context-free grammar

                                                                                                                   Usage: python kgp.py [options] [source]

                                                                                                                   Options:
                                                                                                                     -g ..., --grammar=...  use specified grammar file or URL
                                                                                                                     -h, --help              show this help
                                                                                                                     -d    show debugging information while parsing

                                                                                                                   Examples:
                                                                                                                     kgp.pygenerates several paragraphs of Kantian philosophy
                                                                                                                     kgp.py -g husserl.xml   generates several paragraphs of Husserl
                                                                                                                     kpg.py "<xref id='paragraph'/>"  generates a paragraph of Kant
                                                                                                                     kgp.py template.xml     reads from template.xml to decide what to generate
                                                                                                                   """
                                                                                                                   from xml.dom import minidom
                                                                                                                   import random
                                                                                                                   import toolbox
                                                                                                                   import sys
                                                                                                                   import getopt

                                                                                                                   _debug = 0

                                                                                                                   class NoSourceError(Exception): pass

                                                                                                                   class KantGenerator:
                                                                                                                       """generates mock philosophy based on a context-free grammar"""

                                                                                                                       def __init__(self, grammar, source=None):
                                                                                                                           self.loadGrammar(grammar)
                                                                                                                           self.loadSource(source and source or self.getDefaultSource())
                                                                                                                           self.refresh()

                                                                                                                       def _load(self, source):
                                                                                                                           """load XML input source, return parsed XML document

                                                                                                                           - a URL of a remote XML file ("http://diveintopython3.org/kant.xml")
                                                                                                                           - a filename of a local XML file ("~/diveintopython3/common/py/kant.xml")
                                                                                                                           - standard input ("-")
                                                                                                                           - the actual XML document, as a string
                                                                                                                           """
                                                                                                                           sock = toolbox.openAnything(source)
                                                                                                                           xmldoc = minidom.parse(sock).documentElement
                                                                                                                           sock.close()
                                                                                                                           return xmldoc

                                                                                                                       def loadGrammar(self, grammar):
                                                                                                                           """load context-free grammar"""
                                                                                                                           self.grammar = self._load(grammar)
                                                                                                                           self.refs = {}
                                                                                                                           for ref in self.grammar.getElementsByTagName("ref"):
                                                                                                                               self.refs[ref.attributes["id"].value] = ref

                                                                                                                       def loadSource(self, source):
                                                                                                                           """load source"""
                                                                                                                           self.source = self._load(source)

                                                                                                                       def getDefaultSource(self):
                                                                                                                           """guess default source of the current grammar

                                                                                                                           The default source will be one of the <ref>s that is not
                                                                                                                           cross-referenced. This sounds complicated but it's not.
                                                                                                                           Example: The default source for kant.xml is
                                                                                                                           "<xref id='section'/>", because 'section' is the one <ref>
                                                                                                                           that is not <xref>'d anywhere in the grammar.
                                                                                                                           In most grammars, the default source will produce the
                                                                                                                           longest (and most interesting) output.
                                                                                                                           """
                                                                                                                           xrefs = {}
                                                                                                                           for xref in self.grammar.getElementsByTagName("xref"):
                                                                                                                               xrefs[xref.attributes["id"].value] = 1
                                                                                                                           xrefs = xrefs.keys()
                                                                                                                           standaloneXrefs = [e for e in self.refs.keys() if e not in xrefs]
                                                                                                                           if not standaloneXrefs:
                                                                                                                               raise NoSourceError, "can't guess source, and no source specified"
                                                                                                                           return '<xref id="%s"/>' % random.choice(standaloneXrefs)

                                                                                                                       def reset(self):
                                                                                                                           """reset parser"""
                                                                                                                           self.pieces = []
                                                                                                                           self.capitalizeNextWord = 0

                                                                                                                       def refresh(self):
                                                                                                                           """reset output buffer, re-parse entire source file, and return output

                                                                                                                           Since parsing involves a good deal of randomness, this is an
                                                                                                                           easy way to get new output without having to reload a grammar file
                                                                                                                           each time.
                                                                                                                           """
                                                                                                                           self.reset()
                                                                                                                           self.parse(self.source)
                                                                                                                           return self.output()

                                                                                                                       def output(self):
                                                                                                                           """output generated text"""
                                                                                                                           return "".join(self.pieces)

                                                                                                                       def randomChildElement(self, node):
                                                                                                                           """choose a random child element of a node

                                                                                                                           This is a utility method used by do_xref and do_choice.
                                                                                                                           """
                                                                                                                           choices = [e for e in node.childNodes
                                                                                                                    if e.nodeType == e.ELEMENT_NODE]
                                                                                                                           chosen = random.choice(choices)
                                                                                                                           if _debug:
                                                                                                                               sys.stderr.write('%s available choices: %s\n' % \
                                                                                                                                   (len(choices), [e.toxml() for e in choices]))
                                                                                                                               sys.stderr.write('Chosen: %s\n' % chosen.toxml())
                                                                                                                           return chosen

                                                                                                                       def parse(self, node):
                                                                                                                           """parse a single XML node

                                                                                                                           A parsed XML document (from minidom.parse) is a tree of nodes
                                                                                                                           of various types. Each node is represented by an instance of the
                                                                                                                           corresponding Python class (Element for a tag, Text for
                                                                                                                           text data, Document for the top-level document). The following
                                                                                                                           statement constructs the name of a class method based on the type
                                                                                                                           of node we're parsing ("parse_Element" for an Element node,
                                                                                                                           "parse_Text" for a Text node, etc.) and then calls the method.
                                                                                                                           """
                                                                                                                           parseMethod = getattr(self, "parse_%s" % node.__class__.__name__)
                                                                                                                           parseMethod(node)

                                                                                                                       def parse_Document(self, node):
                                                                                                                           """parse the document node

                                                                                                                           The document node by itself isn't interesting (to us), but
                                                                                                                           its only child, node.documentElement, is: it's the root node
                                                                                                                           of the grammar.
                                                                                                                           """
                                                                                                                           self.parse(node.documentElement)

                                                                                                                       def parse_Text(self, node):
                                                                                                                           """parse a text node

                                                                                                                           The text of a text node is usually added to the output buffer
                                                                                                                           verbatim. The one exception is that <p class='sentence'> sets
                                                                                                                           a flag to capitalize the first letter of the next word. If
                                                                                                                           that flag is set, we capitalize the text and reset the flag.
                                                                                                                           """
                                                                                                                           text = node.data
                                                                                                                           if self.capitalizeNextWord:
                                                                                                                               self.pieces.append(text[0].upper())
                                                                                                                               self.pieces.append(text[1:])
                                                                                                                               self.capitalizeNextWord = 0
                                                                                                                           else:
                                                                                                                               self.pieces.append(text)

                                                                                                                       def parse_Element(self, node):
                                                                                                                           """parse an element

                                                                                                                           An XML element corresponds to an actual tag in the source:
                                                                                                                           <xref id='...'>, <p chance='...'>, <choice>, etc.
                                                                                                                           Each element type is handled in its own method. Like we did in
                                                                                                                           parse(), we construct a method name based on the name of the
                                                                                                                           element ("do_xref" for an <xref> tag, etc.) and
                                                                                                                           call the method.
                                                                                                                           """
                                                                                                                           handlerMethod = getattr(self, "do_%s" % node.tagName)
                                                                                                                           handlerMethod(node)

                                                                                                                       def parse_Comment(self, node):
                                                                                                                           """parse a comment

                                                                                                                           The grammar can contain XML comments, but we ignore them
                                                                                                                           """
                                                                                                                           pass

                                                                                                                       def do_xref(self, node):
                                                                                                                           """handle <xref id='...'> tag

                                                                                                                           An <xref id='...'> tag is a cross-reference to a <ref id='...'>
                                                                                                                           tag. <xref id='sentence'/> evaluates to a randomly chosen child of
                                                                                                                           <ref id='sentence'>.
                                                                                                                           """
                                                                                                                           id = node.attributes["id"].value
                                                                                                                           self.parse(self.randomChildElement(self.refs[id]))

                                                                                                                       def do_p(self, node):
                                                                                                                           """handle <p> tag

                                                                                                                           The <p> tag is the core of the grammar. It can contain almost
                                                                                                                           anything: freeform text, <choice> tags, <xref> tags, even other
                                                                                                                           <p> tags. If a "class='sentence'" attribute is found, a flag
                                                                                                                           is set and the next word will be capitalized. If a "chance='X'"
                                                                                                                           attribute is found, there is an X% chance that the tag will be
                                                                                                                           evaluated (and therefore a (100-X)% chance that it will be
                                                                                                                           completely ignored)
                                                                                                                           """
                                                                                                                           keys = node.attributes.keys()
                                                                                                                           if "class" in keys:
                                                                                                                               if node.attributes["class"].value == "sentence":
                                                                                                                                   self.capitalizeNextWord = 1
                                                                                                                           if "chance" in keys:
                                                                                                                               chance = int(node.attributes["chance"].value)
                                                                                                                               doit = (chance > random.randrange(100))
                                                                                                                           else:
                                                                                                                               doit = 1
                                                                                                                           if doit:
                                                                                                                               for child in node.childNodes: self.parse(child)

                                                                                                                       def do_choice(self, node):
                                                                                                                           """handle <choice> tag

                                                                                                                           A <choice> tag contains one or more <p> tags. One <p> tag
                                                                                                                           is chosen at random and evaluated; the rest are ignored.
                                                                                                                           """
                                                                                                                           self.parse(self.randomChildElement(node))

                                                                                                                   def usage():
                                                                                                                       print __doc__

                                                                                                                   def main(argv):
                                                                                                                       grammar = "kant.xml"
                                                                                                                       try:
                                                                                                                           opts, args = getopt.getopt(argv, "hg:d", ["help", "grammar="])
                                                                                                                       except getopt.GetoptError:
                                                                                                                           usage()
                                                                                                                           sys.exit(2)
                                                                                                                       for opt, arg in opts:
                                                                                                                           if opt in ("-h", "--help"):
                                                                                                                               usage()
                                                                                                                               sys.exit()
                                                                                                                           elif opt == '-d':
                                                                                                                               global _debug
                                                                                                                               _debug = 1
                                                                                                                           elif opt in ("-g", "--grammar"):
                                                                                                                               grammar = arg

                                                                                                                       source = "".join(args)

                                                                                                                       k = KantGenerator(grammar, source)
                                                                                                                       print k.output()

                                                                                                                   if __name__ == "__main__":
                                                                                                                       main(sys.argv[1:])

                                                                                                               ### Example 9.2. `toolbox.py`

                                                                                                                   """Miscellaneous utility functions"""

                                                                                                                   def openAnything(source):
                                                                                                                       """URI, filename, or string --> stream

                                                                                                                       This function lets you define parsers that take any input source
                                                                                                                       (URL, pathname to local or network file, or actual data as a string)
                                                                                                                       and deal with it in a uniform manner. Returned object is guaranteed
                                                                                                                       to have all the basic stdio read methods (read, readline, readlines).
                                                                                                                       Just .close() the object when you're done with it.

                                                                                                                       Examples:
                                                                                                                       >>> from xml.dom import minidom
                                                                                                                       >>> sock = openAnything("http://localhost/kant.xml")
                                                                                                                       >>> doc = minidom.parse(sock)
                                                                                                                       >>> sock.close()
                                                                                                                       >>> sock = openAnything("c:\\inetpub\\wwwroot\\kant.xml")
                                                                                                                       >>> doc = minidom.parse(sock)
                                                                                                                       >>> sock.close()
                                                                                                                       >>> sock = openAnything("<ref id='conjunction'><text>and</text><text>or</text></ref>")
                                                                                                                       >>> doc = minidom.parse(sock)
                                                                                                                       >>> sock.close()
                                                                                                                       """
                                                                                                                       if hasattr(source, "read"):
                                                                                                                           return source

                                                                                                                       if source == '-':
                                                                                                                           import sys
                                                                                                                           return sys.stdin

                                                                                                                       # try to open with urllib (if source is http, ftp, or file URL)
                                                                                                                       import urllib
                                                                                                                       try:
                                                                                                                           return urllib.urlopen(source)
                                                                                                                       except (IOError, OSError):
                                                                                                                           pass

                                                                                                                       # try to open with native open function (if source is pathname)
                                                                                                                       try:
                                                                                                                           return open(source)
                                                                                                                       except (IOError, OSError):
                                                                                                                           pass

                                                                                                                       # treat source as string
                                                                                                                       import StringIO
                                                                                                                       return StringIO.StringIO(str(source))

                                                                                                               Run the program `kgp.py` by itself, and it will parse the default XML-based grammar, in `kant.xml`, and print several paragraphs worth of philosophy in the style of Immanuel Kant.

                                                                                                               ### Example 9.3. Sample output of `kgp.py`

                                                                                                                   [you@localhost kgp]$ python kgp.py
                                                                                                                        As is shown in the writings of Hume, our a priori concepts, in
                                                                                                                   reference to ends, abstract from all content of knowledge; in the study
                                                                                                                   of space, the discipline of human reason, in accordance with the
                                                                                                                   principles of philosophy, is the clue to the discovery of the
                                                                                                                   Transcendental Deduction. The transcendental aesthetic, in all
                                                                                                                   theoretical sciences, occupies part of the sphere of human reason
                                                                                                                   concerning the existence of our ideas in general; still, the
                                                                                                                   never-ending regress in the series of empirical conditions constitutes
                                                                                                                   the whole content for the transcendental unity of apperception. What
                                                                                                                   we have alone been able to show is that, even as this relates to the
                                                                                                                   architectonic of human reason, the Ideal may not contradict itself, but
                                                                                                                   it is still possible that it may be in contradictions with the
                                                                                                                   employment of the pure employment of our hypothetical judgements, but
                                                                                                                   natural causes (and I assert that this is the case) prove the validity
                                                                                                                   of the discipline of pure reason. As we have already seen, time (and
                                                                                                                   it is obvious that this is true) proves the validity of time, and the
                                                                                                                   architectonic of human reason, in the full sense of these terms,
                                                                                                                   abstracts from all content of knowledge. I assert, in the case of the
                                                                                                                   discipline of practical reason, that the Antinomies are just as
                                                                                                                   necessary as natural causes, since knowledge of the phenomena is a
                                                                                                                   posteriori.
                                                                                                                       The discipline of human reason, as I have elsewhere shown, is by
                                                                                                                   its very nature contradictory, but our ideas exclude the possibility of
                                                                                                                   the Antinomies. We can deduce that, on the contrary, the pure
                                                                                                                   employment of philosophy, on the contrary, is by its very nature
                                                                                                                   contradictory, but our sense perceptions are a representation of, in
                                                                                                                   the case of space, metaphysics. The thing in itself is a
                                                                                                                   representation of philosophy. Applied logic is the clue to the
                                                                                                                   discovery of natural causes. However, what we have alone been able to
                                                                                                                   show is that our ideas, in other words, should only be used as a canon
                                                                                                                   for the Ideal, because of our necessary ignorance of the conditions.

                                                                                                                   [...snip...]

                                                                                                               This is, of course, complete gibberish. Well, not complete gibberish. It is syntactically and grammatically correct (although very verbose -- Kant wasn't what you would call a get-to-the-point kind of guy). Some of it may actually be true (or at least the sort of thing that Kant would have agreed with), some of it is blatantly false, and most of it is simply incoherent. But all of it is in the style of Immanuel Kant.

                                                                                                               Let me repeat that this is much, much funnier if you are now or have ever been a philosophy major.

                                                                                                               The interesting thing about this program is that there is nothing Kant-specific about it. All the content in the previous example was derived from the grammar file, `kant.xml`. If you tell the program to use a different grammar file (which you can specify on the command line), the output will be completely different.

                                                                                                               ### Example 9.4. Simpler output from `kgp.py`

                                                                                                                   [you@localhost kgp]$ python kgp.py -g binary.xml
                                                                                                                   00101001
                                                                                                                   [you@localhost kgp]$ python kgp.py -g binary.xml
                                                                                                                   10110100

                                                                                                               You will take a closer look at the structure of the grammar file later in this chapter. For now, all you need to know is that the grammar file defines the structure of the output, and the `kgp.py` program reads through the grammar and makes random decisions about which words to plug in where.

                                                                                                               ## 9.2. Packages

                                                                                                               Actually parsing an XML document is very simple: one line of code. However, before you get to that line of code, you need to take a short detour to talk about packages.

                                                                                                               ### Example 9.5. Loading an XML document (a sneak peek)

                                                                                                                   >>> from xml.dom import minidom ①
                                                                                                                   >>> xmldoc = minidom.parse('~/diveintopython3/common/py/kgp/binary.xml')

                                                                                                               1. This is a syntax you haven't seen before. It looks almost like the `from module import` you know and love, but the `"."` gives it away as something above and beyond a simple import. In fact, `xml` is what is known as a package, `dom` is a nested package within `xml`, and `minidom` is a module within `xml.dom`.

                                                                                                                  That sounds complicated, but it's really not. Looking at the actual implementation may help. Packages are little more than directories of modules; nested packages are subdirectories. The modules within a package (or a nested package) are still just `.py` files, like always, except that they're in a subdirectory instead of the main `lib/` directory of your Python installation.

                                                                                                                  ### Example 9.6. File layout of a package

                                                                                                                      Python21/           root Python installation (home of the executable)
                                                                                                                      |
                                                                                                                      +--lib/             library directory (home of the standard library modules)
                                                                                                                         |
                                                                                                                         +-- xml/         xml package (really just a directory with other stuff in it)
                                                                                                                             |
                                                                                                                             +--sax/      xml.sax package (again, just a directory)
                                                                                                                             |
                                                                                                                             +--dom/      xml.dom package (contains minidom.py)
                                                                                                                             |
                                                                                                                             +--parsers/  xml.parsers package (used internally)

                                                                                                                  So when you say `from xml.dom import minidom`, Python figures out that that means “look in the `xml` directory for a `dom` directory, and look in that for the `minidom` module, and import it as `minidom`”. But Python is even smarter than that; not only can you import entire modules contained within a package, you can selectively import specific classes or functions from a module contained within a package. You can also import the package itself as a module. The syntax is all the same; Python figures out what you mean based on the file layout of the package, and automatically does the right thing.

                                                                                                                  ### Example 9.7. Packages are modules, too

                                                                                                                      >>> from xml.dom import minidom         ①
                                                                                                                      >>> minidom
                                                                                                                      <module 'xml.dom.minidom' from 'C:\Python21\lib\xml\dom\minidom.pyc'>
                                                                                                                      >>> minidom.Element
                                                                                                                      <class xml.dom.minidom.Element at 01095744>
                                                                                                                      >>> from xml.dom.minidom import Element ②
                                                                                                                      >>> Element
                                                                                                                      <class xml.dom.minidom.Element at 01095744>
                                                                                                                      >>> minidom.Element
                                                                                                                      <class xml.dom.minidom.Element at 01095744>
                                                                                                                      >>> from xml import dom                 ③
                                                                                                                      >>> dom
                                                                                                                      <module 'xml.dom' from 'C:\Python21\lib\xml\dom\__init__.pyc'>
                                                                                                                      >>> import xml        ④
                                                                                                                      >>> xml
                                                                                                                      <module 'xml' from 'C:\Python21\lib\xml\__init__.pyc'>

                                                                                                                  1. Here you're importing a module (`minidom`) from a nested package (`xml.dom`). The result is that `minidom` is imported into your namespace, and in order to reference classes within the `minidom` module (like `Element`), you need to preface them with the module name.
                                                                                                                  2. Here you are importing a class (`Element`) from a module (`minidom`) from a nested package (`xml.dom`). The result is that `Element` is imported directly into your namespace. Note that this does not interfere with the previous import; the `Element` class can now be referenced in two ways (but it's all still the same class).
                                                                                                                  3. Here you are importing the `dom` package (a nested package of `xml`) as a module in and of itself. Any level of a package can be treated as a module, as you'll see in a moment. It can even have its own attributes and methods, just the modules you've seen before.
                                                                                                                  4. Here you are importing the root level `xml` package as a module.

                                                                                                                     So how can a package (which is just a directory on disk) be imported and treated as a module (which is always a file on disk)? The answer is the magical `__init__.py` file. You see, packages are not simply directories; they are directories with a specific file, `__init__.py`, inside. This file defines the attributes and methods of the package. For instance, `xml.dom` contains a `Node` class, which is defined in `xml/dom/__init__.py`. When you import a package as a module (like `dom` from `xml`), you're really importing its `__init__.py` file.

                                                                                                                     A package is a directory with the special `__init__.py` file in it. The `__init__.py` file defines the attributes and methods of the package. It doesn't need to define anything; it can just be an empty file, but it has to exist. But if `__init__.py` doesn't exist, the directory is just a directory, not a package, and it can't be imported or contain modules or nested packages.

                                                                                                                     So why bother with packages? Well, they provide a way to logically group related modules. Instead of having an `xml` package with `sax` and `dom` packages inside, the authors could have chosen to put all the `sax` functionality in `xmlsax.py` and all the `dom` functionality in `xmldom.py`, or even put all of it in a single module. But that would have been unwieldy (as of this writing, the XML package has over 3000 lines of code) and difficult to manage (separate source files mean multiple people can work on different areas simultaneously).

                                                                                                                     If you ever find yourself writing a large subsystem in Python (or, more likely, when you realize that your small subsystem has grown into a large one), invest some time designing a good package architecture. It's one of the many things Python is good at, so take advantage of it.

                                                                                                                     ## 9.3. Parsing XML

                                                                                                                     As I was saying, actually parsing an XML document is very simple: one line of code. Where you go from there is up to you.

                                                                                                                     ### Example 9.8. Loading an XML document (for real this time)

                                                                                                                         >>> from xml.dom import minidom      ①
                                                                                                                         >>> xmldoc = minidom.parse('~/diveintopython3/common/py/kgp/binary.xml')  ②
                                                                                                                         >>> xmldoc         ③
                                                                                                                         <xml.dom.minidom.Document instance at 010BE87C>
                                                                                                                         >>> print xmldoc.toxml()             ④
                                                                                                                         <?xml version="1.0" ?>
                                                                                                                         <grammar>
                                                                                                                         <ref id="bit">
                                                                                                                           <p>0</p>
                                                                                                                           <p>1</p>
                                                                                                                         </ref>
                                                                                                                         <ref id="byte">
                                                                                                                           <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
                                                                                                                         <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
                                                                                                                         </ref>
                                                                                                                         </grammar>

                                                                                                                     1. As you saw in the previous section, this imports the `minidom` module from the `xml.dom` package.
                                                                                                                     2. Here is the one line of code that does all the work: `minidom.parse` takes one argument and returns a parsed representation of the XML document. The argument can be many things; in this case, it's simply a filename of an XML document on my local disk. (To follow along, you'll need to change the path to point to your downloaded examples directory.) But you can also pass a file object, or even a file-like object. You'll take advantage of this flexibility later in this chapter.
                                                                                                                     3. The object returned from `minidom.parse` is a `Document` object, a descendant of the `Node` class. This `Document` object is the root level of a complex tree-like structure of interlocking Python objects that completely represent the XML document you passed to `minidom.parse`.
                                                                                                                     4. `toxml` is a method of the `Node` class (and is therefore available on the `Document` object you got from `minidom.parse`). `toxml` prints out the XML that this `Node` represents. For the `Document` node, this prints out the entire XML document.

                                                                                                                        Now that you have an XML document in memory, you can start traversing through it.

                                                                                                                        ### Example 9.9. Getting child nodes

                                                                                                                            >>> xmldoc.childNodes    ①
                                                                                                                            [<DOM Element: grammar at 17538908>]
                                                                                                                            >>> xmldoc.childNodes[0] ②
                                                                                                                            <DOM Element: grammar at 17538908>
                                                                                                                            >>> xmldoc.firstChild    ③
                                                                                                                            <DOM Element: grammar at 17538908>

                                                                                                                        1. Every `Node` has a `childNodes` attribute, which is a list of the `Node` objects. A `Document` always has only one child node, the root element of the XML document (in this case, the `grammar` element).
                                                                                                                        2. To get the first (and in this case, the only) child node, just use regular list syntax. Remember, there is nothing special going on here; this is just a regular Python list of regular Python objects.
                                                                                                                        3. Since getting the first child node of a node is a useful and common activity, the `Node` class has a `firstChild` attribute, which is synonymous with `childNodes[0]`. (There is also a `lastChild` attribute, which is synonymous with `childNodes[-1]`.)

                                                                                                                           ### Example 9.10. `toxml` works on any node

                                                                                                                               >>> grammarNode = xmldoc.firstChild
                                                                                                                               >>> print grammarNode.toxml() ①
                                                                                                                               <grammar>
                                                                                                                               <ref id="bit">
                                                                                                                                 <p>0</p>
                                                                                                                                 <p>1</p>
                                                                                                                               </ref>
                                                                                                                               <ref id="byte">
                                                                                                                                 <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
                                                                                                                               <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
                                                                                                                               </ref>
                                                                                                                               </grammar>

                                                                                                                           1. Since the `toxml` method is defined in the `Node` class, it is available on any XML node, not just the `Document` element.

                                                                                                                              ### Example 9.11. Child nodes can be text

                                                                                                                                  >>> grammarNode.childNodes①
                                                                                                                                  [<DOM Text node "\n">, <DOM Element: ref at 17533332>, \
                                                                                                                                  <DOM Text node "\n">, <DOM Element: ref at 17549660>, <DOM Text node "\n">]
                                                                                                                                  >>> print grammarNode.firstChild.toxml()    ②

                                                                                                                                  >>> print grammarNode.childNodes[1].toxml() ③
                                                                                                                                  <ref id="bit">
                                                                                                                                    <p>0</p>
                                                                                                                                    <p>1</p>
                                                                                                                                  </ref>
                                                                                                                                  >>> print grammarNode.childNodes[3].toxml() ④
                                                                                                                                  <ref id="byte">
                                                                                                                                    <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
                                                                                                                                  <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
                                                                                                                                  </ref>
                                                                                                                                  >>> print grammarNode.lastChild.toxml()     ⑤

                                                                                                                              1. Looking at the XML in `binary.xml`, you might think that the `grammar` has only two child nodes, the two `ref` elements. But you're missing something: the carriage returns! After the `'<grammar>'` and before the first `'<ref>'` is a carriage return, and this text counts as a child node of the `grammar` element. Similarly, there is a carriage return after each `'</ref>'`; these also count as child nodes. So `grammar.childNodes` is actually a list of 5 objects: 3 `Text` objects and 2 `Element` objects.
                                                                                                                              2. The first child is a `Text` object representing the carriage return after the `'<grammar>'` tag and before the first `'<ref>'` tag.
                                                                                                                              3. The second child is an `Element` object representing the first `ref` element.
                                                                                                                              4. The fourth child is an `Element` object representing the second `ref` element.
                                                                                                                              5. The last child is a `Text` object representing the carriage return after the `'</ref>'` end tag and before the `'</grammar>'` end tag.

                                                                                                                                 ### Example 9.12. Drilling down all the way to text

                                                                                                                                     >>> grammarNode
                                                                                                                                     <DOM Element: grammar at 19167148>
                                                                                                                                     >>> refNode = grammarNode.childNodes[1] ①
                                                                                                                                     >>> refNode
                                                                                                                                     <DOM Element: ref at 17987740>
                                                                                                                                     >>> refNode.childNodes②
                                                                                                                                     [<DOM Text node "\n">, <DOM Text node "  ">, <DOM Element: p at 19315844>, \
                                                                                                                                     <DOM Text node "\n">, <DOM Text node "  ">, \
                                                                                                                                     <DOM Element: p at 19462036>, <DOM Text node "\n">]
                                                                                                                                     >>> pNode = refNode.childNodes[2]
                                                                                                                                     >>> pNode
                                                                                                                                     <DOM Element: p at 19315844>
                                                                                                                                     >>> print pNode.toxml()                 ③
                                                                                                                                     <p>0</p>
                                                                                                                                     >>> pNode.firstChild  ④
                                                                                                                                     <DOM Text node "0">
                                                                                                                                     >>> pNode.firstChild.data               ⑤
                                                                                                                                     u'0'

                                                                                                                                 1. As you saw in the previous example, the first `ref` element is `grammarNode.childNodes[1]`, since childNodes[0] is a `Text` node for the carriage return.
                                                                                                                                 2. The `ref` element has its own set of child nodes, one for the carriage return, a separate one for the spaces, one for the `p` element, and so forth.
                                                                                                                                 3. You can even use the `toxml` method here, deeply nested within the document.
                                                                                                                                 4. The `p` element has only one child node (you can't tell that from this example, but look at `pNode.childNodes` if you don't believe me), and it is a `Text` node for the single character `'0'`.
                                                                                                                                 5. The `.data` attribute of a `Text` node gives you the actual string that the text node represents. But what is that `'u'` in front of the string? The answer to that deserves its own section. (Unicode stuff was here)

                                                                                                                                    Remember I said Python usually converted unicode to ASCII whenever it needed to make a regular string out of a unicode string? Well, this default encoding scheme is an option which you can customize.

                                                                                                                                    ### Example 9.15. `sitecustomize.py`

                                                                                                                                        # sitecustomize.py ①
                                                                                                                                        # this file can be anywhere in your Python path,
                                                                                                                                        # but it usually goes in ${pythondir}/lib/site-packages/
                                                                                                                                        import sys
                                                                                                                                        sys.setdefaultencoding('iso-8859-1') ②

                                                                                                                                    1. `sitecustomize.py` is a special script; Python will try to import it on startup, so any code in it will be run automatically. As the comment mentions, it can go anywhere (as long as `import` can find it), but it usually goes in the `site-packages` directory within your Python `lib` directory.
                                                                                                                                    2. `setdefaultencoding` function sets, well, the default encoding. This is the encoding scheme that Python will try to use whenever it needs to auto-coerce a unicode string into a regular string.

                                                                                                                                       ### Example 9.16. Effects of setting the default encoding

                                                                                                                                           >>> import sys
                                                                                                                                           >>> sys.getdefaultencoding() ①
                                                                                                                                           'iso-8859-1'
                                                                                                                                           >>> s = u'La Pe\xf1a'
                                                                                                                                           >>> print s②
                                                                                                                                           La Peña

                                                                                                                                       1. This example assumes that you have made the changes listed in the previous example to your `sitecustomize.py` file, and restarted Python. If your default encoding still says `'ascii'`, you didn't set up your `sitecustomize.py` properly, or you didn't restart Python. The default encoding can only be changed during Python startup; you can't change it later. (Due to some wacky programming tricks that I won't get into right now, you can't even call `sys.setdefaultencoding` after Python has started up. Dig into `site.py` and search for “`setdefaultencoding`” to find out how.)
                                                                                                                                       2. Now that the default encoding scheme includes all the characters you use in your string, Python has no problem auto-coercing the string and printing it. (More Unicode stuff was here)

                                                                                                                                          ## 9.5. Searching for elements

                                                                                                                                          Traversing XML documents by stepping through each node can be tedious. If you're looking for something in particular, buried deep within your XML document, there is a shortcut you can use to find it quickly: `getElementsByTagName`.

                                                                                                                                          For this section, you'll be using the `binary.xml` grammar file, which looks like this:

                                                                                                                                          ### Example 9.20. `binary.xml`

                                                                                                                                              <?xml version="1.0"?>
                                                                                                                                              <!DOCTYPE grammar PUBLIC "-//diveintopython3.org//DTD Kant Generator Pro v1.0//EN" "kgp.dtd">
                                                                                                                                              <grammar>
                                                                                                                                              <ref id="bit">
                                                                                                                                                <p>0</p>
                                                                                                                                                <p>1</p>
                                                                                                                                              </ref>
                                                                                                                                              <ref id="byte">
                                                                                                                                                <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
                                                                                                                                              <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
                                                                                                                                              </ref>
                                                                                                                                              </grammar>

                                                                                                                                          It has two `ref`s, `'bit'` and `'byte'`. A `bit` is either a `'0'` or `'1'`, and a `byte` is 8 `bit`s.

                                                                                                                                          ### Example 9.21. Introducing `getElementsByTagName`

                                                                                                                                              >>> from xml.dom import minidom
                                                                                                                                              >>> xmldoc = minidom.parse('binary.xml')
                                                                                                                                              >>> reflist = xmldoc.getElementsByTagName('ref') ①
                                                                                                                                              >>> reflist
                                                                                                                                              [<DOM Element: ref at 136138108>, <DOM Element: ref at 136144292>]
                                                                                                                                              >>> print reflist[0].toxml()
                                                                                                                                              <ref id="bit">
                                                                                                                                                <p>0</p>
                                                                                                                                                <p>1</p>
                                                                                                                                              </ref>
                                                                                                                                              >>> print reflist[1].toxml()
                                                                                                                                              <ref id="byte">
                                                                                                                                                <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
                                                                                                                                              <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
                                                                                                                                              </ref>

                                                                                                                                          1. `getElementsByTagName` takes one argument, the name of the element you wish to find. It returns a list of `Element` objects, corresponding to the XML elements that have that name. In this case, you find two `ref` elements.

                                                                                                                                             ### Example 9.22. Every element is searchable

                                                                                                                                                 >>> firstref = reflist[0]    ①
                                                                                                                                                 >>> print firstref.toxml()
                                                                                                                                                 <ref id="bit">
                                                                                                                                                   <p>0</p>
                                                                                                                                                   <p>1</p>
                                                                                                                                                 </ref>
                                                                                                                                                 >>> plist = firstref.getElementsByTagName("p") ②
                                                                                                                                                 >>> plist
                                                                                                                                                 [<DOM Element: p at 136140116>, <DOM Element: p at 136142172>]
                                                                                                                                                 >>> print plist[0].toxml()   ③
                                                                                                                                                 <p>0</p>
                                                                                                                                                 >>> print plist[1].toxml()
                                                                                                                                                 <p>1</p>

                                                                                                                                             1. Continuing from the previous example, the first object in your reflist is the `'bit'` `ref` element.
                                                                                                                                             2. You can use the same `getElementsByTagName` method on this `Element` to find all the `<p>` elements within the `'bit'` `ref` element.
                                                                                                                                             3. Just as before, the `getElementsByTagName` method returns a list of all the elements it found. In this case, you have two, one for each bit.

                                                                                                                                                ### Example 9.23. Searching is actually recursive

                                                                                                                                                    >>> plist = xmldoc.getElementsByTagName("p") ①
                                                                                                                                                    >>> plist
                                                                                                                                                    [<DOM Element: p at 136140116>, <DOM Element: p at 136142172>, <DOM Element: p at 136146124>]
                                                                                                                                                    >>> plist[0].toxml()       ②
                                                                                                                                                    '<p>0</p>'
                                                                                                                                                    >>> plist[1].toxml()
                                                                                                                                                    '<p>1</p>'
                                                                                                                                                    >>> plist[2].toxml()       ③
                                                                                                                                                    '<p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
                                                                                                                                                    <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>'

                                                                                                                                                1. Note carefully the difference between this and the previous example. Previously, you were searching for `p` elements within firstref, but here you are searching for `p` elements within xmldoc, the root-level object that represents the entire XML document. This does find the `p` elements nested within the `ref` elements within the root `grammar` element.
                                                                                                                                                2. The first two `p` elements are within the first `ref` (the `'bit'` `ref`).
                                                                                                                                                3. The last `p` element is the one within the second `ref` (the `'byte'` `ref`).

                                                                                                                                                   ## 9.6. Accessing element attributes

                                                                                                                                                   XML elements can have one or more attributes, and it is incredibly simple to access them once you have parsed an XML document.

                                                                                                                                                   For this section, you'll be using the `binary.xml` grammar file that you saw in the previous section.

                                                                                                                                                   This section may be a little confusing, because of some overlapping terminology. Elements in an XML document have attributes, and Python objects also have attributes. When you parse an XML document, you get a bunch of Python objects that represent all the pieces of the XML document, and some of these Python objects represent attributes of the XML elements. But the (Python) objects that represent the (XML) attributes also have (Python) attributes, which are used to access various parts of the (XML) attribute that the object represents. I told you it was confusing. I am open to suggestions on how to distinguish these more clearly.

                                                                                                                                                   ### Example 9.24. Accessing element attributes

                                                                                                                                                       >>> xmldoc = minidom.parse('binary.xml')
                                                                                                                                                       >>> reflist = xmldoc.getElementsByTagName('ref')
                                                                                                                                                       >>> bitref = reflist[0]
                                                                                                                                                       >>> print bitref.toxml()
                                                                                                                                                       <ref id="bit">
                                                                                                                                                         <p>0</p>
                                                                                                                                                         <p>1</p>
                                                                                                                                                       </ref>
                                                                                                                                                       >>> bitref.attributes          ①
                                                                                                                                                       <xml.dom.minidom.NamedNodeMap instance at 0x81e0c9c>
                                                                                                                                                       >>> bitref.attributes.keys()   ② ③
                                                                                                                                                       [u'id']
                                                                                                                                                       >>> bitref.attributes.values() ④
                                                                                                                                                       [<xml.dom.minidom.Attr instance at 0x81d5044>]
                                                                                                                                                       >>> bitref.attributes["id"]    ⑤
                                                                                                                                                       <xml.dom.minidom.Attr instance at 0x81d5044>

                                                                                                                                                   1. Each `Element` object has an attribute called `attributes`, which is a `NamedNodeMap` object. This sounds scary, but it's not, because a `NamedNodeMap` is an object that acts like a dictionary, so you already know how to use it.
                                                                                                                                                   2. Treating the `NamedNodeMap` as a dictionary, you can get a list of the names of the attributes of this element by using `attributes.keys()`. This element has only one attribute, `'id'`.
                                                                                                                                                   3. Attribute names, like all other text in an XML document, are stored in unicode.
                                                                                                                                                   4. Again treating the `NamedNodeMap` as a dictionary, you can get a list of the values of the attributes by using `attributes.values()`. The values are themselves objects, of type `Attr`. You'll see how to get useful information out of this object in the next example.
                                                                                                                                                   5. Still treating the `NamedNodeMap` as a dictionary, you can access an individual attribute by name, using normal dictionary syntax. (Readers who have been paying extra-close attention will already know how the `NamedNodeMap` class accomplishes this neat trick: by defining a `__getitem__` special method. Other readers can take comfort in the fact that they don't need to understand how it works in order to use it effectively.)

                                                                                                                                                      ### Example 9.25. Accessing individual attributes

                                                                                                                                                          >>> a = bitref.attributes["id"]
                                                                                                                                                          >>> a
                                                                                                                                                          <xml.dom.minidom.Attr instance at 0x81d5044>
                                                                                                                                                          >>> a.name  ①
                                                                                                                                                          u'id'
                                                                                                                                                          >>> a.value ②
                                                                                                                                                          u'bit'

                                                                                                                                                      1. The `Attr` object completely represents a single XML attribute of a single XML element. The name of the attribute (the same name as you used to find this object in the `bitref.attributes` `NamedNodeMap` pseudo-dictionary) is stored in `a.name`.
                                                                                                                                                      2. The actual text value of this XML attribute is stored in `a.value`.

                                                                                                                                                         Like a dictionary, attributes of an XML element have no ordering. Attributes may happen to be listed in a certain order in the original XML document, and the `Attr` objects may happen to be listed in a certain order when the XML document is parsed into Python objects, but these orders are arbitrary and should carry no special meaning. You should always access individual attributes by name, like the keys of a dictionary.

                                                                                                                                                         ## 9.7. Segue

                                                                                                                                                         OK, that's it for the hard-core XML stuff. The next chapter will continue to use these same example programs, but focus on other aspects that make the program more flexible: using streams for input processing, using `getattr` for method dispatching, and using command-line flags to allow users to reconfigure the program without changing the code.

                                                                                                                                                         Before moving on to the next chapter, you should be comfortable doing all of these things:

                                                                                                                                                         - Parsing XML documents using `minidom`, searching through the parsed document, and accessing arbitrary element attributes and element children
                                                                                                                                                         - Organizing complex libraries into packages
                                                                                                                                                         - Converting unicode strings to different character encodings

                                                                                                                                                         [5] This, sadly, is still an oversimplification. Unicode now has been extended to handle ancient Chinese, Korean, and Japanese texts, which had so many different characters that the 2-byte unicode system could not represent them all. But Python doesn't currently support that out of the box, and I don't know if there is a project afoot to add it. You've reached the limits of my expertise, sorry.

                                                                                                                                                         ## Chapter 10. Scripts and Streams

                                                                                                                                                         ## 10.1. Abstracting input sources

                                                                                                                                                         One of Python's greatest strengths is its dynamic binding, and one powerful use of dynamic binding is the file-like object.

                                                                                                                                                         Many functions which require an input source could simply take a filename, go open the file for reading, read it, and close it when they're done. But they don't. Instead, they take a file-like object.

                                                                                                                                                         In the simplest case, a file-like object is any object with a `read` method with an optional size parameter, which returns a string. When called with no size parameter, it reads everything there is to read from the input source and returns all the data as a single string. When called with a size parameter, it reads that much from the input source and returns that much data; when called again, it picks up where it left off and returns the next chunk of data.

                                                                                                                                                         This is how reading from real files works; the difference is that you're not limiting yourself to real files. The input source could be anything: a file on disk, a web page, even a hard-coded string. As long as you pass a file-like object to the function, and the function simply calls the object's `read` method, the function can handle any kind of input source without specific code to handle each kind.

                                                                                                                                                         In case you were wondering how this relates to XML processing, `minidom.parse` is one such function which can take a file-like object.

                                                                                                                                                         ### Example 10.1. Parsing XML from a file

                                                                                                                                                             >>> from xml.dom import minidom
                                                                                                                                                             >>> fsock = open('binary.xml')    ①
                                                                                                                                                             >>> xmldoc = minidom.parse(fsock) ②
                                                                                                                                                             >>> fsock.close()                 ③
                                                                                                                                                             >>> print xmldoc.toxml()          ④
                                                                                                                                                             <?xml version="1.0" ?>
                                                                                                                                                             <grammar>
                                                                                                                                                             <ref id="bit">
                                                                                                                                                               <p>0</p>
                                                                                                                                                               <p>1</p>
                                                                                                                                                             </ref>
                                                                                                                                                             <ref id="byte">
                                                                                                                                                               <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
                                                                                                                                                             <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
                                                                                                                                                             </ref>
                                                                                                                                                             </grammar>

                                                                                                                                                         1. First, you open the file on disk. This gives you a file object.
                                                                                                                                                         2. You pass the file object to `minidom.parse`, which calls the `read` method of fsock and reads the XML document from the file on disk.
                                                                                                                                                         3. Be sure to call the `close` method of the file object after you're done with it. `minidom.parse` will not do this for you.
                                                                                                                                                         4. Calling the `toxml()` method on the returned XML document prints out the entire thing.

                                                                                                                                                            Well, that all seems like a colossal waste of time. After all, you've already seen that `minidom.parse` can simply take the filename and do all the opening and closing nonsense automatically. And it's true that if you know you're just going to be parsing a local file, you can pass the filename and `minidom.parse` is smart enough to Do The Right Thing™. But notice how similar -- and easy -- it is to parse an XML document straight from the Internet.

                                                                                                                                                            ### Example 10.2. Parsing XML from a URL

                                                                                                                                                                >>> import urllib
                                                                                                                                                                >>> usock = urllib.urlopen('http://slashdot.org/slashdot.rdf') ①
                                                                                                                                                                >>> xmldoc = minidom.parse(usock)            ②
                                                                                                                                                                >>> usock.close()          ③
                                                                                                                                                                >>> print xmldoc.toxml()   ④
                                                                                                                                                                <?xml version="1.0" ?>
                                                                                                                                                                <rdf:RDF xmlns="http://my.netscape.com/rdf/simple/0.9/"
                                                                                                                                                                 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">

                                                                                                                                                                <channel>
                                                                                                                                                                <title>Slashdot</title>
                                                                                                                                                                <link>http://slashdot.org/</link>
                                                                                                                                                                <description>News for nerds, stuff that matters</description>
                                                                                                                                                                </channel>

                                                                                                                                                                <image>
                                                                                                                                                                <title>Slashdot</title>
                                                                                                                                                                <url>http://images.slashdot.org/topics/topicslashdot.gif</url>
                                                                                                                                                                <link>http://slashdot.org/</link>
                                                                                                                                                                </image>

                                                                                                                                                                <item>
                                                                                                                                                                <title>To HDTV or Not to HDTV?</title>
                                                                                                                                                                <link>http://slashdot.org/article.pl?sid=01/12/28/0421241</link>
                                                                                                                                                                </item>

                                                                                                                                                                [...snip...]

                                                                                                                                                            1. As you saw in a previous chapter, `urlopen` takes a web page URL and returns a file-like object. Most importantly, this object has a `read` method which returns the HTML source of the web page.
                                                                                                                                                            2. Now you pass the file-like object to `minidom.parse`, which obediently calls the `read` method of the object and parses the XML data that the `read` method returns. The fact that this XML data is now coming straight from a web page is completely irrelevant. `minidom.parse` doesn't know about web pages, and it doesn't care about web pages; it just knows about file-like objects.
                                                                                                                                                            3. As soon as you're done with it, be sure to close the file-like object that `urlopen` gives you.
                                                                                                                                                            4. By the way, this URL is real, and it really is XML. It's an XML representation of the current headlines on Slashdot, a technical news and gossip site.

                                                                                                                                                               ### Example 10.3. Parsing XML from a string (the easy but inflexible way)

                                                                                                                                                                   >>> contents = "<grammar><ref id='bit'><p>0</p><p>1</p></ref></grammar>"
                                                                                                                                                                   >>> xmldoc = minidom.parseString(contents) ①
                                                                                                                                                                   >>> print xmldoc.toxml()
                                                                                                                                                                   <?xml version="1.0" ?>
                                                                                                                                                                   <grammar><ref id="bit"><p>0</p><p>1</p></ref></grammar>

                                                                                                                                                               1. `minidom` has a method, `parseString`, which takes an entire XML document as a string and parses it. You can use this instead of `minidom.parse` if you know you already have your entire XML document in a string.

                                                                                                                                                                  OK, so you can use the `minidom.parse` function for parsing both local files and remote URLs, but for parsing strings, you use... a different function. That means that if you want to be able to take input from a file, a URL, or a string, you'll need special logic to check whether it's a string, and call the `parseString` function instead. How unsatisfying.

                                                                                                                                                                  If there were a way to turn a string into a file-like object, then you could simply pass this object to `minidom.parse`. And in fact, there is a module specifically designed for doing just that: `StringIO`.

                                                                                                                                                                  ### Example 10.4. Introducing `StringIO`

                                                                                                                                                                      >>> contents = "<grammar><ref id='bit'><p>0</p><p>1</p></ref></grammar>"
                                                                                                                                                                      >>> import StringIO
                                                                                                                                                                      >>> ssock = StringIO.StringIO(contents)   ①
                                                                                                                                                                      >>> ssock.read()        ②
                                                                                                                                                                      "<grammar><ref id='bit'><p>0</p><p>1</p></ref></grammar>"
                                                                                                                                                                      >>> ssock.read()        ③
                                                                                                                                                                      ''
                                                                                                                                                                      >>> ssock.seek(0)       ④
                                                                                                                                                                      >>> ssock.read(15)      ⑤
                                                                                                                                                                      '<grammar><ref i'
                                                                                                                                                                      >>> ssock.read(15)
                                                                                                                                                                      "d='bit'><p>0</p"
                                                                                                                                                                      >>> ssock.read()
                                                                                                                                                                      '><p>1</p></ref></grammar>'
                                                                                                                                                                      >>> ssock.close()       ⑥

                                                                                                                                                                  1. The `StringIO` module contains a single class, also called `StringIO`, which allows you to turn a string into a file-like object. The `StringIO` class takes the string as a parameter when creating an instance.
                                                                                                                                                                  2. Now you have a file-like object, and you can do all sorts of file-like things with it. Like `read`, which returns the original string.
                                                                                                                                                                  3. Calling `read` again returns an empty string. This is how real file objects work too; once you read the entire file, you can't read any more without explicitly seeking to the beginning of the file. The `StringIO` object works the same way.
                                                                                                                                                                  4. You can explicitly seek to the beginning of the string, just like seeking through a file, by using the `seek` method of the `StringIO` object.
                                                                                                                                                                  5. You can also read the string in chunks, by passing a size parameter to the `read` method.
                                                                                                                                                                  6. At any time, `read` will return the rest of the string that you haven't read yet. All of this is exactly how file objects work; hence the term file-like object.

                                                                                                                                                                     ### Example 10.5. Parsing XML from a string (the file-like object way)

                                                                                                                                                                         >>> contents = "<grammar><ref id='bit'><p>0</p><p>1</p></ref></grammar>"
                                                                                                                                                                         >>> ssock = StringIO.StringIO(contents)
                                                                                                                                                                         >>> xmldoc = minidom.parse(ssock) ①
                                                                                                                                                                         >>> ssock.close()
                                                                                                                                                                         >>> print xmldoc.toxml()
                                                                                                                                                                         <?xml version="1.0" ?>
                                                                                                                                                                         <grammar><ref id="bit"><p>0</p><p>1</p></ref></grammar>

                                                                                                                                                                     1. Now you can pass the file-like object (really a `StringIO`) to `minidom.parse`, which will call the object's `read` method and happily parse away, never knowing that its input came from a hard-coded string.

                                                                                                                                                                        So now you know how to use a single function, `minidom.parse`, to parse an XML document stored on a web page, in a local file, or in a hard-coded string. For a web page, you use `urlopen` to get a file-like object; for a local file, you use `open`; and for a string, you use `StringIO`. Now let's take it one step further and generalize these differences as well.

                                                                                                                                                                        ### Example 10.6. `openAnything`

                                                                                                                                                                            def openAnything(source):①
                                                                                                                                                                                # try to open with urllib (if source is http, ftp, or file URL)
                                                                                                                                                                                import urllib
                                                                                                                                                                                try:
                                                                                                                                                                                    return urllib.urlopen(source)      ②
                                                                                                                                                                                except (IOError, OSError):
                                                                                                                                                                                    pass

                                                                                                                                                                                # try to open with native open function (if source is pathname)
                                                                                                                                                                                try:
                                                                                                                                                                                    return open(source)                ③
                                                                                                                                                                                except (IOError, OSError):
                                                                                                                                                                                    pass

                                                                                                                                                                                # treat source as string
                                                                                                                                                                                import StringIO
                                                                                                                                                                                return StringIO.StringIO(str(source))  ④

                                                                                                                                                                        1. The `openAnything` function takes a single parameter, source, and returns a file-like object. source is a string of some sort; it can either be a URL (like `'http://slashdot.org/slashdot.rdf'`), a full or partial pathname to a local file (like `'binary.xml'`), or a string that contains actual XML data to be parsed.
                                                                                                                                                                        2. First, you see if source is a URL. You do this through brute force: you try to open it as a URL and silently ignore errors caused by trying to open something which is not a URL. This is actually elegant in the sense that, if `urllib` ever supports new types of URLs in the future, you will also support them without recoding. If `urllib` is able to open source, then the `return` kicks you out of the function immediately and the following `try` statements never execute.
                                                                                                                                                                        3. On the other hand, if `urllib` yelled at you and told you that source wasn't a valid URL, you assume it's a path to a file on disk and try to open it. Again, you don't do anything fancy to check whether source is a valid filename or not (the rules for valid filenames vary wildly between different platforms anyway, so you'd probably get them wrong anyway). Instead, you just blindly open the file, and silently trap any errors.
                                                                                                                                                                        4. By this point, you need to assume that source is a string that has hard-coded data in it (since nothing else worked), so you use `StringIO` to create a file-like object out of it and return that. (In fact, since you're using the `str` function, source doesn't even need to be a string; it could be any object, and you'll use its string representation, as defined by its `__str__` special method.)

                                                                                                                                                                           Now you can use this `openAnything` function in conjunction with `minidom.parse` to make a function that takes a source that refers to an XML document somehow (either as a URL, or a local filename, or a hard-coded XML document in a string) and parses it.

                                                                                                                                                                           ### Example 10.7. Using `openAnything` in `kgp.py`

                                                                                                                                                                               class KantGenerator:
                                                                                                                                                                                   def _load(self, source):
                                                                                                                                                                                       sock = toolbox.openAnything(source)
                                                                                                                                                                                       xmldoc = minidom.parse(sock).documentElement
                                                                                                                                                                                       sock.close()
                                                                                                                                                                                       return xmldoc

                                                                                                                                                                           ## 10.2. Standard input, output, and error

                                                                                                                                                                           UNIX users are already familiar with the concept of standard input, standard output, and standard error. This section is for the rest of you.

                                                                                                                                                                           Standard output and standard error (commonly abbreviated `stdout` and `stderr`) are pipes that are built into every UNIX system. When you `print` something, it goes to the `stdout` pipe; when your program crashes and prints out debugging information (like a traceback in Python), it goes to the `stderr` pipe. Both of these pipes are ordinarily just connected to the terminal window where you are working, so when a program prints, you see the output, and when a program crashes, you see the debugging information. (If you're working on a system with a window-based Python IDE, `stdout` and `stderr` default to your “Interactive Window”.)

                                                                                                                                                                           ### Example 10.8. Introducing `stdout` and `stderr`

                                                                                                                                                                               >>> for i in range(3):
                                                                                                                                                                               ...    print 'Dive in'             ①
                                                                                                                                                                               Dive in
                                                                                                                                                                               Dive in
                                                                                                                                                                               Dive in
                                                                                                                                                                               >>> import sys
                                                                                                                                                                               >>> for i in range(3):
                                                                                                                                                                               ...    sys.stdout.write('Dive in') ②
                                                                                                                                                                               Dive inDive inDive in
                                                                                                                                                                               >>> for i in range(3):
                                                                                                                                                                               ...    sys.stderr.write('Dive in') ③
                                                                                                                                                                               Dive inDive inDive in

                                                                                                                                                                           1. As you saw in Example 6.9, “Simple Counters”, you can use Python's built-in `range function to build simple counter loops that repeat something a set number of times.
                                                                                                                                                                           2. `stdout is a file-like object; calling its `write function will print out whatever string you give it. In fact, this is what the `print function really does; it adds a carriage return to the end of the string you're printing, and calls `sys.stdout.write.
                                                                                                                                                                           3. In the simplest case, `stdout and `stderr send their output to the same place: the Python IDE (if you're in one), or the terminal (if you're running Python from the command line). Like `stdout, `stderr does not add carriage returns for you; if you want them, add them yourself.

                                                                                                                                                                              `stdout and `stderr are both file-like objects, like the ones you discussed in Section 10.1, “Abstracting input sources”, but they are both write-only. They have no `read method, only `write. Still, they are file-like objects, and you can assign any other file- or file-like object to them to redirect their output.

                                                                                                                                                                              ### Example 10.9. Redirecting output

                                                                                                                                                                              `[you@localhost kgp]$ python stdout.py Dive in `[you@localhost kgp]$ cat out.log This message will be logged instead of displayed

                                                                                                                                                                              (On Windows, you can use `type instead of `cat to display the contents of a file.)

                                                                                                                                                                              If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                              `#stdout.py import sys print 'Dive in' ① saveout = sys.stdout ② fsock = open('out.log', 'w') ③ sys.stdout = fsock ④ print 'This message will be logged instead of displayed' ⑤ sys.stdout = saveout ⑥ fsock.close() ⑦
                                                                                                                                                                           4. This will print to the IDE “Interactive Window” (or the terminal, if running the script from the command line).
                                                                                                                                                                           5. Always save `stdout before redirecting it, so you can set it back to normal later.
                                                                                                                                                                           6. Open a file for writing. If the file doesn't exist, it will be created. If the file does exist, it will be overwritten.
                                                                                                                                                                           7. Redirect all further output to the new file you just opened.
                                                                                                                                                                           8. This will be “printed” to the log file only; it will not be visible in the IDE window or on the screen.
                                                                                                                                                                           9. Set `stdout back to the way it was before you mucked with it.
                                                                                                                                                                           10. Close the log file.

                                                                                                                                                                               Redirecting `stderr works exactly the same way, using `sys.stderr instead of `sys.stdout.

                                                                                                                                                                               ### Example 10.10. Redirecting error information

                                                                                                                                                                               `[you@localhost kgp]$ python stderr.py `[you@localhost kgp]$ cat error.log `Traceback (most recent line last): File "stderr.py", line 5, in ? raise Exception, 'this error will be logged' Exception: this error will be logged

                                                                                                                                                                               If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                               `#stderr.py import sys fsock = open('error.log', 'w') ① sys.stderr = fsock ② raise Exception, 'this error will be logged' ③ ④
                                                                                                                                                                           11. Open the log file where you want to store debugging information.
                                                                                                                                                                           12. Redirect standard error by assigning the file object of the newly-opened log file to `stderr.
                                                                                                                                                                           13. Raise an exception. Note from the screen output that this does not print anything on screen. All the normal traceback information has been written to `error.log.
                                                                                                                                                                           14. Also note that you're not explicitly closing your log file, nor are you setting `stderr back to its original value. This is fine, since once the program crashes (because of the exception), Python will clean up and close the file for us, and it doesn't make any difference that `stderr is never restored, since, as I mentioned, the program crashes and Python ends. Restoring the original is more important for `stdout, if you expect to go do other stuff within the same script afterwards.

                                                                                                                                                                               Since it is so common to write error messages to standard error, there is a shorthand syntax that can be used instead of going through the hassle of redirecting it outright.

                                                                                                                                                                               ### Example 10.11. Printing to `stderr

                                                                                                                                                                               `>>> `print 'entering function' entering function `>>> `import sys `>>> `print >> sys.stderr, 'entering function' ① entering function
                                                                                                                                                                           15. This shorthand syntax of the `print statement can be used to write to any open file, or file-like object. In this case, you can redirect a single `print statement to `stderr without affecting subsequent `print statements.

                                                                                                                                                                               Standard input, on the other hand, is a read-only file object, and it represents the data flowing into the program from some previous program. This will likely not make much sense to classic Mac OS users, or even Windows users unless you were ever fluent on the MS-DOS command line. The way it works is that you can construct a chain of commands in a single line, so that one program's output becomes the input for the next program in the chain. The first program simply outputs to standard output (without doing any special redirecting itself, just doing normal `print statements or whatever), and the next program reads from standard input, and the operating system takes care of connecting one program's output to the next program's input.

                                                                                                                                                                               ### Example 10.12. Chaining commands

                                                                                                                                                                               `[you@localhost kgp]$ python kgp.py -g binary.xml ① 01100111 `[you@localhost kgp]$ cat binary.xml ② `<?xml version="1.0"?> <!DOCTYPE grammar PUBLIC "-//diveintopython3.org//DTD Kant Generator Pro v1.0//EN" "kgp.dtd"> <grammar> <ref id="bit"> <p>0</p> <p>1</p> </ref> <ref id="byte"> <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\ <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p> </ref> </grammar> `[you@localhost kgp]$ cat binary.xml | python kgp.py -g - ③ ④ 10110001
                                                                                                                                                                           16. As you saw in Section 9.1, “Diving in”, this will print a string of eight random bits, `0 or `1.
                                                                                                                                                                           17. This simply prints out the entire contents of `binary.xml. (Windows users should use `type instead of `cat.)
                                                                                                                                                                           18. This prints the contents of `binary.xml, but the “`|” character, called the “pipe” character, means that the contents will not be printed to the screen. Instead, they will become the standard input of the next command, which in this case calls your Python script.
                                                                                                                                                                           19. Instead of specifying a module (like `binary.xml), you specify “`-”, which causes your script to load the grammar from standard input instead of from a specific file on disk. (More on how this happens in the next example.) So the effect is the same as the first syntax, where you specified the grammar filename directly, but think of the expansion possibilities here. Instead of simply doing `cat binary.xml, you could run a script that dynamically generates the grammar, then you can pipe it into your script. It could come from anywhere: a database, or some grammar-generating meta-script, or whatever. The point is that you don't need to change your `kgp.py script at all to incorporate any of this functionality. All you need to do is be able to take grammar files from standard input, and you can separate all the other logic into another program.

                                                                                                                                                                               So how does the script “know” to read from standard input when the grammar file is “`-”? It's not magic; it's just code.

                                                                                                                                                                               ### Example 10.13. Reading from standard input in `kgp.py

                                                                                                                                                                               `def openAnything(source): if source == "-": ① import sys return sys.stdin # try to open with urllib (if source is http, ftp, or file URL) import urllib try: [... snip ...]
                                                                                                                                                                           20. This is the `openAnything function from `toolbox.py, which you previously examined in Section 10.1, “Abstracting input sources”. All you've done is add three lines of code at the beginning of the function to check if the source is “`-”; if so, you return `sys.stdin. Really, that's it! Remember, `stdin is a file-like object with a `read method, so the rest of the code (in `kgp.py, where you call `openAnything) doesn't change a bit.

                                                                                                                                                                               ## 10.3. Caching node lookups

                                                                                                                                                                               `kgp.py employs several tricks which may or may not be useful to you in your XML processing. The first one takes advantage of the consistent structure of the input documents to build a cache of nodes.

                                                                                                                                                                               A grammar file defines a series of `ref elements. Each `ref contains one or more `p elements, which can contain a lot of different things, including `xrefs. Whenever you encounter an `xref, you look for a corresponding `ref element with the same `id attribute, and choose one of the `ref element's children and parse it. (You'll see how this random choice is made in the next section.)

                                                                                                                                                                               This is how you build up the grammar: define `ref elements for the smallest pieces, then define `ref elements which "include" the first `ref elements by using `xref, and so forth. Then you parse the "largest" reference and follow each `xref, and eventually output real text. The text you output depends on the (random) decisions you make each time you fill in an `xref, so the output is different each time.

                                                                                                                                                                               This is all very flexible, but there is one downside: performance. When you find an `xref and need to find the corresponding `ref element, you have a problem. The `xref has an `id attribute, and you want to find the `ref element that has that same `id attribute, but there is no easy way to do that. The slow way to do it would be to get the entire list of `ref elements each time, then manually loop through and look at each `id attribute. The fast way is to do that once and build a cache, in the form of a dictionary.

                                                                                                                                                                               ### Example 10.14. `loadGrammar

                                                                                                                                                                               `def loadGrammar(self, grammar): self.grammar = self._load(grammar) self.refs = {} ① for ref in self.grammar.getElementsByTagName("ref"): ② self.refs[ref.attributes["id"].value] = ref ③ ④
                                                                                                                                                                           21. Start by creating an empty dictionary, self.refs.
                                                                                                                                                                           22. As you saw in Section 9.5, “Searching for elements”, `getElementsByTagName returns a list of all the elements of a particular name. You easily can get a list of all the `ref elements, then simply loop through that list.
                                                                                                                                                                           23. As you saw in Section 9.6, “Accessing element attributes”, you can access individual attributes of an element by name, using standard dictionary syntax. So the keys of the self.refs dictionary will be the values of the `id attribute of each `ref element.
                                                                                                                                                                           24. The values of the self.refs dictionary will be the `ref elements themselves. As you saw in Section 9.3, “Parsing XML”, each element, each node, each comment, each piece of text in a parsed XML document is an object.

                                                                                                                                                                               Once you build this cache, whenever you come across an `xref and need to find the `ref element with the same `id attribute, you can simply look it up in self.refs.

                                                                                                                                                                               ### Example 10.15. Using the `ref element cache

                                                                                                                                                                               `def do_xref(self, node): id = node.attributes["id"].value self.parse(self.randomChildElement(self.refs[id]))

                                                                                                                                                                               You'll explore the `randomChildElement function in the next section.

                                                                                                                                                                               ## 10.4. Finding direct children of a node

                                                                                                                                                                               Another useful techique when parsing XML documents is finding all the direct child elements of a particular element. For instance, in the grammar files, a `ref element can have several `p elements, each of which can contain many things, including other `p elements. You want to find just the `p elements that are children of the `ref, not `p elements that are children of other `p elements.

                                                                                                                                                                               You might think you could simply use `getElementsByTagName for this, but you can't. `getElementsByTagName searches recursively and returns a single list for all the elements it finds. Since `p elements can contain other `p elements, you can't use `getElementsByTagName, because it would return nested `p elements that you don't want. To find only direct child elements, you'll need to do it yourself.

                                                                                                                                                                               ### Example 10.16. Finding direct child elements

                                                                                                                                                                               `def randomChildElement(self, node): choices = [e for e in node.childNodes if e.nodeType == e.ELEMENT_NODE] ① ② ③ chosen = random.choice(choices) ④ return chosen
                                                                                                                                                                           25. As you saw in Example 9.9, “Getting child nodes”, the `childNodes attribute returns a list of all the child nodes of an element.
                                                                                                                                                                           26. However, as you saw in Example 9.11, “Child nodes can be text”, the list returned by `childNodes contains all different types of nodes, including text nodes. That's not what you're looking for here. You only want the children that are elements.
                                                                                                                                                                           27. Each node has a nodeType attribute, which can be `ELEMENT_NODE, `TEXT_NODE, `COMMENT_NODE, or any number of other values. The complete list of possible values is in the `__init__.py file in the `xml.dom package. (See Section 9.2, “Packages” for more on packages.) But you're just interested in nodes that are elements, so you can filter the list to only include those nodes whose nodeType is `ELEMENT_NODE.
                                                                                                                                                                           28. Once you have a list of actual elements, choosing a random one is easy. Python comes with a module called `random which includes several useful functions. The `random.choice function takes a list of any number of items and returns a random item. For example, if the `ref elements contains several `p elements, then choices would be a list of `p elements, and chosen would end up being assigned exactly one of them, selected at random.

                                                                                                                                                                               ## 10.5. Creating separate handlers by node type

                                                                                                                                                                               The third useful XML processing tip involves separating your code into logical functions, based on node types and element names. Parsed XML documents are made up of various types of nodes, each represented by a Python object. The root level of the document itself is represented by a `Document object. The `Document then contains one or more `Element objects (for actual XML tags), each of which may contain other `Element objects, `Text objects (for bits of text), or `Comment objects (for embedded comments). Python makes it easy to write a dispatcher to separate the logic for each node type.

                                                                                                                                                                               ### Example 10.17. Class names of parsed XML objects

                                                                                                                                                                               `>>> `from xml.dom import minidom `>>> `xmldoc = minidom.parse('kant.xml') ① `>>> `xmldoc <xml.dom.minidom.Document instance at 0x01359DE8> `>>> `xmldoc.__class__ ② <class xml.dom.minidom.Document at 0x01105D40> `>>> `xmldoc.__class__.__name__ ③ 'Document'
                                                                                                                                                                           29. Assume for a moment that `kant.xml is in the current directory.
                                                                                                                                                                           30. As you saw in Section 9.2, “Packages”, the object returned by parsing an XML document is a `Document object, as defined in the `minidom.py in the `xml.dom package. As you saw in Section 5.4, “Instantiating Classes”, `__class__ is built-in attribute of every Python object.
                                                                                                                                                                           31. Furthermore, `__name__ is a built-in attribute of every Python class, and it is a string. This string is not mysterious; it's the same as the class name you type when you define a class yourself. (See Section 5.3, “Defining Classes”.)

                                                                                                                                                                               Fine, so now you can get the class name of any particular XML node (since each XML node is represented as a Python object). How can you use this to your advantage to separate the logic of parsing each node type? The answer is `getattr, which you first saw in Section 4.4, “Getting Object References With getattr”.

                                                                                                                                                                               ### Example 10.18. `parse, a generic XML node dispatcher

                                                                                                                                                                               `def parse(self, node): parseMethod = getattr(self, "parse_%s" % node.__class__.__name__) ① ② parseMethod(node) ③
                                                                                                                                                                           32. First off, notice that you're constructing a larger string based on the class name of the node you were passed (in the node argument). So if you're passed a `Document node, you're constructing the string `'parse_Document', and so forth.
                                                                                                                                                                           33. Now you can treat that string as a function name, and get a reference to the function itself using `getattr
                                                                                                                                                                           34. Finally, you can call that function and pass the node itself as an argument. The next example shows the definitions of each of these functions.

                                                                                                                                                                               ### Example 10.19. Functions called by the `parse dispatcher

                                                                                                                                                                               `def parse_Document(self, node): ① self.parse(node.documentElement) def parse_Text(self, node): ② text = node.data if self.capitalizeNextWord: self.pieces.append(text[0].upper()) self.pieces.append(text[1:]) self.capitalizeNextWord = 0 else: self.pieces.append(text) def parse_Comment(self, node): ③ pass def parse_Element(self, node): ④ handlerMethod = getattr(self, "do_%s" % node.tagName) handlerMethod(node)
                                                                                                                                                                           35. `parse_Document is only ever called once, since there is only one `Document node in an XML document, and only one `Document object in the parsed XML representation. It simply turns around and parses the root element of the grammar file.
                                                                                                                                                                           36. `parse_Text is called on nodes that represent bits of text. The function itself does some special processing to handle automatic capitalization of the first word of a sentence, but otherwise simply appends the represented text to a list.
                                                                                                                                                                           37. `parse_Comment is just a `pass, since you don't care about embedded comments in the grammar files. Note, however, that you still need to define the function and explicitly make it do nothing. If the function did not exist, the generic `parse function would fail as soon as it stumbled on a comment, because it would try to find the non-existent `parse_Comment function. Defining a separate function for every node type, even ones you don't use, allows the generic `parse function to stay simple and dumb.
                                                                                                                                                                           38. The `parse_Element method is actually itself a dispatcher, based on the name of the element's tag. The basic idea is the same: take what distinguishes elements from each other (their tag names) and dispatch to a separate function for each of them. You construct a string like `'do_xref' (for an `<xref> tag), find a function of that name, and call it. And so forth for each of the other tag names that might be found in the course of parsing a grammar file (`<p> tags, `<choice> tags).

                                                                                                                                                                               In this example, the dispatch functions `parse and `parse_Element simply find other methods in the same class. If your processing is very complex (or you have many different tag names), you could break up your code into separate modules, and use dynamic importing to import each module and call whatever functions you needed. Dynamic importing will be discussed in Chapter 16, Functional Programming.

                                                                                                                                                                               ## 10.6. Handling command-line arguments

                                                                                                                                                                               Python fully supports creating programs that can be run on the command line, complete with command-line arguments and either short- or long-style flags to specify various options. None of this is XML-specific, but this script makes good use of command-line processing, so it seemed like a good time to mention it.

                                                                                                                                                                               It's difficult to talk about command-line processing without understanding how command-line arguments are exposed to your Python program, so let's write a simple program to see them.

                                                                                                                                                                               ### Example 10.20. Introducing sys.argv

                                                                                                                                                                               If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                               `#argecho.py import sys for arg in sys.argv: ① print arg
                                                                                                                                                                           39. Each command-line argument passed to the program will be in sys.argv, which is just a list. Here you are printing each argument on a separate line.

                                                                                                                                                                               ### Example 10.21. The contents of sys.argv

                                                                                                                                                                               `[you@localhost py]$ python argecho.py ① argecho.py `[you@localhost py]$ python argecho.py abc def ② `argecho.py abc def `[you@localhost py]$ python argecho.py --help ③ `argecho.py --help `[you@localhost py]$ python argecho.py -m kant.xml ④ `argecho.py -m kant.xml
                                                                                                                                                                           40. The first thing to know about sys.argv is that it contains the name of the script you're calling. You will actually use this knowledge to your advantage later, in Chapter 16, Functional Programming. Don't worry about it for now.
                                                                                                                                                                           41. Command-line arguments are separated by spaces, and each shows up as a separate element in the sys.argv list.
                                                                                                                                                                           42. Command-line flags, like `--help, also show up as their own element in the sys.argv list.
                                                                                                                                                                           43. To make things even more interesting, some command-line flags themselves take arguments. For instance, here you have a flag (`-m) which takes an argument (`kant.xml). Both the flag itself and the flag's argument are simply sequential elements in the sys.argv list. No attempt is made to associate one with the other; all you get is a list.

                                                                                                                                                                               So as you can see, you certainly have all the information passed on the command line, but then again, it doesn't look like it's going to be all that easy to actually use it. For simple programs that only take a single argument and have no flags, you can simply use `sys.argv[1] to access the argument. There's no shame in this; I do it all the time. For more complex programs, you need the `getopt module.

                                                                                                                                                                               ### Example 10.22. Introducing `getopt

                                                                                                                                                                               `def main(argv): grammar = "kant.xml" ① try: opts, args = getopt.getopt(argv, "hg:d", ["help", "grammar="]) ② except getopt.GetoptError: ③ usage() ④ sys.exit(2) ... if __name__ == "__main__": main(sys.argv[1:])
                                                                                                                                                                           44. First off, look at the bottom of the example and notice that you're calling the `main function with `sys.argv[1:]. Remember, `sys.argv[0] is the name of the script that you're running; you don't care about that for command-line processing, so you chop it off and pass the rest of the list.
                                                                                                                                                                           45. This is where all the interesting processing happens. The `getopt function of the `getopt module takes three parameters: the argument list (which you got from `sys.argv[1:]), a string containing all the possible single-character command-line flags that this program accepts, and a list of longer command-line flags that are equivalent to the single-character versions. This is quite confusing at first glance, and is explained in more detail below.
                                                                                                                                                                           46. If anything goes wrong trying to parse these command-line flags, `getopt will raise an exception, which you catch. You told `getopt all the flags you understand, so this probably means that the end user passed some command-line flag that you don't understand.
                                                                                                                                                                           47. As is standard practice in the UNIX world, when the script is passed flags it doesn't understand, you print out a summary of proper usage and exit gracefully. Note that I haven't shown the `usage function here. You would still need to code that somewhere and have it print out the appropriate summary; it's not automatic.

                                                                                                                                                                               So what are all those parameters you pass to the `getopt function? Well, the first one is simply the raw list of command-line flags and arguments (not including the first element, the script name, which you already chopped off before calling the `main function). The second is the list of short command-line flags that the script accepts.

                                                                                                                                                                               ### `"hg:d"
                                                                                                                                                                               `-h
                                                                                                                                                                               print usage summary
                                                                                                                                                                               `-g ...
                                                                                                                                                                               use specified grammar file or URL
                                                                                                                                                                               `-d
                                                                                                                                                                               show debugging information while parsing

                                                                                                                                                                               The first and third flags are simply standalone flags; you specify them or you don't, and they do things (print help) or change state (turn on debugging). However, the second flag (`-g) must be followed by an argument, which is the name of the grammar file to read from. In fact it can be a filename or a web address, and you don't know which yet (you'll figure it out later), but you know it has to be something. So you tell `getopt this by putting a colon after the `g in that second parameter to the `getopt function.

                                                                                                                                                                               To further complicate things, the script accepts either short flags (like `-h) or long flags (like `--help), and you want them to do the same thing. This is what the third parameter to `getopt is for, to specify a list of the long flags that correspond to the short flags you specified in the second parameter.

                                                                                                                                                                               ### `["help", "grammar="]
                                                                                                                                                                               `--help
                                                                                                                                                                               print usage summary
                                                                                                                                                                               `--grammar ...
                                                                                                                                                                               use specified grammar file or URL

                                                                                                                                                                               Three things of note here:
                                                                                                                                                                           48. All long flags are preceded by two dashes on the command line, but you don't include those dashes when calling `getopt. They are understood.
                                                                                                                                                                           49. The `--grammar flag must always be followed by an additional argument, just like the `-g flag. This is notated by an equals sign, `"grammar=".
                                                                                                                                                                           50. The list of long flags is shorter than the list of short flags, because the `-d flag does not have a corresponding long version. This is fine; only `-d will turn on debugging. But the order of short and long flags needs to be the same, so you'll need to specify all the short flags that do have corresponding long flags first, then all the rest of the short flags.

                                                                                                                                                                           Confused yet? Let's look at the actual code and see if it makes sense in context.

                                                                                                                                                                           ### Example 10.23. Handling command-line arguments in `kgp.py

                                                                                                                                                                               def main(argv):        ①
                                                                                                                                                                                   grammar = "kant.xml"
                                                                                                                                                                                   try:
                                                                                                                                                                                       opts, args = getopt.getopt(argv, "hg:d", ["help", "grammar="])
                                                                                                                                                                                   except getopt.GetoptError:
                                                                                                                                                                                       usage()
                                                                                                                                                                                       sys.exit(2)
                                                                                                                                                                                   for opt, arg in opts:                ②
                                                                                                                                                                                       if opt in ("-h", "--help"):      ③
                                                                                                                                                                                           usage()
                                                                                                                                                                                           sys.exit()
                                                                                                                                                                                       elif opt == '-d':                ④
                                                                                                                                                                                           global _debug
                                                                                                                                                                                           _debug = 1
                                                                                                                                                                                       elif opt in ("-g", "--grammar"): ⑤
                                                                                                                                                                                           grammar = arg

                                                                                                                                                                                   source = "".join(args)               ⑥

                                                                                                                                                                                   k = KantGenerator(grammar, source)
                                                                                                                                                                                   print k.output()

                                                                                                                                                                           The grammar variable will keep track of the grammar file you're using. You initialize it here in case it's not specified on the command line (using either the `-g or the `--grammar flag).
                                                                                                                                                                           The opts variable that you get back from `getopt contains a list of tuples: flag and argument. If the flag doesn't take an argument, then arg will simply be `None. This makes it easier to loop through the flags.
                                                                                                                                                                           `getopt validates that the command-line flags are acceptable, but it doesn't do any sort of conversion between short and long flags. If you specify the `-h flag, opt will contain `"-h"; if you specify the `--help flag, opt will contain `"--help". So you need to check for both.
                                                                                                                                                                           Remember, the `-d flag didn't have a corresponding long flag, so you only need to check for the short form. If you find it, you set a global variable that you'll refer to later to print out debugging information. (I used this during the development of the script. What, you thought all these examples worked on the first try?)
                                                                                                                                                                           If you find a grammar file, either with a `-g flag or a `--grammar flag, you save the argument that followed it (stored in arg) into the grammar variable, overwriting the default that you initialized at the top of the `main function.
                                                                                                                                                                           That's it. You've looped through and dealt with all the command-line flags. That means that anything left must be command-line arguments. These come back from the `getopt function in the args variable. In this case, you're treating them as source material for the parser. If there are no command-line arguments specified, args will be an empty list, and source will end up as the empty string.

                                                                                                                                                                           ## 10.7. Putting it all together

                                                                                                                                                                           You've covered a lot of ground. Let's step back and see how all the pieces fit together.

                                                                                                                                                                           To start with, this is a script that takes its arguments on the command line, using the `getopt module.

                                                                                                                                                                           `def main(argv): ... try: opts, args = getopt.getopt(argv, "hg:d", ["help", "grammar="]) except getopt.GetoptError: ... for opt, arg in opts: ...

                                                                                                                                                                           You create a new instance of the `KantGenerator class, and pass it the grammar file and source that may or may not have been specified on the command line.

                                                                                                                                                                           `k = KantGenerator(grammar, source)

                                                                                                                                                                           The `KantGenerator instance automatically loads the grammar, which is an XML file. You use your custom `openAnything function to open the file (which could be stored in a local file or a remote web server), then use the built-in `minidom parsing functions to parse the XML into a tree of Python objects.

                                                                                                                                                                           `def _load(self, source): sock = toolbox.openAnything(source) xmldoc = minidom.parse(sock).documentElement sock.close()

                                                                                                                                                                           Oh, and along the way, you take advantage of your knowledge of the structure of the XML document to set up a little cache of references, which are just elements in the XML document.

                                                                                                                                                                           `def loadGrammar(self, grammar): for ref in self.grammar.getElementsByTagName("ref"): self.refs[ref.attributes["id"].value] = ref

                                                                                                                                                                           If you specified some source material on the command line, you use that; otherwise you rip through the grammar looking for the "top-level" reference (that isn't referenced by anything else) and use that as a starting point.

                                                                                                                                                                           `def getDefaultSource(self): xrefs = {} for xref in self.grammar.getElementsByTagName("xref"): xrefs[xref.attributes["id"].value] = 1 xrefs = xrefs.keys() standaloneXrefs = [e for e in self.refs.keys() if e not in xrefs] return '<xref id="%s"/>' % random.choice(standaloneXrefs)

                                                                                                                                                                           Now you rip through the source material. The source material is also XML, and you parse it one node at a time. To keep the code separated and more maintainable, you use separate handlers for each node type.

                                                                                                                                                                           `def parse_Element(self, node): handlerMethod = getattr(self, "do_%s" % node.tagName) handlerMethod(node)

                                                                                                                                                                           You bounce through the grammar, parsing all the children of each `p element,

                                                                                                                                                                           `def do_p(self, node): ... if doit: for child in node.childNodes: self.parse(child)

                                                                                                                                                                           replacing `choice elements with a random child,

                                                                                                                                                                           `def do_choice(self, node): self.parse(self.randomChildElement(node))

                                                                                                                                                                           and replacing `xref elements with a random child of the corresponding `ref element, which you previously cached.

                                                                                                                                                                           `def do_xref(self, node): id = node.attributes["id"].value self.parse(self.randomChildElement(self.refs[id]))

                                                                                                                                                                           Eventually, you parse your way down to plain text,

                                                                                                                                                                           `def parse_Text(self, node): text = node.data ... self.pieces.append(text)

                                                                                                                                                                           which you print out.

                                                                                                                                                                           `def main(argv): ... k = KantGenerator(grammar, source) print k.output()

                                                                                                                                                                           ## 10.8. Summary

                                                                                                                                                                           Python comes with powerful libraries for parsing and manipulating XML documents. The `minidom takes an XML file and parses it into Python objects, providing for random access to arbitrary elements. Furthermore, this chapter shows how Python can be used to create a "real" standalone command-line script, complete with command-line flags, command-line arguments, error handling, even the ability to take input from the piped result of a previous program.

                                                                                                                                                                           Before moving on to the next chapter, you should be comfortable doing all of these things:
                                                                                                                                                                           Chaining programs with standard input and output
                                                                                                                                                                           Defining dynamic dispatchers with `getattr.
                                                                                                                                                                           Using command-line flags and validating them with `getopt

                                                                                                                                                                           ## Chapter 11. HTTP Web Services

                                                                                                                                                                           ## 11.1. Diving in

                                                                                                                                                                           You've learned about HTML processing and XML processing, and along the way you saw how to download a web page and how to parse XML from a URL, but let's dive into the more general topic of HTTP web services.

                                                                                                                                                                           Simply stated, HTTP web services are programmatic ways of sending and receiving data from remote servers using the operations of HTTP directly. If you want to get data from the server, use a straight HTTP GET; if you want to send new data to the server, use HTTP POST. (Some more advanced HTTP web service APIs also define ways of modifying existing data and deleting data, using HTTP PUT and HTTP DELETE.) In other words, the “verbs” built into the HTTP protocol (GET, POST, PUT, and DELETE) map directly to application-level operations for receiving, sending, modifying, and deleting data.

                                                                                                                                                                           The main advantage of this approach is simplicity, and its simplicity has proven popular with a lot of different sites. Data -- usually XML data -- can be built and stored statically, or generated dynamically by a server-side script, and all major languages include an HTTP library for downloading it. Debugging is also easier, because you can load up the web service in any web browser and see the raw data. Modern browsers will even nicely format and pretty-print XML data for you, to allow you to quickly navigate through it.

                                                                                                                                                                           Examples of pure XML-over-HTTP web services:
                                                                                                                                                                           Amazon API allows you to retrieve product information from the Amazon.com online store.
                                                                                                                                                                           National Weather Service (United States) and Hong Kong Observatory (Hong Kong) offer weather alerts as a web service.
                                                                                                                                                                           Atom API for managing web-based content.
                                                                                                                                                                           Syndicated feeds from weblogs and news sites bring you up-to-the-minute news from a variety of sites.

                                                                                                                                                                           In later chapters, you'll explore APIs which use HTTP as a transport for sending and receiving data, but don't map application semantics to the underlying HTTP semantics. (They tunnel everything over HTTP POST.) But this chapter will concentrate on using HTTP GET to get data from a remote server, and you'll explore several HTTP features you can use to get the maximum benefit out of pure HTTP web services.

                                                                                                                                                                           Here is a more advanced version of the `openanything module that you saw in the previous chapter:

                                                                                                                                                                           ### Example 11.1. `openanything.py

                                                                                                                                                                           If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                           `import urllib2, urlparse, gzip from StringIO import StringIO USER_AGENT = 'OpenAnything/1.0 +http://diveintopython3.org/http_web_services/' class SmartRedirectHandler(urllib2.HTTPRedirectHandler): def http_error_301(self, req, fp, code, msg, headers): result = urllib2.HTTPRedirectHandler.http_error_301( self, req, fp, code, msg, headers) result.status = code return result def http_error_302(self, req, fp, code, msg, headers): result = urllib2.HTTPRedirectHandler.http_error_302( self, req, fp, code, msg, headers) result.status = code return result class DefaultErrorHandler(urllib2.HTTPDefaultErrorHandler): def http_error_default(self, req, fp, code, msg, headers): result = urllib2.HTTPError( req.get_full_url(), code, msg, headers, fp) result.status = code return result def openAnything(source, etag=None, lastmodified=None, agent=USER_AGENT): '''URL, filename, or string --> stream This function lets you define parsers that take any input source (URL, pathname to local or network file, or actual data as a string) and deal with it in a uniform manner. Returned object is guaranteed to have all the basic stdio read methods (read, readline, readlines). Just .close() the object when you're done with it. If the etag argument is supplied, it will be used as the value of an If-None-Match request header. If the lastmodified argument is supplied, it must be a formatted date/time string in GMT (as returned in the Last-Modified header of a previous request). The formatted date/time will be used as the value of an If-Modified-Since request header. If the agent argument is supplied, it will be used as the value of a User-Agent request header. ''' if hasattr(source, 'read'): return source if source == '-': return sys.stdin if urlparse.urlparse(source)[0] == 'http': # open URL with urllib2 request = urllib2.Request(source) request.add_header('User-Agent', agent) if etag: request.add_header('If-None-Match', etag) if lastmodified: request.add_header('If-Modified-Since', lastmodified) request.add_header('Accept-encoding', 'gzip') opener = urllib2.build_opener(SmartRedirectHandler(), DefaultErrorHandler()) return opener.open(request) # try to open with native open function (if source is a filename) try: return open(source) except (IOError, OSError): pass # treat source as string return StringIO(str(source)) def fetch(source, etag=None, last_modified=None, agent=USER_AGENT): '''Fetch data and metadata from a URL, file, stream, or string''' result = {} f = openAnything(source, etag, last_modified, agent) result['data'] = f.read() if hasattr(f, 'headers'): # save ETag, if the server sent one result['etag'] = f.headers.get('ETag') # save Last-Modified header, if the server sent one result['lastmodified'] = f.headers.get('Last-Modified') if f.headers.get('content-encoding', '') == 'gzip': # data came back gzip-compressed, decompress it result['data'] = gzip.GzipFile(fileobj=StringIO(result['data']])).read() if hasattr(f, 'url'): result['url'] = f.url result['status'] = 200 if hasattr(f, 'status'): result['status'] = f.status f.close() return result

                                                                                                                                                                           ### Further reading
                                                                                                                                                                           Paul Prescod believes that pure HTTP web services are the future of the Internet.

                                                                                                                                                                           ## 11.2. How not to fetch data over HTTP

                                                                                                                                                                           Let's say you want to download a resource over HTTP, such as a syndicated Atom feed. But you don't just want to download it once; you want to download it over and over again, every hour, to get the latest news from the site that's offering the news feed. Let's do it the quick-and-dirty way first, and then see how you can do better.

                                                                                                                                                                           ### Example 11.2. Downloading a feed the quick-and-dirty way

                                                                                                                                                                           `>>> `import urllib `>>> `data = urllib.urlopen('http://diveintomark.org/xml/atom.xml').read() ① `>>> `print data `<?xml version="1.0" encoding="iso-8859-1"?> <feed version="0.3" xmlns="http://purl.org/atom/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xml:lang="en"> <title mode="escaped">dive into mark</title> <link rel="alternate" type="text/html" href="http://diveintomark.org/"/> <-- rest of feed omitted for brevity -->
                                                                                                                                                                           Downloading anything over HTTP is incredibly easy in Python; in fact, it's a one-liner. The `urllib module has a handy `urlopen function that takes the address of the page you want, and returns a file-like object that you can just `read() from to get the full contents of the page. It just can't get much easier.

                                                                                                                                                                           So what's wrong with this? Well, for a quick one-off during testing or development, there's nothing wrong with it. I do it all the time. I wanted the contents of the feed, and I got the contents of the feed. The same technique works for any web page. But once you start thinking in terms of a web service that you want to access on a regular basis -- and remember, you said you were planning on retrieving this syndicated feed once an hour -- then you're being inefficient, and you're being rude.

                                                                                                                                                                           Let's talk about some of the basic features of HTTP.

                                                                                                                                                                           ## 11.3. Features of HTTP

                                                                                                                                                                           There are five important features of HTTP which you should support.

                                                                                                                                                                           ### 11.3.1. `User-Agent

                                                                                                                                                                           The `User-Agent is simply a way for a client to tell a server who it is when it requests a web page, a syndicated feed, or any sort of web service over HTTP. When the client requests a resource, it should always announce who it is, as specifically as possible. This allows the server-side administrator to get in touch with the client-side developer if anything is going fantastically wrong.

                                                                                                                                                                           By default, Python sends a generic `User-Agent: `Python-urllib/1.15. In the next section, you'll see how to change this to something more specific.

                                                                                                                                                                           ### 11.3.2. Redirects

                                                                                                                                                                           Sometimes resources move around. Web sites get reorganized, pages move to new addresses. Even web services can reorganize. A syndicated feed at `http://example.com/index.xml might be moved to `http://example.com/xml/atom.xml. Or an entire domain might move, as an organization expands and reorganizes; for instance, `http://www.example.com/index.xml might be redirected to `http://server-farm-1.example.com/index.xml.

                                                                                                                                                                           Every time you request any kind of resource from an HTTP server, the server includes a status code in its response. Status code `200 means “everything's normal, here's the page you asked for”. Status code `404 means “page not found”. (You've probably seen 404 errors while browsing the web.)

                                                                                                                                                                           HTTP has two different ways of signifying that a resource has moved. Status code `302 is a temporary redirect; it means “oops, that got moved over here temporarily” (and then gives the temporary address in a `Location: header). Status code `301 is a permanent redirect; it means “oops, that got moved permanently” (and then gives the new address in a `Location: header). If you get a `302 status code and a new address, the HTTP specification says you should use the new address to get what you asked for, but the next time you want to access the same resource, you should retry the old address. But if you get a `301 status code and a new address, you're supposed to use the new address from then on.

                                                                                                                                                                           `urllib.urlopen will automatically “follow” redirects when it receives the appropriate status code from the HTTP server, but unfortunately, it doesn't tell you when it does so. You'll end up getting data you asked for, but you'll never know that the underlying library “helpfully” followed a redirect for you. So you'll continue pounding away at the old address, and each time you'll get redirected to the new address. That's two round trips instead of one: not very efficient! Later in this chapter, you'll see how to work around this so you can deal with permanent redirects properly and efficiently.

                                                                                                                                                                           ### 11.3.3. `Last-Modified/`If-Modified-Since

                                                                                                                                                                           Some data changes all the time. The home page of CNN.com is constantly updating every few minutes. On the other hand, the home page of Google.com only changes once every few weeks (when they put up a special holiday logo, or advertise a new service). Web services are no different; usually the server knows when the data you requested last changed, and HTTP provides a way for the server to include this last-modified date along with the data you requested.

                                                                                                                                                                           If you ask for the same data a second time (or third, or fourth), you can tell the server the last-modified date that you got last time: you send an `If-Modified-Since header with your request, with the date you got back from the server last time. If the data hasn't changed since then, the server sends back a special HTTP status code `304, which means “this data hasn't changed since the last time you asked for it”. Why is this an improvement? Because when the server sends a `304, it doesn't re-send the data. All you get is the status code. So you don't need to download the same data over and over again if it hasn't changed; the server assumes you have the data cached locally.

                                                                                                                                                                           All modern web browsers support last-modified date checking. If you've ever visited a page, re-visited the same page a day later and found that it hadn't changed, and wondered why it loaded so quickly the second time -- this could be why. Your web browser cached the contents of the page locally the first time, and when you visited the second time, your browser automatically sent the last-modified date it got from the server the first time. The server simply says `304: Not Modified, so your browser knows to load the page from its cache. Web services can be this smart too.

                                                                                                                                                                           Python's URL library has no built-in support for last-modified date checking, but since you can add arbitrary headers to each request and read arbitrary headers in each response, you can add support for it yourself.

                                                                                                                                                                           ### 11.3.4. `ETag/`If-None-Match

                                                                                                                                                                           ETags are an alternate way to accomplish the same thing as the last-modified date checking: don't re-download data that hasn't changed. The way it works is, the server sends some sort of hash of the data (in an `ETag header) along with the data you requested. Exactly how this hash is determined is entirely up to the server. The second time you request the same data, you include the ETag hash in an `If-None-Match: header, and if the data hasn't changed, the server will send you back a `304 status code. As with the last-modified date checking, the server just sends the `304; it doesn't send you the same data a second time. By including the ETag hash in your second request, you're telling the server that there's no need to re-send the same data if it still matches this hash, since you still have the data from the last time.

                                                                                                                                                                           Python's URL library has no built-in support for ETags, but you'll see how to add it later in this chapter.

                                                                                                                                                                           ### 11.3.5. Compression

                                                                                                                                                                           The last important HTTP feature is gzip compression. When you talk about HTTP web services, you're almost always talking about moving XML back and forth over the wire. XML is text, and quite verbose text at that, and text generally compresses well. When you request a resource over HTTP, you can ask the server that, if it has any new data to send you, to please send it in compressed format. You include the `Accept-encoding: gzip header in your request, and if the server supports compression, it will send you back gzip-compressed data and mark it with a `Content-encoding: gzip header.

                                                                                                                                                                           Python's URL library has no built-in support for gzip compression per se, but you can add arbitrary headers to the request. And Python comes with a separate `gzip module, which has functions you can use to decompress the data yourself.

                                                                                                                                                                           Note that our little one-line script to download a syndicated feed did not support any of these HTTP features. Let's see how you can improve it.

                                                                                                                                                                           ## 11.4. Debugging HTTP web services

                                                                                                                                                                           First, let's turn on the debugging features of Python's HTTP library and see what's being sent over the wire. This will be useful throughout the chapter, as you add more and more features.

                                                                                                                                                                           ### Example 11.3. Debugging HTTP

                                                                                                                                                                           `>>> `import httplib `>>> `httplib.HTTPConnection.debuglevel = 1 ① `>>> `import urllib `>>> `feeddata = urllib.urlopen('http://diveintomark.org/xml/atom.xml').read() connect: (diveintomark.org, 80) ② send: ' GET /xml/atom.xml HTTP/1.0 ③ Host: diveintomark.org ④ User-agent: Python-urllib/1.15 ⑤ ' reply: 'HTTP/1.1 200 OK\r\n' ⑥ header: Date: Wed, 14 Apr 2004 22:27:30 GMT header: Server: Apache/2.0.49 (Debian GNU/Linux) header: Content-Type: application/atom+xml header: Last-Modified: Wed, 14 Apr 2004 22:14:38 GMT ⑦ header: ETag: "e8284-68e0-4de30f80" ⑧ header: Accept-Ranges: bytes header: Content-Length: 26848 header: Connection: close
                                                                                                                                                                           `urllib relies on another standard Python library, `httplib. Normally you don't need to `import httplib directly (`urllib does that automatically), but you will here so you can set the debugging flag on the `HTTPConnection class that `urllib uses internally to connect to the HTTP server. This is an incredibly useful technique. Some other Python libraries have similar debug flags, but there's no particular standard for naming them or turning them on; you need to read the documentation of each library to see if such a feature is available.
                                                                                                                                                                           Now that the debugging flag is set, information on the the HTTP request and response is printed out in real time. The first thing it tells you is that you're connecting to the server `diveintomark.org on port 80, which is the standard port for HTTP.
                                                                                                                                                                           When you request the Atom feed, `urllib sends three lines to the server. The first line specifies the HTTP verb you're using, and the path of the resource (minus the domain name). All the requests in this chapter will use `GET, but in the next chapter on SOAP, you'll see that it uses `POST for everything. The basic syntax is the same, regardless of the verb.
                                                                                                                                                                           The second line is the `Host header, which specifies the domain name of the service you're accessing. This is important, because a single HTTP server can host multiple separate domains. My server currently hosts 12 domains; other servers can host hundreds or even thousands.
                                                                                                                                                                           The third line is the `User-Agent header. What you see here is the generic `User-Agent that the `urllib library adds by default. In the next section, you'll see how to customize this to be more specific.
                                                                                                                                                                           The server replies with a status code and a bunch of headers (and possibly some data, which got stored in the feeddata variable). The status code here is `200, meaning “everything's normal, here's the data you requested”. The server also tells you the date it responded to your request, some information about the server itself, and the content type of the data it's giving you. Depending on your application, this might be useful, or not. It's certainly reassuring that you thought you were asking for an Atom feed, and lo and behold, you're getting an Atom feed (`application/atom+xml, which is the registered content type for Atom feeds).
                                                                                                                                                                           The server tells you when this Atom feed was last modified (in this case, about 13 minutes ago). You can send this date back to the server the next time you request the same feed, and the server can do last-modified checking.
                                                                                                                                                                           The server also tells you that this Atom feed has an ETag hash of `"e8284-68e0-4de30f80". The hash doesn't mean anything by itself; there's nothing you can do with it, except send it back to the server the next time you request this same feed. Then the server can use it to tell you if the data has changed or not.

                                                                                                                                                                           ## 11.5. Setting the `User-Agent

                                                                                                                                                                           The first step to improving your HTTP web services client is to identify yourself properly with a `User-Agent. To do that, you need to move beyond the basic `urllib and dive into `urllib2.

                                                                                                                                                                           ### Example 11.4. Introducing `urllib2

                                                                                                                                                                           `>>> `import httplib `>>> `httplib.HTTPConnection.debuglevel = 1 ① `>>> `import urllib2 `>>> `request = urllib2.Request('http://diveintomark.org/xml/atom.xml') ② `>>> `opener = urllib2.build_opener() ③ `>>> `feeddata = opener.open(request).read() ④ connect: (diveintomark.org, 80) send: ' GET /xml/atom.xml HTTP/1.0 Host: diveintomark.org User-agent: Python-urllib/2.1 ' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Wed, 14 Apr 2004 23:23:12 GMT header: Server: Apache/2.0.49 (Debian GNU/Linux) header: Content-Type: application/atom+xml header: Last-Modified: Wed, 14 Apr 2004 22:14:38 GMT header: ETag: "e8284-68e0-4de30f80" header: Accept-Ranges: bytes header: Content-Length: 26848 header: Connection: close
                                                                                                                                                                           If you still have your Python IDE open from the previous section's example, you can skip this, but this turns on HTTP debugging so you can see what you're actually sending over the wire, and what gets sent back.
                                                                                                                                                                           Fetching an HTTP resource with `urllib2 is a three-step process, for good reasons that will become clear shortly. The first step is to create a `Request object, which takes the URL of the resource you'll eventually get around to retrieving. Note that this step doesn't actually retrieve anything yet.
                                                                                                                                                                           The second step is to build a URL opener. This can take any number of handlers, which control how responses are handled. But you can also build an opener without any custom handlers, which is what you're doing here. You'll see how to define and use custom handlers later in this chapter when you explore redirects.
                                                                                                                                                                           The final step is to tell the opener to open the URL, using the `Request object you created. As you can see from all the debugging information that gets printed, this step actually retrieves the resource and stores the returned data in feeddata.

                                                                                                                                                                           ### Example 11.5. Adding headers with the `Request

                                                                                                                                                                           `>>> `request ① <urllib2.Request instance at 0x00250AA8> `>>> `request.get_full_url() http://diveintomark.org/xml/atom.xml `>>> `request.add_header('User-Agent', `... `'OpenAnything/1.0 +http://diveintopython3.org/') ② `>>> `feeddata = opener.open(request).read() ③ connect: (diveintomark.org, 80) send: ' GET /xml/atom.xml HTTP/1.0 Host: diveintomark.org User-agent: OpenAnything/1.0 +http://diveintopython3.org/ ④ ' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Wed, 14 Apr 2004 23:45:17 GMT header: Server: Apache/2.0.49 (Debian GNU/Linux) header: Content-Type: application/atom+xml header: Last-Modified: Wed, 14 Apr 2004 22:14:38 GMT header: ETag: "e8284-68e0-4de30f80" header: Accept-Ranges: bytes header: Content-Length: 26848 header: Connection: close
                                                                                                                                                                           You're continuing from the previous example; you've already created a `Request object with the URL you want to access.
                                                                                                                                                                           Using the `add_header method on the `Request object, you can add arbitrary HTTP headers to the request. The first argument is the header, the second is the value you're providing for that header. Convention dictates that a `User-Agent should be in this specific format: an application name, followed by a slash, followed by a version number. The rest is free-form, and you'll see a lot of variations in the wild, but somewhere it should include a URL of your application. The `User-Agent is usually logged by the server along with other details of your request, and including a URL of your application allows server administrators looking through their access logs to contact you if something is wrong.
                                                                                                                                                                           The opener object you created before can be reused too, and it will retrieve the same feed again, but with your custom `User-Agent header.
                                                                                                                                                                           And here's you sending your custom `User-Agent, in place of the generic one that Python sends by default. If you look closely, you'll notice that you defined a `User-Agent header, but you actually sent a `User-agent header. See the difference? `urllib2 changed the case so that only the first letter was capitalized. It doesn't really matter; HTTP specifies that header field names are completely case-insensitive.

                                                                                                                                                                           ## 11.6. Handling `Last-Modified and `ETag

                                                                                                                                                                           Now that you know how to add custom HTTP headers to your web service requests, let's look at adding support for `Last-Modified and `ETag headers.

                                                                                                                                                                           These examples show the output with debugging turned off. If you still have it turned on from the previous section, you can turn it off by setting `httplib.HTTPConnection.debuglevel = 0. Or you can just leave debugging on, if that helps you.

                                                                                                                                                                           ### Example 11.6. Testing `Last-Modified

                                                                                                                                                                           `>>> `import urllib2 `>>> `request = urllib2.Request('http://diveintomark.org/xml/atom.xml') `>>> `opener = urllib2.build_opener() `>>> `firstdatastream = opener.open(request) `>>> `firstdatastream.headers.dict ① `{'date': 'Thu, 15 Apr 2004 20:42:41 GMT', 'server': 'Apache/2.0.49 (Debian GNU/Linux)', 'content-type': 'application/atom+xml', 'last-modified': 'Thu, 15 Apr 2004 19:45:21 GMT', 'etag': '"e842a-3e53-55d97640"', 'content-length': '15955', 'accept-ranges': 'bytes', 'connection': 'close'} `>>> `request.add_header('If-Modified-Since', `... firstdatastream.headers.get('Last-Modified')) ② `>>> `seconddatastream = opener.open(request) ③ `Traceback (most recent call last): File "<stdin>", line 1, in ? File "c:\python23\lib\urllib2.py", line 326, in open '_open', req) File "c:\python23\lib\urllib2.py", line 306, in _call_chain result = func(*args) File "c:\python23\lib\urllib2.py", line 901, in http_open return self.do_open(httplib.HTTP, req) File "c:\python23\lib\urllib2.py", line 895, in do_open return self.parent.error('http', req, fp, code, msg, hdrs) File "c:\python23\lib\urllib2.py", line 352, in error return self._call_chain(*args) File "c:\python23\lib\urllib2.py", line 306, in _call_chain result = func(*args) File "c:\python23\lib\urllib2.py", line 412, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 304: Not Modified
                                                                                                                                                                           Remember all those HTTP headers you saw printed out when you turned on debugging? This is how you can get access to them programmatically: firstdatastream.headers is an object that acts like a dictionary and allows you to get any of the individual headers returned from the HTTP server.
                                                                                                                                                                           On the second request, you add the `If-Modified-Since header with the last-modified date from the first request. If the data hasn't changed, the server should return a `304 status code.
                                                                                                                                                                           Sure enough, the data hasn't changed. You can see from the traceback that `urllib2 throws a special exception, `HTTPError, in response to the `304 status code. This is a little unusual, and not entirely helpful. After all, it's not an error; you specifically asked the server not to send you any data if it hadn't changed, and the data didn't change, so the server told you it wasn't sending you any data. That's not an error; that's exactly what you were hoping for.

                                                                                                                                                                           `urllib2 also raises an `HTTPError exception for conditions that you would think of as errors, such as `404 (page not found). In fact, it will raise `HTTPError for any status code other than `200 (OK), `301 (permanent redirect), or `302 (temporary redirect). It would be more helpful for your purposes to capture the status code and simply return it, without throwing an exception. To do that, you'll need to define a custom URL handler.

                                                                                                                                                                           ### Example 11.7. Defining URL handlers

                                                                                                                                                                           This custom URL handler is part of `openanything.py.

                                                                                                                                                                           `class DefaultErrorHandler(urllib2.HTTPDefaultErrorHandler): ① def http_error_default(self, req, fp, code, msg, headers): ② result = urllib2.HTTPError( req.get_full_url(), code, msg, headers, fp) result.status = code ③ return result
                                                                                                                                                                           `urllib2 is designed around URL handlers. Each handler is just a class that can define any number of methods. When something happens -- like an HTTP error, or even a `304 code -- `urllib2 introspects into the list of defined handlers for a method that can handle it. You used a similar introspection in Chapter 9, XML Processing to define handlers for different node types, but `urllib2 is more flexible, and introspects over as many handlers as are defined for the current request.
                                                                                                                                                                           `urllib2 searches through the defined handlers and calls the `http_error_default method when it encounters a `304 status code from the server. By defining a custom error handler, you can prevent `urllib2 from raising an exception. Instead, you create the `HTTPError object, but return it instead of raising it.
                                                                                                                                                                           This is the key part: before returning, you save the status code returned by the HTTP server. This will allow you easy access to it from the calling program.

                                                                                                                                                                           ### Example 11.8. Using custom URL handlers

                                                                                                                                                                           `>>> `request.headers ① {'If-modified-since': 'Thu, 15 Apr 2004 19:45:21 GMT'} `>>> `import openanything `>>> `opener = urllib2.build_opener( `... openanything.DefaultErrorHandler()) ② `>>> `seconddatastream = opener.open(request) `>>> `seconddatastream.status ③ 304 `>>> `seconddatastream.read() ④ ''
                                                                                                                                                                           You're continuing the previous example, so the `Request object is already set up, and you've already added the `If-Modified-Since header.
                                                                                                                                                                           This is the key: now that you've defined your custom URL handler, you need to tell `urllib2 to use it. Remember how I said that `urllib2 broke up the process of accessing an HTTP resource into three steps, and for good reason? This is why building the URL opener is its own step, because you can build it with your own custom URL handlers that override `urllib2's default behavior.
                                                                                                                                                                           Now you can quietly open the resource, and what you get back is an object that, along with the usual headers (use seconddatastream.headers.dict to acess them), also contains the HTTP status code. In this case, as you expected, the status is `304, meaning this data hasn't changed since the last time you asked for it.
                                                                                                                                                                           Note that when the server sends back a `304 status code, it doesn't re-send the data. That's the whole point: to save bandwidth by not re-downloading data that hasn't changed. So if you actually want that data, you'll need to cache it locally the first time you get it.

                                                                                                                                                                           Handling `ETag works much the same way, but instead of checking for `Last-Modified and sending `If-Modified-Since, you check for `ETag and send `If-None-Match. Let's start with a fresh IDE session.

                                                                                                                                                                           ### Example 11.9. Supporting `ETag/`If-None-Match

                                                                                                                                                                           `>>> `import urllib2, openanything `>>> `request = urllib2.Request('http://diveintomark.org/xml/atom.xml') `>>> `opener = urllib2.build_opener( `... openanything.DefaultErrorHandler()) `>>> `firstdatastream = opener.open(request) `>>> `firstdatastream.headers.get('ETag') ① '"e842a-3e53-55d97640"' `>>> `firstdata = firstdatastream.read() `>>> `print firstdata ② `<?xml version="1.0" encoding="iso-8859-1"?> <feed version="0.3" xmlns="http://purl.org/atom/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xml:lang="en"> <title mode="escaped">dive into mark</title> <link rel="alternate" type="text/html" href="http://diveintomark.org/"/> <-- rest of feed omitted for brevity --> `>>> `request.add_header('If-None-Match', `... firstdatastream.headers.get('ETag')) ③ `>>> `seconddatastream = opener.open(request) `>>> `seconddatastream.status ④ 304 `>>> `seconddatastream.read() ⑤ ''
                                                                                                                                                                           Using the firstdatastream.headers pseudo-dictionary, you can get the `ETag returned from the server. (What happens if the server didn't send back an `ETag? Then this line would return `None.)
                                                                                                                                                                           OK, you got the data.
                                                                                                                                                                           Now set up the second call by setting the `If-None-Match header to the `ETag you got from the first call.
                                                                                                                                                                           The second call succeeds quietly (without throwing an exception), and once again you see that the server has sent back a `304 status code. Based on the `ETag you sent the second time, it knows that the data hasn't changed.
                                                                                                                                                                           Regardless of whether the `304 is triggered by `Last-Modified date checking or `ETag hash matching, you'll never get the data along with the `304. That's the whole point.

                                                                                                                                                         In these examples, the HTTP server has supported both `Last-Modified` and `ETag` headers, but not all servers do. As a web services client, you should be prepared to support both, but you must code defensively in case a server only supports one or the other, or neither.

                                                                                                                                                         ## 11.7. Handling redirects

                                                                                                                                                         You can support permanent and temporary redirects using a different kind of custom URL handler.

                                                                                                                                                         First, let's see why a redirect handler is necessary in the first place.

                                                                                                                                                         ### Example 11.10. Accessing web services without a redirect handler

                                                                                                                                                             >>> import urllib2, httplib
                                                                                                                                                             >>> httplib.HTTPConnection.debuglevel = 1           ①
                                                                                                                                                             >>> request = urllib2.Request(
                                                                                                                                                             ...    'http://diveintomark.org/redir/example301.xml') ②
                                                                                                                                                             >>> opener = urllib2.build_opener()
                                                                                                                                                             >>> f = opener.open(request)
                                                                                                                                                             connect: (diveintomark.org, 80)
                                                                                                                                                             send: '
                                                                                                                                                             GET /redir/example301.xml HTTP/1.0
                                                                                                                                                             Host: diveintomark.org
                                                                                                                                                             User-agent: Python-urllib/2.1
                                                                                                                                                             '
                                                                                                                                                             reply: 'HTTP/1.1 301 Moved Permanently\r\n'             ③
                                                                                                                                                             header: Date: Thu, 15 Apr 2004 22:06:25 GMT
                                                                                                                                                             header: Server: Apache/2.0.49 (Debian GNU/Linux)
                                                                                                                                                             header: Location: http://diveintomark.org/xml/atom.xml  ④
                                                                                                                                                             header: Content-Length: 338
                                                                                                                                                             header: Connection: close
                                                                                                                                                             header: Content-Type: text/html; charset=iso-8859-1
                                                                                                                                                             connect: (diveintomark.org, 80)
                                                                                                                                                             send: '
                                                                                                                                                             GET /xml/atom.xml HTTP/1.0            ⑤
                                                                                                                                                             Host: diveintomark.org
                                                                                                                                                             User-agent: Python-urllib/2.1
                                                                                                                                                             '
                                                                                                                                                             reply: 'HTTP/1.1 200 OK\r\n'
                                                                                                                                                             header: Date: Thu, 15 Apr 2004 22:06:25 GMT
                                                                                                                                                             header: Server: Apache/2.0.49 (Debian GNU/Linux)
                                                                                                                                                             header: Last-Modified: Thu, 15 Apr 2004 19:45:21 GMT
                                                                                                                                                             header: ETag: "e842a-3e53-55d97640"
                                                                                                                                                             header: Accept-Ranges: bytes
                                                                                                                                                             header: Content-Length: 15955
                                                                                                                                                             header: Connection: close
                                                                                                                                                             header: Content-Type: application/atom+xml
                                                                                                                                                             >>> f.url           ⑥
                                                                                                                                                             'http://diveintomark.org/xml/atom.xml'
                                                                                                                                                             >>> f.headers.dict
                                                                                                                                                             {'content-length': '15955',
                                                                                                                                                             'accept-ranges': 'bytes',
                                                                                                                                                             'server': 'Apache/2.0.49 (Debian GNU/Linux)',
                                                                                                                                                             'last-modified': 'Thu, 15 Apr 2004 19:45:21 GMT',
                                                                                                                                                             'connection': 'close',
                                                                                                                                                             'etag': '"e842a-3e53-55d97640"',
                                                                                                                                                             'date': 'Thu, 15 Apr 2004 22:06:25 GMT',
                                                                                                                                                             'content-type': 'application/atom+xml'}
                                                                                                                                                             >>> f.status
                                                                                                                                                             Traceback (most recent call last):
                                                                                                                                                               File "<stdin>", line 1, in ?
                                                                                                                                                             AttributeError: addinfourl instance has no attribute 'status'

                                                                                                                                                         1. You'll be better able to see what's happening if you turn on debugging.
                                                                                                                                                         2. This is a URL which I have set up to permanently redirect to my Atom feed at `http://diveintomark.org/xml/atom.xml`.
                                                                                                                                                         3. Sure enough, when you try to download the data at that address, the server sends back a `301` status code, telling you that the resource has moved permanently.
                                                                                                                                                         4. The server also sends back a `Location:` header that gives the new address of this data.
                                                                                                                                                         5. `urllib2` notices the redirect status code and automatically tries to retrieve the data at the new location specified in the `Location:` header.
                                                                                                                                                         6. The object you get back from the opener contains the new permanent address and all the headers returned from the second request (retrieved from the new permanent address). But the status code is missing, so you have no way of knowing programmatically whether this redirect was temporary or permanent. And that matters very much: if it was a temporary redirect, then you should continue to ask for the data at the old location. But if it was a permanent redirect (as this was), you should ask for the data at the new location from now on.

                                                                                                                                                            This is suboptimal, but easy to fix. `urllib2` doesn't behave exactly as you want it to when it encounters a `301` or `302`, so let's override its behavior. How? With a custom URL handler, just like you did to handle `304` codes.

                                                                                                                                                            ### Example 11.11. Defining the redirect handler

                                                                                                                                                            This class is defined in `openanything.py`.

                                                                                                                                                                class SmartRedirectHandler(urllib2.HTTPRedirectHandler):     ①
                                                                                                                                                                    def http_error_301(self, req, fp, code, msg, headers):
                                                                                                                                                                        result = urllib2.HTTPRedirectHandler.http_error_301( ②
                                                                                                                                                                            self, req, fp, code, msg, headers)
                                                                                                                                                                        result.status = code               ③
                                                                                                                                                                        return result

                                                                                                                                                                    def http_error_302(self, req, fp, code, msg, headers):   ④
                                                                                                                                                                        result = urllib2.HTTPRedirectHandler.http_error_302(
                                                                                                                                                                            self, req, fp, code, msg, headers)
                                                                                                                                                                        result.status = code
                                                                                                                                                                        return result

                                                                                                                                                            1. Redirect behavior is defined in `urllib2` in a class called `HTTPRedirectHandler`. You don't want to completely override the behavior, you just want to extend it a little, so you'll subclass `HTTPRedirectHandler` so you can call the ancestor class to do all the hard work.
                                                                                                                                                            2. When it encounters a `301` status code from the server, `urllib2` will search through its handlers and call the `http_error_301` method. The first thing ours does is just call the `http_error_301` method in the ancestor, which handles the grunt work of looking for the `Location:` header and following the redirect to the new address.
                                                                                                                                                            3. Here's the key: before you return, you store the status code (`301`), so that the calling program can access it later.
                                                                                                                                                            4. Temporary redirects (status code `302`) work the same way: override the `http_error_302` method, call the ancestor, and save the status code before returning.

                                                                                                                                                               So what has this bought us? You can now build a URL opener with the custom redirect handler, and it will still automatically follow redirects, but now it will also expose the redirect status code.

                                                                                                                                                               ### Example 11.12. Using the redirect handler to detect permanent redirects

                                                                                                                                                                   >>> request = urllib2.Request('http://diveintomark.org/redir/example301.xml')
                                                                                                                                                                   >>> import openanything, httplib
                                                                                                                                                                   >>> httplib.HTTPConnection.debuglevel = 1
                                                                                                                                                                   >>> opener = urllib2.build_opener(
                                                                                                                                                                   ...    openanything.SmartRedirectHandler())           ①
                                                                                                                                                                   >>> f = opener.open(request)
                                                                                                                                                                   connect: (diveintomark.org, 80)
                                                                                                                                                                   send: 'GET /redir/example301.xml HTTP/1.0
                                                                                                                                                                   Host: diveintomark.org
                                                                                                                                                                   User-agent: Python-urllib/2.1
                                                                                                                                                                   '
                                                                                                                                                                   reply: 'HTTP/1.1 301 Moved Permanently\r\n'            ②
                                                                                                                                                                   header: Date: Thu, 15 Apr 2004 22:13:21 GMT
                                                                                                                                                                   header: Server: Apache/2.0.49 (Debian GNU/Linux)
                                                                                                                                                                   header: Location: http://diveintomark.org/xml/atom.xml
                                                                                                                                                                   header: Content-Length: 338
                                                                                                                                                                   header: Connection: close
                                                                                                                                                                   header: Content-Type: text/html; charset=iso-8859-1
                                                                                                                                                                   connect: (diveintomark.org, 80)
                                                                                                                                                                   send: '
                                                                                                                                                                   GET /xml/atom.xml HTTP/1.0
                                                                                                                                                                   Host: diveintomark.org
                                                                                                                                                                   User-agent: Python-urllib/2.1
                                                                                                                                                                   '
                                                                                                                                                                   reply: 'HTTP/1.1 200 OK\r\n'
                                                                                                                                                                   header: Date: Thu, 15 Apr 2004 22:13:21 GMT
                                                                                                                                                                   header: Server: Apache/2.0.49 (Debian GNU/Linux)
                                                                                                                                                                   header: Last-Modified: Thu, 15 Apr 2004 19:45:21 GMT
                                                                                                                                                                   header: ETag: "e842a-3e53-55d97640"
                                                                                                                                                                   header: Accept-Ranges: bytes
                                                                                                                                                                   header: Content-Length: 15955
                                                                                                                                                                   header: Connection: close
                                                                                                                                                                   header: Content-Type: application/atom+xml

                                                                                                                                                                   >>> f.status       ③
                                                                                                                                                                   301
                                                                                                                                                                   >>> f.url
                                                                                                                                                                   'http://diveintomark.org/xml/atom.xml'

                                                                                                                                                               1. First, build a URL opener with the redirect handler you just defined.
                                                                                                                                                               2. You sent off a request, and you got a `301` status code in response. At this point, the `http_error_301` method gets called. You call the ancestor method, which follows the redirect and sends a request at the new location (`http://diveintomark.org/xml/atom.xml`).
                                                                                                                                                               3. This is the payoff: now, not only do you have access to the new URL, but you have access to the redirect status code, so you can tell that this was a permanent redirect. The next time you request this data, you should request it from the new location (`http://diveintomark.org/xml/atom.xml`, as specified in f.url). If you had stored the location in a configuration file or a database, you need to update that so you don't keep pounding the server with requests at the old address. It's time to update your address book.

                                                                                                                                                                  The same redirect handler can also tell you that you shouldn't update your address book.

                                                                                                                                                                  ### Example 11.13. Using the redirect handler to detect temporary redirects

                                                                                                                                                                      >>> request = urllib2.Request(
                                                                                                                                                                      ...    'http://diveintomark.org/redir/example302.xml')   ①
                                                                                                                                                                      >>> f = opener.open(request)
                                                                                                                                                                      connect: (diveintomark.org, 80)
                                                                                                                                                                      send: '
                                                                                                                                                                      GET /redir/example302.xml HTTP/1.0
                                                                                                                                                                      Host: diveintomark.org
                                                                                                                                                                      User-agent: Python-urllib/2.1
                                                                                                                                                                      '
                                                                                                                                                                      reply: 'HTTP/1.1 302 Found\r\n'         ②
                                                                                                                                                                      header: Date: Thu, 15 Apr 2004 22:18:21 GMT
                                                                                                                                                                      header: Server: Apache/2.0.49 (Debian GNU/Linux)
                                                                                                                                                                      header: Location: http://diveintomark.org/xml/atom.xml
                                                                                                                                                                      header: Content-Length: 314
                                                                                                                                                                      header: Connection: close
                                                                                                                                                                      header: Content-Type: text/html; charset=iso-8859-1
                                                                                                                                                                      connect: (diveintomark.org, 80)
                                                                                                                                                                      send: '
                                                                                                                                                                      GET /xml/atom.xml HTTP/1.0              ③
                                                                                                                                                                      Host: diveintomark.org
                                                                                                                                                                      User-agent: Python-urllib/2.1
                                                                                                                                                                      '
                                                                                                                                                                      reply: 'HTTP/1.1 200 OK\r\n'
                                                                                                                                                                      header: Date: Thu, 15 Apr 2004 22:18:21 GMT
                                                                                                                                                                      header: Server: Apache/2.0.49 (Debian GNU/Linux)
                                                                                                                                                                      header: Last-Modified: Thu, 15 Apr 2004 19:45:21 GMT
                                                                                                                                                                      header: ETag: "e842a-3e53-55d97640"
                                                                                                                                                                      header: Accept-Ranges: bytes
                                                                                                                                                                      header: Content-Length: 15955
                                                                                                                                                                      header: Connection: close
                                                                                                                                                                      header: Content-Type: application/atom+xml
                                                                                                                                                                      >>> f.status          ④
                                                                                                                                                                      302
                                                                                                                                                                      >>> f.url
                                                                                                                                                                      http://diveintomark.org/xml/atom.xml

                                                                                                                                                                  1. This is a sample URL I've set up that is configured to tell clients to temporarily redirect to `http://diveintomark.org/xml/atom.xml`.
                                                                                                                                                                  2. The server sends back a `302` status code, indicating a temporary redirect. The temporary new location of the data is given in the `Location:` header.
                                                                                                                                                                  3. `urllib2` calls your `http_error_302` method, which calls the ancestor method of the same name in `urllib2.HTTPRedirectHandler`, which follows the redirect to the new location. Then your `http_error_302` method stores the status code (`302`) so the calling application can get it later.
                                                                                                                                                                  4. And here you are, having successfully followed the redirect to `http://diveintomark.org/xml/atom.xml`. f.status tells you that this was a temporary redirect, which means that you should continue to request data from the original address (`http://diveintomark.org/redir/example302.xml`). Maybe it will redirect next time too, but maybe not. Maybe it will redirect to a different address. It's not for you to say. The server said this redirect was only temporary, so you should respect that. And now you're exposing enough information that the calling application can respect that.

                                                                                                                                                                     ## 11.8. Handling compressed data

                                                                                                                                                                     The last important HTTP feature you want to support is compression. Many web services have the ability to send data compressed, which can cut down the amount of data sent over the wire by 60% or more. This is especially true of XML web services, since XML data compresses very well.

                                                                                                                                                                     Servers won't give you compressed data unless you tell them you can handle it.

                                                                                                                                                                     ### Example 11.14. Telling the server you would like compressed data

                                                                                                                                                                         >>> import urllib2, httplib
                                                                                                                                                                         >>> httplib.HTTPConnection.debuglevel = 1
                                                                                                                                                                         >>> request = urllib2.Request('http://diveintomark.org/xml/atom.xml')
                                                                                                                                                                         >>> request.add_header('Accept-encoding', 'gzip')        ①
                                                                                                                                                                         >>> opener = urllib2.build_opener()
                                                                                                                                                                         >>> f = opener.open(request)
                                                                                                                                                                         connect: (diveintomark.org, 80)
                                                                                                                                                                         send: '
                                                                                                                                                                         GET /xml/atom.xml HTTP/1.0
                                                                                                                                                                         Host: diveintomark.org
                                                                                                                                                                         User-agent: Python-urllib/2.1
                                                                                                                                                                         Accept-encoding: gzip②
                                                                                                                                                                         '
                                                                                                                                                                         reply: 'HTTP/1.1 200 OK\r\n'
                                                                                                                                                                         header: Date: Thu, 15 Apr 2004 22:24:39 GMT
                                                                                                                                                                         header: Server: Apache/2.0.49 (Debian GNU/Linux)
                                                                                                                                                                         header: Last-Modified: Thu, 15 Apr 2004 19:45:21 GMT
                                                                                                                                                                         header: ETag: "e842a-3e53-55d97640"
                                                                                                                                                                         header: Accept-Ranges: bytes
                                                                                                                                                                         header: Vary: Accept-Encoding
                                                                                                                                                                         header: Content-Encoding: gzip         ③
                                                                                                                                                                         header: Content-Length: 6289           ④
                                                                                                                                                                         header: Connection: close
                                                                                                                                                                         header: Content-Type: application/atom+xml

                                                                                                                                                                     1. This is the key: once you've created your `Request` object, add an `Accept-encoding` header to tell the server you can accept gzip-encoded data. `gzip` is the name of the compression algorithm you're using. In theory there could be other compression algorithms, but `gzip` is the compression algorithm used by 99% of web servers.
                                                                                                                                                                     2. There's your header going across the wire.
                                                                                                                                                                     3. And here's what the server sends back: the `Content-Encoding: gzip` header means that the data you're about to receive has been gzip-compressed.
                                                                                                                                                                     4. The `Content-Length` header is the length of the compressed data, not the uncompressed data. As you'll see in a minute, the actual length of the uncompressed data was 15955, so gzip compression cut your bandwidth by over 60%!

                                                                                                                                                                        ### Example 11.15. Decompressing the data

                                                                                                                                                                            >>> compresseddata = f.read()            ①
                                                                                                                                                                            >>> len(compresseddata)
                                                                                                                                                                            6289
                                                                                                                                                                            >>> import StringIO
                                                                                                                                                                            >>> compressedstream = StringIO.StringIO(compresseddata)   ②
                                                                                                                                                                            >>> import gzip
                                                                                                                                                                            >>> gzipper = gzip.GzipFile(fileobj=compressedstream)      ③
                                                                                                                                                                            >>> data = gzipper.read()                ④
                                                                                                                                                                            >>> print data         ⑤
                                                                                                                                                                            <?xml version="1.0" encoding="iso-8859-1"?>
                                                                                                                                                                            <feed version="0.3"
                                                                                                                                                                              xmlns="http://purl.org/atom/ns#"
                                                                                                                                                                              xmlns:dc="http://purl.org/dc/elements/1.1/"
                                                                                                                                                                              xml:lang="en">
                                                                                                                                                                              <title mode="escaped">dive into mark</title>
                                                                                                                                                                              <link rel="alternate" type="text/html" href="http://diveintomark.org/"/>
                                                                                                                                                                              <-- rest of feed omitted for brevity -->
                                                                                                                                                                            >>> len(data)
                                                                                                                                                                            15955

                                                                                                                                                                        1. Continuing from the previous example, f is the file-like object returned from the URL opener. Using its `read()` method would ordinarily get you the uncompressed data, but since this data has been gzip-compressed, this is just the first step towards getting the data you really want.
                                                                                                                                                                        2. OK, this step is a little bit of messy workaround. Python has a `gzip` module, which reads (and actually writes) gzip-compressed files on disk. But you don't have a file on disk, you have a gzip-compressed buffer in memory, and you don't want to write out a temporary file just so you can uncompress it. So what you're going to do is create a file-like object out of the in-memory data (compresseddata), using the `StringIO` module. You first saw the `StringIO` module in the previous chapter, but now you've found another use for it.
                                                                                                                                                                        3. Now you can create an instance of `GzipFile`, and tell it that its “file” is the file-like object compressedstream.
                                                                                                                                                                        4. This is the line that does all the actual work: “reading” from `GzipFile` will decompress the data. Strange? Yes, but it makes sense in a twisted kind of way. gzipper is a file-like object which represents a gzip-compressed file. That “file” is not a real file on disk, though; gzipper is really just “reading” from the file-like object you created with `StringIO` to wrap the compressed data, which is only in memory in the variable compresseddata. And where did that compressed data come from? You originally downloaded it from a remote HTTP server by “reading” from the file-like object you built with `urllib2.build_opener`. And amazingly, this all just works. Every step in the chain has no idea that the previous step is faking it.
                                                                                                                                                                        5. Look ma, real data. (15955 bytes of it, in fact.)

                                                                                                                                                                           “But wait!” I hear you cry. “This could be even easier!” I know what you're thinking. You're thinking that opener.open returns a file-like object, so why not cut out the `StringIO` middleman and just pass f directly to `GzipFile`? OK, maybe you weren't thinking that, but don't worry about it, because it doesn't work.

                                                                                                                                                                           ### Example 11.16. Decompressing the data directly from the server

                                                                                                                                                                               >>> f = opener.open(request)①
                                                                                                                                                                               >>> f.headers.get('Content-Encoding')         ②
                                                                                                                                                                               'gzip'
                                                                                                                                                                               >>> data = gzip.GzipFile(fileobj=f).read()    ③
                                                                                                                                                                               Traceback (most recent call last):
                                                                                                                                                                                 File "<stdin>", line 1, in ?
                                                                                                                                                                                 File "c:\python23\lib\gzip.py", line 217, in read
                                                                                                                                                                                   self._read(readsize)
                                                                                                                                                                                 File "c:\python23\lib\gzip.py", line 252, in _read
                                                                                                                                                                                   pos = self.fileobj.tell()   # Save current position
                                                                                                                                                                               AttributeError: addinfourl instance has no attribute 'tell'

                                                                                                                                                                           1. Continuing from the previous example, you already have a `Request` object set up with an `Accept-encoding: gzip` header.
                                                                                                                                                                           2. Simply opening the request will get you the headers (though not download any data yet). As you can see from the returned `Content-Encoding` header, this data has been sent gzip-compressed.
                                                                                                                                                                           3. Since `opener.open` returns a file-like object, and you know from the headers that when you read it, you're going to get gzip-compressed data, why not simply pass that file-like object directly to `GzipFile`? As you “read” from the `GzipFile` instance, it will “read” compressed data from the remote HTTP server and decompress it on the fly. It's a good idea, but unfortunately it doesn't work. Because of the way gzip compression works, `GzipFile` needs to save its position and move forwards and backwards through the compressed file. This doesn't work when the “file” is a stream of bytes coming from a remote server; all you can do with it is retrieve bytes one at a time, not move back and forth through the data stream. So the inelegant hack of using `StringIO` is the best solution: download the compressed data, create a file-like object out of it with `StringIO`, and then decompress the data from that.

                                                                                                                                                                              ## 11.9. Putting it all together

                                                                                                                                                                              You've seen all the pieces for building an intelligent HTTP web services client. Now let's see how they all fit together.

                                                                                                                                                                              ### Example 11.17. The `openanything` function

                                                                                                                                                                              This function is defined in `openanything.py`.

                                                                                                                                                                                  def openAnything(source, etag=None, lastmodified=None, agent=USER_AGENT):
                                                                                                                                                                                      # non-HTTP code omitted for brevity
                                                                                                                                                                                      if urlparse.urlparse(source)[0] == 'http':   ①
                                                                                                                                                                                          # open URL with urllib2
                                                                                                                                                                                          request = urllib2.Request(source)
                                                                                                                                                                                          request.add_header('User-Agent', agent)  ②
                                                                                                                                                                                          if etag:
                                                                                                                                                                                              request.add_header('If-None-Match', etag)              ③
                                                                                                                                                                                          if lastmodified:
                                                                                                                                                                                              request.add_header('If-Modified-Since', lastmodified)  ④
                                                                                                                                                                                          request.add_header('Accept-encoding', 'gzip')              ⑤
                                                                                                                                                                                          opener = urllib2.build_opener(SmartRedirectHandler(), DefaultErrorHandler()) ⑥
                                                                                                                                                                                          return opener.open(request)              ⑦

                                                                                                                                                                              1. `urlparse` is a handy utility module for, you guessed it, parsing URLs. It's primary function, also called `urlparse`, takes a URL and splits it into a tuple of (scheme, domain, path, params, query string parameters, and fragment identifier). Of these, the only thing you care about is the scheme, to make sure that you're dealing with an HTTP URL (which `urllib2` can handle).
                                                                                                                                                                              2. You identify yourself to the HTTP server with the `User-Agent` passed in by the calling function. If no `User-Agent` was specified, you use a default one defined earlier in the `openanything.py` module. You never use the default one defined by `urllib2`.
                                                                                                                                                                              3. If an `ETag` hash was given, send it in the `If-None-Match` header.
                                                                                                                                                                              4. If a last-modified date was given, send it in the `If-Modified-Since` header.
                                                                                                                                                                              5. Tell the server you would like compressed data if possible.
                                                                                                                                                                              6. Build a URL opener that uses both of the custom URL handlers: `SmartRedirectHandler` for handling `301` and `302` redirects, and `DefaultErrorHandler` for handling `304`, `404`, and other error conditions gracefully.
                                                                                                                                                                              7. That's it! Open the URL and return a file-like object to the caller.

                                                                                                                                                                                 ### Example 11.18. The `fetch function

                                                                                                                                                                                 This function is defined in `openanything.py.

                                                                                                                                                                                     def fetch(source, etag=None, last_modified=None, agent=USER_AGENT):
                                                                                                                                                                                         '''Fetch data and metadata from a URL, file, stream, or string'''
                                                                                                                                                                                         result = {}
                                                                                                                                                                                         f = openAnything(source, etag, last_modified, agent)              ①
                                                                                                                                                                                         result['data'] = f.read()     ②
                                                                                                                                                                                         if hasattr(f, 'headers'):
                                                                                                                                                                                             # save ETag, if the server sent one
                                                                                                                                                                                             result['etag'] = f.headers.get('ETag')      ③
                                                                                                                                                                                             # save Last-Modified header, if the server sent one
                                                                                                                                                                                             result['lastmodified'] = f.headers.get('Last-Modified')       ④
                                                                                                                                                                                             if f.headers.get('content-encoding', '') == 'gzip':           ⑤
                                                                                                                                                                                                 # data came back gzip-compressed, decompress it
                                                                                                                                                                                                 result['data'] = gzip.GzipFile(fileobj=StringIO(result['data']])).read()
                                                                                                                                                                                         if hasattr(f, 'url'):         ⑥
                                                                                                                                                                                             result['url'] = f.url
                                                                                                                                                                                             result['status'] = 200
                                                                                                                                                                                         if hasattr(f, 'status'):      ⑦
                                                                                                                                                                                             result['status'] = f.status
                                                                                                                                                                                         f.close()
                                                                                                                                                                                         return result

                                                                                                                                                                                 First, you call the `openAnything function with a URL, `ETag hash, `Last-Modified date, and `User-Agent.
                                                                                                                                                                                 Read the actual data returned from the server. This may be compressed; if so, you'll decompress it later.
                                                                                                                                                                                 Save the `ETag hash returned from the server, so the calling application can pass it back to you next time, and you can pass it on to `openAnything, which can stick it in the `If-None-Match header and send it to the remote server.
                                                                                                                                                                                 Save the `Last-Modified date too.
                                                                                                                                                                                 If the server says that it sent compressed data, decompress it.
                                                                                                                                                                                 If you got a URL back from the server, save it, and assume that the status code is `200 until you find out otherwise.
                                                                                                                                                                                 If one of the custom URL handlers captured a status code, then save that too.

                                                                                                                                                                                 ### Example 11.19. Using `openanything.py

                                                                                                                                                                                 `>>> `import openanything `>>> `useragent = 'MyHTTPWebServicesApp/1.0' `>>> `url = 'http://diveintopython3.org/redir/example301.xml' `>>> `params = openanything.fetch(url, agent=useragent) ① `>>> `params ② `{'url': 'http://diveintomark.org/xml/atom.xml', 'lastmodified': 'Thu, 15 Apr 2004 19:45:21 GMT', 'etag': '"e842a-3e53-55d97640"', 'status': 301, 'data': '<?xml version="1.0" encoding="iso-8859-1"?> <feed version="0.3" <-- rest of data omitted for brevity -->'} `>>> `if params['status'] == 301:③ `... url = params['url'] `>>> `newparams = openanything.fetch( `... url, params['etag'], params['lastmodified'], useragent) ④ `>>> `newparams `{'url': 'http://diveintomark.org/xml/atom.xml', 'lastmodified': None, 'etag': '"e842a-3e53-55d97640"', 'status': 304, 'data': ''} ⑤
                                                                                                                                                                                 The very first time you fetch a resource, you don't have an `ETag hash or `Last-Modified date, so you'll leave those out. (They're optional parameters.)
                                                                                                                                                                                 What you get back is a dictionary of several useful headers, the HTTP status code, and the actual data returned from the server. `openanything handles the gzip compression internally; you don't care about that at this level.
                                                                                                                                                                                 If you ever get a `301 status code, that's a permanent redirect, and you need to update your URL to the new address.
                                                                                                                                                                                 The second time you fetch the same resource, you have all sorts of information to pass back: a (possibly updated) URL, the `ETag from the last time, the `Last-Modified date from the last time, and of course your `User-Agent.
                                                                                                                                                                                 What you get back is again a dictionary, but the data hasn't changed, so all you got was a `304 status code and no data.

                                                                                                                                                                                 ## 11.10. Summary

                                                                                                                                                                                 The `openanything.py and its functions should now make perfect sense.

                                                                                                                                                                                 There are 5 important features of HTTP web services that every client should support:
                                                                                                                                                                                 Identifying your application by setting a proper `User-Agent.
                                                                                                                                                                                 Handling permanent redirects properly.
                                                                                                                                                                                 Supporting `Last-Modified date checking to avoid re-downloading data that hasn't changed.
                                                                                                                                                                                 Supporting `ETag hashes to avoid re-downloading data that hasn't changed.
                                                                                                                                                                                 Supporting gzip compression to reduce bandwidth even when data has changed.

                                                                                                                                                                                 ## Chapter 13. Unit Testing

                                                                                                                                                                                 ## 13.1. Introduction to Roman numerals

                                                                                                                                                                                 In previous chapters, you “dived in” by immediately looking at code and trying to understand it as quickly as possible. Now that you have some Python under your belt, you're going to step back and look at the steps that happen before the code gets written.

                                                                                                                                                                                 In the next few chapters, you're going to write, debug, and optimize a set of utility functions to convert to and from Roman numerals. You saw the mechanics of constructing and validating Roman numerals in Section 7.3, “Case Study: Roman Numerals”, but now let's step back and consider what it would take to expand that into a two-way utility.

                                                                                                                                                                                 The rules for Roman numerals lead to a number of interesting observations:
                                                                                                                                                                                 There is only one correct way to represent a particular number as Roman numerals.
                                                                                                                                                                                 The converse is also true: if a string of characters is a valid Roman numeral, it represents only one number (i.e. it can only be read one way).
                                                                                                                                                                                 There is a limited range of numbers that can be expressed as Roman numerals, specifically `1 through `3999. (The Romans did have several ways of expressing larger numbers, for instance by having a bar over a numeral to represent that its normal value should be multiplied by `1000, but you're not going to deal with that. For the purposes of this chapter, let's stipulate that Roman numerals go from `1 to `3999.)
                                                                                                                                                                                 There is no way to represent `0 in Roman numerals. (Amazingly, the ancient Romans had no concept of `0 as a number. Numbers were for counting things you had; how can you count what you don't have?)
                                                                                                                                                                                 There is no way to represent negative numbers in Roman numerals.
                                                                                                                                                                                 There is no way to represent fractions or non-integer numbers in Roman numerals.

                                                                                                                                                                                 Given all of this, what would you expect out of a set of functions to convert to and from Roman numerals?

                                                                                                                                                                                 ### `roman.py requirements
                                                                                                                                                                              8. `to_roman()` should return the Roman numeral representation for all integers `1` to `3999`.
                                                                                                                                                                              9. `to_roman()` should fail when given an integer outside the range `1` to `3999`.
                                                                                                                                                                              10. `to_roman()` should fail when given a non-integer number.
                                                                                                                                                                              11. `from_roman()` should take a valid Roman numeral and return the number that it represents.
                                                                                                                                                                              12. `from_roman()` should fail when given an invalid Roman numeral.
                                                                                                                                                                              13. If you take a number, convert it to Roman numerals, then convert that back to a number, you should end up with the number you started with. So `from_roman(to_roman(n)) == n` for all n in `1..3999`.
                                                                                                                                                                              14. `to_roman()` should always return a Roman numeral using uppercase letters.
                                                                                                                                                                              15. `from_roman()` should only accept uppercase Roman numerals (i.e. it should fail when given lowercase input).

                                                                                                                                                                              ### Further reading

                                                                                                                                                                              - This site has more on Roman numerals, including a fascinating history of how Romans and other civilizations really used them (short answer: haphazardly and inconsistently).

                                                                                                                                                                              ## 13.5. Testing for failure

                                                                                                                                                                              It is not enough to test that functions succeed when given good input; you must also test that they fail when given bad input. And not just any sort of failure; they must fail in the way you expect.

                                                                                                                                                                              Remember the other requirements for `to_roman()`:

                                                                                                                                                                              2. `to_roman() should fail when given an integer outside the range `1 to `3999.
                                                                                                                                                                              3. `to_roman() should fail when given a non-integer number.

                                                                                                                                                                              In Python, functions indicate failure by raising exceptions, and the `unittest` module provides methods for testing whether a function raises a particular exception when given bad input.

                                                                                                                                                                              ### Example 13.3. Testing bad input to `to_roman()

                                                                                                                                                                                  class ToRomanBadInput(unittest.TestCase):
                                                                                                                                                                                      def testTooLarge(self):
                                                                                                                                                                                          """to_roman should fail with large input"""
                                                                                                                                                                                          self.assertRaises(roman.OutOfRangeError, roman.to_roman, 4000) ①

                                                                                                                                                                                      def testZero(self):
                                                                                                                                                                                          """to_roman should fail with 0 input"""
                                                                                                                                                                                          self.assertRaises(roman.OutOfRangeError, roman.to_roman, 0)    ②

                                                                                                                                                                                      def testNegative(self):
                                                                                                                                                                                          """to_roman should fail with negative input"""
                                                                                                                                                                                          self.assertRaises(roman.OutOfRangeError, roman.to_roman, -1)

                                                                                                                                                                                      def testNonInteger(self):
                                                                                                                                                                                          """to_roman should fail with non-integer input"""
                                                                                                                                                                                          self.assertRaises(roman.NotIntegerError, roman.to_roman, 0.5)  ③

                                                                                                                                                                              The `TestCase class of the `unittest provides the `assertRaises method, which takes the following arguments: the exception you're expecting, the function you're testing, and the arguments you're passing that function. (If the function you're testing takes more than one argument, pass them all to `assertRaises, in order, and it will pass them right along to the function you're testing.) Pay close attention to what you're doing here: instead of calling `to_roman() directly and manually checking that it raises a particular exception (by wrapping it in a `try...except block), `assertRaises has encapsulated all of that for us. All you do is give it the exception (`roman.OutOfRangeError), the function (`to_roman()), and `to_roman()'s arguments (`4000), and `assertRaises takes care of calling `to_roman() and checking to make sure that it raises `roman.OutOfRangeError. (Also note that you're passing the `to_roman() function itself as an argument; you're not calling it, and you're not passing the name of it as a string. Have I mentioned recently how handy it is that everything in Python is an object, including functions and exceptions?)
                                                                                                                                                                              Along with testing numbers that are too large, you need to test numbers that are too small. Remember, Roman numerals cannot express `0 or negative numbers, so you have a test case for each of those (`testZero and `testNegative). In `testZero, you are testing that `to_roman() raises a `roman.OutOfRangeError exception when called with `0; if it does not raise a `roman.OutOfRangeError (either because it returns an actual value, or because it raises some other exception), this test is considered failed.
                                                                                                                                                                              Requirement #3 specifies that `to_roman() cannot accept a non-integer number, so here you test to make sure that `to_roman() raises a `roman.NotIntegerError exception when called with `0.5. If `to_roman() does not raise a `roman.NotIntegerError, this test is considered failed.

                                                                                                                                                                              The next two requirements are similar to the first three, except they apply to `from_roman() instead of `to_roman():
                                                                                                                                                                              `from_roman() should take a valid Roman numeral and return the number that it represents.
                                                                                                                                                                              `from_roman() should fail when given an invalid Roman numeral.

                                                                                                                                                                              Requirement #4 is handled in the same way as requirement #1, iterating through a sampling of known values and testing each in turn. Requirement #5 is handled in the same way as requirements #2 and #3, by testing a series of bad inputs and making sure `from_roman() raises the appropriate exception.

                                                                                                                                                                              ### Example 13.4. Testing bad input to `from_roman()

                                                                                                                                                                              `class FromRomanBadInput(unittest.TestCase): def testTooManyRepeatedNumerals(self): """from_roman should fail with too many repeated numerals""" for s in ('MMMM', 'DD', 'CCCC', 'LL', 'XXXX', 'VV', 'IIII'): self.assertRaises(roman.InvalidRomanNumeralError, roman.from_roman, s) ① def testRepeatedPairs(self): """from_roman should fail with repeated pairs of numerals""" for s in ('CMCM', 'CDCD', 'XCXC', 'XLXL', 'IXIX', 'IVIV'): self.assertRaises(roman.InvalidRomanNumeralError, roman.from_roman, s) def testMalformedAntecedent(self): """from_roman should fail with malformed antecedents""" for s in ('IIMXCC', 'VX', 'DCM', 'CMM', 'IXIV', 'MCMC', 'XCX', 'IVI', 'LM', 'LD', 'LC'): self.assertRaises(roman.InvalidRomanNumeralError, roman.from_roman, s)
                                                                                                                                                                           4. Not much new to say about these; the pattern is exactly the same as the one you used to test bad input to `to_roman()`. I will briefly note that you have another exception: `roman.InvalidRomanNumeralError`. That makes a total of three custom exceptions that will need to be defined in `roman.py` (along with `roman.OutOfRangeError` and `roman.NotIntegerError`). You'll see how to define these custom exceptions when you actually start writing `roman.py`, later in this chapter.

                                                                                                                                                                              ## 13.6. Testing for sanity

                                                                                                                                                                              Often, you will find that a unit of code contains a set of reciprocal functions, usually in the form of conversion functions where one converts A to B and the other converts B to A. In these cases, it is useful to create a “sanity check” to make sure that you can convert A to B and back to A without losing precision, incurring rounding errors, or triggering any other sort of bug.

                                                                                                                                                                              Consider this requirement:

                                                                                                                                                                              6. If you take a number, convert it to Roman numerals, then convert that back to a number, you should end up with the number you started with. So `from_roman(to_roman(n)) == n` for all n in `1..3999`.

                                                                                                                                                                              ### Example 13.5. Testing `to_roman()` against `from_roman()`

                                                                                                                                                                                  class SanityCheck(unittest.TestCase):
                                                                                                                                                                                      def testSanity(self):
                                                                                                                                                                                          """from_roman(to_roman(n))==n for all n"""
                                                                                                                                                                                          for integer in range(1, 4000):        ① ②
                                                                                                                                                                                              numeral = roman.to_roman(integer)
                                                                                                                                                                                              result = roman.from_roman(numeral)
                                                                                                                                                                                              self.assertEqual(integer, result) ③

                                                                                                                                                                              1. You've seen the `range function before, but here it is called with two arguments, which returns a list of integers starting at the first argument (`1`) and counting consecutively up to but not including the second argument (`4000`). Thus, `1..3999`, which is the valid range for converting to Roman numerals.
                                                                                                                                                                              2. I just wanted to mention in passing that integer is not a keyword in Python; here it's just a variable name like any other.
                                                                                                                                                                              3. The actual testing logic here is straightforward: take a number (integer), convert it to a Roman numeral (numeral), then convert it back to a number (result) and make sure you end up with the same number you started with. If not, `assertEqual` will raise an exception and the test will immediately be considered failed. If all the numbers match, `assertEqual` will always return silently, the entire `testSanity` method will eventually return silently, and the test will be considered passed.

                                                                                                                                                                                 The last two requirements are different from the others because they seem both arbitrary and trivial:

                                                                                                                                                                              4. `to_roman()` should always return a Roman numeral using uppercase letters.
                                                                                                                                                                              5. `from_roman()` should only accept uppercase Roman numerals (i.e. it should fail when given lowercase input).

                                                                                                                                                                              In fact, they are somewhat arbitrary. You could, for instance, have stipulated that `from_roman()` accept lowercase and mixed case input. But they are not completely arbitrary; if `to_roman()` is always returning uppercase output, then `from_roman()` must at least accept uppercase input, or the “sanity check” (requirement #6) would fail. The fact that it only accepts uppercase input is arbitrary, but as any systems integrator will tell you, case always matters, so it's worth specifying the behavior up front. And if it's worth specifying, it's worth testing.

                                                                                                                                                                              ### Example 13.6. Testing for case

                                                                                                                                                                                  class CaseCheck(unittest.TestCase):
                                                                                                                                                                                      def testToRomanCase(self):
                                                                                                                                                                                          """to_roman should always return uppercase"""
                                                                                                                                                                                          for integer in range(1, 4000):
                                                                                                                                                                                              numeral = roman.to_roman(integer)
                                                                                                                                                                                              self.assertEqual(numeral, numeral.upper())         ①

                                                                                                                                                                                      def testFromRomanCase(self):
                                                                                                                                                                                          """from_roman should only accept uppercase input"""
                                                                                                                                                                                          for integer in range(1, 4000):
                                                                                                                                                                                              numeral = roman.to_roman(integer)
                                                                                                                                                                                              roman.from_roman(numeral.upper()) ② ③
                                                                                                                                                                                              self.assertRaises(roman.InvalidRomanNumeralError,
                                                                                                                                                                                              roman.from_roman, numeral.lower())   ④

                                                                                                                                                                              1. The most interesting thing about this test case is all the things it doesn't test. It doesn't test that the value returned from `to_roman() is right or even consistent; those questions are answered by separate test cases. You have a whole test case just to test for uppercase-ness. You might be tempted to combine this with the sanity check, since both run through the entire range of values and call `to_roman(). [6] But that would violate one of the fundamental rules: each test case should answer only a single question. Imagine that you combined this case check with the sanity check, and then that test case failed. You would need to do further analysis to figure out which part of the test case failed to determine what the problem was. If you need to analyze the results of your unit testing just to figure out what they mean, it's a sure sign that you've mis-designed your test cases.
                                                                                                                                                                              2. There's a similar lesson to be learned here: even though “you know” that `to_roman() always returns uppercase, you are explicitly converting its return value to uppercase here to test that `from_roman() accepts uppercase input. Why? Because the fact that `to_roman() always returns uppercase is an independent requirement. If you changed that requirement so that, for instance, it always returned lowercase, the `testToRomanCase test case would need to change, but this test case would still work. This was another of the fundamental rules: each test case must be able to work in isolation from any of the others. Every test case is an island.
                                                                                                                                                                              3. Note that you're not assigning the return value of `from_roman() to anything. This is legal syntax in Python; if a function returns a value but nobody's listening, Python just throws away the return value. In this case, that's what you want. This test case doesn't test anything about the return value; it just tests that `from_roman() accepts the uppercase input without raising an exception.
                                                                                                                                                                              4. This is a complicated line, but it's very similar to what you did in the `ToRomanBadInput and `FromRomanBadInput tests. You are testing to make sure that calling a particular function (`roman.from_roman) with a particular value (`numeral.lower(), the lowercase version of the current Roman numeral in the loop) raises a particular exception (`roman.InvalidRomanNumeralError). If it does (each time through the loop), the test passes; if even one time it does something else (like raises a different exception, or returning a value without raising an exception at all), the test fails.

                                                                                                                                                                                 In the next chapter, you'll see how to write code that passes these tests.

                                                                                                                                                                                 [6] “I can resist everything except temptation.” --Oscar Wilde

                                                                                                                                                                                 ## Chapter 14. Test-First Programming

                                                                                                                                                                                 ## 14.1. `roman.py, stage 1

                                                                                                                                                                                 Now that the unit tests are complete, it's time to start writing the code that the test cases are attempting to test. You're going to do this in stages, so you can see all the unit tests fail, then watch them pass one by one as you fill in the gaps in `roman.py.

                                                                                                                                                                                 ### Example 14.1. `roman1.py

                                                                                                                                                                                 This file is available in `py/roman/stage1/ in the examples directory.

                                                                                                                                                                                 If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                                 `"""Convert to and from Roman numerals""" #Define exceptions class RomanError(Exception): pass ① class OutOfRangeError(RomanError): pass ② class NotIntegerError(RomanError): pass class InvalidRomanNumeralError(RomanError): pass ③ def to_roman(n): """convert integer to Roman numeral""" pass ④ def from_roman(s): """convert Roman numeral to integer""" pass
                                                                                                                                                                              5. This is how you define your own custom exceptions in Python. Exceptions are classes, and you create your own by subclassing existing exceptions. It is strongly recommended (but not required) that you subclass `Exception, which is the base class that all built-in exceptions inherit from. Here I am defining `RomanError (inherited from `Exception) to act as the base class for all my other custom exceptions to follow. This is a matter of style; I could just as easily have inherited each individual exception from the `Exception class directly.
                                                                                                                                                                              6. The `OutOfRangeError and `NotIntegerError exceptions will eventually be used by `to_roman() to flag various forms of invalid input, as specified in `ToRomanBadInput.
                                                                                                                                                                              7. The `InvalidRomanNumeralError exception will eventually be used by `from_roman() to flag invalid input, as specified in `FromRomanBadInput.
                                                                                                                                                                              8. At this stage, you want to define the API of each of your functions, but you don't want to code them yet, so you stub them out using the Python reserved word `pass.

                                                                                                                                                                                 Now for the big moment (drum roll please): you're finally going to run the unit test against this stubby little module. At this point, every test case should fail. In fact, if any test case passes in stage 1, you should go back to `romantest.py and re-evaluate why you coded a test so useless that it passes with do-nothing functions.
                                                                                                                                                                              9. At this stage, you want to define the API of each of your functions, but you don't want to code them yet, so you stub them out using the Python reserved word `pass.

                                                                                                                                                                                 Run `romantest1.py with the `-v command-line option, which will give more verbose output so you can see exactly what's going on as each test case runs. With any luck, your output should look like this:

                                                                                                                                                                                 ### Example 14.2. Output of `romantest1.py against `roman1.py

                                                                                                                                                                                 `from_roman should only accept uppercase input ... ERROR to_roman should always return uppercase ... ERROR from_roman should fail with malformed antecedents ... FAIL from_roman should fail with repeated pairs of numerals ... FAIL from_roman should fail with too many repeated numerals ... FAIL from_roman should give known result with known input ... FAIL to_roman should give known result with known input ... FAIL from_roman(to_roman(n))==n for all n ... FAIL to_roman should fail with non-integer input ... FAIL to_roman should fail with negative input ... FAIL to_roman should fail with large input ... FAIL to_roman should fail with 0 input ... FAIL ====================================================================== ERROR: from_roman should only accept uppercase input ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 154, in testFromRomanCase roman1.from_roman(numeral.upper()) AttributeError: 'None' object has no attribute 'upper'`====================================================================== ERROR: to_roman should always return uppercase ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 148, in testToRomanCase self.assertEqual(numeral, numeral.upper()) AttributeError: 'None' object has no attribute 'upper'`====================================================================== FAIL: from_roman should fail with malformed antecedents ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 133, in testMalformedAntecedent self.assertRaises(roman1.InvalidRomanNumeralError, roman1.from_roman, s) File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises raise self.failureException, excName AssertionError: InvalidRomanNumeralError`====================================================================== FAIL: from_roman should fail with repeated pairs of numerals ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 127, in testRepeatedPairs self.assertRaises(roman1.InvalidRomanNumeralError, roman1.from_roman, s) File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises raise self.failureException, excName AssertionError: InvalidRomanNumeralError`====================================================================== FAIL: from_roman should fail with too many repeated numerals ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 122, in testTooManyRepeatedNumerals self.assertRaises(roman1.InvalidRomanNumeralError, roman1.from_roman, s) File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises raise self.failureException, excName AssertionError: InvalidRomanNumeralError`====================================================================== FAIL: from_roman should give known result with known input ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 99, in testFromRomanKnownValues self.assertEqual(integer, result) File "c:\python21\lib\unittest.py", line 273, in failUnlessEqual raise self.failureException, (msg or '%s != %s' % (first, second)) AssertionError: 1 != None`====================================================================== FAIL: to_roman should give known result with known input ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 93, in testToRomanKnownValues self.assertEqual(numeral, result) File "c:\python21\lib\unittest.py", line 273, in failUnlessEqual raise self.failureException, (msg or '%s != %s' % (first, second)) AssertionError: I != None`====================================================================== FAIL: from_roman(to_roman(n))==n for all n ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 141, in testSanity self.assertEqual(integer, result) File "c:\python21\lib\unittest.py", line 273, in failUnlessEqual raise self.failureException, (msg or '%s != %s' % (first, second)) AssertionError: 1 != None`====================================================================== FAIL: to_roman should fail with non-integer input ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 116, in testNonInteger self.assertRaises(roman1.NotIntegerError, roman1.to_roman, 0.5) File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises raise self.failureException, excName AssertionError: NotIntegerError`====================================================================== FAIL: to_roman should fail with negative input ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 112, in testNegative self.assertRaises(roman1.OutOfRangeError, roman1.to_roman, -1) File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises raise self.failureException, excName AssertionError: OutOfRangeError`====================================================================== FAIL: to_roman should fail with large input ---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 104, in testTooLarge self.assertRaises(roman1.OutOfRangeError, roman1.to_roman, 4000) File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises raise self.failureException, excName AssertionError: OutOfRangeError`====================================================================== FAIL: to_roman should fail with 0 input ①`---------------------------------------------------------------------- `Traceback (most recent call last): File "C:\docbook\dip\py\roman\stage1\romantest1.py", line 108, in testZero self.assertRaises(roman1.OutOfRangeError, roman1.to_roman, 0) File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises raise self.failureException, excName AssertionError: OutOfRangeError ②`---------------------------------------------------------------------- Ran 12 tests in 0.040s ③ `FAILED (failures=10, errors=2) ④

                                                                                                                                                                                 ## 14.2. `roman.py, stage 2

                                                                                                                                                                                 Now that you have the framework of the `roman module laid out, it's time to start writing code and passing test cases.

                                                                                                                                                                                 ### Example 14.3. `roman2.py

                                                                                                                                                                                 This file is available in `py/roman/stage2/ in the examples directory.

                                                                                                                                                                                 If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                                 `"""Convert to and from Roman numerals""" #Define exceptions class RomanError(Exception): pass class OutOfRangeError(RomanError): pass class NotIntegerError(RomanError): pass class InvalidRomanNumeralError(RomanError): pass #Define digit mapping romanNumeralMap = (('M', 1000), ① ('CM', 900), ('D', 500), ('CD', 400), ('C', 100), ('XC', 90), ('L', 50), ('XL', 40), ('X', 10), ('IX', 9), ('V', 5), ('IV', 4), ('I', 1)) def to_roman(n): """convert integer to Roman numeral""" result = "" for numeral, integer in romanNumeralMap: while n >= integer: ② result += numeral n -= integer return result def from_roman(s): """convert Roman numeral to integer""" pass
                                                                                                                                                                              10. romanNumeralMap is a tuple of tuples which defines three things:
                                                                                                                                                                              11. The character representations of the most basic Roman numerals. Note that this is not just the single-character Roman numerals; you're also defining two-character pairs like `CM (“one hundred less than one thousand”); this will make the `to_roman() code simpler later.
                                                                                                                                                                              12. The order of the Roman numerals. They are listed in descending value order, from `M all the way down to `I.
                                                                                                                                                                              13. The value of each Roman numeral. Each inner tuple is a pair of `(numeral, value).
                                                                                                                                                                           5. Here's where your rich data structure pays off, because you don't need any special logic to handle the subtraction rule. To convert to Roman numerals, you simply iterate through romanNumeralMap looking for the largest integer value less than or equal to the input. Once found, you add the Roman numeral representation to the end of the output, subtract the corresponding integer value from the input, lather, rinse, repeat.

                                                                                                                                                                              ### Example 14.4. How `to_roman()` works

                                                                                                                                                                              If you're not clear how `to_roman()` works, add a `print` statement to the end of the `while` loop:

                                                                                                                                                                                          while n >= integer:
                                                                                                                                                                                              result += numeral
                                                                                                                                                                                              n -= integer
                                                                                                                                                                                              print 'subtracting', integer, 'from input, adding', numeral, 'to output'

                                                                                                                                                                                  >>> import roman2
                                                                                                                                                                                  >>> roman2.to_roman(1424)
                                                                                                                                                                                  subtracting 1000 from input, adding M to output
                                                                                                                                                                                  subtracting 400 from input, adding CD to output
                                                                                                                                                                                  subtracting 10 from input, adding X to output
                                                                                                                                                                                  subtracting 10 from input, adding X to output
                                                                                                                                                                                  subtracting 4 from input, adding IV to output
                                                                                                                                                                                  'MCDXXIV'

                                                                                                                                                                              So `to_roman()` appears to work, at least in this manual spot check. But will it pass the unit testing? Well no, not entirely.

                                                                                                                                                                              ### Example 14.5. Output of `romantest2.py` against `roman2.py`

                                                                                                                                                                              Remember to run `romantest2.py` with the `-v` command-line flag to enable verbose mode.

                                                                                                                                                                                  from_roman should only accept uppercase input ... FAIL
                                                                                                                                                                                  to_roman should always return uppercase ... ok①
                                                                                                                                                                                  from_roman should fail with malformed antecedents ... FAIL
                                                                                                                                                                                  from_roman should fail with repeated pairs of numerals ... FAIL
                                                                                                                                                                                  from_roman should fail with too many repeated numerals ... FAIL
                                                                                                                                                                                  from_roman should give known result with known input ... FAIL
                                                                                                                                                                                  to_roman should give known result with known input ... ok       ②
                                                                                                                                                                                  from_roman(to_roman(n))==n for all n ... FAIL
                                                                                                                                                                                  to_roman should fail with non-integer input ... FAIL            ③
                                                                                                                                                                                  to_roman should fail with negative input ... FAIL
                                                                                                                                                                                  to_roman should fail with large input ... FAIL
                                                                                                                                                                                  to_roman should fail with 0 input ... FAIL

                                                                                                                                                                              1. `to_roman()` does, in fact, always return uppercase, because romanNumeralMap defines the Roman numeral representations as uppercase. So this test passes already.
                                                                                                                                                                              2. Here's the big news: this version of the `to_roman()` function passes the known values test. Remember, it's not comprehensive, but it does put the function through its paces with a variety of good inputs, including inputs that produce every single-character Roman numeral, the largest possible input (`3999`), and the input that produces the longest possible Roman numeral (`3888`). At this point, you can be reasonably confident that the function works for any good input value you could throw at it.
                                                                                                                                                                              3. However, the function does not “work” for bad values; it fails every single bad input test. That makes sense, because you didn't include any checks for bad input. Those test cases look for specific exceptions to be raised (via `assertRaises`), and you're never raising them. You'll do that in the next stage.

                                                                                                                                                                                 Here's the rest of the output of the unit test, listing the details of all the failures. You're down to 10.

                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: from_roman should only accept uppercase input
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 156, in testFromRomanCase
                                                                                                                                                                                         roman2.from_roman, numeral.lower())
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: from_roman should fail with malformed antecedents
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 133, in testMalformedAntecedent
                                                                                                                                                                                         self.assertRaises(roman2.InvalidRomanNumeralError, roman2.from_roman, s)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: from_roman should fail with repeated pairs of numerals
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 127, in testRepeatedPairs
                                                                                                                                                                                         self.assertRaises(roman2.InvalidRomanNumeralError, roman2.from_roman, s)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: from_roman should fail with too many repeated numerals
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 122, in testTooManyRepeatedNumerals
                                                                                                                                                                                         self.assertRaises(roman2.InvalidRomanNumeralError, roman2.from_roman, s)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: from_roman should give known result with known input
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 99, in testFromRomanKnownValues
                                                                                                                                                                                         self.assertEqual(integer, result)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 273, in failUnlessEqual
                                                                                                                                                                                         raise self.failureException, (msg or '%s != %s' % (first, second))
                                                                                                                                                                                     AssertionError: 1 != None
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: from_roman(to_roman(n))==n for all n
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 141, in testSanity
                                                                                                                                                                                         self.assertEqual(integer, result)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 273, in failUnlessEqual
                                                                                                                                                                                         raise self.failureException, (msg or '%s != %s' % (first, second))
                                                                                                                                                                                     AssertionError: 1 != None
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: to_roman should fail with non-integer input
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 116, in testNonInteger
                                                                                                                                                                                         self.assertRaises(roman2.NotIntegerError, roman2.to_roman, 0.5)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: NotIntegerError
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: to_roman should fail with negative input
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 112, in testNegative
                                                                                                                                                                                         self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, -1)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: OutOfRangeError
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: to_roman should fail with large input
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 104, in testTooLarge
                                                                                                                                                                                         self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 4000)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: OutOfRangeError
                                                                                                                                                                                     ======================================================================
                                                                                                                                                                                     FAIL: to_roman should fail with 0 input
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Traceback (most recent call last):
                                                                                                                                                                                       File "C:\docbook\dip\py\roman\stage2\romantest2.py", line 108, in testZero
                                                                                                                                                                                         self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 0)
                                                                                                                                                                                       File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                         raise self.failureException, excName
                                                                                                                                                                                     AssertionError: OutOfRangeError
                                                                                                                                                                                     ----------------------------------------------------------------------
                                                                                                                                                                                     Ran 12 tests in 0.320s

                                                                                                                                                                                     FAILED (failures=10)

                                                                                                                                                                                 ## 14.3. `roman.py, stage 3

                                                                                                                                                                                 Now that `to_roman() behaves correctly with good input (integers from `1 to `3999), it's time to make it behave correctly with bad input (everything else).

                                                                                                                                                                                 ### Example 14.6. `roman3.py

                                                                                                                                                                                 This file is available in `py/roman/stage3/ in the examples directory.

                                                                                                                                                                                 If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                                 `"""Convert to and from Roman numerals""" #Define exceptions class RomanError(Exception): pass class OutOfRangeError(RomanError): pass class NotIntegerError(RomanError): pass class InvalidRomanNumeralError(RomanError): pass #Define digit mapping romanNumeralMap = (('M', 1000), ('CM', 900), ('D', 500), ('CD', 400), ('C', 100), ('XC', 90), ('L', 50), ('XL', 40), ('X', 10), ('IX', 9), ('V', 5), ('IV', 4), ('I', 1)) def to_roman(n): """convert integer to Roman numeral""" if not (0 < n < 4000): ① raise OutOfRangeError, "number out of range (must be 1..3999)" ② if int(n) <> n: ③ raise NotIntegerError, "non-integers can not be converted" result = "" ④ for numeral, integer in romanNumeralMap: while n >= integer: result += numeral n -= integer return result def from_roman(s): """convert Roman numeral to integer""" pass
                                                                                                                                                                              4. This is a nice Pythonic shortcut: multiple comparisons at once. This is equivalent to `if not ((0 < n) and (n < 4000))`, but it's much easier to read. This is the range check, and it should catch inputs that are too large, negative, or zero.
                                                                                                                                                                              5. You raise exceptions yourself with the `raise` statement. You can raise any of the built-in exceptions, or you can raise any of your custom exceptions that you've defined. The second parameter, the error message, is optional; if given, it is displayed in the traceback that is printed if the exception is never handled.
                                                                                                                                                                              6. This is the non-integer check. Non-integers can not be converted to Roman numerals.
                                                                                                                                                                              7. The rest of the function is unchanged.

                                                                                                                                                                                 ### Example 14.7. Watching `to_roman() handle bad input

                                                                                                                                                                                 `>>> `import roman3 `>>> `roman3.to_roman(4000) `Traceback (most recent call last): File "<interactive input>", line 1, in ? File "roman3.py", line 27, in to_roman raise OutOfRangeError, "number out of range (must be 1..3999)" OutOfRangeError: number out of range (must be 1..3999) `>>> `roman3.to_roman(1.5) `Traceback (most recent call last): File "<interactive input>", line 1, in ? File "roman3.py", line 29, in to_roman raise NotIntegerError, "non-integers can not be converted" NotIntegerError: non-integers can not be converted

                                                                                                                                                                                 ### Example 14.8. Output of `romantest3.py against `roman3.py

                                                                                                                                                                                 `from_roman should only accept uppercase input ... FAIL to_roman should always return uppercase ... ok from_roman should fail with malformed antecedents ... FAIL from_roman should fail with repeated pairs of numerals ... FAIL from_roman should fail with too many repeated numerals ... FAIL from_roman should give known result with known input ... FAIL to_roman should give known result with known input ... ok ① `from_roman(to_roman(n))==n for all n ... FAIL to_roman should fail with non-integer input ... ok ② `to_roman should fail with negative input ... ok ③ `to_roman should fail with large input ... ok to_roman should fail with 0 input ... ok
                                                                                                                                                                              8. `to_roman()` still passes the known values test, which is comforting. All the tests that passed in stage 2 still pass, so the latest code hasn't broken anything.
                                                                                                                                                                              9. More exciting is the fact that all of the bad input tests now pass. This test, `testNonInteger`, passes because of the `int(n) <> n` check. When a non-integer is passed to `to_roman()`, the `int(n) <> n` check notices it and raises the `NotIntegerError` exception, which is what `testNonInteger` is looking for.
                                                                                                                                                                              10. This test, `testNegative`, passes because of the `not (0 < n < 4000)` check, which raises an `OutOfRangeError` exception, which is what `testNegative` is looking for.

                                                                                                                                                                                      ======================================================================
                                                                                                                                                                                      FAIL: from_roman should only accept uppercase input
                                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                                      Traceback (most recent call last):
                                                                                                                                                                                        File "C:\docbook\dip\py\roman\stage3\romantest3.py", line 156, in testFromRomanCase
                                                                                                                                                                                          roman3.from_roman, numeral.lower())
                                                                                                                                                                                        File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                          raise self.failureException, excName
                                                                                                                                                                                      AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                      ======================================================================
                                                                                                                                                                                      FAIL: from_roman should fail with malformed antecedents
                                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                                      Traceback (most recent call last):
                                                                                                                                                                                        File "C:\docbook\dip\py\roman\stage3\romantest3.py", line 133, in testMalformedAntecedent
                                                                                                                                                                                          self.assertRaises(roman3.InvalidRomanNumeralError, roman3.from_roman, s)
                                                                                                                                                                                        File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                          raise self.failureException, excName
                                                                                                                                                                                      AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                      ======================================================================
                                                                                                                                                                                      FAIL: from_roman should fail with repeated pairs of numerals
                                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                                      Traceback (most recent call last):
                                                                                                                                                                                        File "C:\docbook\dip\py\roman\stage3\romantest3.py", line 127, in testRepeatedPairs
                                                                                                                                                                                          self.assertRaises(roman3.InvalidRomanNumeralError, roman3.from_roman, s)
                                                                                                                                                                                        File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                          raise self.failureException, excName
                                                                                                                                                                                      AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                      ======================================================================
                                                                                                                                                                                      FAIL: from_roman should fail with too many repeated numerals
                                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                                      Traceback (most recent call last):
                                                                                                                                                                                        File "C:\docbook\dip\py\roman\stage3\romantest3.py", line 122, in testTooManyRepeatedNumerals
                                                                                                                                                                                          self.assertRaises(roman3.InvalidRomanNumeralError, roman3.from_roman, s)
                                                                                                                                                                                        File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                                          raise self.failureException, excName
                                                                                                                                                                                      AssertionError: InvalidRomanNumeralError
                                                                                                                                                                                      ======================================================================
                                                                                                                                                                                      FAIL: from_roman should give known result with known input
                                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                                      Traceback (most recent call last):
                                                                                                                                                                                        File "C:\docbook\dip\py\roman\stage3\romantest3.py", line 99, in testFromRomanKnownValues
                                                                                                                                                                                          self.assertEqual(integer, result)
                                                                                                                                                                                        File "c:\python21\lib\unittest.py", line 273, in failUnlessEqual
                                                                                                                                                                                          raise self.failureException, (msg or '%s != %s' % (first, second))
                                                                                                                                                                                      AssertionError: 1 != None
                                                                                                                                                                                      ======================================================================
                                                                                                                                                                                      FAIL: from_roman(to_roman(n))==n for all n
                                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                                      Traceback (most recent call last):
                                                                                                                                                                                        File "C:\docbook\dip\py\roman\stage3\romantest3.py", line 141, in testSanity
                                                                                                                                                                                          self.assertEqual(integer, result)
                                                                                                                                                                                        File "c:\python21\lib\unittest.py", line 273, in failUnlessEqual
                                                                                                                                                                                          raise self.failureException, (msg or '%s != %s' % (first, second))
                                                                                                                                                                                      AssertionError: 1 != None
                                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                                      Ran 12 tests in 0.401s

                                                                                                                                                                                      FAILED (failures=6) ①

                                                                                                                                                                                  You're down to 6 failures, and all of them involve `from_roman(): the known values test, the three separate bad input tests, the case check, and the sanity check. That means that `to_roman() has passed all the tests it can pass by itself. (It's involved in the sanity check, but that also requires that `from_roman() be written, which it isn't yet.) Which means that you must stop coding `to_roman() now. No tweaking, no twiddling, no extra checks “just in case”. Stop. Now. Back away from the keyboard.

                                                                                                                                                         The most important thing that comprehensive unit testing can tell you is when to stop coding. When all the unit tests for a function pass, stop coding the function. When all the unit tests for an entire module pass, stop coding the module.

                                                                                                                                                         ## 14.4. `roman.py`, stage 4

                                                                                                                                                         Now that `to_roman()` is done, it's time to start coding `from_roman()`. Thanks to the rich data structure that maps individual Roman numerals to integer values, this is no more difficult than the `to_roman()` function.

                                                                                                                                                         ### Example 14.9. `roman4.py`

                                                                                                                                                         This file is available in `py/roman/stage4/` in the examples directory.

                                                                                                                                                         If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                             """Convert to and from Roman numerals"""

                                                                                                                                                             #Define exceptions
                                                                                                                                                             class RomanError(Exception): pass
                                                                                                                                                             class OutOfRangeError(RomanError): pass
                                                                                                                                                             class NotIntegerError(RomanError): pass
                                                                                                                                                             class InvalidRomanNumeralError(RomanError): pass

                                                                                                                                                             #Define digit mapping
                                                                                                                                                             romanNumeralMap = (('M',  1000),
                                                                                                                                                              ('CM', 900),
                                                                                                                                                              ('D',  500),
                                                                                                                                                              ('CD', 400),
                                                                                                                                                              ('C',  100),
                                                                                                                                                              ('XC', 90),
                                                                                                                                                              ('L',  50),
                                                                                                                                                              ('XL', 40),
                                                                                                                                                              ('X',  10),
                                                                                                                                                              ('IX', 9),
                                                                                                                                                              ('V',  5),
                                                                                                                                                              ('IV', 4),
                                                                                                                                                              ('I',  1))

                                                                                                                                                             # to_roman function omitted for clarity (it hasn't changed)

                                                                                                                                                             def from_roman(s):
                                                                                                                                                                 """convert Roman numeral to integer"""
                                                                                                                                                                 result = 0
                                                                                                                                                                 index = 0
                                                                                                                                                                 for numeral, integer in romanNumeralMap:
                                                                                                                                                                     while s[index:index+len(numeral)] == numeral: ①
                                                                                                                                                                         result += integer
                                                                                                                                                                         index += len(numeral)
                                                                                                                                                                 return result

                                                                                                                                                         1. The pattern here is the same as `to_roman()`. You iterate through your Roman numeral data structure (a tuple of tuples), and instead of matching the highest integer values as often as possible, you match the “highest” Roman numeral character strings as often as possible.

                                                                                                                                                            ### Example 14.10. How `from_roman()` works

                                                                                                                                                            If you're not clear how `from_roman()` works, add a `print` statement to the end of the `while` loop:

                                                                                                                                                                        while s[index:index+len(numeral)] == numeral:
                                                                                                                                                                            result += integer
                                                                                                                                                                            index += len(numeral)
                                                                                                                                                                            print 'found', numeral, 'of length', len(numeral), ', adding', integer

                                                                                                                                                                >>> import roman4
                                                                                                                                                                >>> roman4.from_roman('MCMLXXII')
                                                                                                                                                                found M , of length 1, adding 1000
                                                                                                                                                                found CM , of length 2, adding 900
                                                                                                                                                                found L , of length 1, adding 50
                                                                                                                                                                found X , of length 1, adding 10
                                                                                                                                                                found X , of length 1, adding 10
                                                                                                                                                                found I , of length 1, adding 1
                                                                                                                                                                found I , of length 1, adding 1
                                                                                                                                                                1972

                                                                                                                                                            ### Example 14.11. Output of `romantest4.py` against `roman4.py`

                                                                                                                                                                from_roman should only accept uppercase input ... FAIL
                                                                                                                                                                to_roman should always return uppercase ... ok
                                                                                                                                                                from_roman should fail with malformed antecedents ... FAIL
                                                                                                                                                                from_roman should fail with repeated pairs of numerals ... FAIL
                                                                                                                                                                from_roman should fail with too many repeated numerals ... FAIL
                                                                                                                                                                from_roman should give known result with known input ... ok ①
                                                                                                                                                                to_roman should give known result with known input ... ok
                                                                                                                                                                from_roman(to_roman(n))==n for all n ... ok②
                                                                                                                                                                to_roman should fail with non-integer input ... ok
                                                                                                                                                                to_roman should fail with negative input ... ok
                                                                                                                                                                to_roman should fail with large input ... ok
                                                                                                                                                                to_roman should fail with 0 input ... ok

                                                                                                                                                            1. Two pieces of exciting news here. The first is that `from_roman()` works for good input, at least for all the known values you test.
                                                                                                                                                            2. The second is that the sanity check also passed. Combined with the known values tests, you can be reasonably sure that both `to_roman()` and `from_roman()` work properly for all possible good values. (This is not guaranteed; it is theoretically possible that `to_roman()` has a bug that produces the wrong Roman numeral for some particular set of inputs, and that `from_roman()` has a reciprocal bug that produces the same wrong integer values for exactly that set of Roman numerals that `to_roman()` generated incorrectly. Depending on your application and your requirements, this possibility may bother you; if so, write more comprehensive test cases until it doesn't bother you.)

                                                                                                                                                                   ======================================================================
                                                                                                                                                                   FAIL: from_roman should only accept uppercase input
                                                                                                                                                                   ----------------------------------------------------------------------
                                                                                                                                                                   Traceback (most recent call last):
                                                                                                                                                                     File "C:\docbook\dip\py\roman\stage4\romantest4.py", line 156, in testFromRomanCase
                                                                                                                                                                       roman4.from_roman, numeral.lower())
                                                                                                                                                                     File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                       raise self.failureException, excName
                                                                                                                                                                   AssertionError: InvalidRomanNumeralError
                                                                                                                                                                   ======================================================================
                                                                                                                                                                   FAIL: from_roman should fail with malformed antecedents
                                                                                                                                                                   ----------------------------------------------------------------------
                                                                                                                                                                   Traceback (most recent call last):
                                                                                                                                                                     File "C:\docbook\dip\py\roman\stage4\romantest4.py", line 133, in testMalformedAntecedent
                                                                                                                                                                       self.assertRaises(roman4.InvalidRomanNumeralError, roman4.from_roman, s)
                                                                                                                                                                     File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                       raise self.failureException, excName
                                                                                                                                                                   AssertionError: InvalidRomanNumeralError
                                                                                                                                                                   ======================================================================
                                                                                                                                                                   FAIL: from_roman should fail with repeated pairs of numerals
                                                                                                                                                                   ----------------------------------------------------------------------
                                                                                                                                                                   Traceback (most recent call last):
                                                                                                                                                                     File "C:\docbook\dip\py\roman\stage4\romantest4.py", line 127, in testRepeatedPairs
                                                                                                                                                                       self.assertRaises(roman4.InvalidRomanNumeralError, roman4.from_roman, s)
                                                                                                                                                                     File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                       raise self.failureException, excName
                                                                                                                                                                   AssertionError: InvalidRomanNumeralError
                                                                                                                                                                   ======================================================================
                                                                                                                                                                   FAIL: from_roman should fail with too many repeated numerals
                                                                                                                                                                   ----------------------------------------------------------------------
                                                                                                                                                                   Traceback (most recent call last):
                                                                                                                                                                     File "C:\docbook\dip\py\roman\stage4\romantest4.py", line 122, in testTooManyRepeatedNumerals
                                                                                                                                                                       self.assertRaises(roman4.InvalidRomanNumeralError, roman4.from_roman, s)
                                                                                                                                                                     File "c:\python21\lib\unittest.py", line 266, in failUnlessRaises
                                                                                                                                                                       raise self.failureException, excName
                                                                                                                                                                   AssertionError: InvalidRomanNumeralError
                                                                                                                                                                   ----------------------------------------------------------------------
                                                                                                                                                                   Ran 12 tests in 1.222s

                                                                                                                                                                   FAILED (failures=4)

                                                                                                                                                               ## 14.5. `roman.py`, stage 5

                                                                                                                                                               Now that `from_roman()` works properly with good input, it's time to fit in the last piece of the puzzle: making it work properly with bad input. That means finding a way to look at a string and determine if it's a valid Roman numeral. This is inherently more difficult than validating numeric input in `to_roman()`, but you have a powerful tool at your disposal: regular expressions.

                                                                                                                                                               If you're not familiar with regular expressions and didn't read Chapter 7, Regular Expressions, now would be a good time.

                                                                                                                                                               As you saw in Section 7.3, “Case Study: Roman Numerals”, there are several simple rules for constructing a Roman numeral, using the letters `M`, `D`, `C`, `L`, `X`, `V`, and `I`. Let's review the rules:

                                                                                                                                                               1. Characters are additive. `I` is `1`, `II` is `2`, and `III` is `3`. `VI` is `6` (literally, “`5` and `1`”), `VII` is `7`, and `VIII` is `8`.
                                                                                                                                                               2. The tens characters (`I`, `X`, `C`, and `M`) can be repeated up to three times. At `4`, you need to subtract from the next highest fives character. You can't represent `4` as `IIII`; instead, it is represented as `IV` (“`1` less than `5`”). `40` is written as `XL` (“`10` less than `50`”), `41` as `XLI`, `42` as `XLII`, `43` as `XLIII`, and then `44` as `XLIV` (“`10` less than `50`, then `1` less than `5`”).
                                                                                                                                                               3. Similarly, at `9`, you need to subtract from the next highest tens character: `8` is `VIII`, but `9` is `IX` (“`1` less than `10`”), not `VIIII` (since the `I` character can not be repeated four times). `90` is `XC`, `900` is `CM`.
                                                                                                                                                               4. The fives characters can not be repeated. `10` is always represented as `X`, never as `VV`. `100` is always `C`, never `LL`.
                                                                                                                                                               5. Roman numerals are always written highest to lowest, and read left to right, so order of characters matters very much. `DC` is `600`; `CD` is a completely different number (`400`, “`100` less than `500`”). `CI` is `101`; `IC` is not even a valid Roman numeral (because you can't subtract `1` directly from `100`; you would need to write it as `XCIX`, “`10` less than `100`, then `1` less than `10`”).

                                                                                                                                                               ### Example 14.12. `roman5.py`

                                                                                                                                                               This file is available in `py/roman/stage5/` in the examples directory.

                                                                                                                                                               If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                   """Convert to and from Roman numerals"""
                                                                                                                                                                   import re

                                                                                                                                                                   #Define exceptions
                                                                                                                                                                   class RomanError(Exception): pass
                                                                                                                                                                   class OutOfRangeError(RomanError): pass
                                                                                                                                                                   class NotIntegerError(RomanError): pass
                                                                                                                                                                   class InvalidRomanNumeralError(RomanError): pass

                                                                                                                                                                   #Define digit mapping
                                                                                                                                                                   romanNumeralMap = (('M',  1000),
                                                                                                                                                                    ('CM', 900),
                                                                                                                                                                    ('D',  500),
                                                                                                                                                                    ('CD', 400),
                                                                                                                                                                    ('C',  100),
                                                                                                                                                                    ('XC', 90),
                                                                                                                                                                    ('L',  50),
                                                                                                                                                                    ('XL', 40),
                                                                                                                                                                    ('X',  10),
                                                                                                                                                                    ('IX', 9),
                                                                                                                                                                    ('V',  5),
                                                                                                                                                                    ('IV', 4),
                                                                                                                                                                    ('I',  1))

                                                                                                                                                                   def to_roman(n):
                                                                                                                                                                       """convert integer to Roman numeral"""
                                                                                                                                                                       if not (0 < n < 4000):
                                                                                                                                                                           raise OutOfRangeError, "number out of range (must be 1..3999)"
                                                                                                                                                                       if int(n) <> n:
                                                                                                                                                                           raise NotIntegerError, "non-integers can not be converted"

                                                                                                                                                                       result = ""
                                                                                                                                                                       for numeral, integer in romanNumeralMap:
                                                                                                                                                                           while n >= integer:
                                                                                                                                                                               result += numeral
                                                                                                                                                                               n -= integer
                                                                                                                                                                       return result

                                                                                                                                                                   #Define pattern to detect valid Roman numerals
                                                                                                                                                                   romanNumeralPattern = '^M?M?M?(CM|CD|D?C?C?C?)(XC|XL|L?X?X?X?)(IX|IV|V?I?I?I?)$' ①

                                                                                                                                                                   def from_roman(s):
                                                                                                                                                                       """convert Roman numeral to integer"""
                                                                                                                                                                       if not re.search(romanNumeralPattern, s):②
                                                                                                                                                                           raise InvalidRomanNumeralError, 'Invalid Roman numeral: %s' % s

                                                                                                                                                                       result = 0
                                                                                                                                                                       index = 0
                                                                                                                                                                       for numeral, integer in romanNumeralMap:
                                                                                                                                                                           while s[index:index+len(numeral)] == numeral:
                                                                                                                                                                               result += integer
                                                                                                                                                                               index += len(numeral)
                                                                                                                                                                       return result

                                                                                                                                                               1. This is just a continuation of the pattern you discussed in Section 7.3, “Case Study: Roman Numerals”. The tens places is either `XC` (`90`), `XL` (`40`), or an optional `L` followed by 0 to 3 optional `X` characters. The ones place is either `IX` (`9`), `IV` (`4`), or an optional `V` followed by 0 to 3 optional `I` characters.
                                                                                                                                                               2. Having encoded all that logic into a regular expression, the code to check for invalid Roman numerals becomes trivial. If `re.search` returns an object, then the regular expression matched and the input is valid; otherwise, the input is invalid.

                                                                                                                                                                  At this point, you are allowed to be skeptical that that big ugly regular expression could possibly catch all the types of invalid Roman numerals. But don't take my word for it, look at the results:

                                                                                                                                                                  ### Example 14.13. Output of `romantest5.py` against `roman5.py`

                                                                                                                                                                      from_roman should only accept uppercase input ... ok          ①
                                                                                                                                                                      to_roman should always return uppercase ... ok
                                                                                                                                                                      from_roman should fail with malformed antecedents ... ok      ②
                                                                                                                                                                      from_roman should fail with repeated pairs of numerals ... ok ③
                                                                                                                                                                      from_roman should fail with too many repeated numerals ... ok
                                                                                                                                                                      from_roman should give known result with known input ... ok
                                                                                                                                                                      to_roman should give known result with known input ... ok
                                                                                                                                                                      from_roman(to_roman(n))==n for all n ... ok
                                                                                                                                                                      to_roman should fail with non-integer input ... ok
                                                                                                                                                                      to_roman should fail with negative input ... ok
                                                                                                                                                                      to_roman should fail with large input ... ok
                                                                                                                                                                      to_roman should fail with 0 input ... ok

                                                                                                                                                                      ----------------------------------------------------------------------
                                                                                                                                                                      Ran 12 tests in 2.864s

                                                                                                                                                                      OK     ④

                                                                                                                                                                  1. One thing I didn't mention about regular expressions is that, by default, they are case-sensitive. Since the regular expression romanNumeralPattern was expressed in uppercase characters, the `re.search` check will reject any input that isn't completely uppercase. So the uppercase input test passes.
                                                                                                                                                                  2. More importantly, the bad input tests pass. For instance, the malformed antecedents test checks cases like `MCMC`. As you've seen, this does not match the regular expression, so `from_roman()` raises an `InvalidRomanNumeralError` exception, which is what the malformed antecedents test case is looking for, so the test passes.
                                                                                                                                                                  3. In fact, all the bad input tests pass. This regular expression catches everything you could think of when you made your test cases.
                                                                                                                                                                  4. And the anticlimax award of the year goes to the word “`OK`”, which is printed by the `unittest` module when all the tests pass.

                                                                                                                                                                     When all of your tests pass, stop coding.

                                                                                                                                                                     ## Chapter 16. Functional Programming

                                                                                                                                                                     ## 16.1. Diving in

                                                                                                                                                                     In Chapter 13, Unit Testing, you learned about the philosophy of unit testing. In Chapter 14, Test-First Programming, you stepped through the implementation of basic unit tests in Python. In Chapter 15, Refactoring, you saw how unit testing makes large-scale refactoring easier. This chapter will build on those sample programs, but here we will focus more on advanced Python-specific techniques, rather than on unit testing itself.

                                                                                                                                                                     The following is a complete Python program that acts as a cheap and simple regression testing framework. It takes unit tests that you've written for individual modules, collects them all into one big test suite, and runs them all at once. I actually use this script as part of the build process for this book; I have unit tests for several of the example programs (not just the `roman.py` module featured in Chapter 13, Unit Testing), and the first thing my automated build script does is run this program to make sure all my examples still work. If this regression test fails, the build immediately stops. I don't want to release non-working examples any more than you want to download them and sit around scratching your head and yelling at your monitor and wondering why they don't work.

                                                                                                                                                                     ### Example 16.1. `regression.py`

                                                                                                                                                                     If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                         """Regression testing framework

                                                                                                                                                                         This module will search for scripts in the same directory named
                                                                                                                                                                         XYZtest.py. Each such script should be a test suite that tests a
                                                                                                                                                                         module through PyUnit. (As of Python 2.1, PyUnit is included in
                                                                                                                                                                         the standard library as "unittest".)  This script will aggregate all
                                                                                                                                                                         found test suites into one big test suite and run them all at once.
                                                                                                                                                                         """

                                                                                                                                                                         import sys, os, re, unittest

                                                                                                                                                                         def regressionTest():
                                                                                                                                                                             path = os.path.abspath(os.path.dirname(sys.argv[0]))
                                                                                                                                                                             files = os.listdir(path)
                                                                                                                                                                             test = re.compile("test\.py$", re.IGNORECASE)
                                                                                                                                                                             files = filter(test.search, files)
                                                                                                                                                                             filenameToModuleName = lambda f: os.path.splitext(f)[0]
                                                                                                                                                                             moduleNames = map(filenameToModuleName, files)
                                                                                                                                                                             modules = map(__import__, moduleNames)
                                                                                                                                                                             load = unittest.defaultTestLoader.loadTestsFromModule
                                                                                                                                                                             return unittest.TestSuite(map(load, modules))

                                                                                                                                                                         if __name__ == "__main__":
                                                                                                                                                                             unittest.main(defaultTest="regressionTest")

                                                                                                                                                                     Running this script in the same directory as the rest of the example scripts that come with this book will find all the unit tests, named ``module`test.py`, run them as a single test, and pass or fail them all at once.

                                                                                                                                                                     ### Example 16.2. Sample output of `regression.py`

                                                                                                                                                                         [you@localhost py]$ python regression.py -v
                                                                                                                                                                         help should fail with no object ... ok           ①
                                                                                                                                                                         help should return known result for apihelper ... ok
                                                                                                                                                                         help should honor collapse argument ... ok
                                                                                                                                                                         help should honor spacing argument ... ok
                                                                                                                                                                         buildConnectionString should fail with list input ... ok           ②
                                                                                                                                                                         buildConnectionString should fail with string input ... ok
                                                                                                                                                                         buildConnectionString should fail with tuple input ... ok
                                                                                                                                                                         buildConnectionString handles empty dictionary ... ok
                                                                                                                                                                         buildConnectionString returns known result with known input ... ok
                                                                                                                                                                         from_roman should only accept uppercase input ... ok                ③
                                                                                                                                                                         to_roman should always return uppercase ... ok
                                                                                                                                                                         from_roman should fail with blank string ... ok
                                                                                                                                                                         from_roman should fail with malformed antecedents ... ok
                                                                                                                                                                         from_roman should fail with repeated pairs of numerals ... ok
                                                                                                                                                                         from_roman should fail with too many repeated numerals ... ok
                                                                                                                                                                         from_roman should give known result with known input ... ok
                                                                                                                                                                         to_roman should give known result with known input ... ok
                                                                                                                                                                         from_roman(to_roman(n))==n for all n ... ok
                                                                                                                                                                         to_roman should fail with non-integer input ... ok
                                                                                                                                                                         to_roman should fail with negative input ... ok
                                                                                                                                                                         to_roman should fail with large input ... ok
                                                                                                                                                                         to_roman should fail with 0 input ... ok
                                                                                                                                                                         kgp a ref test ... ok
                                                                                                                                                                         kgp b ref test ... ok
                                                                                                                                                                         kgp c ref test ... ok
                                                                                                                                                                         kgp d ref test ... ok
                                                                                                                                                                         kgp e ref test ... ok
                                                                                                                                                                         kgp f ref test ... ok
                                                                                                                                                                         kgp g ref test ... ok

                                                                                                                                                                         ----------------------------------------------------------------------
                                                                                                                                                                         Ran 29 tests in 2.799s

                                                                                                                                                                         OK

                                                                                                                                                                     1. The first 5 tests are from `apihelpertest.py`, which tests the example script from Chapter 4, The Power Of Introspection.
                                                                                                                                                                     2. The next 5 tests are from `odbchelpertest.py`, which tests the example script from Chapter 2, Your First Python Program.
                                                                                                                                                                     3. The rest are from `romantest.py`, which you studied in depth in Chapter 13, Unit Testing.

                                                                                                                                                                        ## 16.2. Finding the path

                                                                                                                                                                        When running Python scripts from the command line, it is sometimes useful to know where the currently running script is located on disk.

                                                                                                                                                                        This is one of those obscure little tricks that is virtually impossible to figure out on your own, but simple to remember once you see it. The key to it is `sys.argv`. As you saw in Chapter 9, XML Processing, this is a list that holds the list of command-line arguments. However, it also holds the name of the running script, exactly as it was called from the command line, and this is enough information to determine its location.

                                                                                                                                                                        ### Example 16.3. `fullpath.py`

                                                                                                                                                                        If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                            import sys, os

                                                                                                                                                                            print 'sys.argv[0] =', sys.argv[0]             ①
                                                                                                                                                                            pathname = os.path.dirname(sys.argv[0])        ②
                                                                                                                                                                            print 'path =', pathname
                                                                                                                                                                            print 'full path =', os.path.abspath(pathname) ③

                                                                                                                                                                        1. Regardless of how you run a script, `sys.argv[0]` will always contain the name of the script, exactly as it appears on the command line. This may or may not include any path information, as you'll see shortly.
                                                                                                                                                                        2. `os.path.dirname` takes a filename as a string and returns the directory path portion. If the given filename does not include any path information, `os.path.dirname` returns an empty string.
                                                                                                                                                                        3. `os.path.abspath` is the key here. It takes a pathname, which can be partial or even blank, and returns a fully qualified pathname.

                                                                                                                                                                           `os.path.abspath` deserves further explanation. It is very flexible; it can take any kind of pathname.

                                                                                                                                                                           ### Example 16.4. Further explanation of `os.path.abspath

                                                                                                                                                                               >>> import os
                                                                                                                                                                               >>> os.getcwd()      ①
                                                                                                                                                                               /home/you
                                                                                                                                                                               >>> os.path.abspath('')                ②
                                                                                                                                                                               /home/you
                                                                                                                                                                               >>> os.path.abspath('.ssh')            ③
                                                                                                                                                                               /home/you/.ssh
                                                                                                                                                                               >>> os.path.abspath('/home/you/.ssh') ④
                                                                                                                                                                               /home/you/.ssh
                                                                                                                                                                               >>> os.path.abspath('.ssh/../foo/')    ⑤
                                                                                                                                                                               /home/you/foo

                                                                                                                                                                           `os.getcwd() returns the current working directory.
                                                                                                                                                                           Calling `os.path.abspath with an empty string returns the current working directory, same as `os.getcwd().
                                                                                                                                                                           Calling `os.path.abspath with a partial pathname constructs a fully qualified pathname out of it, based on the current working directory.
                                                                                                                                                                           Calling `os.path.abspath with a full pathname simply returns it.
                                                                                                                                                                           `os.path.abspath also normalizes the pathname it returns. Note that this example worked even though I don't actually have a 'foo' directory. `os.path.abspath never checks your actual disk; this is all just string manipulation.

                                                                                                                                                                     The pathnames and filenames you pass to `os.path.abspath` do not need to exist.

                                                                                                                                                                     `os.path.abspath` not only constructs full path names, it also normalizes them. That means that if you are in the `/usr/` directory, `os.path.abspath('bin/../local/bin')` will return `/usr/local/bin`. It normalizes the path by making it as simple as possible. If you just want to normalize a pathname like this without turning it into a full pathname, use `os.path.normpath` instead.

                                                                                                                                                                     ### Example 16.5. Sample output from `fullpath.py`

                                                                                                                                                                         [you@localhost py]$ python /home/you/diveintopython3/common/py/fullpath.py ①
                                                                                                                                                                         sys.argv[0] = /home/you/diveintopython3/common/py/fullpath.py
                                                                                                                                                                         path = /home/you/diveintopython3/common/py
                                                                                                                                                                         full path = /home/you/diveintopython3/common/py
                                                                                                                                                                         [you@localhost diveintopython3]$ python common/py/fullpath.py               ②
                                                                                                                                                                         sys.argv[0] = common/py/fullpath.py
                                                                                                                                                                         path = common/py
                                                                                                                                                                         full path = /home/you/diveintopython3/common/py
                                                                                                                                                                         [you@localhost diveintopython3]$ cd common/py
                                                                                                                                                                         [you@localhost py]$ python fullpath.py ③
                                                                                                                                                                         sys.argv[0] = fullpath.py
                                                                                                                                                                         path =
                                                                                                                                                                         full path = /home/you/diveintopython3/common/py

                                                                                                                                                                     1. In the first case, `sys.argv[0]` includes the full path of the script. You can then use the `os.path.dirname` function to strip off the script name and return the full directory name, and `os.path.abspath` simply returns what you give it.
                                                                                                                                                                     2. If the script is run by using a partial pathname, `sys.argv[0]` will still contain exactly what appears on the command line. `os.path.dirname` will then give you a partial pathname (relative to the current directory), and `os.path.abspath` will construct a full pathname from the partial pathname.
                                                                                                                                                                     3. If the script is run from the current directory without giving any path, `os.path.dirname` will simply return an empty string. Given an empty string, `os.path.abspath` returns the current directory, which is what you want, since the script was run from the current directory.

                                                                                                                                                                        Like the other functions in the `os` and `os.path` modules, `os.path.abspath` is cross-platform. Your results will look slightly different than my examples if you're running on Windows (which uses backslash as a path separator) or Mac OS (which uses colons), but they'll still work. That's the whole point of the `os` module.

                                                                                                                                                                        Addendum. One reader was dissatisfied with this solution, and wanted to be able to run all the unit tests in the current directory, not the directory where `regression.py` is located. He suggests this approach instead:

                                                                                                                                                                        ### Example 16.6. Running scripts in the current directory

                                                                                                                                                                            import sys, os, re, unittest

                                                                                                                                                                            def regressionTest():
                                                                                                                                                                                path = os.getcwd()       ①
                                                                                                                                                                                sys.path.append(path)    ②
                                                                                                                                                                                files = os.listdir(path) ③

                                                                                                                                                                        1. Instead of setting path to the directory where the currently running script is located, you set it to the current working directory instead. This will be whatever directory you were in before you ran the script, which is not necessarily the same as the directory the script is in. (Read that sentence a few times until you get it.)
                                                                                                                                                                        2. Append this directory to the Python library search path, so that when you dynamically import the unit test modules later, Python can find them. You didn't need to do this when path was the directory of the currently running script, because Python always looks in that directory.
                                                                                                                                                                        3. The rest of the function is the same.

                                                                                                                                                                           This technique will allow you to re-use this `regression.py script on multiple projects. Just put the script in a common directory, then change to the project's directory before running it. All of that project's unit tests will be found and tested, instead of the unit tests in the common directory where `regression.py is located.

                                                                                                                                                                           ## 16.3. Filtering lists revisited

                                                                                                                                                                           You're already familiar with using list comprehensions to filter lists. There is another way to accomplish this same thing, which some people feel is more expressive.

                                                                                                                                                                           Python has a built-in `filter function which takes two arguments, a function and a list, and returns a list. [7] The function passed as the first argument to `filter must itself take one argument, and the list that `filter returns will contain all the elements from the list passed to `filter for which the function passed to `filter returns true.

                                                                                                                                                                           Got all that? It's not as difficult as it sounds.

                                                                                                                                                                           ### Example 16.7. Introducing `filter

                                                                                                                                                                           `>>> `def odd(n): ① `... return n % 2 `... `>>> `li = [1, 2, 3, 5, 9, 10, 256, -3] `>>> `filter(odd, li) ② [1, 3, 5, 9, -3] `>>> `[e for e in li if odd(e)] ③ `>>> `filteredList = [] `>>> `for n in li: ④ `... if odd(n): `... filteredList.append(n) `... `>>> `filteredList [1, 3, 5, 9, -3]
                                                                                                                                                                        4. `odd uses the built-in mod function “`%” to return `True if n is odd and `False if n is even.
                                                                                                                                                                        5. `filter takes two arguments, a function (`odd) and a list (li). It loops through the list and calls `odd with each element. If `odd returns a true value (remember, any non-zero value is true in Python), then the element is included in the returned list, otherwise it is filtered out. The result is a list of only the odd numbers from the original list, in the same order as they appeared in the original.
                                                                                                                                                                        6. You could accomplish the same thing using list comprehensions, as you saw in Section 4.5, “Filtering Lists”.
                                                                                                                                                                        7. You could also accomplish the same thing with a `for loop. Depending on your programming background, this may seem more “straightforward”, but functions like `filter are much more expressive. Not only is it easier to write, it's easier to read, too. Reading the `for loop is like standing too close to a painting; you see all the details, but it may take a few seconds to be able to step back and see the bigger picture: “Oh, you're just filtering the list!”

                                                                                                                                                                           ### Example 16.8. `filter in `regression.py

                                                                                                                                                                           `files = os.listdir(path) ① test = re.compile("test\.py$", re.IGNORECASE) ② files = filter(test.search, files) ③
                                                                                                                                                                        8. As you saw in Section 16.2, “Finding the path”, path may contain the full or partial pathname of the directory of the currently running script, or it may contain an empty string if the script is being run from the current directory. Either way, files will end up with the names of the files in the same directory as this script you're running.
                                                                                                                                                                        9. This is a compiled regular expression. As you saw in Section 15.3, “Refactoring”, if you're going to use the same regular expression over and over, you should compile it for faster performance. The compiled object has a `search method which takes a single argument, the string to search. If the regular expression matches the string, the `search method returns a `Match object containing information about the regular expression match; otherwise it returns `None, the Python null value.
                                                                                                                                                                        10. For each element in the files list, you're going to call the `search method of the compiled regular expression object, test. If the regular expression matches, the method will return a `Match object, which Python considers to be true, so the element will be included in the list returned by `filter. If the regular expression does not match, the `search method will return `None, which Python considers to be false, so the element will not be included.

                                                                                                                                                                            Historical note. Versions of Python prior to 2.0 did not have list comprehensions, so you couldn't filter using list comprehensions; the `filter function was the only game in town. Even with the introduction of list comprehensions in 2.0, some people still prefer the old-style `filter (and its companion function, `map, which you'll see later in this chapter). Both techniques work at the moment, so which one you use is a matter of style. There is discussion that `map and `filter might be deprecated in a future version of Python, but no decision has been made.

                                                                                                                                                                            ### Example 16.9. Filtering using list comprehensions instead

                                                                                                                                                                            `files = os.listdir(path) test = re.compile("test\.py$", re.IGNORECASE) files = [f for f in files if test.search(f)] ①
                                                                                                                                                                        11. This will accomplish exactly the same result as using the `filter function. Which way is more expressive? That's up to you.

                                                                                                                                                                            ## 16.4. Mapping lists revisited

                                                                                                                                                                            You're already familiar with using list comprehensions to map one list into another. There is another way to accomplish the same thing, using the built-in `map function. It works much the same way as the `filter function.

                                                                                                                                                                            ### Example 16.10. Introducing `map

                                                                                                                                                                            `>>> `def double(n): `... return n*2 `... `>>> `li = [1, 2, 3, 5, 9, 10, 256, -3] `>>> `map(double, li) ① [2, 4, 6, 10, 18, 20, 512, -6] `>>> `[double(n) for n in li] ② [2, 4, 6, 10, 18, 20, 512, -6] `>>> `newlist = [] `>>> `for n in li: ③ `... newlist.append(double(n)) `... `>>> `newlist [2, 4, 6, 10, 18, 20, 512, -6]
                                                                                                                                                                        12. `map takes a function and a list[8] and returns a new list by calling the function with each element of the list in order. In this case, the function simply multiplies each element by 2.
                                                                                                                                                                        13. You could accomplish the same thing with a list comprehension. List comprehensions were first introduced in Python 2.0; `map has been around forever.
                                                                                                                                                                        14. You could, if you insist on thinking like a Visual Basic programmer, use a `for loop to accomplish the same thing.

                                                                                                                                                                            ### Example 16.11. `map with lists of mixed datatypes

                                                                                                                                                                            `>>> `li = [5, 'a', (2, 'b')] `>>> `map(double, li) ① [10, 'aa', (2, 'b', 2, 'b')]
                                                                                                                                                                        15. As a side note, I'd like to point out that `map works just as well with lists of mixed datatypes, as long as the function you're using correctly handles each type. In this case, the `double function simply multiplies the given argument by 2, and Python Does The Right Thing depending on the datatype of the argument. For integers, this means actually multiplying it by 2; for strings, it means concatenating the string with itself; for tuples, it means making a new tuple that has all of the elements of the original, then all of the elements of the original again.

                                                                                                                                                                            All right, enough play time. Let's look at some real code.

                                                                                                                                                                            ### Example 16.12. `map in `regression.py

                                                                                                                                                                            `filenameToModuleName = lambda f: os.path.splitext(f)[0] ① moduleNames = map(filenameToModuleName, files) ②
                                                                                                                                                                        16. As you saw in Section 4.7, “Using lambda Functions”, `lambda defines an inline function. And as you saw in Example 6.17, “Splitting Pathnames”, `os.path.splitext takes a filename and returns a tuple `(name, extension). So `filenameToModuleName is a function which will take a filename and strip off the file extension, and return just the name.
                                                                                                                                                                        17. Calling `map takes each filename listed in files, passes it to the function `filenameToModuleName, and returns a list of the return values of each of those function calls. In other words, you strip the file extension off of each filename, and store the list of all those stripped filenames in moduleNames.

                                                                                                                                                                            As you'll see in the rest of the chapter, you can extend this type of data-centric thinking all the way to the final goal, which is to define and execute a single test suite that contains the tests from all of those individual test suites.

                                                                                                                                                                            ## 16.5. Data-centric programming

                                                                                                                                                                            By now you're probably scratching your head wondering why this is better than using `for loops and straight function calls. And that's a perfectly valid question. Mostly, it's a matter of perspective. Using `map and `filter forces you to center your thinking around your data.

                                                                                                                                                                            In this case, you started with no data at all; the first thing you did was get the directory path of the current script, and got a list of files in that directory. That was the bootstrap, and it gave you real data to work with: a list of filenames.

                                                                                                                                                                            However, you knew you didn't care about all of those files, only the ones that were actually test suites. You had too much data, so you needed to `filter it. How did you know which data to keep? You needed a test to decide, so you defined one and passed it to the `filter function. In this case you used a regular expression to decide, but the concept would be the same regardless of how you constructed the test.

                                                                                                                                                                            Now you had the filenames of each of the test suites (and only the test suites, since everything else had been filtered out), but you really wanted module names instead. You had the right amount of data, but it was in the wrong format. So you defined a function that would transform a single filename into a module name, and you mapped that function onto the entire list. From one filename, you can get a module name; from a list of filenames, you can get a list of module names.

                                                                                                                                                                            Instead of `filter, you could have used a `for loop with an `if statement. Instead of `map, you could have used a `for loop with a function call. But using `for loops like that is busywork. At best, it simply wastes time; at worst, it introduces obscure bugs. For instance, you need to figure out how to test for the condition “is this file a test suite?” anyway; that's the application-specific logic, and no language can write that for us. But once you've figured that out, do you really want go to all the trouble of defining a new empty list and writing a `for loop and an `if statement and manually calling `append to add each element to the new list if it passes the condition and then keeping track of which variable holds the new filtered data and which one holds the old unfiltered data? Why not just define the test condition, then let Python do the rest of that work for us?

                                                                                                                                                                            Oh sure, you could try to be fancy and delete elements in place without creating a new list. But you've been burned by that before. Trying to modify a data structure that you're looping through can be tricky. You delete an element, then loop to the next element, and suddenly you've skipped one. Is Python one of the languages that works that way? How long would it take you to figure it out? Would you remember for certain whether it was safe the next time you tried? Programmers spend so much time and make so many mistakes dealing with purely technical issues like this, and it's all pointless. It doesn't advance your program at all; it's just busywork.

                                                                                                                                                                            I resisted list comprehensions when I first learned Python, and I resisted `filter and `map even longer. I insisted on making my life more difficult, sticking to the familiar way of `for loops and `if statements and step-by-step code-centric programming. And my Python programs looked a lot like Visual Basic programs, detailing every step of every operation in every function. And they had all the same types of little problems and obscure bugs. And it was all pointless.

                                                                                                                                                                            Let it all go. Busywork code is not important. Data is important. And data is not difficult. It's only data. If you have too much, filter it. If it's not what you want, map it. Focus on the data; leave the busywork behind.

                                                                                                                                                                            ## 16.6. Dynamically importing modules

                                                                                                                                                                            OK, enough philosophizing. Let's talk about dynamically importing modules.

                                                                                                                                                                            First, let's look at how you normally import modules. The `import module syntax looks in the search path for the named module and imports it by name. You can even import multiple modules at once this way, with a comma-separated list. You did this on the very first line of this chapter's script.

                                                                                                                                                                            ### Example 16.13. Importing multiple modules at once

                                                                                                                                                                            `import sys, os, re, unittest ①
                                                                                                                                                                        18. This imports four modules at once: `sys (for system functions and access to the command line parameters), `os (for operating system functions like directory listings), `re (for regular expressions), and `unittest (for unit testing).

                                                                                                                                                                            Now let's do the same thing, but with dynamic imports.

                                                                                                                                                                            ### Example 16.14. Importing modules dynamically

                                                                                                                                                                            `>>> `sys = __import__('sys') ① `>>> `os = __import__('os') `>>> `re = __import__('re') `>>> `unittest = __import__('unittest') `>>> `sys ② `>>> `<module 'sys' (built-in)> `>>> `os `>>> `<module 'os' from '/usr/local/lib/python2.2/os.pyc'>
                                                                                                                                                                        19. The built-in `__import__ function accomplishes the same goal as using the `import statement, but it's an actual function, and it takes a string as an argument.
                                                                                                                                                                        20. The variable sys is now the `sys module, just as if you had said `import sys. The variable os is now the `os module, and so forth.

                                                                                                                                                                            So `__import__ imports a module, but takes a string argument to do it. In this case the module you imported was just a hard-coded string, but it could just as easily be a variable, or the result of a function call. And the variable that you assign the module to doesn't need to match the module name, either. You could import a series of modules and assign them to a list.

                                                                                                                                                                            ### Example 16.15. Importing a list of modules dynamically

                                                                                                                                                                            `>>> `moduleNames = ['sys', 'os', 're', 'unittest'] ① `>>> `moduleNames ['sys', 'os', 're', 'unittest'] `>>> `modules = map(__import__, moduleNames) ② `>>> `modules ③ `[<module 'sys' (built-in)>, <module 'os' from 'c:\Python22\lib\os.pyc'>, <module 're' from 'c:\Python22\lib\re.pyc'>, <module 'unittest' from 'c:\Python22\lib\unittest.pyc'>] `>>> `modules[0].version ④ '2.2.2 (#37, Nov 26 2002, 10:24:37) [MSC 32 bit (Intel)]' `>>> `import sys `>>> `sys.version '2.2.2 (#37, Nov 26 2002, 10:24:37) [MSC 32 bit (Intel)]'
                                                                                                                                                                        21. moduleNames is just a list of strings. Nothing fancy, except that the strings happen to be names of modules that you could import, if you wanted to.
                                                                                                                                                                        22. Surprise, you wanted to import them, and you did, by mapping the `__import__ function onto the list. Remember, this takes each element of the list (moduleNames) and calls the function (`__import__) over and over, once with each element of the list, builds a list of the return values, and returns the result.
                                                                                                                                                                        23. So now from a list of strings, you've created a list of actual modules. (Your paths may be different, depending on your operating system, where you installed Python, the phase of the moon, etc.)
                                                                                                                                                                        24. To drive home the point that these are real modules, let's look at some module attributes. Remember, modules[0] is the `sys module, so modules[0].version is sys.version. All the other attributes and methods of these modules are also available. There's nothing magic about the `import statement, and there's nothing magic about modules. Modules are objects. Everything is an object.

                                                                                                                                                                            Now you should be able to put this all together and figure out what most of this chapter's code sample is doing.

                                                                                                                                                                            ## 16.7. Putting it all together

                                                                                                                                                                            You've learned enough now to deconstruct the first seven lines of this chapter's code sample: reading a directory and importing selected modules within it.

                                                                                                                                                                            ### Example 16.16. The `regressionTest function

                                                                                                                                                                            `def regressionTest(): path = os.path.abspath(os.path.dirname(sys.argv[0])) files = os.listdir(path) test = re.compile("test\.py$", re.IGNORECASE) files = filter(test.search, files) filenameToModuleName = lambda f: os.path.splitext(f)[0] moduleNames = map(filenameToModuleName, files) modules = map(__import__, moduleNames) load = unittest.defaultTestLoader.loadTestsFromModule return unittest.TestSuite(map(load, modules))

                                                                                                                                                                            Let's look at it line by line, interactively. Assume that the current directory is `c:\diveintopython3\py, which contains the examples that come with this book, including this chapter's script. As you saw in Section 16.2, “Finding the path”, the script directory will end up in the path variable, so let's start hard-code that and go from there.

                                                                                                                                                                            ### Example 16.17. Step 1: Get all the files

                                                                                                                                                                            `>>> `import sys, os, re, unittest `>>> `path = r'c:\diveintopython3\py' `>>> `files = os.listdir(path) `>>> `files ① `['BaseHTMLProcessor.py', 'LICENSE.txt', 'apihelper.py', 'apihelpertest.py', 'argecho.py', 'autosize.py', 'builddialectexamples.py', 'dialect.py', 'fileinfo.py', 'fullpath.py', 'kgptest.py', 'makerealworddoc.py', 'odbchelper.py', 'odbchelpertest.py', 'parsephone.py', 'piglatin.py', 'plural.py', 'pluraltest.py', 'pyfontify.py', 'regression.py', 'roman.py', 'romantest.py', 'uncurly.py', 'unicode2koi8r.py', 'urllister.py', 'kgp', 'plural', 'roman', 'colorize.py']
                                                                                                                                                                        25. files is a list of all the files and directories in the script's directory. (If you've been running some of the examples already, you may also see some `.pyc files in there as well.)

                                                                                                                                                                            ### Example 16.18. Step 2: Filter to find the files you care about

                                                                                                                                                                            `>>> `test = re.compile("test\.py$", re.IGNORECASE) ① `>>> `files = filter(test.search, files) ② `>>> `files ③ ['apihelpertest.py', 'kgptest.py', 'odbchelpertest.py', 'pluraltest.py', 'romantest.py']
                                                                                                                                                                        26. This regular expression will match any string that ends with `test.py. Note that you need to escape the period, since a period in a regular expression usually means “match any single character”, but you actually want to match a literal period instead.
                                                                                                                                                                        27. The compiled regular expression acts like a function, so you can use it to filter the large list of files and directories, to find the ones that match the regular expression.
                                                                                                                                                                        28. And you're left with the list of unit testing scripts, because they were the only ones named `SOMETHINGtest.py.

                                                                                                                                                                            ### Example 16.19. Step 3: Map filenames to module names

                                                                                                                                                                            `>>> `filenameToModuleName = lambda f: os.path.splitext(f)[0] ① `>>> `filenameToModuleName('romantest.py') ② 'romantest' `>>> `filenameToModuleName('odchelpertest.py') 'odbchelpertest' `>>> `moduleNames = map(filenameToModuleName, files) ③ `>>> `moduleNames ④ ['apihelpertest', 'kgptest', 'odbchelpertest', 'pluraltest', 'romantest']
                                                                                                                                                                        29. As you saw in Section 4.7, “Using lambda Functions”, `lambda is a quick-and-dirty way of creating an inline, one-line function. This one takes a filename with an extension and returns just the filename part, using the standard library function `os.path.splitext that you saw in Example 6.17, “Splitting Pathnames”.
                                                                                                                                                                        30. filenameToModuleName is a function. There's nothing magic about `lambda functions as opposed to regular functions that you define with a `def statement. You can call the filenameToModuleName function like any other, and it does just what you wanted it to do: strips the file extension off of its argument.
                                                                                                                                                                        31. Now you can apply this function to each file in the list of unit test files, using `map.
                                                                                                                                                                        32. And the result is just what you wanted: a list of modules, as strings.

                                                                                                                                                                            ### Example 16.20. Step 4: Mapping module names to modules

                                                                                                                                                                            `>>> `modules = map(__import__, moduleNames)① `>>> `modules ② `[<module 'apihelpertest' from 'apihelpertest.py'>, <module 'kgptest' from 'kgptest.py'>, <module 'odbchelpertest' from 'odbchelpertest.py'>, <module 'pluraltest' from 'pluraltest.py'>, <module 'romantest' from 'romantest.py'>] `>>> `modules[-1] ③ <module 'romantest' from 'romantest.py'>
                                                                                                                                                                        33. As you saw in Section 16.6, “Dynamically importing modules”, you can use a combination of `map and `__import__ to map a list of module names (as strings) into actual modules (which you can call or access like any other module).
                                                                                                                                                                        34. modules is now a list of modules, fully accessible like any other module.
                                                                                                                                                                        35. The last module in the list is the `romantest module, just as if you had said `import romantest.

                                                                                                                                                                            ### Example 16.21. Step 5: Loading the modules into a test suite

                                                                                                                                                                            `>>> `load = unittest.defaultTestLoader.loadTestsFromModule `>>> `map(load, modules) ① `[<unittest.TestSuite tests=[ <unittest.TestSuite tests=[<apihelpertest.BadInput testMethod=testNoObject>]>, <unittest.TestSuite tests=[<apihelpertest.KnownValues testMethod=testApiHelper>]>, <unittest.TestSuite tests=[ <apihelpertest.ParamChecks testMethod=testCollapse>, <apihelpertest.ParamChecks testMethod=testSpacing>]>, ... ] ] `>>> `unittest.TestSuite(map(load, modules)) ②
                                                                                                                                                                        36. These are real module objects. Not only can you access them like any other module, instantiate classes and call functions, you can also introspect into the module to figure out which classes and functions it has in the first place. That's what the `loadTestsFromModule method does: it introspects into each module and returns a `unittest.TestSuite object for each module. Each `TestSuite object actually contains a list of `TestSuite objects, one for each `TestCase class in your module, and each of those `TestSuite objects contains a list of tests, one for each test method in your module.
                                                                                                                                                                        37. Finally, you wrap the list of `TestSuite objects into one big test suite. The `unittest module has no problem traversing this tree of nested test suites within test suites; eventually it gets down to an individual test method and executes it, verifies that it passes or fails, and moves on to the next one.

                                                                                                                                                                            This introspection process is what the `unittest module usually does for us. Remember that magic-looking `unittest.main() function that our individual test modules called to kick the whole thing off? `unittest.main() actually creates an instance of `unittest.TestProgram, which in turn creates an instance of a `unittest.defaultTestLoader and loads it up with the module that called it. (How does it get a reference to the module that called it if you don't give it one? By using the equally-magic `__import__('__main__') command, which dynamically imports the currently-running module. I could write a book on all the tricks and techniques used in the `unittest module, but then I'd never finish this one.)

                                                                                                                                                                            ### Example 16.22. Step 6: Telling `unittest to use your test suite

                                                                                                                                                                            `if __name__ == "__main__": unittest.main(defaultTest="regressionTest") ①
                                                                                                                                                                        38. Instead of letting the `unittest module do all its magic for us, you've done most of it yourself. You've created a function (`regressionTest) that imports the modules yourself, calls `unittest.defaultTestLoader yourself, and wraps it all up in a test suite. Now all you need to do is tell `unittest that, instead of looking for tests and building a test suite in the usual way, it should just call the `regressionTest function, which returns a ready-to-use `TestSuite.

                                                                                                                                                                            ## 16.8. Summary

                                                                                                                                                                            The `regression.py program and its output should now make perfect sense.

                                                                                                                                                                            You should now feel comfortable doing all of these things:
                                                                                                                                                                        39. Manipulating path information from the command line.
                                                                                                                                                                        40. Filtering lists using `filter instead of list comprehensions.
                                                                                                                                                                        41. Mapping lists using `map instead of list comprehensions.
                                                                                                                                                                        42. Dynamically importing modules.

                                                                                                                                                                            [7] Technically, the second argument to `filter can be any sequence, including lists, tuples, and custom classes that act like lists by defining the `__getitem__ special method. If possible, `filter will return the same datatype as you give it, so filtering a list returns a list, but filtering a tuple returns a tuple.

                                                                                                                                                                            [8] Again, I should point out that `map can take a list, a tuple, or any object that acts like a sequence. See previous footnote about `filter.

                                                                                                                                                                            ## Chapter 18. Performance Tuning

                                                                                                                                                                            Performance tuning is a many-splendored thing. Just because Python is an interpreted language doesn't mean you shouldn't worry about code optimization. But don't worry about it too much.

                                                                                                                                                                            ## 18.1. Diving in

                                                                                                                                                                            There are so many pitfalls involved in optimizing your code, it's hard to know where to start.

                                                                                                                                                                            Let's start here: are you sure you need to do it at all? Is your code really so bad? Is it worth the time to tune it? Over the lifetime of your application, how much time is going to be spent running that code, compared to the time spent waiting for a remote database server, or waiting for user input?

                                                                                                                                                                            Second, are you sure you're done coding? Premature optimization is like spreading frosting on a half-baked cake. You spend hours or days (or more) optimizing your code for performance, only to discover it doesn't do what you need it to do. That's time down the drain.

                                                                                                                                                                            This is not to say that code optimization is worthless, but you need to look at the whole system and decide whether it's the best use of your time. Every minute you spend optimizing code is a minute you're not spending adding new features, or writing documentation, or playing with your kids, or writing unit tests.

                                                                                                                                                                            Oh yes, unit tests. It should go without saying that you need a complete set of unit tests before you begin performance tuning. The last thing you need is to introduce new bugs while fiddling with your algorithms.

                                                                                                                                                                            With these caveats in place, let's look at some techniques for optimizing Python code. The code in question is an implementation of the Soundex algorithm. Soundex was a method used in the early 20th century for categorizing surnames in the United States census. It grouped similar-sounding names together, so even if a name was misspelled, researchers had a chance of finding it. Soundex is still used today for much the same reason, although of course we use computerized database servers now. Most database servers include a Soundex function.

                                                                                                                                                                            There are several subtle variations of the Soundex algorithm. This is the one used in this chapter:
                                                                                                                                                                        43. Keep the first letter of the name as-is.
                                                                                                                                                                        44. Convert the remaining letters to digits, according to a specific table:
                                                                                                                                                                        45. B, F, P, and V become 1.
                                                                                                                                                                        46. C, G, J, K, Q, S, X, and Z become 2.
                                                                                                                                                                        47. D and T become 3.
                                                                                                                                                                        48. L becomes 4.
                                                                                                                                                                        49. M and N become 5.
                                                                                                                                                                        50. R becomes 6.
                                                                                                                                                                        51. All other letters become 9.
                                                                                                                                                                        52. Remove consecutive duplicates.
                                                                                                                                                                        53. Remove all 9s altogether.
                                                                                                                                                                        54. If the result is shorter than four characters (the first letter plus three digits), pad the result with trailing zeros.
                                                                                                                                                                        55. if the result is longer than four characters, discard everything after the fourth character.

                                                                                                                                                                        For example, my name, `Pilgrim`, becomes P942695. That has no consecutive duplicates, so nothing to do there. Then you remove the 9s, leaving P4265. That's too long, so you discard the excess character, leaving P426.

                                                                                                                                                                        Another example: `Woo` becomes W99, which becomes W9, which becomes W, which gets padded with zeros to become W000.

                                                                                                                                                                        Here's a first attempt at a Soundex function:

                                                                                                                                                                        ### Example 18.1. `soundex/stage1/soundex1a.py

                                                                                                                                                                        If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                            import string, re

                                                                                                                                                                            charToSoundex = {"A": "9",
                                                                                                                                                                                             "B": "1",
                                                                                                                                                                                             "C": "2",
                                                                                                                                                                                             "D": "3",
                                                                                                                                                                                             "E": "9",
                                                                                                                                                                                             "F": "1",
                                                                                                                                                                                             "G": "2",
                                                                                                                                                                                             "H": "9",
                                                                                                                                                                                             "I": "9",
                                                                                                                                                                                             "J": "2",
                                                                                                                                                                                             "K": "2",
                                                                                                                                                                                             "L": "4",
                                                                                                                                                                                             "M": "5",
                                                                                                                                                                                             "N": "5",
                                                                                                                                                                                             "O": "9",
                                                                                                                                                                                             "P": "1",
                                                                                                                                                                                             "Q": "2",
                                                                                                                                                                                             "R": "6",
                                                                                                                                                                                             "S": "2",
                                                                                                                                                                                             "T": "3",
                                                                                                                                                                                             "U": "9",
                                                                                                                                                                                             "V": "1",
                                                                                                                                                                                             "W": "9",
                                                                                                                                                                                             "X": "2",
                                                                                                                                                                                             "Y": "9",
                                                                                                                                                                                             "Z": "2"}

                                                                                                                                                                            def soundex(source):
                                                                                                                                                                                "convert string to Soundex equivalent"

                                                                                                                                                                                # Soundex requirements:
                                                                                                                                                                                # source string must be at least 1 character
                                                                                                                                                                                # and must consist entirely of letters
                                                                                                                                                                                allChars = string.uppercase + string.lowercase
                                                                                                                                                                                if not re.search('^[%s]+$' % allChars, source):
                                                                                                                                                                                    return "0000"

                                                                                                                                                                                # Soundex algorithm:
                                                                                                                                                                                # 1. make first character uppercase
                                                                                                                                                                                source = source[0].upper() + source[1:]

                                                                                                                                                                                # 2. translate all other characters to Soundex digits
                                                                                                                                                                                digits = source[0]
                                                                                                                                                                                for s in source[1:]:
                                                                                                                                                                                    s = s.upper()
                                                                                                                                                                                    digits += charToSoundex[s]

                                                                                                                                                                                # 3. remove consecutive duplicates
                                                                                                                                                                                digits2 = digits[0]
                                                                                                                                                                                for d in digits[1:]:
                                                                                                                                                                                    if digits2[-1] != d:
                                                                                                                                                                                        digits2 += d

                                                                                                                                                                                # 4. remove all "9"s
                                                                                                                                                                                digits3 = re.sub('9', '', digits2)

                                                                                                                                                                                # 5. pad end with "0"s to 4 characters
                                                                                                                                                                                while len(digits3) < 4:
                                                                                                                                                                                    digits3 += "0"

                                                                                                                                                                                # 6. return first 4 characters
                                                                                                                                                                                return digits3[:4]

                                                                                                                                                                            if __name__ == '__main__':
                                                                                                                                                                                from timeit import Timer
                                                                                                                                                                                names = ('Woo', 'Pilgrim', 'Flingjingwaller')
                                                                                                                                                                                for name in names:
                                                                                                                                                                                    statement = "soundex('%s')" % name
                                                                                                                                                                                    t = Timer(statement, "from __main__ import soundex")
                                                                                                                                                                                    print name.ljust(15), soundex(name), min(t.repeat())

                                                                                                                                                                        ### Further Reading on Soundex

                                                                                                                                                                     4. Soundexing and Genealogy gives a chronology of the evolution of the Soundex and its regional variations.

                                                                                                                                                                        ## 18.2. Using the `timeit` Module

                                                                                                                                                                        The most important thing you need to know about optimizing Python code is that you shouldn't write your own timing function.

                                                                                                                                                                        Timing short pieces of code is incredibly complex. How much processor time is your computer devoting to running this code? Are there things running in the background? Are you sure? Every modern computer has background processes running, some all the time, some intermittently. Cron jobs fire off at consistent intervals; background services occasionally “wake up” to do useful things like check for new mail, connect to instant messaging servers, check for application updates, scan for viruses, check whether a disk has been inserted into your CD drive in the last 100 nanoseconds, and so on. Before you start your timing tests, turn everything off and disconnect from the network. Then turn off all the things you forgot to turn off the first time, then turn off the service that's incessantly checking whether the network has come back yet, then ...

                                                                                                                                                                        And then there's the matter of the variations introduced by the timing framework itself. Does the Python interpreter cache method name lookups? Does it cache code block compilations? Regular expressions? Will your code have side effects if run more than once? Don't forget that you're dealing with small fractions of a second, so small mistakes in your timing framework will irreparably skew your results.

                                                                                                                                                                        The Python community has a saying: “Python comes with batteries included.” Don't write your own timing framework. Python 2.3 comes with a perfectly good one called `timeit`.

                                                                                                                                                                        ### Example 18.2. Introducing `timeit`

                                                                                                                                                                        If you have not already done so, you can download this and other examples used in this book.

                                                                                                                                                                            >>> import timeit
                                                                                                                                                                            >>> t = timeit.Timer("soundex.soundex('Pilgrim')",
                                                                                                                                                                            ...    "import soundex")   ①
                                                                                                                                                                            >>> t.timeit()              ②
                                                                                                                                                                            8.21683733547
                                                                                                                                                                            >>> t.repeat(3, 2000000)    ③
                                                                                                                                                                            [16.48319309109, 16.46128984923, 16.44203948912]

                                                                                                                                                                        1. The `timeit` module defines one class, `Timer`, which takes two arguments. Both arguments are strings. The first argument is the statement you wish to time; in this case, you are timing a call to the Soundex function within the `soundex` with an argument of `'Pilgrim'`. The second argument to the `Timer` class is the import statement that sets up the environment for the statement. Internally, `timeit` sets up an isolated virtual environment, manually executes the setup statement (importing the `soundex` module), then manually compiles and executes the timed statement (calling the Soundex function).
                                                                                                                                                                        2. Once you have the `Timer` object, the easiest thing to do is call `timeit()`, which calls your function 1 million times and returns the number of seconds it took to do it.
                                                                                                                                                                        3. The other major method of the `Timer` object is `repeat()`, which takes two optional arguments. The first argument is the number of times to repeat the entire test, and the second argument is the number of times to call the timed statement within each test. Both arguments are optional, and they default to `3` and `1000000` respectively. The `repeat()` method returns a list of the times each test cycle took, in seconds.

                                                                                                                                                                           > ☞You can use the `timeit module on the command line to test an existing Python program, without modifying the code. See http://docs.python.org/lib/node396.html for documentation on the command-line flags.

                                                                                                                                                                           > Note that `repeat() returns a list of times. The times will almost never be identical, due to slight variations in how much processor time the Python interpreter is getting (and those pesky background processes that you can't get rid of). Your first thought might be to say “Let's take the average and call that The True Number.”

                                                                                                                                                                           > In fact, that's almost certainly wrong. The tests that took longer didn't take longer because of variations in your code or in the Python interpreter; they took longer because of those pesky background processes, or other factors outside of the Python interpreter that you can't fully eliminate. If the different timing results differ by more than a few percent, you still have too much variability to trust the results. Otherwise, take the minimum time and discard the rest.

                                                                                                                                                                           > Python has a handy `min function that takes a list and returns the smallest value:

                                                                                                                                                                           >     >>> min(t.repeat(3, 1000000))
                                                                                                                                                                           >     8.22203948912

                                                                                                                                                                           > > ☞The `timeit module only works if you already know what piece of code you need to optimize. If you have a larger Python program and don't know where your performance problems are, check out the `hotshot module.

                                                                                                                                                                           > > ## 18.3. Optimizing Regular Expressions

                                                                                                                                                                           > > The first thing the Soundex function checks is whether the input is a non-empty string of letters. What's the best way to do this?

                                                                                                                                                                           > > If you answered “regular expressions”, go sit in the corner and contemplate your bad instincts. Regular expressions are almost never the right answer; they should be avoided whenever possible. Not only for performance reasons, but simply because they're difficult to debug and maintain. Also for performance reasons.

                                                                                                                                                                           > > This code fragment from `soundex/stage1/soundex1a.py checks whether the function argument source is a word made entirely of letters, with at least one letter (not the empty string):

                                                                                                                                                                           > > `allChars = string.uppercase + string.lowercase if not re.search('^[%s]+$' % allChars, source): return "0000"

                                                                                                                                                                           > > How does `soundex1a.py perform? For convenience, the `__main__ section of the script contains this code that calls the `timeit module, sets up a timing test with three different names, tests each name three times, and displays the minimum time for each:

                                                                                                                                                                           > > `if __name__ == '__main__': from timeit import Timer names = ('Woo', 'Pilgrim', 'Flingjingwaller') for name in names: statement = "soundex('%s')" % name t = Timer(statement, "from __main__ import soundex") print name.ljust(15), soundex(name), min(t.repeat())

                                                                                                                                                                           > > So how does `soundex1a.py perform with this regular expression?

                                                                                                                                                                           > > `C:\samples\soundex\stage1>python soundex1a.py `Woo W000 19.3356647283 Pilgrim P426 24.0772053431 Flingjingwaller F452 35.0463220884

                                                                                                                                                                           > > As you might expect, the algorithm takes significantly longer when called with longer names. There will be a few things we can do to narrow that gap (make the function take less relative time for longer input), but the nature of the algorithm dictates that it will never run in constant time.

                                                                                                                                                                           > > The other thing to keep in mind is that we are testing a representative sample of names. `Woo is a kind of trivial case, in that it gets shorted down to a single letter and then padded with zeros. `Pilgrim is a normal case, of average length and a mixture of significant and ignored letters. `Flingjingwaller is extraordinarily long and contains consecutive duplicates. Other tests might also be helpful, but this hits a good range of different cases.

                                                                                                                                                                           > > So what about that regular expression? Well, it's inefficient. Since the expression is testing for ranges of characters (`A-Z in uppercase, and `a-z in lowercase), we can use a shorthand regular expression syntax. Here is `soundex/stage1/soundex1b.py:

                                                                                                                                                                           > > `if not re.search('^[A-Za-z]+$', source): return "0000"

                                                                                                                                                                           > > `timeit says `soundex1b.py is slightly faster than `soundex1a.py, but nothing to get terribly excited about:

                                                                                                                                                                           > > `C:\samples\soundex\stage1>python soundex1b.py `Woo W000 17.1361133887 Pilgrim P426 21.8201693232 Flingjingwaller F452 32.7262294509

                                                                                                                                                                           > > We saw in Section 15.3, “Refactoring” that regular expressions can be compiled and reused for faster results. Since this regular expression never changes across function calls, we can compile it once and use the compiled version. Here is `soundex/stage1/soundex1c.py:

                                                                                                                                                                           > > `isOnlyChars = re.compile('^[A-Za-z]+$').search def soundex(source): if not isOnlyChars(source): return "0000"

                                                                                                                                                                           > > Using a compiled regular expression in `soundex1c.py is significantly faster:

                                                                                                                                                                           > > `C:\samples\soundex\stage1>python soundex1c.py `Woo W000 14.5348347346 Pilgrim P426 19.2784703084 Flingjingwaller F452 30.0893873383

                                                                                                                                                                           > > But is this the wrong path? The logic here is simple: the input source needs to be non-empty, and it needs to be composed entirely of letters. Wouldn't it be faster to write a loop checking each character, and do away with regular expressions altogether?

                                                                                                                                                                           > > Here is `soundex/stage1/soundex1d.py:

                                                                                                                                                                           > > `if not source: return "0000" for c in source: if not ('A' <= c <= 'Z') and not ('a' <= c <= 'z'): return "0000"

                                                                                                                                                                           > > It turns out that this technique in `soundex1d.py is not faster than using a compiled regular expression (although it is faster than using a non-compiled regular expression):

                                                                                                                                                                           > > `C:\samples\soundex\stage1>python soundex1d.py `Woo W000 15.4065058548 Pilgrim P426 22.2753567842 Flingjingwaller F452 37.5845122774

                                                                                                                                                                           > > Why isn't `soundex1d.py faster? The answer lies in the interpreted nature of Python. The regular expression engine is written in C, and compiled to run natively on your computer. On the other hand, this loop is written in Python, and runs through the Python interpreter. Even though the loop is relatively simple, it's not simple enough to make up for the overhead of being interpreted. Regular expressions are never the right answer... except when they are.

                                                                                                                                                                           > > It turns out that Python offers an obscure string method. You can be excused for not knowing about it, since it's never been mentioned in this book. The method is called `isalpha(), and it checks whether a string contains only letters.

                                                                                                                                                                           > > This is `soundex/stage1/soundex1e.py:

                                                                                                                                                                           > > `if (not source) and (not source.isalpha()): return "0000"

                                                                                                                                                                           > > How much did we gain by using this specific method in `soundex1e.py? Quite a bit.

                                                                                                                                                                           > > `C:\samples\soundex\stage1>python soundex1e.py `Woo W000 13.5069504644 Pilgrim P426 18.2199394057 Flingjingwaller F452 28.9975225902

                                                                                                                                                                           > > ### Example 18.3. Best Result So Far: `soundex/stage1/soundex1e.py

                                                                                                                                                                           > > `import string, re charToSoundex = {"A": "9", "B": "1", "C": "2", "D": "3", "E": "9", "F": "1", "G": "2", "H": "9", "I": "9", "J": "2", "K": "2", "L": "4", "M": "5", "N": "5", "O": "9", "P": "1", "Q": "2", "R": "6", "S": "2", "T": "3", "U": "9", "V": "1", "W": "9", "X": "2", "Y": "9", "Z": "2"} def soundex(source): if (not source) and (not source.isalpha()): return "0000" source = source[0].upper() + source[1:] digits = source[0] for s in source[1:]: s = s.upper() digits += charToSoundex[s] digits2 = digits[0] for d in digits[1:]: if digits2[-1] != d: digits2 += d digits3 = re.sub('9', '', digits2) while len(digits3) < 4: digits3 += "0" return digits3[:4] if __name__ == '__main__': from timeit import Timer names = ('Woo', 'Pilgrim', 'Flingjingwaller') for name in names: statement = "soundex('%s')" % name t = Timer(statement, "from __main__ import soundex") print name.ljust(15), soundex(name), min(t.repeat())

                                                                                                                                                                           > > ## 18.4. Optimizing Dictionary Lookups

                                                                                                                                                                           > > The second step of the Soundex algorithm is to convert characters to digits in a specific pattern. What's the best way to do this?

                                                                                                                                                                           > > The most obvious solution is to define a dictionary with individual characters as keys and their corresponding digits as values, and do dictionary lookups on each character. This is what we have in `soundex/stage1/soundex1c.py (the current best result so far):

                                                                                                                                                                           > > `charToSoundex = {"A": "9", "B": "1", "C": "2", "D": "3", "E": "9", "F": "1", "G": "2", "H": "9", "I": "9", "J": "2", "K": "2", "L": "4", "M": "5", "N": "5", "O": "9", "P": "1", "Q": "2", "R": "6", "S": "2", "T": "3", "U": "9", "V": "1", "W": "9", "X": "2", "Y": "9", "Z": "2"} def soundex(source): # ... input check omitted for brevity ... source = source[0].upper() + source[1:] digits = source[0] for s in source[1:]: s = s.upper() digits += charToSoundex[s]

                                                                                                                                                                           > > You timed `soundex1c.py already; this is how it performs:

                                                                                                                                                                           > > `C:\samples\soundex\stage1>python soundex1c.py `Woo W000 14.5341678901 Pilgrim P426 19.2650071448 Flingjingwaller F452 30.1003563302

                                                                                                                                                                           > > This code is straightforward, but is it the best solution? Calling `upper() on each individual character seems inefficient; it would probably be better to call `upper() once on the entire string.

                                                                                                                                                                           > > Then there's the matter of incrementally building the digits string. Incrementally building strings like this is horribly inefficient; internally, the Python interpreter needs to create a new string each time through the loop, then discard the old one.

                                                                                                                                                                           > > Python is good at lists, though. It can treat a string as a list of characters automatically. And lists are easy to combine into strings again, using the string method `join().

                                                                                                                                                                           > > Here is `soundex/stage2/soundex2a.py, which converts letters to digits by using ↦ and `lambda:

                                                                                                                                                                           > > `def soundex(source): # ... source = source.upper() digits = source[0] + "".join(map(lambda c: charToSoundex[c], source[1:]))

                                                                                                                                                                           > > Surprisingly, `soundex2a.py is not faster:

                                                                                                                                                                           > > `C:\samples\soundex\stage2>python soundex2a.py `Woo W000 15.0097526362 Pilgrim P426 19.254806407 Flingjingwaller F452 29.3790847719

                                                                                                                                                                           > > The overhead of the anonymous `lambda function kills any performance you gain by dealing with the string as a list of characters.

                                                                                                                                                                           > > `soundex/stage2/soundex2b.py uses a list comprehension instead of ↦ and `lambda:

                                                                                                                                                                           > > `source = source.upper() digits = source[0] + "".join([charToSoundex[c] for c in source[1:]])

                                                                                                                                                                           > > Using a list comprehension in `soundex2b.py is faster than using ↦ and `lambda in `soundex2a.py, but still not faster than the original code (incrementally building a string in `soundex1c.py):

                                                                                                                                                                           > > `C:\samples\soundex\stage2>python soundex2b.py `Woo W000 13.4221324219 Pilgrim P426 16.4901234654 Flingjingwaller F452 25.8186157738

                                                                                                                                                                           > > It's time for a radically different approach. Dictionary lookups are a general purpose tool. Dictionary keys can be any length string (or many other data types), but in this case we are only dealing with single-character keys and single-character values. It turns out that Python has a specialized function for handling exactly this situation: the `string.maketrans function.

                                                                                                                                                                           > > This is `soundex/stage2/soundex2c.py:

                                                                                                                                                                           > > `allChar = string.uppercase + string.lowercase charToSoundex = string.maketrans(allChar, "91239129922455912623919292" * 2) def soundex(source): # ... digits = source[0].upper() + source[1:].translate(charToSoundex)

                                                                                                                                                                           > > What the heck is going on here? `string.maketrans creates a translation matrix between two strings: the first argument and the second argument. In this case, the first argument is the string `ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz, and the second argument is the string `9123912992245591262391929291239129922455912623919292. See the pattern? It's the same conversion pattern we were setting up longhand with a dictionary. A maps to 9, B maps to 1, C maps to 2, and so forth. But it's not a dictionary; it's a specialized data structure that you can access using the string method `translate, which translates each character into the corresponding digit, according to the matrix defined by `string.maketrans.

                                                                                                                                                                           > > `timeit shows that `soundex2c.py is significantly faster than defining a dictionary and looping through the input and building the output incrementally:

                                                                                                                                                                           > > `C:\samples\soundex\stage2>python soundex2c.py `Woo W000 11.437645008 Pilgrim P426 13.2825062962 Flingjingwaller F452 18.5570110168

                                                                                                                                                                           > > You're not going to get much better than that. Python has a specialized function that does exactly what you want to do; use it and move on.

                                                                                                                                                                           > > ### Example 18.4. Best Result So Far: `soundex/stage2/soundex2c.py

                                                                                                                                                                           > > `import string, re allChar = string.uppercase + string.lowercase charToSoundex = string.maketrans(allChar, "91239129922455912623919292" * 2) isOnlyChars = re.compile('^[A-Za-z]+$').search def soundex(source): if not isOnlyChars(source): return "0000" digits = source[0].upper() + source[1:].translate(charToSoundex) digits2 = digits[0] for d in digits[1:]: if digits2[-1] != d: digits2 += d digits3 = re.sub('9', '', digits2) while len(digits3) < 4: digits3 += "0" return digits3[:4] if __name__ == '__main__': from timeit import Timer names = ('Woo', 'Pilgrim', 'Flingjingwaller') for name in names: statement = "soundex('%s')" % name t = Timer(statement, "from __main__ import soundex") print name.ljust(15), soundex(name), min(t.repeat())

                                                                                                                                                                           > > ## 18.5. Optimizing List Operations

                                                                                                                                                                           > > The third step in the Soundex algorithm is eliminating consecutive duplicate digits. What's the best way to do this?

                                                                                                                                                                           > > Here's the code we have so far, in `soundex/stage2/soundex2c.py:

                                                                                                                                                                           > > `digits2 = digits[0] for d in digits[1:]: if digits2[-1] != d: digits2 += d

                                                                                                                                                                           > > Here are the performance results for `soundex2c.py:

                                                                                                                                                                           > > `C:\samples\soundex\stage2>python soundex2c.py `Woo W000 12.6070768771 Pilgrim P426 14.4033353401 Flingjingwaller F452 19.7774882003

                                                                                                                                                                           > > The first thing to consider is whether it's efficient to check digits[-1] each time through the loop. Are list indexes expensive? Would we be better off maintaining the last digit in a separate variable, and checking that instead?

                                                                                                                                                                           > > To answer this question, here is `soundex/stage3/soundex3a.py:

                                                                                                                                                                           > > `digits2 = '' last_digit = '' for d in digits: if d != last_digit: digits2 += d last_digit = d

                                                                                                                                                                           > > `soundex3a.py does not run any faster than `soundex2c.py, and may even be slightly slower (although it's not enough of a difference to say for sure):

                                                                                                                                                                           > > `C:\samples\soundex\stage3>python soundex3a.py `Woo W000 11.5346048171 Pilgrim P426 13.3950636184 Flingjingwaller F452 18.6108927252

                                                                                                                                                                           > > Why isn't `soundex3a.py faster? It turns out that list indexes in Python are extremely efficient. Repeatedly accessing digits2[-1] is no problem at all. On the other hand, manually maintaining the last seen digit in a separate variable means we have two variable assignments for each digit we're storing, which wipes out any small gains we might have gotten from eliminating the list lookup.

                                                                                                                                                                           > > Let's try something radically different. If it's possible to treat a string as a list of characters, it should be possible to use a list comprehension to iterate through the list. The problem is, the code needs access to the previous character in the list, and that's not easy to do with a straightforward list comprehension.

                                                                                                                                                                           > > However, it is possible to create a list of index numbers using the built-in `range() function, and use those index numbers to progressively search through the list and pull out each character that is different from the previous character. That will give you a list of characters, and you can use the string method `join() to reconstruct a string from that.

                                                                                                                                                                           > > Here is `soundex/stage3/soundex3b.py:

                                                                                                                                                                           > > `digits2 = "".join([digits[i] for i in range(len(digits)) if i == 0 or digits[i-1] != digits[i]])

                                                                                                                                                                           > > Is this faster? In a word, no.

                                                                                                                                                                           > > `C:\samples\soundex\stage3>python soundex3b.py `Woo W000 14.2245271396 Pilgrim P426 17.8337165757 Flingjingwaller F452 25.9954005327

                                                                                                                                                                           > > It's possible that the techniques so far as have been “string-centric”. Python can convert a string into a list of characters with a single command: `list('abc') returns `['a', 'b', 'c']. Furthermore, lists can be modified in place very quickly. Instead of incrementally building a new list (or string) out of the source string, why not move elements around within a single list?

                                                                                                                                                                           > > Here is `soundex/stage3/soundex3c.py, which modifies a list in place to remove consecutive duplicate elements:

                                                                                                                                                                           > > `digits = list(source[0].upper() + source[1:].translate(charToSoundex)) i=0 for item in digits: if item==digits[i]: continue i+=1 digits[i]=item del digits[i+1:] digits2 = "".join(digits)

                                                                                                                                                                           > > Is this faster than `soundex3a.py or `soundex3b.py? No, in fact it's the slowest method yet:

                                                                                                                                                                           > > `C:\samples\soundex\stage3>python soundex3c.py `Woo W000 14.1662554878 Pilgrim P426 16.0397885765 Flingjingwaller F452 22.1789341942

                                                                                                                                                                           > > We haven't made any progress here at all, except to try and rule out several “clever” techniques. The fastest code we've seen so far was the original, most straightforward method (`soundex2c.py). Sometimes it doesn't pay to be clever.

                                                                                                                                                                           > > ### Example 18.5. Best Result So Far: `soundex/stage2/soundex2c.py

                                                                                                                                                                           > > `import string, re allChar = string.uppercase + string.lowercase charToSoundex = string.maketrans(allChar, "91239129922455912623919292" * 2) isOnlyChars = re.compile('^[A-Za-z]+$').search def soundex(source): if not isOnlyChars(source): return "0000" digits = source[0].upper() + source[1:].translate(charToSoundex) digits2 = digits[0] for d in digits[1:]: if digits2[-1] != d: digits2 += d digits3 = re.sub('9', '', digits2) while len(digits3) < 4: digits3 += "0" return digits3[:4] if __name__ == '__main__': from timeit import Timer names = ('Woo', 'Pilgrim', 'Flingjingwaller') for name in names: statement = "soundex('%s')" % name t = Timer(statement, "from __main__ import soundex") print name.ljust(15), soundex(name), min(t.repeat())

                                                                                                                                                                           > > ## 18.6. Optimizing String Manipulation

                                                                                                                                                                           > > The final step of the Soundex algorithm is padding short results with zeros, and truncating long results. What is the best way to do this?

                                                                                                                                                                           > > This is what we have so far, taken from `soundex/stage2/soundex2c.py:

                                                                                                                                                                           > > `digits3 = re.sub('9', '', digits2) while len(digits3) < 4: digits3 += "0" return digits3[:4]

                                                                                                                                                                           > > These are the results for `soundex2c.py:

                                                                                                                                                                           > > `C:\samples\soundex\stage2>python soundex2c.py `Woo W000 12.6070768771 Pilgrim P426 14.4033353401 Flingjingwaller F452 19.7774882003

                                                                                                                                                                           > > The first thing to consider is replacing that regular expression with a loop. This code is from `soundex/stage4/soundex4a.py:

                                                                                                                                                                           > > `digits3 = '' for d in digits2: if d != '9': digits3 += d

                                                                                                                                                                           > > Is `soundex4a.py faster? Yes it is:

                                                                                                                                                                           > > `C:\samples\soundex\stage4>python soundex4a.py `Woo W000 6.62865531792 Pilgrim P426 9.02247576158 Flingjingwaller F452 13.6328416042

                                                                                                                                                                           > > But wait a minute. A loop to remove characters from a string? We can use a simple string method for that. Here's `soundex/stage4/soundex4b.py:

                                                                                                                                                                           > > `digits3 = digits2.replace('9', '')

                                                                                                                                                                           > > Is `soundex4b.py faster? That's an interesting question. It depends on the input:

                                                                                                                                                                           > > `C:\samples\soundex\stage4>python soundex4b.py `Woo W000 6.75477414029 Pilgrim P426 7.56652144337 Flingjingwaller F452 10.8727729362

                                                                                                                                                                           > > The string method in `soundex4b.py is faster than the loop for most names, but it's actually slightly slower than `soundex4a.py in the trivial case (of a very short name). Performance optimizations aren't always uniform; tuning that makes one case faster can sometimes make other cases slower. In this case, the majority of cases will benefit from the change, so let's leave it at that, but the principle is an important one to remember.

                                                                                                                                                                           > > Last but not least, let's examine the final two steps of the algorithm: padding short results with zeros, and truncating long results to four characters. The code you see in `soundex4b.py does just that, but it's horribly inefficient. Take a look at `soundex/stage4/soundex4c.py to see why:

                                                                                                                                                                           > > `digits3 += '000' return digits3[:4]

                                                                                                                                                                           > > Why do we need a `while loop to pad out the result? We know in advance that we're going to truncate the result to four characters, and we know that we already have at least one character (the initial letter, which is passed unchanged from the original source variable). That means we can simply add three zeros to the output, then truncate it. Don't get stuck in a rut over the exact wording of the problem; looking at the problem slightly differently can lead to a simpler solution.

                                                                                                                                                                           > > How much speed do we gain in `soundex4c.py by dropping the `while loop? It's significant:

                                                                                                                                                                           > > `C:\samples\soundex\stage4>python soundex4c.py `Woo W000 4.89129791636 Pilgrim P426 7.30642134685 Flingjingwaller F452 10.689832367

                                                                                                                                                                           > > Finally, there is still one more thing you can do to these three lines of code to make them faster: you can combine them into one line. Take a look at `soundex/stage4/soundex4d.py:

                                                                                                                                                                           > > `return (digits2.replace('9', '') + '000')[:4]

                                                                                                                                                                           > > Putting all this code on one line in `soundex4d.py is barely faster than `soundex4c.py:

                                                                                                                                                                           > > `C:\samples\soundex\stage4>python soundex4d.py `Woo W000 4.93624105857 Pilgrim P426 7.19747593619 Flingjingwaller F452 10.5490700634

                                                                                                                                                                           > > It is also significantly less readable, and for not much performance gain. Is that worth it? I hope you have good comments. Performance isn't everything. Your optimization efforts must always be balanced against threats to your program's readability and maintainability.

                                                                                                                                                                           > > ## 18.7. Summary

                                                                                                                                                                           > > This chapter has illustrated several important aspects of performance tuning in Python, and performance tuning in general.

                                                                                                                                                                        4. If you need to choose between regular expressions and writing a loop, choose regular expressions. The regular expression engine is compiled in C and runs natively on your computer; your loop is written in Python and runs through the Python interpreter.
                                                                                                                                                                        5. If you need to choose between regular expressions and string methods, choose string methods. Both are compiled in C, so choose the simpler one.
                                                                                                                                                                        6. General-purpose dictionary lookups are fast, but specialtiy functions such as `string.maketrans` and string methods such as `isalpha()` are faster. If Python has a custom-tailored function for you, use it.
                                                                                                                                                                        7. Don't be too clever. Sometimes the most obvious algorithm is also the fastest.
                                                                                                                                                                        8. Don't sweat it too much. Performance isn't everything.

                                                                                                                                                                           I can't emphasize that last point strongly enough. Over the course of this chapter, you made this function three times faster and saved 20 seconds over 1 million function calls. Great. Now think: over the course of those million function calls, how many seconds will your surrounding application wait for a database connection? Or wait for disk I/O? Or wait for user input? Don't spend too much time over-optimizing one algorithm, or you'll ignore obvious improvements somewhere else. Develop an instinct for the sort of code that Python runs well, correct obvious blunders if you find them, and leave the rest alone.