Introducing chardet: a mini-FAQ
@@ -111,8 +114,8 @@ RefactoringTool: Skipping implicit fixer: ws_comma
__version__ = "1.0.1"
def detect(aBuf):
-- import universaldetector
-+ from . import universaldetector
+Well, that wasn’t so hard. Just a few imports and print statements to convert. Time to run the new version. Do you think it’ll work? @@ -648,7 +651,7 @@ import sys
There are variations of this problem scattered throughout the chardet library. In some places it’s "import constants, sys"; in other places, it’s "import constants, re". The fix is the same: manually split the import statement into two lines, one for the relative import, the other for the absolute import.
Onward!
Name 'file' is not defined
-FIXME intro +
And here we go again, running test.py to try to execute our test cases…
C:\home\chardet> python test.py tests\*\* tests\ascii\howto.diveintomark.org.xml @@ -661,7 +664,7 @@ NameError: name 'file' is not defined
for line in open(f, 'rb'):And that’s all I have to say about that.
Can’t use a string pattern on a bytes-like object
-FIXME intro +
Now things are starting to get interesting. And by “interesting,” I mean “confusing as all hell.”
C:\home\chardet> python test.py tests\*\* tests\ascii\howto.diveintomark.org.xml @@ -671,8 +674,8 @@ NameError: name 'file' is not definedFile "C:\home\chardet\chardet\universaldetector.py", line 98, in feed if self._highBitDetector.search(aBuf): TypeError: can't use a string pattern on a bytes-like object -
Now things are starting to get interesting. And by “interesting,” I mean “confusing as all hell.” -
First, let’s see what self._highBitDetector is. It’s defined in the __init__ method of the UniversalDetector class: +
+
To debug this, let’s see what self._highBitDetector is. It’s defined in the __init__ method of the UniversalDetector class:
class UniversalDetector:
def __init__(self):
@@ -687,7 +690,7 @@ TypeError: can't use a string pattern on a bytes-like objectAnd what is aBuf? Let’s backtrack further to a place that calls UniversalDetector.feed(). One place that calls it is the test harness, test.py.
+
And what is aBuf? Let’s backtrack further to a place that calls UniversalDetector.feed(). One place that calls it is the test harness, test.py.
u = UniversalDetector()
.
@@ -695,7 +698,7 @@ TypeError: can't use a string pattern on a bytes-like objectAnd here we find our answer: in the UniversalDetector.feed() method, aBuf is a line read from a file on disk. Look carefully at the parameters used to open the file: 'rb'. 'r' is for “read”; OK, big deal, we’re reading the file. Ah, but 'b' is for “binary.” Without the 'b' flag, this for loop would read the file, line by line, and convert each line into a string -- an array of Unicode characters -- according to the system default character encoding. (You could override the system encoding with another parameter to open(), but never mind that for now.) But with the 'b' flag, this for loop reads the file, line by line, and stores each line exactly as it appears in the file, as an array of bytes. That byte array gets passed to UniversalDetector.feed(), and eventually gets passed to the pre-compiled regular expression, self._highBitDetector, to search for high-bit... characters. But we don’t have characters; we have bytes. Oops.
+
And here we find our answer: in the UniversalDetector.feed() method, aBuf is a line read from a file on disk. Look carefully at the parameters used to open the file: 'rb'. 'r' is for “read”; OK, big deal, we’re reading the file. Ah, but 'b' is for “binary.” Without the 'b' flag, this for loop would read the file, line by line, and convert each line into a string -- an array of Unicode characters -- according to the system default character encoding. (You could override the system encoding with another parameter to open(), but never mind that for now.) But with the 'b' flag, this for loop reads the file, line by line, and stores each line exactly as it appears in the file, as an array of bytes. That byte array gets passed to UniversalDetector.feed(), and eventually gets passed to the pre-compiled regular expression, self._highBitDetector, to search for high-bit... characters. But we don’t have characters; we have bytes. Oops.
What we need this regular expression to search is not an array of characters, but an array of bytes.
Once you realize that, the solution is not difficult. Regular expressions defined with strings can search strings. Regular expressions defined with byte arrays can search byte arrays. To define a byte array pattern, we simply change the type of the argument we use to define the regular expression to a byte array. So instead of this:
self._highBitDetector = re.compile(r'[\x80-\xFF]')... + +
There's an unfortunate clash of coding style and Python interpreter here. The TypeError could be anywhere on that line, but the traceback doesn't tell you exactly where it is. It could be in the first conditional or the second, and the traceback would look the same. To narrow it down, you should split the line in half, like this:
+
+
elif (self._mInputState == ePureAscii) and \
+ self._escDetector.search(self._mLastChar + aBuf):And re-run the test:
+ +skip over this command output listing +
C:\home\chardet> python test.py tests\*\* +tests\ascii\howto.diveintomark.org.xml +Traceback (most recent call last): + File "test.py", line 10, in <module> + u.feed(line) + File "C:\home\chardet\chardet\universaldetector.py", line 101, in feed + self._escDetector.search(self._mLastChar + aBuf): +TypeError: Can't convert 'bytes' object to str implicitly+ +
Aha! The problem was not in the first conditional (self._mInputState == ePureAscii) but in the second one. So what could cause a TypeError there? Perhaps you're thinking that the search() method is expecting a value of a different type, but that wouldn't generate this traceback. Python functions can take any value; if you pass the right number of arguments, the function will execute. It may crash if you pass it a value of a different type than it's expecting, but if that happened, the traceback would point to somewhere inside the function. But this traceback says it never got as far as calling the search() method. So the problem must be in that + operation, as it's trying to construct the value that it will eventually pass to the search() method.
+
+
We know from previous debugging that aBuf is a byte array. So what is self._mLastChar? It's an instance variable, defined in the reset() method, which is actually called from the __init__() method.
+
+
class UniversalDetector:
+ def __init__(self):
+ self._highBitDetector = re.compile(b'[\x80-\xFF]')
+ self._escDetector = re.compile(b'(\033|~{)')
+ self._mEscCharSetProber = None
+ self._mCharSetProbers = []
+ self.reset()
+
+ def reset(self):
+ self.result = {'encoding': None, 'confidence': 0.0}
+ self.done = False
+ self._mStart = True
+ self._mGotData = False
+ self._mInputState = ePureAscii
+ self._mLastChar = ''And now we have our answer. Do you see it? self._mLastChar is a string, but aBuf is a byte array. And you can't concatenate a string to a byte array — not even a zero-length string. + +
So what is self._mLastChar anyway? The answer is in the feed() method, just a few lines down from where the trackback occurred.
+
+
if self._mInputState == ePureAscii:
+ if self._highBitDetector.search(aBuf):
+ self._mInputState = eHighbyte
+ elif (self._mInputState == ePureAscii) and \
+ self._escDetector.search(self._mLastChar + aBuf):
+ self._mInputState = eEscAscii
+
+self._mLastChar = aBuf[-1]The calling function calls this feed() method over and over again with a few bytes at a time. The method processes the bytes it was given (passed in as aBuf), then stores the last byte in self._mLastChar in case it's needed during the next call. (In a multi-byte encoding, the feed() method might get called with half of a character, then called again with the other half.) But because aBuf is now a byte array instead of a string, self._mLastChar needs to be a byte array as well. Thus:
+
+
def reset(self):
+ .
+ .
+ .
+- self._mLastChar = ''
++ self._mLastChar = b''TypeError: unsupported operand type(s) for +: 'int' and 'bytes'
+ +I have good news, and I have bad news. The good news is we're making progress… + +
skip over this command listing +
C:\home\chardet> python test.py tests\*\* +tests\ascii\howto.diveintomark.org.xml +Traceback (most recent call last): + File "test.py", line 10, in <module> + u.feed(line) + File "C:\home\chardet\chardet\universaldetector.py", line 101, in feed + self._escDetector.search(self._mLastChar + aBuf): +TypeError: unsupported operand type(s) for +: 'int' and 'bytes'+ +
…The bad news is it doesn't always feel like progress. + +
But this is progress! Really! Even though the traceback calls out the same line of code, it's a different error than it used to be. Progress! So what's the problem now? The last time I checked, this line of code didn't try to concatenate an int with a byte array (bytes). In fact, you just spent a lot of time ensuring that self._mLastChar was a byte array. How did it turn into an int?
+
+
The answer lies not in the previous lines of code, but in the following lines. + +
if self._mInputState == ePureAscii:
+ if self._highBitDetector.search(aBuf):
+ self._mInputState = eHighbyte
+ elif (self._mInputState == ePureAscii) and \
+ self._escDetector.search(self._mLastChar + aBuf):
+ self._mInputState = eEscAscii
+
+self._mLastChar = aBuf[-1]This error doesn't occur the first time the feed() method gets called; it occurs the second time, after self._mLastChar has been set to the last byte of aBuf. Well, what's the problem with that? Getting a single element from a byte array yields an integer, not a byte array. To see the difference, follow me to the interactive shell:
+
+
skip over this interpreter listing +
+>>> aBuf = b'\xEF\xBB\xBF' ① +>>> len(aBuf) +3 +>>> mLastChar = aBuf[-1] +>>> mLastChar ② +191 +>>> type(mLastChar) ③ +<class 'int'> +>>> mLastChar + aBuf ④ +Traceback (most recent call last): + File "+", line 1, in <module> +TypeError: unsupported operand type(s) for +: 'int' and 'bytes' +>>> mLastChar = aBuf[-1:] ⑤ +>>> mLastChar +b'\xbf' +>>> mLastChar + aBuf ⑥ +b'\xbf\xef\xbb\xbf'
-
+
- Define a byte array of 3 bytes. +
- The last element of the byte array is 191. +
- That's an integer. +
- Concatenating an integer with a byte array doesn't work. You've now replicated the error you just found in
universaldetector.py. + - Ah, here's the fix. Instead of taking the last element of the byte array, use list slicing to create a new byte array containing just the last element. That is, start with the last element and continue the slice until the end of the byte array. Now mLastChar is a byte array of length 1. +
- Concatenating a byte array of length 1 with a byte array of length 3 returns a new byte array of length 4. +
So, to ensure that the feed() method in universaldetector.py continues to work no matter how often it's called, you need to initialize self._mLastChar as a 0-length byte array, then make sure it stays a byte array.
+
+
self._escDetector.search(self._mLastChar + aBuf):
+ self._mInputState = eEscAscii
+
+- self._mLastChar = aBuf[-1]
++ self._mLastChar = aBuf[-1:]TypeError: ord() expected string of length 1, but int found
+ +Tired yet? You're almost there… + +
skip over this command output listing +
C:\home\chardet> python test.py tests\*\* +tests\ascii\howto.diveintomark.org.xml ascii with confidence 1.0 +tests\Big5\0804.blogspot.com.xml +Traceback (most recent call last): + File "test.py", line 10, in <module> + u.feed(line) + File "C:\home\chardet\chardet\universaldetector.py", line 116, in feed + if prober.feed(aBuf) == constants.eFoundIt: + File "C:\home\chardet\chardet\charsetgroupprober.py", line 60, in feed + st = prober.feed(aBuf) + File "C:\home\chardet\chardet\utf8prober.py", line 53, in feed + codingState = self._mCodingSM.next_state(c) + File "C:\home\chardet\chardet\codingstatemachine.py", line 43, in next_state + byteCls = self._mModel['classTable'][ord(c)] +TypeError: ord() expected string of length 1, but int found+ +
FIXME + +
# codingstatemachine.py
+def next_state(self, c):
+ # for each byte we get its class
+ # if it is first byte, we also get byte length
+ byteCls = self._mModel['classTable'][ord(c)]FIXME [aBuf is a byte array, so c is an int, not a 1-character string. IOW, there's no need to call the ord() function because c is already an int!]
+
+
# utf8prober.py
+def feed(self, aBuf):
+ for c in aBuf:
+ codingState = self._mCodingSM.next_state(c)FIXME [wrapup or deleteme] + +
TypeError: unorderable types: int() >= str()
+ +FIXME [let's go again] + +
skip over this command output listing +
C:\home\chardet> python test.py tests\*\* +tests\ascii\howto.diveintomark.org.xml ascii with confidence 1.0 +tests\Big5\0804.blogspot.com.xml +Traceback (most recent call last): + File "test.py", line 10, in <module> + u.feed(line) + File "C:\home\chardet\chardet\universaldetector.py", line 116, in feed + if prober.feed(aBuf) == constants.eFoundIt: + File "C:\home\chardet\chardet\charsetgroupprober.py", line 60, in feed + st = prober.feed(aBuf) + File "C:\home\chardet\chardet\sjisprober.py", line 68, in feed + self._mContextAnalyzer.feed(self._mLastChar[2 - charLen :], charLen) + File "C:\home\chardet\chardet\jpcntx.py", line 145, in feed + order, charLen = self.get_order(aBuf[i:i+2]) + File "C:\home\chardet\chardet\jpcntx.py", line 176, in get_order + if ((aStr[0] >= '\x81') and (aStr[0] <= '\x9F')) or \ +TypeError: unorderable types: int() >= str()+ +
FIXME +
© 2001–4, 2009 ℳark Pilgrim, CC-BY-SA-3.0 diff --git a/chardet/chardet/__init__.pyc b/chardet/chardet/__init__.pyc index a991c54..d5024df 100644 Binary files a/chardet/chardet/__init__.pyc and b/chardet/chardet/__init__.pyc differ diff --git a/chardet/chardet/big5freq.pyc b/chardet/chardet/big5freq.pyc index 7552252..6f6a6af 100644 Binary files a/chardet/chardet/big5freq.pyc and b/chardet/chardet/big5freq.pyc differ diff --git a/chardet/chardet/big5prober.pyc b/chardet/chardet/big5prober.pyc index 522ab5d..5cd8f7d 100644 Binary files a/chardet/chardet/big5prober.pyc and b/chardet/chardet/big5prober.pyc differ diff --git a/chardet/chardet/chardistribution.pyc b/chardet/chardet/chardistribution.pyc index 4dc7697..c9ed078 100644 Binary files a/chardet/chardet/chardistribution.pyc and b/chardet/chardet/chardistribution.pyc differ diff --git a/chardet/chardet/charsetgroupprober.pyc b/chardet/chardet/charsetgroupprober.pyc index ed6b38f..9887ef5 100644 Binary files a/chardet/chardet/charsetgroupprober.pyc and b/chardet/chardet/charsetgroupprober.pyc differ diff --git a/chardet/chardet/charsetprober.pyc b/chardet/chardet/charsetprober.pyc index f1bd064..9550a50 100644 Binary files a/chardet/chardet/charsetprober.pyc and b/chardet/chardet/charsetprober.pyc differ diff --git a/chardet/chardet/codingstatemachine.pyc b/chardet/chardet/codingstatemachine.pyc index 9f44854..e02d8bb 100644 Binary files a/chardet/chardet/codingstatemachine.pyc and b/chardet/chardet/codingstatemachine.pyc differ diff --git a/chardet/chardet/constants.pyc b/chardet/chardet/constants.pyc index e7cd4fc..ae83f2c 100644 Binary files a/chardet/chardet/constants.pyc and b/chardet/chardet/constants.pyc differ diff --git a/chardet/chardet/escprober.pyc b/chardet/chardet/escprober.pyc index 5564ff1..fb025d0 100644 Binary files a/chardet/chardet/escprober.pyc and b/chardet/chardet/escprober.pyc differ diff --git a/chardet/chardet/escsm.pyc b/chardet/chardet/escsm.pyc index 1b97ea4..bc66e0b 100644 Binary files a/chardet/chardet/escsm.pyc and b/chardet/chardet/escsm.pyc differ diff --git a/chardet/chardet/eucjpprober.pyc b/chardet/chardet/eucjpprober.pyc index 3810fde..019edc8 100644 Binary files a/chardet/chardet/eucjpprober.pyc and b/chardet/chardet/eucjpprober.pyc differ diff --git a/chardet/chardet/euckrfreq.pyc b/chardet/chardet/euckrfreq.pyc index ba11667..2b7c2c4 100644 Binary files a/chardet/chardet/euckrfreq.pyc and b/chardet/chardet/euckrfreq.pyc differ diff --git a/chardet/chardet/euckrprober.pyc b/chardet/chardet/euckrprober.pyc index d9e33a5..ba8b4ac 100644 Binary files a/chardet/chardet/euckrprober.pyc and b/chardet/chardet/euckrprober.pyc differ diff --git a/chardet/chardet/euctwfreq.pyc b/chardet/chardet/euctwfreq.pyc index ee0826e..e586f56 100644 Binary files a/chardet/chardet/euctwfreq.pyc and b/chardet/chardet/euctwfreq.pyc differ diff --git a/chardet/chardet/euctwprober.pyc b/chardet/chardet/euctwprober.pyc index 2133083..41d576a 100644 Binary files a/chardet/chardet/euctwprober.pyc and b/chardet/chardet/euctwprober.pyc differ diff --git a/chardet/chardet/gb2312freq.pyc b/chardet/chardet/gb2312freq.pyc index c3e5c5a..4df57b1 100644 Binary files a/chardet/chardet/gb2312freq.pyc and b/chardet/chardet/gb2312freq.pyc differ diff --git a/chardet/chardet/gb2312prober.pyc b/chardet/chardet/gb2312prober.pyc index 89356b6..c200bb9 100644 Binary files a/chardet/chardet/gb2312prober.pyc and b/chardet/chardet/gb2312prober.pyc differ diff --git a/chardet/chardet/hebrewprober.pyc b/chardet/chardet/hebrewprober.pyc index 5546ae7..5788cf3 100644 Binary files a/chardet/chardet/hebrewprober.pyc and b/chardet/chardet/hebrewprober.pyc differ diff --git a/chardet/chardet/jisfreq.pyc b/chardet/chardet/jisfreq.pyc index 4259ec3..899285b 100644 Binary files a/chardet/chardet/jisfreq.pyc and b/chardet/chardet/jisfreq.pyc differ diff --git a/chardet/chardet/jpcntx.pyc b/chardet/chardet/jpcntx.pyc index 7adf578..455f6e8 100644 Binary files a/chardet/chardet/jpcntx.pyc and b/chardet/chardet/jpcntx.pyc differ diff --git a/chardet/chardet/langbulgarianmodel.pyc b/chardet/chardet/langbulgarianmodel.pyc index 5fc684c..65b8196 100644 Binary files a/chardet/chardet/langbulgarianmodel.pyc and b/chardet/chardet/langbulgarianmodel.pyc differ diff --git a/chardet/chardet/langcyrillicmodel.pyc b/chardet/chardet/langcyrillicmodel.pyc index 41d0a9f..ad465ee 100644 Binary files a/chardet/chardet/langcyrillicmodel.pyc and b/chardet/chardet/langcyrillicmodel.pyc differ diff --git a/chardet/chardet/langgreekmodel.pyc b/chardet/chardet/langgreekmodel.pyc index 55aa44b..0012470 100644 Binary files a/chardet/chardet/langgreekmodel.pyc and b/chardet/chardet/langgreekmodel.pyc differ diff --git a/chardet/chardet/langhebrewmodel.pyc b/chardet/chardet/langhebrewmodel.pyc index 0b9e814..369dc9e 100644 Binary files a/chardet/chardet/langhebrewmodel.pyc and b/chardet/chardet/langhebrewmodel.pyc differ diff --git a/chardet/chardet/langhungarianmodel.pyc b/chardet/chardet/langhungarianmodel.pyc index b2f02c1..cf25b16 100644 Binary files a/chardet/chardet/langhungarianmodel.pyc and b/chardet/chardet/langhungarianmodel.pyc differ diff --git a/chardet/chardet/langthaimodel.pyc b/chardet/chardet/langthaimodel.pyc index c29e8de..38b86d7 100644 Binary files a/chardet/chardet/langthaimodel.pyc and b/chardet/chardet/langthaimodel.pyc differ diff --git a/chardet/chardet/latin1prober.pyc b/chardet/chardet/latin1prober.pyc index 869b031..4b38eff 100644 Binary files a/chardet/chardet/latin1prober.pyc and b/chardet/chardet/latin1prober.pyc differ diff --git a/chardet/chardet/mbcharsetprober.pyc b/chardet/chardet/mbcharsetprober.pyc index 3b796fa..ba52fba 100644 Binary files a/chardet/chardet/mbcharsetprober.pyc and b/chardet/chardet/mbcharsetprober.pyc differ diff --git a/chardet/chardet/mbcsgroupprober.pyc b/chardet/chardet/mbcsgroupprober.pyc index 8eed604..5f03d2c 100644 Binary files a/chardet/chardet/mbcsgroupprober.pyc and b/chardet/chardet/mbcsgroupprober.pyc differ diff --git a/chardet/chardet/mbcssm.pyc b/chardet/chardet/mbcssm.pyc index 1db05cc..b04c2da 100644 Binary files a/chardet/chardet/mbcssm.pyc and b/chardet/chardet/mbcssm.pyc differ diff --git a/chardet/chardet/sbcharsetprober.pyc b/chardet/chardet/sbcharsetprober.pyc index c8b8672..b0bad59 100644 Binary files a/chardet/chardet/sbcharsetprober.pyc and b/chardet/chardet/sbcharsetprober.pyc differ diff --git a/chardet/chardet/sbcsgroupprober.pyc b/chardet/chardet/sbcsgroupprober.pyc index 857deca..6ec3e06 100644 Binary files a/chardet/chardet/sbcsgroupprober.pyc and b/chardet/chardet/sbcsgroupprober.pyc differ diff --git a/chardet/chardet/sjisprober.pyc b/chardet/chardet/sjisprober.pyc index 6a81164..f5b3a96 100644 Binary files a/chardet/chardet/sjisprober.pyc and b/chardet/chardet/sjisprober.pyc differ diff --git a/chardet/chardet/universaldetector.pyc b/chardet/chardet/universaldetector.pyc index 43c3a05..73669a3 100644 Binary files a/chardet/chardet/universaldetector.pyc and b/chardet/chardet/universaldetector.pyc differ diff --git a/chardet/chardet/utf8prober.pyc b/chardet/chardet/utf8prober.pyc index 9811837..9303ce9 100644 Binary files a/chardet/chardet/utf8prober.pyc and b/chardet/chardet/utf8prober.pyc differ diff --git a/chardet/python3-conversion-notes.txt b/chardet/python3-conversion-notes.txt index 3f7c7e7..5f74a32 100644 --- a/chardet/python3-conversion-notes.txt +++ b/chardet/python3-conversion-notes.txt @@ -10,11 +10,9 @@ import sys * test.py: change file() to open() * universaldetector.py: change r'' strings to b'' byte arrays in self._highBitDetector, self._escDetector regular expressions -- charsetprober.py: change regular expression-based replace to use b'' byte arrays instead of strings -- universaldetector.py: change self._mLastChar from a r'' string to a b'' byte array -- mbcharsetprober.py: change self._mLastChar from a list of two 1-character strings to a list of two ints -- universaldetector.py: getting a single element from a byte array yields an integer, not a byte, so change syntax to make sure we self._mLastChar is always a byte +* universaldetector.py: change self._mLastChar from a '' string to a b'' byte array +* universaldetector.py: getting a single element from a byte array yields an integer, not a byte, so change syntax to make sure we self._mLastChar is always a byte old: self._mLastChar = aBuf[-1] new: @@ -25,4 +23,8 @@ - jpcntx.py, chardistribution.py (editorial): global search-and-replace "aStr" --> "aBuf" to make it clear that we're passing around a byte array - sbcharsetprober.py, latin1prober.py: change ord(c) to c since it's already an int (iterating through a byte array) +- (not sure where this fits) mbcharsetprober.py: change self._mLastChar from a list of two 1-character strings to a list of two ints + +- (not sure where this fits) charsetprober.py: change regular expression-based replace to use b'' byte arrays instead of strings + - latin1prober.py: refactor reduce(operator.add, ...) to use a for loop instead diff --git a/dip2 b/dip2 index 3fa4ea4..4c15027 100644 --- a/dip2 +++ b/dip2 @@ -291,14 +291,14 @@
The first thing you need to do with Python is install it. Or do you?
If you're using an account on a hosted server, your ISP may have already installed Python. Most popular Linux distributions come with Python in the default installation. Mac OS X 10.2 and later includes a command-line version of Python, although you'll probably want to install a version that includes a more Mac-like graphical interface.
Windows does not come with any version of Python, but don't despair! There are several ways to point-and-click your way to Python on Windows. -
As you can see already, Python runs on a great many operating systems. The full list includes Windows, Mac OS, Mac OS X, and all varieties of free UNIX-compatible systems like Linux. There are also versions that run on Sun Solaris, AS/400, Amiga, OS/2, BeOS, and a plethora +
As you can see already, Python runs on a great many operating systems. The full list includes Windows, Mac OS, Mac OS X, and all varieties of free UNIX-compatible systems like Linux. There are also versions that run on Sun Solaris, AS/400, Amiga, OS/2, BeOS, and a plethora of other platforms you've probably never even heard of.
What's more, Python programs written on one platform can, with a little care, run on any supported platform. For instance, I regularly develop Python programs on Windows and later deploy them on Linux.
So back to the question that started this section, “Which Python is right for you?” The answer is whichever one runs on the computer you already have.
1.2. Python on Windows
On Windows, you have a couple choices for installing Python. -
ActiveState makes a Windows installer for Python called ActivePython, which includes a complete version of Python, an IDE with a Python-aware code editor, plus some Windows extensions for Python that allow complete access to Windows-specific services, APIs, and the Windows Registry. -
ActivePython is freely downloadable, although it is not open source. It is the IDE I used to learn Python, and I recommend you try it unless you have a specific reason not to. One such reason might be that ActiveState is generally +
ActiveState makes a Windows installer for Python called ActivePython, which includes a complete version of Python, an IDE with a Python-aware code editor, plus some Windows extensions for Python that allow complete access to Windows-specific services, APIs, and the Windows Registry. +
ActivePython is freely downloadable, although it is not open source. It is the IDE I used to learn Python, and I recommend you try it unless you have a specific reason not to. One such reason might be that ActiveState is generally several months behind in updating their ActivePython installer when new version of Python are released. If you absolutely need the latest version of Python and ActivePython is still a version behind as you read this, you'll want to use the second option for installing Python on Windows.
The second option is the “official” Python installer, distributed by the people who develop Python itself. It is freely downloadable and open source, and it is always current with the latest version of Python.
1.3. Python on Mac OS X
On Mac OS X, you have two choices for installing Python: install it, or don't install it. You probably want to install it.
Mac OS X 10.2 and later comes with a command-line version of Python preinstalled. If you are comfortable with the command line, you can use this version for the first third of the book. However, -the preinstalled version does not come with an XML parser, so when you get to the XML chapter, you'll need to install the full version. +the preinstalled version does not come with an XML parser, so when you get to the XML chapter, you'll need to install the full version.
Rather than using the preinstalled version, you'll probably want to install the latest version, which also comes with a graphical interactive shell.
Double-click PythonIDE to launch Python.
-
The MacPython IDE should display a splash screen, then take you to the interactive shell. If the interactive shell does not appear, select +
The MacPython IDE should display a splash screen, then take you to the interactive shell. If the interactive shell does not appear, select Window->Python Interactive (Cmd-0). The opening window will look something like this:
Python 2.3 (#2, Jul 30 2003, 11:45:28) @@ -475,7 +475,7 @@ Type "help", "copyright", "credits", or "license" for more information.Double-click
Python IDEto launch Python. -The MacPython IDE should display a splash screen, and then take you to the interactive shell. If the interactive shell does not appear, select +
The MacPython IDE should display a splash screen, and then take you to the interactive shell. If the interactive shell does not appear, select Window->Python Interactive (Cmd-0). You'll see a screen like this:
Python 2.3 (#2, Jul 30 2003, 11:45:28) @@ -486,7 +486,7 @@ MacPython IDE 1.0.11.5. Python on RedHat Linux
Installing under UNIX-compatible operating systems such as Linux is easy if you're willing to install a binary package. Pre-built binary packages are available for most popular Linux distributions. Or you can always compile from source. -
Download the latest Python RPM by going to http://www.python.org/ftp/python/ and selecting the highest version number listed, then selecting the
rpms/directory within that. Then download the RPM with the highest version number. You can install it with the rpm command, as shown here: +Download the latest Python RPM by going to http://www.python.org/ftp/python/ and selecting the highest version number listed, then selecting the
rpms/directory within that. Then download the RPM with the highest version number. You can install it with the rpm command, as shown here:Example 1.2. Installing on RedHat Linux 9
localhost:~$ su - Password: [enter your root password] @@ -516,9 +516,9 @@ Type "help", "copyright", "credits", or "license" for more information.Whoops! Just typing python gives you the older version of Python -- the one that was installed by default. That's not the one you want. At the time of this writing, the newest version is called python2.3. You'll probably want to change the path on the first line of the sample scripts to point to the newer version. This is the complete path of the newer version of Python that you just installed. Use this on the #!line (the first line of each script) to ensure that scripts are running under the latest version of Python, and be sure to type python2.3 to get into the interactive shell. -1.6. Python on Debian GNU/Linux
-If you are lucky enough to be running Debian GNU/Linux, you install Python through the apt command. -
Example 1.3. Installing on Debian GNU/Linux
+1.6. Python on Debian GNU/Linux
+If you are lucky enough to be running Debian GNU/Linux, you install Python through the apt command. +
Example 1.3. Installing on Debian GNU/Linux
localhost:~$ su - Password: [enter your root password] localhost:~# apt-get install python @@ -640,16 +640,16 @@ if __name__ == "__main__": print buildConnectionString(myParams)Now run this program and see what happens.
-
In the ActivePython IDE on Windows, you can run the Python program you're editing by choosing + In the ActivePython IDE on Windows, you can run the Python program you're editing by choosing File->Run... (Ctrl-R). Output is displayed in the interactive window. -
In the Python IDE on Mac OS, you can run a Python program with -Python->Run window... (Cmd-R), but there is an important option you must set first. Open the .pyfile in the IDE, pop up the options menu by clicking the black triangle in the upper-right corner of the window, and make sure the Run as __main__ option is checked. This is a per-file setting, but you'll only need to do it once per file. +In the Python IDE on Mac OS, you can run a Python program with +Python->Run window... (Cmd-R), but there is an important option you must set first. Open the .pyfile in the IDE, pop up the options menu by clicking the black triangle in the upper-right corner of the window, and make sure the Run as __main__ option is checked. This is a per-file setting, but you'll only need to do it once per file.-
On UNIX-compatible systems (including Mac OS X), you can run a Python program from the command line: python odbchelper.pyThe id="odbchelper.output" output of
odbchelper.pywill look like this:server=mpilgrim;uid=sa;database=master;pwd=secret2.2. Declaring Functions
-Python has functions like most other languages, but it does not have separate header files like C++ or
interface/implementationsections like Pascal. When you need a function, just declare it, like this: +On UNIX-compatible systems (including Mac OS X), you can run a Python program from the command line: python odbchelper.pyThe id="odbchelper.output" output of
odbchelper.pywill look like this:server=mpilgrim;uid=sa;database=master;pwd=secret2.2. Declaring Functions
+Python has functions like most other languages, but it does not have separate header files like C++ or
interface/implementationsections like Pascal. When you need a function, just declare it, like this:def buildConnectionString(params):Note that the keyword
defstarts the function declaration, followed by the function name, followed by the arguments in parentheses. Multiple arguments (not shown here) are separated with commas. @@ -661,7 +661,7 @@ In fact, every Python function returns a value; if the function ever executes aThe argument,
params, doesn't specify a datatype. In Python, variables are never explicitly typed. Python figures out what type a variable is and keeps track of it internally.-
In Java, C++, and other statically-typed languages, you must specify the datatype of the function return value and each function argument. + In Java, C++, and other statically-typed languages, you must specify the datatype of the function return value and each function argument. In Python, you never explicitly specify the datatype of anything. Based on what value you assign, Python keeps track of the datatype internally. 2.2.1. How Python's Datatypes Compare to Other Programming Languages
An erudite reader sent me this explanation of how Python compares to other programming languages: @@ -669,7 +669,7 @@ In fact, every Python function returns a value; if the function ever executes a
- statically typed language
- A language in which types are fixed at compile time. Most statically typed languages enforce this by requiring you to declare - all variables with their datatypes before using them. Java and C are statically typed languages. + all variables with their datatypes before using them. Java and C are statically typed languages.
- dynamically typed language
- A language in which types are discovered at execution time; the opposite of statically typed. VBScript and Python are dynamically typed, because they figure out what type a variable is when you first assign it a value. @@ -698,7 +698,7 @@ def buildConnectionString(params): need to give your function a
docstring, but you always should. I know you've heard this in every programming class you've ever taken, but Python gives you an added incentive: thedocstringis available at runtime as an attribute of the function.-
Many Python IDEs use the docstringto provide context-sensitive documentation, so that when you type a function name, itsdocstringappears as a tooltip. This can be incredibly helpful, but it's only as good as thedocstrings you write. +Many Python IDEs use the docstringto provide context-sensitive documentation, so that when you type a function name, itsdocstringappears as a tooltip. This can be incredibly helpful, but it's only as good as thedocstrings you write. @@ -712,7 +712,7 @@ need to give your function adocstring, but you always should. I kn if __name__ == "__main__":Some quick observations before you get to the good stuff. First, parentheses are not required around the
ifexpression. Second, theifstatement ends with a colon, and is followed by indented code.-
Like C, Python uses ==for comparison and=for assignment. Unlike C, Python does not support in-line assignment, so there's no chance of accidentally assigning the value you thought you were comparing. +Like C, Python uses ==for comparison and=for assignment. Unlike C, Python does not support in-line assignment, so there's no chance of accidentally assigning the value you thought you were comparing.So why is this particular
ifstatement a trick? Modules are objects, and all modules have a built-in attribute__name__. A module's__name__depends on how you're using the module. If youimportthe module, then__name__is the module's filename, without a directory path or file extension. But you can also run the module directly as a standalone program, in which case__name__will be a special default value,__main__.>>> import odbchelper @@ -746,7 +746,7 @@ if __name__ == "__main__":Also notice that the variable assignment is one command split over several lines, with a backslash (“
\”) serving as a line-continuation marker.-
When a command is split among several lines with the line-continuation marker (“ \”), the continued lines can be indented in any manner; Python's normally stringent indentation rules do not apply. If your Python IDE auto-indents the continued line, you should probably accept its default unless you have a burning reason not to. +When a command is split among several lines with the line-continuation marker (“ \”), the continued lines can be indented in any manner; Python's normally stringent indentation rules do not apply. If your Python IDE auto-indents the continued line, you should probably accept its default unless you have a burning reason not to.Strictly speaking, expressions in parentheses, straight brackets, or curly braces (like defining a dictionary) can be split into multiple lines with or without the line continuation character (“
\”). I like to include the backslash even when it's not required because I think it makes the code easier to read, but that's a matter of style.Third, you never declared the variable myParams, you just assigned a value to it. This is like VBScript without the
option explicitoption. Luckily, unlike VBScript, Python will not allow you to reference a variable that has never been assigned a value; trying to do so will raise an exception. @@ -770,7 +770,7 @@ NameError: There is no variable named 'x' 'e'
- v is a tuple of three elements, and
(x, y, z)is a tuple of three variables. Assigning one to the other assigns each of the values of v to each of the variables, in order. -This has all sorts of uses. I often want to assign names to a range of values. In C, you would use
enumand manually list each constant and its associated value, which seems especially tedious when the values are consecutive. +This has all sorts of uses. I often want to assign names to a range of values. In C, you would use
enumand manually list each constant and its associated value, which seems especially tedious when the values are consecutive. In Python, you can use the built-inrangefunction with multi-variable assignment to quickly assign consecutive values.Example 3.20. Assigning Consecutive Values
>>> range(7) ① [0, 1, 2, 3, 4, 5, 6] @@ -784,7 +784,7 @@ NameError: There is no variable named 'x'
- The built-in
rangefunction returns a list of integers. In its simplest form, it takes an upper limit and returns a zero-based list counting up to but not including the upper limit. (If you like, you can pass other parameters to specify a base other than0and a step other than1. You canprint range.__doc__for details.) -- MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, and SUNDAY are the variables you're defining. (This example came from the
calendarmodule, a fun little module that prints calendars, like the UNIX programcal. Thecalendarmodule defines integer constants for days of the week.) +- MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, and SUNDAY are the variables you're defining. (This example came from the
calendarmodule, a fun little module that prints calendars, like the UNIX programcal. Thecalendarmodule defines integer constants for days of the week.)- Now each variable has its value: MONDAY is
0, TUESDAY is1, and so forth.You can also use multi-variable assignment to build functions that return multiple values, simply by returning a tuple of all the values. The caller can treat it as a tuple, or assign the values to individual variables. Many standard Python libraries do this, including the
osmodule, which you'll discuss in Chapter 6. @@ -801,7 +801,7 @@ NameError: There is no variable named 'x' to insert values into a string with the%splaceholder.-
String formatting in Python uses the same syntax as the sprintffunction in C. +String formatting in Python uses the same syntax as the sprintffunction in C.Example 3.21. Introducing String Formatting
>>> k = "uid" >>> v = "sa" >>> "%s=%s" % (k, v) ① @@ -833,7 +833,7 @@ TypeError: cannot concatenate 'str' and 'int' objectsString formatting works with integers by specifying%dinstead of%s.- Trying to concatenate a string with a non-string raises an exception. Unlike string formatting, string concatenation works only when everything is already a string. -
As with
printfin C, string formatting in Python is like a Swiss Army knife. There are options galore, and modifier strings to specially format many different types of values. +As with
printfin C, string formatting in Python is like a Swiss Army knife. There are options galore, and modifier strings to specially format many different types of values.Example 3.23. Formatting Numbers
>>> print "Today's stock price: %f" % 50.4625 ① 50.462500 @@ -852,7 +852,7 @@ TypeError: cannot concatenate 'str' and 'int' objects- Python Library Reference summarizes all the string formatting format characters. -
- Effective AWK Programming discusses all the format characters and advanced string formatting techniques like specifying width, precision, and zero-padding. +
- Effective AWK Programming discusses all the format characters and advanced string formatting techniques like specifying width, precision, and zero-padding.
3.6. Mapping Lists
@@ -953,7 +953,7 @@ calledsplit.- Python Library Reference documents the
stringmodule. -- The Whole Python FAQ explains why
joinis a string method instead of a list method. +- The Whole Python FAQ explains why
joinis a string method instead of a list method.3.7.1. Historical Note on String Methods
@@ -981,9 +981,9 @@ if __name__ == "__main__":Before diving into the next chapter, make sure you're comfortable doing all of these things:
-
- Using the Python IDE to test expressions interactively +
- Using the Python IDE to test expressions interactively -
- Writing Python programs and running them from within your IDE, or from the command line +
- Writing Python programs and running them from within your IDE, or from the command line
- Importing modules and calling their functions @@ -1029,7 +1029,7 @@ if __name__ == "__main__": ④ ⑤The
if __name__trick allows this program do something useful when run by itself, without interfering with its use as a module for other programs. In this case, the program simply prints out thedocstringof theinfofunction.ifstatements use==for comparison, and parentheses are not required. -The
infofunction is designed to be used by you, the programmer, while working in the Python IDE. It takes any object that has functions or methods (like a module, which has functions, or a list, which has methods) and +The
infofunction is designed to be used by you, the programmer, while working in the Python IDE. It takes any object that has functions or methods (like a module, which has functions, or a list, which has methods) and prints out the functions and theirdocstrings.Example 4.2. Sample Usage of
apihelper.py>>> from apihelper import info >>> li = [] @@ -1054,7 +1054,7 @@ buildConnectionString Build a connection string from a dictionary Retur Returns string.4.2. Using Optional and Named Arguments
Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default - value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part. + value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part.
Here is an example of
info, a function with two optional arguments:def info(object, spacing=10, collapse=1):spacing and collapse are optional, because they have default values defined. object is required, because it has no default value. If
infois called with only one argument, spacing defaults to10and collapse defaults to1. Ifinfois called with two arguments, collapse still defaults to1. @@ -1247,7 +1247,7 @@ True4.4.2.
getattrAs a DispatcherA common usage pattern of
getattris as a dispatcher. For example, if you had a program that could output data in a variety of different formats, you could define separate functions for each output format and use a single dispatch function to call the right one. -For example, let's imagine a program that prints site statistics in HTML, XML, and plain text formats. The choice of output format could be specified on the command line, or stored in a configuration +
For example, let's imagine a program that prints site statistics in HTML, XML, and plain text formats. The choice of output format could be specified on the command line, or stored in a configuration file. A
statsoutmodule defines three functions,output_html,output_xml, andoutput_text. Then the main program defines a single output function, like this:Example 4.12. Creating a Dispatcher with
getattrimport statsout @@ -1345,7 +1345,7 @@ thepopmethod of a list) and user-defined (like thebuildCon- If all values are false,
orreturns the last value.orevaluates'', which is false, then[], which is false, then{}, which is false, and returns{}.- Note that
orevaluates values only until it finds one that is true in a boolean context, and then it ignores the rest. This distinction is important if some values can have side effects. Here, the functionsidefxis never called, becauseorevaluates'a', which is true, and returns'a'immediately. -If you're a C hacker, you are certainly familiar with the
bool ? a : bexpression, which evaluates to a ifboolis true, and b otherwise. Because of the wayandandorwork in Python, you can accomplish the same thing. +If you're a C hacker, you are certainly familiar with the
bool ? a : bexpression, which evaluates to a ifboolis true, and b otherwise. Because of the wayandandorwork in Python, you can accomplish the same thing.4.6.1. Using the
and-orTrickExample 4.17. Introducing the
and-orTrick>>> a = "first" >>> b = "second" @@ -1355,17 +1355,17 @@ thepopmethod of a list) and user-defined (like thebuildCon 'second'-
- This syntax looks similar to the
bool ? a : bexpression in C. The entire expression is evaluated from left to right, so theandis evaluated first.1 and 'first'evalutes to'first', then'first' or 'second'evalutes to'first'. +- This syntax looks similar to the
bool ? a : bexpression in C. The entire expression is evaluated from left to right, so theandis evaluated first.1 and 'first'evalutes to'first', then'first' or 'second'evalutes to'first'.0 and 'first'evalutes toFalse, and then0 or 'second'evaluates to'second'.However, since this Python expression is simply boolean logic, and not a special construct of the language, there is one extremely important difference - between this
and-ortrick in Python and thebool ? a : bsyntax in C. If the value of a is false, the expression will not work as you would expect it to. (Can you tell I was bitten by this? More than once?) + between thisand-ortrick in Python and thebool ? a : bsyntax in C. If the value of a is false, the expression will not work as you would expect it to. (Can you tell I was bitten by this? More than once?)Example 4.18. When the
and-orTrick Fails>>> a = "" >>> b = "second" >>> 1 and a or b ① 'second'
- Since a is an empty string, which Python considers false in a boolean context,
1 and ''evalutes to'', and then'' or 'second'evalutes to'second'. Oops! That's not what you wanted. -The
and-ortrick,bool and a or b, will not work like the C expressionbool ? a : bwhen a is false in a boolean context. +The
and-ortrick,bool and a or b, will not work like the C expressionbool ? a : bwhen a is false in a boolean context.The real trick behind the
and-ortrick, then, is to make sure that the value of a is never false. One common way of doing this is to turn a into[a]and b into[b], then taking the first element of the returned list, which will be either a or b.Example 4.19. Using the
and-orTrick Safely>>> a = "" >>> b = "second" @@ -1436,9 +1436,9 @@ a test
- Python Knowledge Base discusses using
lambdato call functions indirectly. -- Python Tutorial shows how to access outside variables from inside a
lambdafunction. (PEP 227 explains how this will change in future versions of Python.) +- Python Tutorial shows how to access outside variables from inside a
lambdafunction. (PEP 227 explains how this will change in future versions of Python.) -- The Whole Python FAQ has examples of obfuscated one-liners using
lambda. +- The Whole Python FAQ has examples of obfuscated one-liners using
lambda.4.8. Putting It All Together
@@ -1479,12 +1479,12 @@ True 'None'-
- You can easily define a function that has no
docstring, so its__doc__attribute isNone. Confusingly, if you evaluate the__doc__attribute directly, the Python IDE prints nothing at all, which makes sense if you think about it, but is still unhelpful. +- You can easily define a function that has no
docstring, so its__doc__attribute isNone. Confusingly, if you evaluate the__doc__attribute directly, the Python IDE prints nothing at all, which makes sense if you think about it, but is still unhelpful.- You can verify that the value of the
__doc__attribute is actuallyNoneby comparing it directly.- The
strfunction takes the null value and returns a string representation of it,'None'.-
In SQL, you must use IS NULLinstead of= NULLto compare a null value. In Python, you can use either== Noneoris None, butis Noneis faster. +In SQL, you must use IS NULLinstead of= NULLto compare a null value. In Python, you can use either== Noneoris None, butis Noneis faster.Now that you are guaranteed to have a string, you can pass the string to processFunc, which you have already defined as a function that either does or doesn't collapse whitespace. Now you see why it was important to use
strto convert aNonevalue into a string representation. processFunc is assuming a string argument and calling itssplitmethod, which would crash if you passed itNonebecauseNonedoesn't have asplitmethod.Stepping back even further, you see that you're using string formatting again to concatenate the return value of processFunc with the return value of method's
ljustmethod. This is a new string method that you haven't seen before.Example 4.24. Introducing
ljust>>> s = 'buildConnectionString' @@ -1703,7 +1703,7 @@ can import individual items or usefrom module import *
from module import *in Python is likeimport module.*in Java;import modulein Python is likeimport modulein Java. -Example 5.2.
import modulevs.from module import>>> import types +Example 5.2.
import modulevs.from module import>>> import types >>> types.FunctionType ① <type 'function'> >>> FunctionType ② @@ -1736,7 +1736,7 @@ NameError: There is no variable named 'FunctionType'Further Reading on Module Importing Techniques
-
- eff-bot has more to say on
import modulevs.from module import. +- eff-bot has more to say on
import modulevs.from module import.- Python Tutorial discusses advanced import techniques, including
from module import *. @@ -1756,10 +1756,10 @@ class Loaf: ①- You probably guessed this, but everything in a class is indented, just like the code within a function,
ifstatement,forloop, and so forth. The first thing not indented is not in the class.-
The passstatement in Python is like an empty set of braces ({}) in Java or C. +The passstatement in Python is like an empty set of braces ({}) in Java or C.Of course, realistically, most classes will be inherited from other classes, and they will define their own class methods and attributes. But as you've just seen, there is nothing that a class absolutely must have, other than a name. In particular, -C++ programmers may find it odd that Python classes don't have explicit constructors and destructors. Python classes do have something similar to a constructor: the
__init__method. +C++ programmers may find it odd that Python classes don't have explicit constructors and destructors. Python classes do have something similar to a constructor: the__init__method.Example 5.4. Defining the
FileInfoClassfrom UserDict import UserDict @@ -1791,7 +1791,7 @@ class FileInfo(UserDict): them optional to the caller. In this case, filename has a default value ofNone, which is the Python null value.-
By convention, the first argument of any Python class method (the reference to the current instance) is called self. This argument fills the role of the reserved wordthisin C++ or Java, butselfis not a reserved word in Python, merely a naming convention. Nonetheless, please don't call it anything butself; this is a very strong convention. +By convention, the first argument of any Python class method (the reference to the current instance) is called self. This argument fills the role of the reserved wordthisin C++ or Java, butselfis not a reserved word in Python, merely a naming convention. Nonetheless, please don't call it anything butself; this is a very strong convention.Example 5.6. Coding the
FileInfoClassclass FileInfo(UserDict): "store file metadata" @@ -1844,7 +1844,7 @@ class FileInfo(UserDict):- Remember when the
__init__method assigned its filename argument toself["name"]? Well, here's the result. The arguments you pass when you create the class instance get sent right along to the__init__method (along with the object reference,self, which Python adds for free).-
In Python, simply call a class as if it were a function to create a new instance of the class. There is no explicit newoperator like C++ or Java. +In Python, simply call a class as if it were a function to create a new instance of the class. There is no explicit newoperator like C++ or Java.5.4.1. Garbage Collection
If creating new instances is easy, destroying them is even easier. In general, there is no need to explicitly free instances, because they are freed automatically when the variables assigned to them go out of scope. Memory leaks are rare in Python. @@ -1874,7 +1874,7 @@ class FileInfo(UserDict):
As you've seen,
FileInfois a class that acts like a dictionary. To explore this further, let's look at theUserDictclass in theUserDictmodule, which is the ancestor of theFileInfoclass. This is nothing special; the class is written in Python and stored in a.pyfile, just like any other Python code. In particular, it's stored in thelibdirectory in your Python installation.-
In the ActivePython IDE on Windows, you can quickly open any module in your library path by selecting + In the ActivePython IDE on Windows, you can quickly open any module in your library path by selecting File->Locate... (Ctrl-L). Example 5.9. Defining the
UserDictClassclass UserDict: ① @@ -1887,24 +1887,24 @@ class UserDict: ①- This is the
__init__method that you overrode in theFileInfoclass. Note that the argument list in this ancestor class is different than the descendant. That's okay; each subclass can have its own set of arguments, as long as it calls the ancestor with the correct arguments. Here the ancestor class has a way to define initial values (by passing a dictionary in the dict argument) which theFileInfodoes not use. -- Python supports data attributes (called “instance variables” in Java and Powerbuilder, and “member variables” in C++). Data attributes are pieces of data held by a specific instance of a class. In this case, each instance of
UserDictwill have a data attribute data. To reference this attribute from code outside the class, you qualify it with the instance name,instance.data, in the same way that you qualify a function with its module name. To reference a data attribute from within the class, +- Python supports data attributes (called “instance variables” in Java and Powerbuilder, and “member variables” in C++). Data attributes are pieces of data held by a specific instance of a class. In this case, each instance of
UserDictwill have a data attribute data. To reference this attribute from code outside the class, you qualify it with the instance name,instance.data, in the same way that you qualify a function with its module name. To reference a data attribute from within the class, you useselfas the qualifier. By convention, all data attributes are initialized to reasonable values in the__init__method. However, this is not required, since data attributes, like local variables, spring into existence when they are first assigned a value.- The
updatemethod is a dictionary duplicator: it copies all the keys and values from one dictionary to another. This does not clear the target dictionary first; if the target dictionary already has some keys, the ones from the source dictionary will be overwritten, but others will be left untouched. Think ofupdateas a merge function, not a copy function.- This is a syntax you may not have seen before (I haven't used it in the examples in this book). It's an
ifstatement, but instead of having an indented block starting on the next line, there is just a single statement on the same line, after the colon. This is perfectly legal syntax, which is just a shortcut you can use when you have only one statement - in a block. (It's like specifying a single statement without braces in C++.) You can use this syntax, or you can have indented code on subsequent lines, but you can't do both for the same block. + in a block. (It's like specifying a single statement without braces in C++.) You can use this syntax, or you can have indented code on subsequent lines, but you can't do both for the same block.-
Java and Powerbuilder support function overloading by argument list, i.e. one class can have multiple methods with the same name but a different number of arguments, or arguments of different types. - Other languages (most notably PL/SQL) even support function overloading by argument name; i.e. one class can have multiple methods with the same name and the same number of arguments of the same type but different argument + Java and Powerbuilder support function overloading by argument list, i.e. one class can have multiple methods with the same name but a different number of arguments, or arguments of different types. + Other languages (most notably PL/SQL) even support function overloading by argument name; i.e. one class can have multiple methods with the same name and the same number of arguments of the same type but different argument names. Python supports neither of these; it has no form of function overloading whatsoever. Methods are defined solely by their name, and there can be only one method per class with a given name. So if a descendant class has an __init__method, it always overrides the ancestor__init__method, even if the descendant defines it with a different argument list. And the same rule applies to any other method.
Guido, the original author of Python, explains method overriding this way: "Derived classes may override methods of their base classes. Because methods have no special privileges when calling other methods of the same object, a method of a base class that calls another method defined - in the same base class, may in fact end up calling a method of a derived class that overrides it. (For C++ programmers: all methods in Python are effectively virtual.)" If that doesn't make sense to you (it confuses the hell out of me), feel free to ignore it. + in the same base class, may in fact end up calling a method of a derived class that overrides it. (For C++ programmers: all methods in Python are effectively virtual.)" If that doesn't make sense to you (it confuses the hell out of me), feel free to ignore it. I just thought I'd pass it along. @@ -2190,7 +2190,7 @@ AttributeError: 'MP3FileInfo' instance has no attribute '__parse'
Like many other programming languages, Python has exception handling via try...exceptblocks.-
Python uses try...exceptto handle exceptions andraiseto generate them. Java and C++ usetry...catchto handle exceptions, andthrowto generate them. +Python uses try...exceptto handle exceptions andraiseto generate them. Java and C++ usetry...catchto handle exceptions, andthrowto generate them.Exceptions are everywhere in Python. Virtually every module in the standard Python library uses them, and Python itself will raise them in a lot of different circumstances. You've already seen them repeatedly throughout this book.
@@ -2205,8 +2205,8 @@ AttributeError: 'MP3FileInfo' instance has no attribute '__parse'
-Mixing datatypes without coercion will raise a TypeErrorexception.In each of these cases, you were simply playing around in the Python IDE: an error occurred, the exception was printed (depending on your IDE, perhaps in an intentionally jarring shade of red), and that was that. This is called an unhandled exception. When the exception was raised, there was no code to explicitly notice it and deal with it, so it bubbled its -way back to the default behavior built in to Python, which is to spit out some debugging information and give up. In the IDE, that's no big deal, but if that happened while your actual Python program was running, the entire program would come to a screeching halt. +
In each of these cases, you were simply playing around in the Python IDE: an error occurred, the exception was printed (depending on your IDE, perhaps in an intentionally jarring shade of red), and that was that. This is called an unhandled exception. When the exception was raised, there was no code to explicitly notice it and deal with it, so it bubbled its +way back to the default behavior built in to Python, which is to spit out some debugging information and give up. In the IDE, that's no big deal, but if that happened while your actual Python program was running, the entire program would come to a screeching halt.
An exception doesn't need result in a complete program crash, though. Exceptions, when raised, can be handled. Sometimes an exception is really because you have a bug in your code (like accessing a variable that doesn't exist), but many times, an exception is something you can anticipate. If you're opening a file, it might not exist. If you're connecting to a database, it might be unavailable, or you might not have the correct security credentials to access it. If you know @@ -2239,7 +2239,7 @@ exceptions, errors occur immediately, and you can handle them in a standard way or to support multiple platforms (where platform-specific code is separated into different modules).
You can also define your own exceptions by creating a class that inherits from the built-in
Exceptionclass, and then raise your exceptions with theraisecommand. See the further reading section if you're interested in doing this.The next example demonstrates how to use an exception to support platform-specific functionality. This code comes from the -
getpassmodule, a wrapper module for getting a password from the user. Getting a password is accomplished differently on UNIX, Windows, and Mac OS platforms, but this code encapsulates all of those differences. +getpassmodule, a wrapper module for getting a password from the user. Getting a password is accomplished differently on UNIX, Windows, and Mac OS platforms, but this code encapsulates all of those differences.Example 6.2. Supporting Platform-Specific Functionality
# Bind the name getpass to the appropriate function try: @@ -2259,9 +2259,9 @@ exceptions, errors occur immediately, and you can handle them in a standard way else: getpass = unix_getpass-
termiosis a UNIX-specific module that provides low-level control over the input terminal. If this module is not available (because it's not +termiosis a UNIX-specific module that provides low-level control over the input terminal. If this module is not available (because it's not on your system, or your system doesn't support it), the import fails and Python raises anImportError, which you catch. -- OK, you didn't have
termios, so let's trymsvcrt, which is a Windows-specific module that provides an API to many useful functions in the Microsoft Visual C++ runtime services. If this import fails, Python will raise anImportError, which you catch. +- OK, you didn't have
termios, so let's trymsvcrt, which is a Windows-specific module that provides an API to many useful functions in the Microsoft Visual C++ runtime services. If this import fails, Python will raise anImportError, which you catch.- If the first two didn't work, you try to import a function from
EasyDialogs, which is a Mac OS-specific module that provides functions to pop up dialog boxes of various types. Once again, if this import fails, Python will raise anImportError, which you catch.- None of these platform-specific modules is available (which is possible, since Python has been ported to a lot of different platforms), so you need to fall back on a default password input function (which is defined elsewhere in the
getpassmodule). Notice what you're doing here: assigning the functiondefault_getpassto the variable getpass. If you read the officialgetpassdocumentation, it tells you that thegetpassmodule defines agetpassfunction. It does this by binding getpass to the correct function for your platform. Then when you call thegetpassfunction, you're really calling a platform-specific function that this code has set up for you. You don't need to know or @@ -2358,7 +2358,7 @@ ValueError: I/O operation on closed file- Just because a file is closed doesn't mean that the file object ceases to exist. The variable f will continue to exist until it goes out of scope or gets manually deleted. However, none of the methods that manipulate an open file will work once the file has been closed; they all raise an exception.
- Calling
closeon a file object whose file is already closed does not raise an exception; it fails silently. -6.2.3. Handling I/O Errors
+6.2.3. Handling I/O Errors
Now you've seen enough to understand the file handling code in the
fileinfo.pysample code from teh previous chapter. This example shows how to safely open and read from a file and gracefully handle errors.Example 6.6. File Objects in
MP3FileInfo@@ -2486,7 +2486,7 @@ USERNAME=mpilgrim [...snip...]
- os.environ is a dictionary of the environment variables defined on your system. In Windows, these are your user and system variables - accessible from MS-DOS. In UNIX, they are the variables exported in your shell's startup scripts. In Mac OS, there is no concept of environment variables, so this dictionary is empty. + accessible from MS-DOS. In UNIX, they are the variables exported in your shell's startup scripts. In Mac OS, there is no concept of environment variables, so this dictionary is empty.
os.environ.items()returns a list of tuples:[(key1, value1), (key2, value2), ...]. Theforloop iterates through this list. The first round, it assignskey1to k andvalue1to v, so k =USERPROFILEand v =C:\Documents and Settings\mpilgrim. In the second round, k gets the second key,OS, and v gets the corresponding value,Windows_NT.- With multi-variable assignment and list comprehensions, you can replace the entire
forloop with a single statement. Whether you actually do this in real code is a matter of personal coding style. I like it because it makes it clear that what I'm doing is mapping a dictionary into a list, then joining the list into a single string. @@ -2530,7 +2530,7 @@ UserDict stat
- The
sysmodule contains system-level information, such as the version of Python you're running (orsys.version), and system-level options such as the maximum allowed recursion depth (sys.version_infoandsys.getrecursionlimit()). -sys.setrecursionlimit()is a dictionary containing all the modules that have ever been imported since Python was started; the key is the module name, the value is the module object. Note that this is more than just the modules your program has imported. Python preloads some modules on startup, and if you're using a Python IDE,sys.modulescontains all the modules imported by all the programs you've run within the IDE. +sys.modulesis a dictionary containing all the modules that have ever been imported since Python was started; the key is the module name, the value is the module object. Note that this is more than just the modules your program has imported. Python preloads some modules on startup, and if you're using a Python IDE,sys.modulescontains all the modules imported by all the programs you've run within the IDE.sys.modulesThis example demonstrates how to use
.sys.modulesExample 6.13. Using
sys.modules>>> import fileinfo ① >>> print '\n'.join(sys.modules.keys()) @@ -2604,7 +2604,7 @@ stat- In this slightly less trivial case,
joinwill add an extra backslash to the pathname before joining it to the filename. I was overjoyed when I discovered this, sinceaddSlashIfNecessaryis one of the stupid little functions I always need to write when building up my toolbox in a new language. Do not write this stupid little function in Python; smart people have already taken care of it for you.expanduserwill expand a pathname that uses~to represent the current user's home directory. This works on any platform where users have a home directory, like Windows, -UNIX, and Mac OS X; it has no effect on Mac OS. +UNIX, and Mac OS X; it has no effect on Mac OS.- Combining these techniques, you can easily construct pathnames for directories and files under the user's home directory.
Example 6.17. Splitting Pathnames
>>> os.path.split("c:\\music\\ap\\mahadeva.mp3") ① ('c:\\music\\ap', 'mahadeva.mp3') @@ -2662,14 +2662,14 @@ def listDirectory(directory, fileExtList): if os.path.splitext(f)[1] in fileExtList] ③ ④ ⑤
os.listdir(directory)returns a list of all the files and folders in directory. -- Iterating through the list with f, you use
os.path.normcase(f)to normalize the case according to operating system defaults.normcaseis a useful little function that compensates for case-insensitive operating systems that think thatmahadeva.mp3andmahadeva.MP3are the same file. For instance, on Windows and Mac OS,normcasewill convert the entire filename to lowercase; on UNIX-compatible systems, it will return the filename unchanged. +- Iterating through the list with f, you use
os.path.normcase(f)to normalize the case according to operating system defaults.normcaseis a useful little function that compensates for case-insensitive operating systems that think thatmahadeva.mp3andmahadeva.MP3are the same file. For instance, on Windows and Mac OS,normcasewill convert the entire filename to lowercase; on UNIX-compatible systems, it will return the filename unchanged.- Iterating through the normalized list with f again, you use
os.path.splitext(f)to split each filename into name and extension.- For each file, you see if the extension is in the list of file extensions you care about (fileExtList, which was passed to the
listDirectoryfunction).- For each file you care about, you use
os.path.join(directory, f)to construct the full pathname of the file, and return a list of the full pathnames.
Whenever possible, you should use the functions in osandos.pathfor file, directory, and path manipulations. These modules are wrappers for platform-specific modules, so functions like -os.path.splitwork on UNIX, Windows, Mac OS, and any other platform supported by Python. +os.path.splitwork on UNIX, Windows, Mac OS, and any other platform supported by Python.There is one other way to get the contents of a directory. It's very powerful, and it uses the sort of wildcards that you may already be familiar with from working on the command line.
Example 6.20. Listing Directories with
glob@@ -2735,7 +2735,7 @@ def listDirectory(directory, fileExtList): ①Note that
listDirectoryis completely generic. It doesn't know ahead of time which types of files it will be getting, or which classes are defined that could potentially handle those files. It inspects the directory for the files to process, and then introspects its own module to see what special handler classes (likeMP3FileInfo) are defined. You can extend this program to handle other types of files simply by defining an appropriately-named class: -HTMLFileInfofor HTML files,DOCFileInfofor Word.docfiles, and so forth.listDirectorywill handle them all, without modification, by handing off the real work to the appropriate classes and collating the results. +HTMLFileInfofor HTML files,DOCFileInfofor Word.docfiles, and so forth.listDirectorywill handle them all, without modification, by handing off the real work to the appropriate classes and collating the results.6.7. Summary
The
fileinfo.pyprogram introduced in Chapter 5 should now make perfect sense.@@ -2829,10 +2829,10 @@ if __name__ == "__main__":-Chapter 8. HTML Processing
+Chapter 8. HTML Processing
8.1. Diving in
-I often see questions on comp.lang.python like “How can I list all the [headers|images|links] in my HTML document?” “How do I parse/translate/munge the text of my HTML document but leave the tags alone?” “How can I add/remove/quote attributes of all my HTML tags at once?” This chapter will answer all of these questions. -
Here is a complete, working Python program in two parts. The first part,
BaseHTMLProcessor.py, is a generic tool to help you process HTML files by walking through the tags and text blocks. The second part,dialect.py, is an example of how to useBaseHTMLProcessor.pyto translate the text of an HTML document but leave the tags alone. Read thedocstrings and comments to get an overview of what's going on. Most of it will seem like black magic, because it's not obvious how +I often see questions on comp.lang.python like “How can I list all the [headers|images|links] in my HTML document?” “How do I parse/translate/munge the text of my HTML document but leave the tags alone?” “How can I add/remove/quote attributes of all my HTML tags at once?” This chapter will answer all of these questions. +
Here is a complete, working Python program in two parts. The first part,
BaseHTMLProcessor.py, is a generic tool to help you process HTML files by walking through the tags and text blocks. The second part,dialect.py, is an example of how to useBaseHTMLProcessor.pyto translate the text of an HTML document but leave the tags alone. Read thedocstrings and comments to get an overview of what's going on. Most of it will seem like black magic, because it's not obvious how any of these class methods ever get called. Don't worry, all will be revealed in due time.Example 8.1.
BaseHTMLProcessor.pyIf you have not already done so, you can download this and other examples used in this book. @@ -3061,7 +3061,7 @@ def test(url): if __name__ == "__main__": test("http://diveintopython3.org/odbchelper_list.html")
Example 8.3. Output of
-dialect.pyRunning this script will translate Section 3.2, “Introducing Lists” into mock Swedish Chef-speak (from The Muppets), mock Elmer Fudd-speak (from Bugs Bunny cartoons), and mock Middle English (loosely based on Chaucer's The Canterbury Tales). If you look at the HTML source of the output pages, you'll see that all the HTML tags and attributes are untouched, but the text between the tags has been “translated” into the mock language. If you look closer, you'll see that, in fact, only the titles and paragraphs were translated; the +
Running this script will translate Section 3.2, “Introducing Lists” into mock Swedish Chef-speak (from The Muppets), mock Elmer Fudd-speak (from Bugs Bunny cartoons), and mock Middle English (loosely based on Chaucer's The Canterbury Tales). If you look at the HTML source of the output pages, you'll see that all the HTML tags and attributes are untouched, but the text between the tags has been “translated” into the mock language. If you look closer, you'll see that, in fact, only the titles and paragraphs were translated; the code listings and screen examples were left untouched.
<div class=abstract> @@ -3072,34 +3072,34 @@ in <span class=application>Powewbuiwdew</span>, bwace youwsewf fow <span class=application>Pydon</span> wists.</p> </div>8.2. Introducing
-sgmllib.pyHTML processing is broken into three steps: breaking down the HTML into its constituent pieces, fiddling with the pieces, and reconstructing the pieces into HTML again. The first step is done by
sgmllib.py, a part of the standard Python library. -The key to understanding this chapter is to realize that HTML is not just text, it is structured text. The structure is derived from the more-or-less-hierarchical sequence of start tags -and end tags. Usually you don't work with HTML this way; you work with it textually in a text editor, or visually in a web browser or web authoring tool.
sgmllib.pypresents HTML structurally. -
sgmllib.pycontains one important class:SGMLParser.SGMLParserparses HTML into useful pieces, like start tags and end tags. As soon as it succeeds in breaking down some data into a useful piece, -it calls a method on itself based on what it found. In order to use the parser, you subclass theSGMLParserclass and override these methods. This is what I meant when I said that it presents HTML structurally: the structure of the HTML determines the sequence of method calls and the arguments passed to each method. -
SGMLParserparses HTML into 8 kinds of data, and calls a separate method for each of them: +HTML processing is broken into three steps: breaking down the HTML into its constituent pieces, fiddling with the pieces, and reconstructing the pieces into HTML again. The first step is done by
sgmllib.py, a part of the standard Python library. +The key to understanding this chapter is to realize that HTML is not just text, it is structured text. The structure is derived from the more-or-less-hierarchical sequence of start tags +and end tags. Usually you don't work with HTML this way; you work with it textually in a text editor, or visually in a web browser or web authoring tool.
sgmllib.pypresents HTML structurally. +
sgmllib.pycontains one important class:SGMLParser.SGMLParserparses HTML into useful pieces, like start tags and end tags. As soon as it succeeds in breaking down some data into a useful piece, +it calls a method on itself based on what it found. In order to use the parser, you subclass theSGMLParserclass and override these methods. This is what I meant when I said that it presents HTML structurally: the structure of the HTML determines the sequence of method calls and the arguments passed to each method. +
SGMLParserparses HTML into 8 kinds of data, and calls a separate method for each of them:
- Start tag
-- An HTML tag that starts a block, like
<html>,<head>,<body>, or<pre>, or a standalone tag like<br>or<img>. When it finds a start tagtagname,SGMLParserwill look for a method calledstart_ortagnamedo_. For instance, when it finds atagname<pre>tag, it will look for astart_preordo_premethod. If found,SGMLParsercalls this method with a list of the tag's attributes; otherwise, it callsunknown_starttagwith the tag name and list of attributes. +- An HTML tag that starts a block, like
<html>,<head>,<body>, or<pre>, or a standalone tag like<br>or<img>. When it finds a start tagtagname,SGMLParserwill look for a method calledstart_ortagnamedo_. For instance, when it finds atagname<pre>tag, it will look for astart_preordo_premethod. If found,SGMLParsercalls this method with a list of the tag's attributes; otherwise, it callsunknown_starttagwith the tag name and list of attributes.- End tag
-- An HTML tag that ends a block, like
</html>,</head>,</body>, or</pre>. When it finds an end tag,SGMLParserwill look for a method calledend_. If found,tagnameSGMLParsercalls this method, otherwise it callsunknown_endtagwith the tag name. +- An HTML tag that ends a block, like
</html>,</head>,</body>, or</pre>. When it finds an end tag,SGMLParserwill look for a method calledend_. If found,tagnameSGMLParsercalls this method, otherwise it callsunknown_endtagwith the tag name.- Character reference
- An escaped character referenced by its decimal or hexadecimal equivalent, like
 . When found,SGMLParsercallshandle_charrefwith the text of the decimal or hexadecimal character equivalent.- Entity reference
-- An HTML entity, like
©. When found,SGMLParsercallshandle_entityrefwith the name of the HTML entity. +- An HTML entity, like
©. When found,SGMLParsercallshandle_entityrefwith the name of the HTML entity.- Comment
-- An HTML comment, enclosed in
<!-- ... -->. When found,SGMLParsercallshandle_commentwith the body of the comment. +- An HTML comment, enclosed in
<!-- ... -->. When found,SGMLParsercallshandle_commentwith the body of the comment.- Processing instruction
-- An HTML processing instruction, enclosed in
<? ... >. When found,SGMLParsercallshandle_piwith the body of the processing instruction. +- An HTML processing instruction, enclosed in
<? ... >. When found,SGMLParsercallshandle_piwith the body of the processing instruction.- Declaration
-- An HTML declaration, such as a
DOCTYPE, enclosed in<! ... >. When found,SGMLParsercallshandle_declwith the body of the declaration. +- An HTML declaration, such as a
DOCTYPE, enclosed in<! ... >. When found,SGMLParsercallshandle_declwith the body of the declaration.- Text data
- A block of text. Anything that doesn't fit into the other 7 categories. When found,
SGMLParsercallshandle_datawith the text. @@ -3108,13 +3108,13 @@ it calls a method on itself based on what it found. In order to use the parser,
Python 2.0 had a bug where SGMLParserwould not recognize declarations at all (handle_declwould never be called), which meant thatDOCTYPEs were silently ignored. This is fixed in Python 2.1. -
sgmllib.pycomes with a test suite to illustrate this. You can runsgmllib.py, passing the name of an HTML file on the command line, and it will print out the tags and other elements as it parses them. It does this by subclassing +
sgmllib.pycomes with a test suite to illustrate this. You can runsgmllib.py, passing the name of an HTML file on the command line, and it will print out the tags and other elements as it parses them. It does this by subclassing theSGMLParserclass and definingunknown_starttag,unknown_endtag,handle_dataand other methods which simply print their arguments.-
In the ActivePython IDE on Windows, you can specify command line arguments in the “Run script” dialog. Separate multiple arguments with spaces. + In the ActivePython IDE on Windows, you can specify command line arguments in the “Run script” dialog. Separate multiple arguments with spaces. Example 8.4. Sample test of
-sgmllib.pyHere is a snippet from the table of contents of the HTML version of this book. Of course your paths may vary. (If you haven't downloaded the HTML version of the book, you can do so at http://diveintopython3.org/. +
Here is a snippet from the table of contents of the HTML version of this book. Of course your paths may vary. (If you haven't downloaded the HTML version of the book, you can do so at http://diveintopython3.org/.
c:\python23\lib> type "c:\downloads\diveintopython3\html\toc\index.html"@@ -3148,11 +3148,11 @@ data: '\n 'Here's the roadmap for the rest of the chapter:
-
- Subclass
SGMLParserto create classes that extract interesting data out of HTML documents. +- Subclass
SGMLParserto create classes that extract interesting data out of HTML documents. -- Subclass
SGMLParserto createBaseHTMLProcessor, which overrides all 8 handler methods and uses them to reconstruct the original HTML from the pieces. +- Subclass
SGMLParserto createBaseHTMLProcessor, which overrides all 8 handler methods and uses them to reconstruct the original HTML from the pieces. -- Subclass
BaseHTMLProcessorto createDialectizer, which adds some methods to process specific HTML tags specially, and overrides thehandle_datamethod to provide a framework for processing the text blocks between the HTML tags. +- Subclass
BaseHTMLProcessorto createDialectizer, which adds some methods to process specific HTML tags specially, and overrides thehandle_datamethod to provide a framework for processing the text blocks between the HTML tags.- Subclass
Dialectizerto create classes that define text processing rules used byDialectizer.handle_data. @@ -3160,9 +3160,9 @@ data: '\n 'Along the way, you'll also learn about
locals,globals, and dictionary-based string formatting. -8.3. Extracting data from HTML documents
-To extract data from HTML documents, subclass the
SGMLParserclass and define methods for each tag or entity you want to capture. -The first step to extracting data from an HTML document is getting some HTML. If you have some HTML lying around on your hard drive, you can use file functions to read it, but the real fun begins when you get HTML from live web pages. +
8.3. Extracting data from HTML documents
+To extract data from HTML documents, subclass the
SGMLParserclass and define methods for each tag or entity you want to capture. +The first step to extracting data from an HTML document is getting some HTML. If you have some HTML lying around on your hard drive, you can use file functions to read it, but the real fun begins when you get HTML from live web pages.
Example 8.5. Introducing
urllib>>> import urllib ① >>> sock = urllib.urlopen("http://diveintopython3.org/") ② @@ -3185,11 +3185,11 @@ data: '\n ' [...snip...]-
- The
urllibmodule is part of the standard Python library. It contains functions for getting information about and actually retrieving data from Internet-based URLs (mainly web pages). -- The simplest use of
urllibis to retrieve the entire text of a web page using theurlopenfunction. Opening a URL is similar to opening a file. The return value ofurlopenis a file-like object, which has some of the same methods as a file object. -- The simplest thing to do with the file-like object returned by
urlopenisread, which reads the entire HTML of the web page into a single string. The object also supportsreadlines, which reads the text line by line into a list. +- The
urllibmodule is part of the standard Python library. It contains functions for getting information about and actually retrieving data from Internet-based URLs (mainly web pages). +- The simplest use of
urllibis to retrieve the entire text of a web page using theurlopenfunction. Opening a URL is similar to opening a file. The return value ofurlopenis a file-like object, which has some of the same methods as a file object. +- The simplest thing to do with the file-like object returned by
urlopenisread, which reads the entire HTML of the web page into a single string. The object also supportsreadlines, which reads the text line by line into a list.- When you're done with the object, make sure to
closeit, just like a normal file object. -- You now have the complete HTML of the home page of
http://diveintopython3.org/in a string, and you're ready to parse it. +- You now have the complete HTML of the home page of
http://diveintopython3.org/in a string, and you're ready to parse it.Example 8.6. Introducing
urllister.pyIf you have not already done so, you can download this and other examples used in this book.
@@ -3207,7 +3207,7 @@ class URLLister(SGMLParser):
resetis called by the__init__method ofSGMLParser, and it can also be called manually once an instance of the parser has been created. So if you need to do any initialization, do it inreset, not in__init__, so that it will be re-initialized properly when someone re-uses a parser instance. -start_ais called bySGMLParserwhenever it finds an<a>tag. The tag may contain anhrefattribute, and/or other attributes, likenameortitle. The attrs parameter is a list of tuples,[(attribute, value), (attribute, value), ...]. Or it may be just an<a>, a valid (if useless) HTML tag, in which case attrs would be an empty list. +start_ais called bySGMLParserwhenever it finds an<a>tag. The tag may contain anhrefattribute, and/or other attributes, likenameortitle. The attrs parameter is a list of tuples,[(attribute, value), (attribute, value), ...]. Or it may be just an<a>, a valid (if useless) HTML tag, in which case attrs would be an empty list.- You can find out whether this
<a>tag has anhrefattribute with a simple multi-variable list comprehension.- String comparisons like
k=='href'are always case-sensitive, but that's safe in this case, becauseSGMLParserconverts attribute names to lowercase while building attrs.Example 8.7. Using
urllister.py@@ -3234,15 +3234,15 @@ download/diveintopython3-common-5.0.zip ... rest of output omitted for brevity ...-
- Call the
feedmethod, defined inSGMLParser, to get HTML into the parser. +- Call the
feedmethod, defined inSGMLParser, to get HTML into the parser. [1] It takes a string, which is whatusock.read()returns. -- Like files, you should
closeyour URL objects as soon as you're done with them. -- You should
closeyour parser object, too, but for a different reason. You've read all the data and fed it to the parser, but thefeedmethod isn't guaranteed to have actually processed all the HTML you give it; it may buffer it, waiting for more. Be sure to callcloseto flush the buffer and force everything to be fully parsed. -- Once the parser is
closed, the parsing is complete, and parser.urls contains a list of all the linked URLs in the HTML document. (Your output may look different, if the download links have been updated by the time you read this.) +- Like files, you should
closeyour URL objects as soon as you're done with them. +- You should
closeyour parser object, too, but for a different reason. You've read all the data and fed it to the parser, but thefeedmethod isn't guaranteed to have actually processed all the HTML you give it; it may buffer it, waiting for more. Be sure to callcloseto flush the buffer and force everything to be fully parsed. +- Once the parser is
closed, the parsing is complete, and parser.urls contains a list of all the linked URLs in the HTML document. (Your output may look different, if the download links have been updated by the time you read this.)8.4. Introducing
BaseHTMLProcessor.py
SGMLParserdoesn't produce anything by itself. It parses and parses and parses, and it calls a method for each interesting thing it - finds, but the methods don't do anything.SGMLParseris an HTML consumer: it takes HTML and breaks it down into small, structured pieces. As you saw in the previous section, you can subclassSGMLParserto define classes that catch specific tags and produce useful things, like a list of all the links on a web page. Now you'll - take this one step further by defining a class that catches everythingSGMLParserthrows at it and reconstructs the complete HTML document. In technical terms, this class will be an HTML producer. + finds, but the methods don't do anything.SGMLParseris an HTML consumer: it takes HTML and breaks it down into small, structured pieces. As you saw in the previous section, you can subclassSGMLParserto define classes that catch specific tags and produce useful things, like a list of all the links on a web page. Now you'll + take this one step further by defining a class that catches everythingSGMLParserthrows at it and reconstructs the complete HTML document. In technical terms, this class will be an HTML producer.
BaseHTMLProcessorsubclassesSGMLParserand provides all 8 essential handler methods:unknown_starttag,unknown_endtag,handle_charref,handle_entityref,handle_comment,handle_pi,handle_decl, andhandle_data.Example 8.8. Introducing
BaseHTMLProcessorclass BaseHTMLProcessor(SGMLParser): @@ -3277,27 +3277,27 @@ class BaseHTMLProcessor(SGMLParser): def handle_decl(self, text): self.pieces.append("<!%(text)s>" % locals())-
reset, called bySGMLParser.__init__, initializes self.pieces as an empty list before calling the ancestor method. self.pieces is a data attribute which will hold the pieces of the HTML document you're constructing. Each handler method will reconstruct the HTML thatSGMLParserparsed, and each method will append that string to self.pieces. Note that self.pieces is a list. You might be tempted to define it as a string and just keep appending each piece to it. That would work, but +reset, called bySGMLParser.__init__, initializes self.pieces as an empty list before calling the ancestor method. self.pieces is a data attribute which will hold the pieces of the HTML document you're constructing. Each handler method will reconstruct the HTML thatSGMLParserparsed, and each method will append that string to self.pieces. Note that self.pieces is a list. You might be tempted to define it as a string and just keep appending each piece to it. That would work, but Python is much more efficient at dealing with lists. -[2]- Since
BaseHTMLProcessordoes not define any methods for specific tags (like thestart_amethod inURLLister),SGMLParserwill callunknown_starttagfor every start tag. This method takes the tag (tag) and the list of attribute name/value pairs (attrs), reconstructs the original HTML, and appends it to self.pieces. The string formatting here is a little strange; you'll untangle that (and also the odd-lookinglocalsfunction) later in this chapter. +[2]- Since
BaseHTMLProcessordoes not define any methods for specific tags (like thestart_amethod inURLLister),SGMLParserwill callunknown_starttagfor every start tag. This method takes the tag (tag) and the list of attribute name/value pairs (attrs), reconstructs the original HTML, and appends it to self.pieces. The string formatting here is a little strange; you'll untangle that (and also the odd-lookinglocalsfunction) later in this chapter.- Reconstructing end tags is much simpler; just take the tag name and wrap it in the
</...>brackets. -- When
SGMLParserfinds a character reference, it callshandle_charrefwith the bare reference. If the HTML document contains the reference , ref will be160. Reconstructing the original complete character reference just involves wrapping ref in&#...;characters. +- When
SGMLParserfinds a character reference, it callshandle_charrefwith the bare reference. If the HTML document contains the reference , ref will be160. Reconstructing the original complete character reference just involves wrapping ref in&#...;characters.- Entity references are similar to character references, but without the hash mark. Reconstructing the original entity reference requires wrapping ref in
&...;characters. (Actually, as an erudite reader pointed out to me, it's slightly more complicated than this. Only certain standard -HTML entites end in a semicolon; other similar-looking entities do not. Luckily for us, the set of standard HTML entities is defined in a dictionary in a Python module calledhtmlentitydefs. Hence the extraifstatement.) +HTML entites end in a semicolon; other similar-looking entities do not. Luckily for us, the set of standard HTML entities is defined in a dictionary in a Python module calledhtmlentitydefs. Hence the extraifstatement.)- Blocks of text are simply appended to self.pieces unaltered. -
- HTML comments are wrapped in
<!--...-->characters. +- HTML comments are wrapped in
<!--...-->characters.- Processing instructions are wrapped in
<?...>characters.-
The HTML specification requires that all non-HTML (like client-side JavaScript) must be enclosed in HTML comments, but not all web pages do this properly (and all modern web browsers are forgiving if they don't). BaseHTMLProcessoris not forgiving; if script is improperly embedded, it will be parsed as if it were HTML. For instance, if the script contains less-than and equals signs,SGMLParsermay incorrectly think that it has found tags and attributes.SGMLParseralways converts tags and attribute names to lowercase, which may break the script, andBaseHTMLProcessoralways encloses attribute values in double quotes (even if the original HTML document used single quotes or no quotes), which will certainly break the script. Always protect your client-side script - within HTML comments. +The HTML specification requires that all non-HTML (like client-side JavaScript) must be enclosed in HTML comments, but not all web pages do this properly (and all modern web browsers are forgiving if they don't). BaseHTMLProcessoris not forgiving; if script is improperly embedded, it will be parsed as if it were HTML. For instance, if the script contains less-than and equals signs,SGMLParsermay incorrectly think that it has found tags and attributes.SGMLParseralways converts tags and attribute names to lowercase, which may break the script, andBaseHTMLProcessoralways encloses attribute values in double quotes (even if the original HTML document used single quotes or no quotes), which will certainly break the script. Always protect your client-side script + within HTML comments.Example 8.9.
BaseHTMLProcessoroutputdef output(self): ① """Return processed HTML as a single string""" return "".join(self.pieces) ②-
- This is the one method in
BaseHTMLProcessorthat is never called by the ancestorSGMLParser. Since the other handler methods store their reconstructed HTML in self.pieces, this function is needed to join all those pieces into one string. As noted before, Python is great at lists and mediocre at strings, so you only create the complete string when somebody explicitly asks for it. +- This is the one method in
BaseHTMLProcessorthat is never called by the ancestorSGMLParser. Since the other handler methods store their reconstructed HTML in self.pieces, this function is needed to join all those pieces into one string. As noted before, Python is great at lists and mediocre at strings, so you only create the complete string when somebody explicitly asks for it.- If you prefer, you could use the
joinmethod of thestringmodule instead:string.join(self.pieces, "")Further reading
@@ -3307,7 +3307,7 @@ Python is much more efficient at dealing with lists.
8.5.
-localsandglobalsLet's digress from HTML processing for a minute and talk about how Python handles variables. Python has two built-in functions,
localsandglobals, which provide dictionary-based access to local and global variables. +Let's digress from HTML processing for a minute and talk about how Python handles variables. Python has two built-in functions,
localsandglobals, which provide dictionary-based access to local and global variables.Remember
locals? You first saw it here:def unknown_starttag(self, tag, attrs): @@ -3458,8 +3458,8 @@ meaningful keys and values already. LikeUsing dictionary-based string formatting with localsis a convenient way of making complex string formatting expressions more readable, but it comes with a price. There is a slight performance hit in making the call tolocals, sincelocalsbuilds a copy of the local namespace.8.7. Quoting attribute values
-A common question on comp.lang.python is “I have a bunch of HTML documents with unquoted attribute values, and I want to properly quote them all. How can I do this?”[4] (This is generally precipitated by a project manager who has found the HTML-is-a-standard religion joining a large project and proclaiming that all pages must validate against an HTML validator. Unquoted attribute values are a common violation of the HTML standard.) Whatever the reason, unquoted attribute values are easy to fix by feeding HTML through
BaseHTMLProcessor. -
BaseHTMLProcessorconsumes HTML (since it's descended fromSGMLParser) and produces equivalent HTML, but the HTML output is not identical to the input. Tags and attribute names will end up in lowercase, even if they started in uppercase +A common question on comp.lang.python is “I have a bunch of HTML documents with unquoted attribute values, and I want to properly quote them all. How can I do this?”[4] (This is generally precipitated by a project manager who has found the HTML-is-a-standard religion joining a large project and proclaiming that all pages must validate against an HTML validator. Unquoted attribute values are a common violation of the HTML standard.) Whatever the reason, unquoted attribute values are easy to fix by feeding HTML through
BaseHTMLProcessor. +
BaseHTMLProcessorconsumes HTML (since it's descended fromSGMLParser) and produces equivalent HTML, but the HTML output is not identical to the input. Tags and attribute names will end up in lowercase, even if they started in uppercase or mixed case, and attribute values will be enclosed in double quotes, even if they started in single quotes or with no quotes at all. It is this last side effect that you can take advantage of.Example 8.16. Quoting attribute values
@@ -3492,10 +3492,10 @@ at all. It is this last side effect that you can take advantage of. </body> </html>-
- Note that the attribute values of the
hrefattributes in the<a>tags are not properly quoted. (Also note that you're using triple quotes for something other than adocstring. And directly in the IDE, no less. They're very useful.) +- Note that the attribute values of the
hrefattributes in the<a>tags are not properly quoted. (Also note that you're using triple quotes for something other than adocstring. And directly in the IDE, no less. They're very useful.)- Feed the parser.
- Using the
outputfunction defined inBaseHTMLProcessor, you get the output as a single string, complete with quoted attribute values. While this may seem anti-climactic, think - about how much has actually happened here:SGMLParserparsed the entire HTML document, breaking it down into tags, refs, data, and so forth;BaseHTMLProcessorused those elements to reconstruct pieces of HTML (which are still stored in parser.pieces, if you want to see them); finally, you calledparser.output, which joined all the pieces of HTML into one string. + about how much has actually happened here:SGMLParserparsed the entire HTML document, breaking it down into tags, refs, data, and so forth;BaseHTMLProcessorused those elements to reconstruct pieces of HTML (which are still stored in parser.pieces, if you want to see them); finally, you calledparser.output, which joined all the pieces of HTML into one string.8.8. Introducing
dialect.py
Dialectizeris a simple (and silly) descendant ofBaseHTMLProcessor. It runs blocks of text through a series of substitutions, but it makes sure that anything within ablock passes through unaltered.<pre>...</pre>To handle the
<pre>blocks, you define two methods inDialectizer:start_preandend_pre. @@ -3508,7 +3508,7 @@ at all. It is this last side effect that you can take advantage of. self.unknown_endtag("pre") ⑤ self.verbatim -= 1⑥-
start_preis called every timeSGMLParserfinds a<pre>tag in the HTML source. (In a minute, you'll see exactly how this happens.) The method takes a single parameter, attrs, which contains the attributes of the tag (if any). attrs is a list of key/value tuples, just likeunknown_starttagtakes. +start_preis called every timeSGMLParserfinds a<pre>tag in the HTML source. (In a minute, you'll see exactly how this happens.) The method takes a single parameter, attrs, which contains the attributes of the tag (if any). attrs is a list of key/value tuples, just likeunknown_starttagtakes.- In the
resetmethod, you initialize a data attribute that serves as a counter for<pre>tags. Every time you hit a<pre>tag, you increment the counter; every time you hit a</pre>tag, you'll decrement the counter. (You could just use this as a flag and set it to1and reset it to0, but it's just as easy to do it this way, and this handles the odd (but possible) case of nested<pre>tags.) In a minute, you'll see how this counter is put to good use.- That's it, that's the only special processing you do for
<pre>tags. Now you pass the list of attributes along tounknown_starttagso it can do the default processing.end_preis called every timeSGMLParserfinds a</pre>tag. Since end tags can not contain attributes, the method takes no parameters. @@ -3563,7 +3563,7 @@ you need to override thehandle_datamethod.- In the ancestor
BaseHTMLProcessor, thehandle_datamethod simply appended the text to the output buffer, self.pieces. Here the logic is only slightly more complicated. If you're in the middle of ablock, self.verbatim will be some value greater than<pre>...</pre>0, and you want to put the text in the output buffer unaltered. Otherwise, you will call a separate method to process the substitutions, then put the result of that into the output buffer. In Python, this is a one-liner, using theand-ortrick.You're close to completely understanding
Dialectizer. The only missing link is the nature of the text substitutions themselves. If you know any Perl, you know that when complex text substitutions are required, the only real solution is regular expressions. The classes -later indialect.pydefine a series of regular expressions that operate on the text between the HTML tags. But you just had a whole chapter on regular expressions. You don't really want to slog through regular expressions again, do you? God knows I don't. I think you've learned enough +later indialect.pydefine a series of regular expressions that operate on the text between the HTML tags. But you just had a whole chapter on regular expressions. You don't really want to slog through regular expressions again, do you? God knows I don't. I think you've learned enough for one chapter.8.9. Putting it all together
It's time to put everything you've learned so far to good use. I hope you were paying attention. @@ -3596,7 +3596,7 @@ def translate(url, dialectName="chef"): ①
Why bother? After all, there are only 3
Dialectizerclasses; why not just use acasestatement? (Well, there's nocasestatement in Python, but why not just use a series ofifstatements?) One reason: extensibility. Thetranslatefunction has absolutely no idea how many Dialectizer classes you've defined. Imagine if you defined a newFooDialectizertomorrow;translatewould work by passing'foo'as the dialectName.Even better, imagine putting
FooDialectizerin a separate module, and importing it withfrom module import. You've already seen that this includes it inglobals(), sotranslatewould still work without modification, even thoughFooDialectizerwas in a separate file.Now imagine that the name of the dialect is coming from somewhere outside the program, maybe from a database or from a user-inputted -value on a form. You can use any number of server-side Python scripting architectures to dynamically generate web pages; this function could take a URL and a dialect name (both strings) in the query string of a web page request, and output the “translated” web page. +value on a form. You can use any number of server-side Python scripting architectures to dynamically generate web pages; this function could take a URL and a dialect name (both strings) in the query string of a web page request, and output the “translated” web page.
Finally, imagine a
Dialectizerframework with a plug-in architecture. You could put eachDialectizerclass in a separate file, leaving only thetranslatefunction indialect.py. Assuming a consistent naming scheme, thetranslatefunction could dynamic import the appropiate class from the appropriate file, given nothing but the dialect name. (You haven't seen dynamic importing yet, but I promise to cover it in a later chapter.) To add a new dialect, you would simply add an appropriately-named file in the plug-ins directory (likefoodialect.pywhich contains theFooDialectizerclass). Calling thetranslatefunction with the dialect name'foo'would find the modulefoodialect.py, import the classFooDialectizer, and away you go. @@ -3606,12 +3606,12 @@ appropriately-named file in the plug-ins directory (likefoodialect.py③-
- After all that imagining, this is going to seem pretty boring, but the
feedfunction is what does the entire transformation. You had the entire HTML source in a single string, so you only had to callfeedonce. However, you can callfeedas often as you want, and the parser will just keep parsing. So if you were worried about memory usage (or you knew you - were going to be dealing with very large HTML pages), you could set this up in a loop, where you read a few bytes of HTML and fed it to the parser. The result would be the same. +- After all that imagining, this is going to seem pretty boring, but the
feedfunction is what does the entire transformation. You had the entire HTML source in a single string, so you only had to callfeedonce. However, you can callfeedas often as you want, and the parser will just keep parsing. So if you were worried about memory usage (or you knew you + were going to be dealing with very large HTML pages), you could set this up in a loop, where you read a few bytes of HTML and fed it to the parser. The result would be the same.- Because
feedmaintains an internal buffer, you should always call the parser'sclosemethod when you're done (even if you fed it all at once, like you did). Otherwise you may find that your output is missing the last few bytes.- Remember,
outputis the function you defined onBaseHTMLProcessorthat joins all the pieces of output you've buffered and returns them in a single string. -And just like that, you've “translated” a web page, given nothing but a URL and the name of a dialect. +
And just like that, you've “translated” a web page, given nothing but a URL and the name of a dialect.
Further reading
@@ -3619,14 +3619,14 @@ appropriately-named file in the plug-ins directory (like
foodialect.py8.10. Summary
-Python provides you with a powerful tool,
sgmllib.py, to manipulate HTML by turning its structure into an object model. You can use this tool in many different ways. +Python provides you with a powerful tool,
sgmllib.py, to manipulate HTML by turning its structure into an object model. You can use this tool in many different ways.-
- parsing the HTML looking for something specific +
- parsing the HTML looking for something specific -
- aggregating the results, like the URL lister +
- aggregating the results, like the URL lister
- altering the structure along the way, like the attribute quoter -
- transforming the HTML into something else by manipulating the text while leaving the tags alone, like the
Dialectizer+- transforming the HTML into something else by manipulating the text while leaving the tags alone, like the
DialectizerAlong with these examples, you should be comfortable doing all of the following things:
@@ -3638,7 +3638,7 @@ appropriately-named file in the plug-ins directory (likefoodialect.py
-[1] The technical term for a parser like
SGMLParseris a consumer: it consumes HTML and breaks it down. Presumably, the namefeedwas chosen to fit into the whole “consumer” motif. Personally, it makes me think of an exhibit in the zoo where there's just a dark cage with no trees or plants or +[1] The technical term for a parser like
SGMLParseris a consumer: it consumes HTML and breaks it down. Presumably, the namefeedwas chosen to fit into the whole “consumer” motif. Personally, it makes me think of an exhibit in the zoo where there's just a dark cage with no trees or plants or evidence of life of any kind, but if you stand perfectly still and look really closely you can make out two beady eyes staring back at you from the far left corner, but you convince yourself that that's just your mind playing tricks on you, and the only way you can tell that the whole thing isn't just an empty cage is a small innocuous sign on the railing that reads, “Do not feed the parser.” But maybe that's just me. In any event, it's an interesting mental image. @@ -3650,18 +3650,18 @@ appropriately-named file in the plug-ins directory (likefoodialect.py[3] I don't get out much.
-[4] All right, it's not that common a question. It's not up there with “What editor should I use to write Python code?” (answer: Emacs) or “Is Python better or worse than Perl?” (answer: “Perl is worse than Python because people wanted it worse.” -Larry Wall, 10/14/1998) But questions about HTML processing pop up in one form or another about once a month, and among those questions, this is a popular one. +
[4] All right, it's not that common a question. It's not up there with “What editor should I use to write Python code?” (answer: Emacs) or “Is Python better or worse than Perl?” (answer: “Perl is worse than Python because people wanted it worse.” -Larry Wall, 10/14/1998) But questions about HTML processing pop up in one form or another about once a month, and among those questions, this is a popular one.
-Chapter 9. XML Processing
+Chapter 9. XML Processing
9.1. Diving in
-These next two chapters are about XML processing in Python. It would be helpful if you already knew what an XML document looks like, that it's made up of structured tags to form a hierarchy of elements, and so on. If this doesn't make -sense to you, there are many XML tutorials that can explain the basics. +
These next two chapters are about XML processing in Python. It would be helpful if you already knew what an XML document looks like, that it's made up of structured tags to form a hierarchy of elements, and so on. If this doesn't make +sense to you, there are many XML tutorials that can explain the basics.
If you're not particularly interested in XML, you should still read these chapters, which cover important topics like Python packages, Unicode, command line arguments, and how to use
getattrfor method dispatching.Being a philosophy major is not required, although if you have ever had the misfortune of being subjected to the writings of Immanuel Kant, you will appreciate the example program a lot more than if you majored in something useful, like computer science. -
There are two basic ways to work with XML. One is called SAX (“Simple API for XML”), and it works by reading the XML a little bit at a time and calling a method for each element it finds. (If you read Chapter 8, HTML Processing, this should sound familiar, because that's how the
sgmllibmodule works.) The other is called DOM (“Document Object Model”), and it works by reading in the entire XML document at once and creating an internal representation of it using native Python classes linked in a tree structure. Python has standard modules for both kinds of parsing, but this chapter will only deal with using the DOM. -The following is a complete Python program which generates pseudo-random output based on a context-free grammar defined in an XML format. Don't worry yet if you don't understand what that means; you'll examine both the program's input and its output +
There are two basic ways to work with XML. One is called SAX (“Simple API for XML”), and it works by reading the XML a little bit at a time and calling a method for each element it finds. (If you read Chapter 8, HTML Processing, this should sound familiar, because that's how the
sgmllibmodule works.) The other is called DOM (“Document Object Model”), and it works by reading in the entire XML document at once and creating an internal representation of it using native Python classes linked in a tree structure. Python has standard modules for both kinds of parsing, but this chapter will only deal with using the DOM. +The following is a complete Python program which generates pseudo-random output based on a context-free grammar defined in an XML format. Don't worry yet if you don't understand what that means; you'll examine both the program's input and its output in more depth throughout these next two chapters.
Example 9.1.
kgp.pyIf you have not already done so, you can download this and other examples used in this book. @@ -3953,7 +3953,7 @@ def openAnything(source): # treat source as string import StringIO return StringIO.StringIO(str(source)) -
Run the program
kgp.pyby itself, and it will parse the default XML-based grammar, inkant.xml, and print several paragraphs worth of philosophy in the style of Immanuel Kant. +Run the program
kgp.pyby itself, and it will parse the default XML-based grammar, inkant.xml, and print several paragraphs worth of philosophy in the style of Immanuel Kant.Example 9.3. Sample output of
kgp.py[you@localhost kgp]$ python kgp.py As is shown in the writings of Hume, our a priori concepts, in reference to ends, abstract from all content of knowledge; in the study @@ -4001,9 +4001,9 @@ completely different. 10110100You will take a closer look at the structure of the grammar file later in this chapter. For now, all you need to know is that the grammar file defines the structure of the output, and the
kgp.pyprogram reads through the grammar and makes random decisions about which words to plug in where.9.2. Packages
-Actually parsing an XML document is very simple: one line of code. However, before you get to that line of code, you need to take a short detour +
Actually parsing an XML document is very simple: one line of code. However, before you get to that line of code, you need to take a short detour to talk about packages. -
Example 9.5. Loading an XML document (a sneak peek)
+Example 9.5. Loading an XML document (a sneak peek)
>>> from xml.dom import minidom ① >>> xmldoc = minidom.parse('~/diveintopython3/common/py/kgp/binary.xml')@@ -4052,13 +4052,13 @@ The answer is the magical
__init__.pyfile. You see, packages are nA package is a directory with the special __init__.pyfile in it. The__init__.pyfile defines the attributes and methods of the package. It doesn't need to define anything; it can just be an empty file, but it has to exist. But if__init__.pydoesn't exist, the directory is just a directory, not a package, and it can't be imported or contain modules or nested packages. -So why bother with packages? Well, they provide a way to logically group related modules. Instead of having an
xmlpackage withsaxanddompackages inside, the authors could have chosen to put all thesaxfunctionality inxmlsax.pyand all thedomfunctionality inxmldom.py, or even put all of it in a single module. But that would have been unwieldy (as of this writing, the XML package has over 3000 lines of code) and difficult to manage (separate source files mean multiple people can work on different +So why bother with packages? Well, they provide a way to logically group related modules. Instead of having an
xmlpackage withsaxanddompackages inside, the authors could have chosen to put all thesaxfunctionality inxmlsax.pyand all thedomfunctionality inxmldom.py, or even put all of it in a single module. But that would have been unwieldy (as of this writing, the XML package has over 3000 lines of code) and difficult to manage (separate source files mean multiple people can work on different areas simultaneously).If you ever find yourself writing a large subsystem in Python (or, more likely, when you realize that your small subsystem has grown into a large one), invest some time designing a good package architecture. It's one of the many things Python is good at, so take advantage of it. -
9.3. Parsing XML
-As I was saying, actually parsing an XML document is very simple: one line of code. Where you go from there is up to you. -
Example 9.8. Loading an XML document (for real this time)
+9.3. Parsing XML
+As I was saying, actually parsing an XML document is very simple: one line of code. Where you go from there is up to you. +
Example 9.8. Loading an XML document (for real this time)
>>> from xml.dom import minidom ① >>> xmldoc = minidom.parse('~/diveintopython3/common/py/kgp/binary.xml') ② >>> xmldoc ③ @@ -4077,11 +4077,11 @@ package architecture. It's one of the many things Python is good at, so take adv </grammar>
- As you saw in the previous section, this imports the
minidommodule from thexml.dompackage. -- Here is the one line of code that does all the work:
minidom.parsetakes one argument and returns a parsed representation of the XML document. The argument can be many things; in this case, it's simply a filename of an XML document on my local disk. (To follow along, you'll need to change the path to point to your downloaded examples directory.) +- Here is the one line of code that does all the work:
minidom.parsetakes one argument and returns a parsed representation of the XML document. The argument can be many things; in this case, it's simply a filename of an XML document on my local disk. (To follow along, you'll need to change the path to point to your downloaded examples directory.) But you can also pass a file object, or even a file-like object. You'll take advantage of this flexibility later in this chapter. -- The object returned from
minidom.parseis aDocumentobject, a descendant of theNodeclass. ThisDocumentobject is the root level of a complex tree-like structure of interlocking Python objects that completely represent the XML document you passed tominidom.parse. -toxmlis a method of theNodeclass (and is therefore available on theDocumentobject you got fromminidom.parse).toxmlprints out the XML that thisNoderepresents. For theDocumentnode, this prints out the entire XML document. -Now that you have an XML document in memory, you can start traversing through it. +
- The object returned from
minidom.parseis aDocumentobject, a descendant of theNodeclass. ThisDocumentobject is the root level of a complex tree-like structure of interlocking Python objects that completely represent the XML document you passed tominidom.parse. +toxmlis a method of theNodeclass (and is therefore available on theDocumentobject you got fromminidom.parse).toxmlprints out the XML that thisNoderepresents. For theDocumentnode, this prints out the entire XML document. +Now that you have an XML document in memory, you can start traversing through it.
Example 9.9. Getting child nodes
>>> xmldoc.childNodes ① [<DOM Element: grammar at 17538908>] @@ -4090,7 +4090,7 @@ package architecture. It's one of the many things Python is good at, so take adv >>> xmldoc.firstChild ③ <DOM Element: grammar at 17538908>-
- Every
Nodehas achildNodesattribute, which is a list of theNodeobjects. ADocumentalways has only one child node, the root element of the XML document (in this case, thegrammarelement). +- Every
Nodehas achildNodesattribute, which is a list of theNodeobjects. ADocumentalways has only one child node, the root element of the XML document (in this case, thegrammarelement).- To get the first (and in this case, the only) child node, just use regular list syntax. Remember, there is nothing special going on here; this is just a regular Python list of regular Python objects.
- Since getting the first child node of a node is a useful and common activity, the
Nodeclass has afirstChildattribute, which is synonymous withchildNodes[0]. (There is also alastChildattribute, which is synonymous withchildNodes[-1].) @@ -4108,7 +4108,7 @@ package architecture. It's one of the many things Python is good at, so take adv </ref> </grammar>-
- Since the
toxmlmethod is defined in theNodeclass, it is available on any XML node, not just theDocumentelement. +- Since the
toxmlmethod is defined in theNodeclass, it is available on any XML node, not just theDocumentelement.Example 9.11. Child nodes can be text
>>> grammarNode.childNodes① [<DOM Text node "\n">, <DOM Element: ref at 17533332>, \ @@ -4132,7 +4132,7 @@ package architecture. It's one of the many things Python is good at, so take adv-
- Looking at the XML in
binary.xml, you might think that thegrammarhas only two child nodes, the tworefelements. But you're missing something: the carriage returns! After the'<grammar>'and before the first'<ref>'is a carriage return, and this text counts as a child node of thegrammarelement. Similarly, there is a carriage return after each'</ref>'; these also count as child nodes. Sogrammar.childNodesis actually a list of 5 objects: 3Textobjects and 2Elementobjects. +- Looking at the XML in
binary.xml, you might think that thegrammarhas only two child nodes, the tworefelements. But you're missing something: the carriage returns! After the'<grammar>'and before the first'<ref>'is a carriage return, and this text counts as a child node of thegrammarelement. Similarly, there is a carriage return after each'</ref>'; these also count as child nodes. Sogrammar.childNodesis actually a list of 5 objects: 3Textobjects and 2Elementobjects.- The first child is a
Textobject representing the carriage return after the'<grammar>'tag and before the first'<ref>'tag.- The second child is an
Elementobject representing the firstrefelement.- The fourth child is an
Elementobject representing the secondrefelement. @@ -4163,7 +4163,7 @@ u'0'- The
pelement has only one child node (you can't tell that from this example, but look atpNode.childNodesif you don't believe me), and it is aTextnode for the single character'0'.- The
.dataattribute of aTextnode gives you the actual string that the text node represents. But what is that'u'in front of the string? The answer to that deserves its own section.9.4. Unicode
-Unicode is a system to represent characters from all the world's different languages. When Python parses an XML document, all data is stored in memory as unicode. +
Unicode is a system to represent characters from all the world's different languages. When Python parses an XML document, all data is stored in memory as unicode.
You'll get to all that in a minute, but first, some background.
Historical note. Before unicode, there were separate character encoding systems for each language, each using the same numbers (0-255) to represent that language's characters. Some languages (like Russian) have multiple conflicting standards about how to represent the @@ -4179,14 +4179,14 @@ mode, so character 241 means something else. And so on.) These are the problems [5] Each 2-byte number represents a unique character used in at least one of the world's languages. (Characters that are used in multiple languages have the same numeric code.) There is exactly 1 number per character, and exactly 1 character per number. Unicode data is never ambiguous. -
Of course, there is still the matter of all these legacy encoding systems. 7-bit ASCII, for instance, which stores English characters as numbers ranging from 0 to 127. (65 is capital “
A”, 97 is lowercase “a”, and so forth.) English has a very simple alphabet, so it can be completely expressed in 7-bit ASCII. Western European languages like French, Spanish, and German all use an encoding system called ISO-8859-1 (also called “latin-1”), which uses the 7-bit ASCII characters for the numbers 0 through 127, but then extends into the 128-255 range for characters like n-with-a-tilde-over-it -(241), and u-with-two-dots-over-it (252). And unicode uses the same characters as 7-bit ASCII for 0 through 127, and the same characters as ISO-8859-1 for 128 through 255, and then extends from there into characters +Of course, there is still the matter of all these legacy encoding systems. 7-bit ASCII, for instance, which stores English characters as numbers ranging from 0 to 127. (65 is capital “
A”, 97 is lowercase “a”, and so forth.) English has a very simple alphabet, so it can be completely expressed in 7-bit ASCII. Western European languages like French, Spanish, and German all use an encoding system called ISO-8859-1 (also called “latin-1”), which uses the 7-bit ASCII characters for the numbers 0 through 127, but then extends into the 128-255 range for characters like n-with-a-tilde-over-it +(241), and u-with-two-dots-over-it (252). And unicode uses the same characters as 7-bit ASCII for 0 through 127, and the same characters as ISO-8859-1 for 128 through 255, and then extends from there into characters for other languages with the remaining numbers, 256 through 65535.When dealing with unicode data, you may at some point need to convert the data back into one of these other legacy encoding systems. For instance, to integrate with some other computer system which expects its data in a specific 1-byte encoding -scheme, or to print it to a non-unicode-aware terminal or printer. Or to store it in an XML document which explicitly specifies the encoding scheme. +scheme, or to print it to a non-unicode-aware terminal or printer. Or to store it in an XML document which explicitly specifies the encoding scheme.
And on that note, let's get back to Python. -
Python has had unicode support throughout the language since version 2.0. The XML package uses unicode to store all parsed XML data, but you can use unicode anywhere. +
Python has had unicode support throughout the language since version 2.0. The XML package uses unicode to store all parsed XML data, but you can use unicode anywhere.
Example 9.13. Introducing unicode
>>> s = u'Dive in' ① >>> s @@ -4194,9 +4194,9 @@ u'Dive in' >>> print s ② Dive in-
- To create a unicode string instead of a regular ASCII string, add the letter “
u” before the string. Note that this particular string doesn't have any non-ASCII characters. That's fine; unicode is a superset of ASCII (a very large superset at that), so any regular ASCII string can also be stored as unicode. -- When printing a string, Python will attempt to convert it to your default encoding, which is usually ASCII. (More on this in a minute.) Since this unicode string is made up of characters that are also ASCII characters, printing it has the same result as printing a normal ASCII string; the conversion is seamless, and if you didn't know that s was a unicode string, you'd never notice the difference. -
Example 9.14. Storing non-ASCII characters
+- To create a unicode string instead of a regular ASCII string, add the letter “
u” before the string. Note that this particular string doesn't have any non-ASCII characters. That's fine; unicode is a superset of ASCII (a very large superset at that), so any regular ASCII string can also be stored as unicode. +- When printing a string, Python will attempt to convert it to your default encoding, which is usually ASCII. (More on this in a minute.) Since this unicode string is made up of characters that are also ASCII characters, printing it has the same result as printing a normal ASCII string; the conversion is seamless, and if you didn't know that s was a unicode string, you'd never notice the difference. +
Example 9.14. Storing non-ASCII characters
>>> s = u'La Pe\xf1a' ① >>> print s ② Traceback (innermost last): @@ -4205,11 +4205,11 @@ UnicodeError: ASCII encoding error: ordinal not in range(128) >>> print s.encode('latin-1') ③ La Peña-
- The real advantage of unicode, of course, is its ability to store non-ASCII characters, like the Spanish “
ñ” (nwith a tilde over it). The unicode character code for the tilde-n is0xf1in hexadecimal (241 in decimal), which you can type like this:\xf1. -- Remember I said that the
- The real advantage of unicode, of course, is its ability to store non-ASCII characters, like the Spanish “
ñ” (nwith a tilde over it). The unicode character code for the tilde-n is0xf1in hexadecimal (241 in decimal), which you can type like this:\xf1. +- Remember I said that the
- Here's where the conversion-from-unicode-to-other-encoding-schemes comes in. s is a unicode string, but
encodemethod, available on every unicode string, to convert the unicode string to a regular string in the given encoding scheme, - which you pass as a parameter. In this case, you're usinglatin-1(also known asiso-8859-1), which includes the tilde-n (whereas the default ASCII encoding scheme did not, since it only includes characters numbered 0 through 127). -Remember I said Python usually converted unicode to ASCII whenever it needed to make a regular string out of a unicode string? Well, this default encoding scheme is an option which + which you pass as a parameter. In this case, you're using
latin-1(also known asiso-8859-1), which includes the tilde-n (whereas the default ASCII encoding scheme did not, since it only includes characters numbered 0 through 127). +Remember I said Python usually converted unicode to ASCII whenever it needed to make a regular string out of a unicode string? Well, this default encoding scheme is an option which you can customize.
Example 9.15.
sitecustomize.py# sitecustomize.py ① @@ -4237,15 +4237,15 @@ La PeñaIf you are going to be storing non-ASCII strings within your Python code, you'll need to specify the encoding of each individual
.pyfile by putting an encoding declaration at the top of each file. This declaration defines the.pyfile to be UTF-8:#!/usr/bin/env python # -*- coding: UTF-8 -*- -Now, what about XML? Well, every XML document is in a specific encoding. Again, ISO-8859-1 is a popular encoding for data in Western European languages. KOI8-R -is popular for Russian texts. The encoding, if specified, is in the header of the XML document. +
Now, what about XML? Well, every XML document is in a specific encoding. Again, ISO-8859-1 is a popular encoding for data in Western European languages. KOI8-R +is popular for Russian texts. The encoding, if specified, is in the header of the XML document.
Example 9.18.
russiansample.xml<?xml version="1.0" encoding="koi8-r"?> ① <preface> <title>Предисловие</title> ② </preface>-
- This is a sample extract from a real Russian XML document; it's part of a Russian translation of this very book. Note the encoding,
koi8-r, specified in the header. +- This is a sample extract from a real Russian XML document; it's part of a Russian translation of this very book. Note the encoding,
koi8-r, specified in the header.- These are Cyrillic characters which, as far as I know, spell the Russian word for “Preface”. If you open this file in a regular text editor, the characters will most likely like gibberish, because they're encoded using the
koi8-rencoding scheme, but they're being displayed iniso-8859-1.Example 9.19. Parsing
russiansample.xml@@ -4267,26 +4267,26 @@ UnicodeError: ASCII encoding error: ordinal not in range(128)- I'm assuming here that you saved the previous example as
russiansample.xmlin the current directory. I am also, for the sake of completeness, assuming that you've changed your default encoding back to'ascii'by removing yoursitecustomize.pyfile, or at least commenting out thesetdefaultencodingline.- Note that the text data of the
titletag (now in the title variable, thanks to that long concatenation of Python functions which I hastily skipped over and, annoyingly, won't explain until the next section) -- the text data inside the -XML document'stitleelement is stored in unicode. -- Printing the title is not possible, because this unicode string contains non-ASCII characters, so Python can't convert it to ASCII because that doesn't make sense. +XML document's
titleelement is stored in unicode. +- Printing the title is not possible, because this unicode string contains non-ASCII characters, so Python can't convert it to ASCII because that doesn't make sense.
- You can, however, explicitly convert it to
koi8-r, in which case you get a (regular, not unicode) string of single-byte characters (f0,d2,c5, and so forth) that are thekoi8-r-encoded versions of the characters in the original unicode string. -- Printing the
koi8-r-encoded string will probably show gibberish on your screen, because your Python IDE is interpreting those characters asiso-8859-1, notkoi8-r. But at least they do print. (And, if you look carefully, it's the same gibberish that you saw when you opened the original -XML document in a non-unicode-aware text editor. Python converted it fromkoi8-rinto unicode when it parsed the XML document, and you've just converted it back.) +- Printing the
koi8-r-encoded string will probably show gibberish on your screen, because your Python IDE is interpreting those characters asiso-8859-1, notkoi8-r. But at least they do print. (And, if you look carefully, it's the same gibberish that you saw when you opened the original +XML document in a non-unicode-aware text editor. Python converted it fromkoi8-rinto unicode when it parsed the XML document, and you've just converted it back.)To sum up, unicode itself is a bit intimidating if you've never seen it before, but unicode data is really very easy to handle -in Python. If your XML documents are all 7-bit ASCII (like the examples in this chapter), you will literally never think about unicode. Python will convert the ASCII data in the XML documents into unicode while parsing, and auto-coerce it back to ASCII whenever necessary, and you'll never even notice. But if you need to deal with that in other languages, Python is ready. +in Python. If your XML documents are all 7-bit ASCII (like the examples in this chapter), you will literally never think about unicode. Python will convert the ASCII data in the XML documents into unicode while parsing, and auto-coerce it back to ASCII whenever necessary, and you'll never even notice. But if you need to deal with that in other languages, Python is ready.
Further reading
- Unicode.org is the home page of the unicode standard, including a brief technical introduction. -
- Unicode Tutorial has some more examples of how to use Python's unicode functions, including how to force Python to coerce unicode into ASCII even when it doesn't really want to. +
- Unicode Tutorial has some more examples of how to use Python's unicode functions, including how to force Python to coerce unicode into ASCII even when it doesn't really want to.
- PEP 263 goes into more detail about how and when to define a character encoding in your
.pyfiles.9.5. Searching for elements
-Traversing XML documents by stepping through each node can be tedious. If you're looking for something in particular, buried deep within - your XML document, there is a shortcut you can use to find it quickly:
getElementsByTagName. +Traversing XML documents by stepping through each node can be tedious. If you're looking for something in particular, buried deep within + your XML document, there is a shortcut you can use to find it quickly:
getElementsByTagName.For this section, you'll be using the
binary.xmlgrammar file, which looks like this:Example 9.20.
binary.xml<?xml version="1.0"?> <!DOCTYPE grammar PUBLIC "-//diveintopython3.org//DTD Kant Generator Pro v1.0//EN" "kgp.dtd"> @@ -4318,7 +4318,7 @@ in Python. If your XML documents are all 7-bit ASCII </ref>-
getElementsByTagNametakes one argument, the name of the element you wish to find. It returns a list ofElementobjects, corresponding to the XML elements that have that name. In this case, you find tworefelements. +getElementsByTagNametakes one argument, the name of the element you wish to find. It returns a list ofElementobjects, corresponding to the XML elements that have that name. In this case, you find tworefelements.Example 9.22. Every element is searchable
>>> firstref = reflist[0] ① >>> print firstref.toxml() @@ -4349,15 +4349,15 @@ in Python. If your XML documents are all 7-bit ASCII '<p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\ <xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>'-
- Note carefully the difference between this and the previous example. Previously, you were searching for
pelements within firstref, but here you are searching forpelements within xmldoc, the root-level object that represents the entire XML document. This does find thepelements nested within therefelements within the rootgrammarelement. +- Note carefully the difference between this and the previous example. Previously, you were searching for
pelements within firstref, but here you are searching forpelements within xmldoc, the root-level object that represents the entire XML document. This does find thepelements nested within therefelements within the rootgrammarelement.- The first two
pelements are within the firstref(the'bit'ref).- The last
pelement is the one within the secondref(the'byte'ref).9.6. Accessing element attributes
-XML elements can have one or more attributes, and it is incredibly simple to access them once you have parsed an XML document. +
XML elements can have one or more attributes, and it is incredibly simple to access them once you have parsed an XML document.
For this section, you'll be using the
binary.xmlgrammar file that you saw in the previous section.-
This section may be a little confusing, because of some overlapping terminology. Elements in an XML document have attributes, and Python objects also have attributes. When you parse an XML document, you get a bunch of Python objects that represent all the pieces of the XML document, and some of these Python objects represent attributes of the XML elements. But the (Python) objects that represent the (XML) attributes also have (Python) attributes, which are used to access various parts of the (XML) attribute that the object represents. I told you it was confusing. I am open to suggestions on how to distinguish these + This section may be a little confusing, because of some overlapping terminology. Elements in an XML document have attributes, and Python objects also have attributes. When you parse an XML document, you get a bunch of Python objects that represent all the pieces of the XML document, and some of these Python objects represent attributes of the XML elements. But the (Python) objects that represent the (XML) attributes also have (Python) attributes, which are used to access various parts of the (XML) attribute that the object represents. I told you it was confusing. I am open to suggestions on how to distinguish these more clearly. Example 9.24. Accessing element attributes
>>> xmldoc = minidom.parse('binary.xml') @@ -4379,7 +4379,7 @@ in Python. If your XML documents are all 7-bit ASCII
- Each
Elementobject has an attribute calledattributes, which is aNamedNodeMapobject. This sounds scary, but it's not, because aNamedNodeMapis an object that acts like a dictionary, so you already know how to use it.- Treating the
NamedNodeMapas a dictionary, you can get a list of the names of the attributes of this element by usingattributes.keys(). This element has only one attribute,'id'. -- Attribute names, like all other text in an XML document, are stored in unicode. +
- Attribute names, like all other text in an XML document, are stored in unicode.
- Again treating the
NamedNodeMapas a dictionary, you can get a list of the values of the attributes by usingattributes.values(). The values are themselves objects, of typeAttr. You'll see how to get useful information out of this object in the next example.- Still treating the
NamedNodeMapas a dictionary, you can access an individual attribute by name, using normal dictionary syntax. (Readers who have been paying extra-close attention will already know how theNamedNodeMapclass accomplishes this neat trick: by defining a__getitem__special method. Other readers can take comfort in the fact that they don't need to understand how it works in order to use it effectively.) @@ -4392,11 +4392,11 @@ u'id' >>> a.value ② u'bit'-
- The
Attrobject completely represents a single XML attribute of a single XML element. The name of the attribute (the same name as you used to find this object in thebitref.attributesNamedNodeMappseudo-dictionary) is stored ina.name. -- The actual text value of this XML attribute is stored in
a.value. +- The
Attrobject completely represents a single XML attribute of a single XML element. The name of the attribute (the same name as you used to find this object in thebitref.attributesNamedNodeMappseudo-dictionary) is stored ina.name. +- The actual text value of this XML attribute is stored in
a.value.-
Like a dictionary, attributes of an XML element have no ordering. Attributes may happen to be listed in a certain order in the original XML document, and the Attrobjects may happen to be listed in a certain order when the XML document is parsed into Python objects, but these orders are arbitrary and should carry no special meaning. You should always access individual attributes +Like a dictionary, attributes of an XML element have no ordering. Attributes may happen to be listed in a certain order in the original XML document, and the Attrobjects may happen to be listed in a certain order when the XML document is parsed into Python objects, but these orders are arbitrary and should carry no special meaning. You should always access individual attributes by name, like the keys of a dictionary.9.7. Segue
OK, that's it for the hard-core XML stuff. The next chapter will continue to use these same example programs, but focus on @@ -4404,7 +4404,7 @@ u'bit'
Before moving on to the next chapter, you should be comfortable doing all of these things:
