2to3False is invalid syntaxconstantsbytes' object to str implicitlychardet: a mini-FAQWhen you think of “text”, you probably think of “characters and symbols I see on my computer screen”. But computers don't deal in characters and symbols; they deal in bits and bytes. Every piece of text you've ever seen on a computer screen is actually stored in a particular character encoding. There are many different character encodings, some optimized for particular languages like Russian or Chinese or English, and others that can be used for multiple languages. Very roughly speaking, the character encoding provides a mapping between the stuff you see on your screen and the stuff your computer actually stores in memory and on disk. -
FIXME intro
+In reality, it's more complicated than that. Many characters are common to multiple encodings, but each encoding may use a different sequence of bytes to actually store those characters in memory or on disk. So you can think of the character encoding as a kind of decryption key for the text. Whenever someone gives you a sequence of bytes and claims it's “text”, you need to know what character encoding they used so you can decode the bytes into characters and display them (or process them, or whatever). -
...
+It means taking a sequence of bytes in an unknown character encoding, and attempting to determine the encoding so you can read the text. It's like cracking a code when you don't have the decryption key.
- In general, yes. However, some encodings are optimized for specific languages, and languages are not random. Some character sequences pop up all the time, while other sequences make no sense. A person fluent in English who opens a newspaper and finds “txzqJv 2!dasd0a QqdKjvz” will instantly recognize that that isn't English (even though it is composed entirely of English letters). By studying lots of “typical” text, a computer algorithm can simulate this kind of fluency and make an educated guess about a text's language.
+ In other words, encoding detection is really language detection, combined with knowledge of which languages tend to use which character encodings.
- We're going to migrate the The This library is a port of the auto-detection code in Mozilla. I have attempted to maintain as much of the original structure as possible (mostly for selfish reasons, to make it easier to maintain the port as the original code evolves). I have also retained the original authors' comments, which are quite extensive and informative.
- You may also be interested in the research paper which led to the Mozilla implementation, A composite approach to language/encoding detection.
+
+ Don't do that. Virtually every format and protocol contains a method for specifying character encoding.
+
+ If text comes with explicit character encoding information, you should use it. If the text has no explicit information, but the relevant standard defines a default encoding, you should use that. (This is harder than it sounds, because standards can overlap. If you fetch an XML document over HTTP, you need to support both standards and figure out which one wins if they give you conflicting information.)
+
+ Despite the complexity, it's worthwhile to follow standards and respect explicit character encoding information. It will almost certainly be faster and more accurate than trying to auto-detect the encoding. It will also make the world a better place, since your program will interoperate with other programs that follow the same standards.
+
+ Sometimes you receive text with verifiably inaccurate encoding information. Or text without any encoding information, and the specified default encoding doesn't work. There are also some poorly designed standards that have no way to specify encoding at all.
+
+ If following the relevant standards gets you nowhere, and you decide that processing the text is more important than maintaining interoperability, then you can try to auto-detect the character encoding as a last resort. An example is my Universal Feed Parser, which calls this auto-detection library only after exhausting all other options.
+
+ This is a brief guide to navigating the code itself.
+
+ The main entry point for the detection algorithm is There are 5 categories of encodings that If the text starts with a BOM, we can reasonably assume that the text is encoded in If the text contains a recognizable escape sequence that might indicate an escaped encoding, Assuming no BOM, The multi-byte encoding prober, Most of the multi-byte encoding probers are inherited from The distribution analyzers (each defined in The case of Japanese is more difficult. Single-character distribution analysis is not always sufficient to distinguish between The single-byte encoding prober, Hebrew is handled as a special case. If the text appears to be Hebrew based on 2-character distribution analysis, If We're going to migrate the The main Now run the Now run the Well, that wasn't so hard. Just a few imports and print statements to convert. Time to run the new version. Do you think it'll work?Isn't that impossible?
-Running
+2to3chardet module from Python 2 to Python 3. Python 3 comes with a utility script to help with this, called 2to3. 2to3 takes your actual Python 2 source code as input, and auto-converts as much as it can to Python 3. [FIXME reference 2to3 chapter once it's done]Who wrote this detection algorithm?
-chardet library is split across several different files, all in the same directory. The 2to3 script makes it easy to convert multiple files at once: just pass a directory as a command line argument, and 2to3 will convert each of the files in turn.Yippie! Screw the standards, I'll just auto-detect everything!
+
+
+
+
+charset parameter in the Content-type header.
+<meta http-equiv="content-type"> element in the <head> of a web page.
+encoding attribute in the XML prolog.
+Why bother with auto-detection if it's slow, inaccurate, and non-standard?
+
+Diving in
+
+universaldetector.py, which has one class, UniversalDetector. (You might think the main entry point is the detect function in chardet/__init__.py, but that's really just a convenience function that creates a UniversalDetector object, calls it, and returns its result.)
+
+UniversalDetector handles:
+
+
+
+
+UTF-n with a BOM. This includes UTF-8, both BE and LE variants of UTF-16, and all 4 byte-order variants of UTF-32.
+ISO-2022-JP (Japanese) and HZ-GB-2312 (Chinese).
+Big5 (Chinese), SHIFT_JIS (Japanese), EUC-KR (Korean), and UTF-8 without a BOM.
+KOI8-R (Russian), windows-1255 (Hebrew), and TIS-620 (Thai).
+windows-1252, which is used primarily on Microsoft Windows by middle managers who wouldn't know a character encoding from a hole in the ground.
+
+
+UTF-n with a BOMUTF-8, UTF-16, or UTF-32. (The BOM will tell us exactly which one; that's what it's for.) This is handled inline in UniversalDetector, which returns the result immediately without any further processing.
+
+Escaped encodings
+
+UniversalDetector creates an EscCharSetProber (defined in escprober.py) and feeds it the text.
+
+EscCharSetProber creates a series of state machines, based on models of HZ-GB-2312, ISO-2022-CN, ISO-2022-JP, and ISO-2022-KR (defined in escsm.py). EscCharSetProber feeds the text to each of these state machines, one byte at a time. If any state machine ends up uniquely identifying the encoding, EscCharSetProber immediately returns the positive result to UniversalDetector, which returns it to the caller. If any state machine hits an illegal sequence, it is dropped and processing continues with the other state machines.
+
+Multi-byte encodings
+
+UniversalDetector checks whether the text contains any high-bit characters. If so, it creates a series of “probers” for detecting multi-byte encodings, single-byte encodings, and as a last resort, windows-1252.
+
+MBCSGroupProber (defined in mbcsgroupprober.py), is really just a shell that manages a group of other probers, one for each multi-byte encoding: Big5, GB2312, EUC-TW, EUC-KR, EUC-JP, SHIFT_JIS, and UTF-8. MBCSGroupProber feeds the text to each of these encoding-specific probers and checks the results. If a prober reports that it has found an illegal byte sequence, it is dropped from further processing (so that, for instance, any subsequent calls to UniversalDetector.feed() will skip that prober). If a prober reports that it is reasonably confident that it has detected the encoding, MBCSGroupProber reports this positive result to UniversalDetector, which reports the result to the caller.
+
+MultiByteCharSetProber (defined in mbcharsetprober.py), and simply hook up the appropriate state machine and distribution analyzer and let MultiByteCharSetProber do the rest of the work. MultiByteCharSetProber runs the text through the encoding-specific state machine, one byte at a time, to look for byte sequences that would indicate a conclusive positive or negative result. At the same time, MultiByteCharSetProber feeds the text to an encoding-specific distribution analyzer.
+
+chardistribution.py) use language-specific models of which characters are used most frequently. Once MultiByteCharSetProber has fed enough text to the distribution analyzer, it calculates a confidence rating based on the number of frequently-used characters, the total number of characters, and a language-specific distribution ratio. If the confidence is high enough, MultiByteCharSetProber returns the result to MBCSGroupProber, which returns it to UniversalDetector, which returns it to the caller.
+
+EUC-JP and SHIFT_JIS, so the SJISProber (defined in sjisprober.py) also uses 2-character distribution analysis. SJISContextAnalysis and EUCJPContextAnalysis (both defined in jpcntx.py and both inheriting from a common JapaneseContextAnalysis class) check the frequency of Hiragana syllabary characters within the text. Once enough text has been processed, they return a confidence level to SJISProber, which checks both analyzers and returns the higher confidence level to MBCSGroupProber.
+
+Single-byte encodings
+
+SBCSGroupProber (defined in sbcsgroupprober.py), is also just a shell that manages a group of other probers, one for each combination of single-byte encoding and language: windows-1251, KOI8-R, ISO-8859-5, MacCyrillic, IBM855, and IBM866 (Russian); ISO-8859-7 and windows-1253 (Greek); ISO-8859-5 and windows-1251 (Bulgarian); ISO-8859-2 and windows-1250 (Hungarian); TIS-620 (Thai); windows-1255 and ISO-8859-8 (Hebrew).
+
+SBCSGroupProber feeds the text to each of these encoding+language-specific probers and checks the results. These probers are all implemented as a single class, SingleByteCharSetProber (defined in sbcharsetprober.py), which takes a language model as an argument. The language model defines how frequently different 2-character sequences appear in typical text. SingleByteCharSetProber processes the text and tallies the most frequently used 2-character sequences. Once enough text has been processed, it calculates a confidence level based on the number of frequently-used sequences, the total number of characters, and a language-specific distribution ratio.
+
+HebrewProber (defined in hebrewprober.py) tries to distinguish between Visual Hebrew (where the source text actually stored “backwards” line-by-line, and then displayed verbatim so it can be read from right to left) and Logical Hebrew (where the source text is stored in reading order and then rendered right-to-left by the client). Because certain characters are encoded differently based on whether they appear in the middle of or at the end of a word, we can make a reasonable guess about direction of the source text, and return the appropriate encoding (windows-1255 for Logical Hebrew, or ISO-8859-8 for Visual Hebrew).
+
+
+
+windows-1252UniversalDetector detects a high-bit character in the text, but none of the other multi-byte or single-byte encoding probers return a confident result, it creates a Latin1Prober (defined in latin1prober.py) to try to detect English text in a windows-1252 encoding. This detection is inherently unreliable, because English letters are encoded in the same way in many different encodings. The only way to distinguish windows-1252 is through commonly used symbols like smart quotes, curly apostrophes, copyright symbols, and the like. Latin1Prober automatically reduces its confidence rating to allow more accurate probers to win if at all possible.
+
+Running
+
+2to3chardet module from Python 2 to Python 3. Python 3 comes with a utility script called 2to3, which takes your actual Python 2 source code as input and auto-converts as much as it can to Python 3. In some cases this is easy -- a function was renamed or moved to a different modules -- but in other cases it can get pretty complex. To get a sense of all that it can do, refer to the appendix, Porting code to Python 3 with 2to3. In this chapter, we'll start by running 2to3 on the chardet package, but as you'll see, there will still be a lot of work to do after the automated tools have performed their magic.
+
+chardet package is split across several different files, all in the same directory. The 2to3 script makes it easy to convert multiple files at once: just pass a directory as a command line argument, and 2to3 will convert each of the files in turn.
+
+C:\home\chardet>python c:\Python30\Tools\Scripts\2to3.py -w chardet\
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
@@ -507,9 +604,9 @@ RefactoringTool: chardet\sjisprober.py
RefactoringTool: chardet\universaldetector.py
RefactoringTool: chardet\utf8prober.py
-2to3 script on the testing harness, test.py.2to3 script on the testing harness, test.py.
-C:\home\chardet>python c:\Python30\Tools\Scripts\2to3.py -w test.py
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
@@ -541,15 +638,15 @@ RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Files that were modified:
RefactoringTool: test.py
-
Well, that wasn't so hard. Just a few imports and print statements to convert. Time to run the new version. Do you think it'll work?
- Now for the real test: running the test harness against the test suite. Since the test suite is designed to cover all the possible code paths, it's a good way to test our ported code to make sure there aren't any bugs lurking anywhere. Now for the real test: running the test harness against the test suite. Since the test suite is designed to cover all the possible code paths, it's a good way to test our ported code to make sure there aren't any bugs lurking anywhere.
+
+ Hmm, a small snag. In Python 3, Hmm, a small snag. In Python 3, This piece of code is designed to allow this library to run under older versions of Python 2. Prior to Python 2.3 [FIXME-LINK], Python had no built-in This piece of code is designed to allow this library to run under older versions of Python 2. Prior to Python 2.3 [FIXME-LINK], Python had no built-in However, Python 3 will always have a However, Python 3 will always have a So this line in So this line in Becomes Becomes
Ah, wasn't that satisfying? The code is shorter and more readable already.
+False is invalid syntaxFixing what
-2to3 can't
-
+False is invalid syntaxC:\home\chardet>python test.py tests\*\*
Traceback (most recent call last):
File "test.py", line 1, in <module>
@@ -559,9 +656,9 @@ RefactoringTool: test.py
^
SyntaxError: invalid syntax
-False is a reserved word, so you can't use it as a variable name. Let's look at constants.py to see where it's defined. Here's the original version from constants.py, before the 2to3 script changed it:False is a reserved word, so you can't use it as a variable name. Let's look at constants.py to see where it's defined. Here's the original version from constants.py, before the 2to3 script changed it:
-
-import __builtin__
if not hasattr(__builtin__, 'False'):
False = 0
@@ -570,27 +667,25 @@ else:
False = __builtin__.False
True = __builtin__.TrueBoolean type. This code detects the absence of the built-in constants True and False, and defines them if necessary.Boolean type. This code detects the absence of the built-in constants True and False, and defines them if necessary.
-Boolean type, so this entire code snippet is unnecessary. The simplest solution is to replace all instances of "constants.True" and "constants.False" with "True" and "False", respectively, then delete this dead code from constants.py.Boolean type, so this entire code snippet is unnecessary. The simplest solution is to replace all instances of "constants.True" and "constants.False" with "True" and "False", respectively, then delete this dead code from constants.py.
-universaldetector.py:universaldetector.py:
-self.done = constants.False
-self.done = False
Ah, wasn't that satisfying? The code is shorter and more readable already.
- Time to run test.py again and see how far it gets. Time to run What's that you say? No module named What's that you say? No module named But wait. Wasn't the But wait. Wasn't the The solution is to split the import statement manually. So this two-in-one import: The solution is to split the import statement manually. So this two-in-one import:
Needs to become two separate imports: Needs to become two separate imports:
There are variations of this problem scattered throughout the There are variations of this problem scattered throughout the Onward!No module named
+constantsNo module named
-constantstest.py again and see how far it gets.
-C:\home\chardet>python test.py tests\*\*
Traceback (most recent call last):
File "test.py", line 1, in <module>
@@ -599,32 +694,30 @@ else:
import constants, sys
ImportError: No module named constants
-constants? Of course there's a module named constants. ... Oh wait, no there isn't. Remember when the 2to3 script fixed up all those import statements? This library has a lot of relative imports -- that is, modules that import other modules within the library. In Python 3, all import statements are absolute by default [FIXME-LINK PEP 0328]. To do relative imports, you need to do something like this instead:constants? Of course there's a module named constants. ... Oh wait, no there isn't. Remember when the 2to3 script fixed up all those import statements? This library has a lot of relative imports -- that is, modules that import other modules within the library. In Python 3, all import statements are absolute by default [FIXME-LINK PEP 0328]. To do relative imports, you need to do something like this instead:
-from . import constants2to3 script supposed to take care of these for you? Well, it did, but this particular import statement combines two different types of imports into one line: a relative import of the constants module within the library, and an absolute import of the sys module that is pre-installed in the Python standard library. In Python 2, you could combine these into one import statement. In Python 3, you can't, and the 2to3 script is not smart enough to split the import statement into two.2to3 script supposed to take care of these for you? Well, it did, but this particular import statement combines two different types of imports into one line: a relative import of the constants module within the library, and an absolute import of the sys module that is pre-installed in the Python standard library. In Python 2, you could combine these into one import statement. In Python 3, you can't, and the 2to3 script is not smart enough to split the import statement into two.
-
-import constants, sys
-from . import constants
import syschardet library. In some places it's "import constants, sys"; in other places, it's "import constants, re". The fix is the same: manually split the import statement into two lines, one for the relative import, the other for the absolute import.chardet library. In some places it's "import constants, sys"; in other places, it's "import constants, re". The fix is the same: manually split the import statement into two lines, one for the relative import, the other for the absolute import.
-
Onward!
- FIXME intro FIXME intro
- This one surprised me, because I've been using this idiom as long as I can remember. In Python 2, the global file() function was an alias for open(), which was the standard way of opening files for reading. In Python 3, the entire system for reading and writing files has been refactored into the This one surprised me, because I've been using this idiom as long as I can remember. In Python 2, the global file() function was an alias for open(), which was the standard way of opening files for reading. In Python 3, the entire system for reading and writing files has been refactored into the Thus, the simplest solution to the problem of the missing file() is to call open() instead: Thus, the simplest solution to the problem of the missing file() is to call open() instead:
And that's all I have to say about that.Name 'file' is not defined
+Name 'file' is not defined
-C:\home\chardet>python test.py tests\*\*
tests\ascii\howto.diveintomark.org.xml
Traceback (most recent call last):
@@ -632,21 +725,19 @@ import sys
for line in file(f, 'rb'):
NameError: name 'file' is not defined
-io module. [FIXME-LINK PEP 3116] I'll cover the new I/O module in more detail in Chapter FIXME, but for now, the important bit is that the global file() function no longer exists. However, the open() function does still exist. (Technically, it's an alias for io.open(), but never mind that right now.)io module. [FIXME-LINK PEP 3116] I'll cover the new I/O module in more detail in Chapter FIXME, but for now, the important bit is that the global file() function no longer exists. However, the open() function does still exist. (Technically, it's an alias for io.open(), but never mind that right now.)
-
-for line in open(f, 'rb'):
And that's all I have to say about that.
- FIXME intro FIXME intro
- Now things are starting to get interesting. And by "interesting," I mean "confusing as all hell." Now things are starting to get interesting. And by "interesting," I mean "confusing as all hell."
- First, let's see what self._highBitDetector is. It's defined in the __init__ method of the UniversalDetector class: First, let's see what self._highBitDetector is. It's defined in the __init__ method of the UniversalDetector class:
- This pre-compiles a regular expression designed to find non-ASCII characters in the range 128-255 (0x80-0xFF). Wait, that's not quite right; I need to be more precise with my terminology. This pattern is designed to find non-ASCII bytes in the range 128-255. This pre-compiles a regular expression designed to find non-ASCII characters in the range 128-255 (0x80-0xFF). Wait, that's not quite right; I need to be more precise with my terminology. This pattern is designed to find non-ASCII bytes in the range 128-255.
- And therein lies the problem. And therein lies the problem.
- In Python 2, a string was an array of bytes whose character encoding was tracked separately. If you wanted Python 2 to keep track of the character encoding, you had to use a Unicode string ( In Python 2, a string was an array of bytes whose character encoding was tracked separately. If you wanted Python 2 to keep track of the character encoding, you had to use a Unicode string ( And what is aBuf? Let's backtrack further to a place that calls UniversalDetector.feed(). One place that calls it is the test harness, And what is aBuf? Let's backtrack further to a place that calls UniversalDetector.feed(). One place that calls it is the test harness, And here we find our answer: in the UniversalDetector.feed() method, aBuf is a line read from a file on disk. Look carefully at the parameters used to open the file: And here we find our answer: in the UniversalDetector.feed() method, aBuf is a line read from a file on disk. Look carefully at the parameters used to open the file: What we need this regular expression to search is not an array of characters, but an array of bytes. What we need this regular expression to search is not an array of characters, but an array of bytes.
- Once you realize that, the solution is not difficult. Regular expressions defined with strings can search strings. Regular expressions defined with byte arrays can search byte arrays. To define a byte array pattern, we simply change the type of the argument we use to define the regular expression to a byte array. So instead of this: Once you realize that, the solution is not difficult. Regular expressions defined with strings can search strings. Regular expressions defined with byte arrays can search byte arrays. To define a byte array pattern, we simply change the type of the argument we use to define the regular expression to a byte array. So instead of this:
We now have this: We now have this:
There is one other case of this same problem, on the very next line: There is one other case of this same problem, on the very next line:
Again, this is going to be used to search a byte array (the same aBuf variable, in fact), so the regular expression pattern needs to be defined as a byte array: Again, this is going to be used to search a byte array (the same aBuf variable, in fact), so the regular expression pattern needs to be defined as a byte array:
Curiouser and curiouser... Curiouser and curiouser...
- ...Can't use a string pattern on a bytes-like object
+Can't use a string pattern on a bytes-like object
-C:\home\chardet>python test.py tests\*\*
tests\ascii\howto.diveintomark.org.xml
Traceback (most recent call last):
@@ -656,22 +747,22 @@ NameError: name 'file' is not defined
if self._highBitDetector.search(aBuf):
TypeError: can't use a string pattern on a bytes-like object
-
-class UniversalDetector:
def __init__(self):
self._highBitDetector = re.compile(r'[\x80-\xFF]')u'') instead. But in Python 3, a string is always what Python 2 called a Unicode string -- that is, an array of Unicode characters (of possibly varying byte lengths). Since this regular expression is defined by a string pattern, it can only be used to search a string -- again, an array of characters. But what we're searching is not a string, it's a byte array. Looking at the traceback, this error occurred in universaldetector.py:u'') instead. But in Python 3, a string is always what Python 2 called a Unicode string -- that is, an array of Unicode characters (of possibly varying byte lengths). Since this regular expression is defined by a string pattern, it can only be used to search a string -- again, an array of characters. But what we're searching is not a string, it's a byte array. Looking at the traceback, this error occurred in universaldetector.py:
-
if self._mInputState == ePureAscii:
if self._highBitDetector.search(aBuf):
-def feed(self, aBuf):
.
.
@@ -679,9 +770,9 @@ TypeError: can't use a string pattern on a bytes-like objecttest.py.test.py.
-
for line in open(f, 'rb'):
u.feed(line)
-u = UniversalDetector()
.
.
@@ -689,33 +780,31 @@ TypeError: can't use a string pattern on a bytes-like object'rb'. 'r' is for "read"; OK, big deal, we're reading the file. Ah, but 'b' is for "bytes." Without the 'b' flag, this for loop would read the file, line by line, and convert each line into a string -- an array of Unicode characters -- according to the system default character encoding. (You could override the system encoding with another parameter to open(), but never mind that for now.) But with the 'b' flag, this for loop reads the file, line by line, and stores each line exactly as it appears in the file, as an array of bytes. That byte array gets passed to UniversalDetector.feed(), and eventually gets passed to the pre-compiled regular expression, self._highBitDetector, to search for high-bit... characters. But we don't have characters; we have bytes. Oops.'rb'. 'r' is for "read"; OK, big deal, we're reading the file. Ah, but 'b' is for "bytes." Without the 'b' flag, this for loop would read the file, line by line, and convert each line into a string -- an array of Unicode characters -- according to the system default character encoding. (You could override the system encoding with another parameter to open(), but never mind that for now.) But with the 'b' flag, this for loop reads the file, line by line, and stores each line exactly as it appears in the file, as an array of bytes. That byte array gets passed to UniversalDetector.feed(), and eventually gets passed to the pre-compiled regular expression, self._highBitDetector, to search for high-bit... characters. But we don't have characters; we have bytes. Oops.
-
-self._highBitDetector = re.compile(r'[\x80-\xFF]')
-self._highBitDetector = re.compile(b'[\x80-\xFF]')
-self._escDetector = re.compile(r'(\033|~{)')
-self._escDetector = re.compile(b'(\033|~{)')Can't convert '
+bytes' object to str implicitlyCan't convert '
-bytes' object to str implicitlyC:\home\chardet>python test.py tests\*\*
tests\ascii\howto.diveintomark.org.xml
Traceback (most recent call last):
@@ -725,11 +814,10 @@ for line in open(f, 'rb'):
elif (self._mInputState == ePureAscii) and self._escDetector.search(self._mLastChar + aBuf):
TypeError: Can't convert 'bytes' object to str implicitly
-
... diff --git a/porting-code-to-python-3-with-2to3.html b/porting-code-to-python-3-with-2to3.html index 143a245..4df7f63 100644 --- a/porting-code-to-python-3-with-2to3.html +++ b/porting-code-to-python-3-with-2to3.html @@ -51,84 +51,78 @@ for (var i = arTables.length - 1; i >= 0; i--) {
2to3print statementhas_key() dictionary methodprint statement
+has_key() dictionary method
+filter() global functionmap() global functionreduce() global function (3.1+)apply() global functionintern() global functionexec statementexecfile statement (3.1+)repr literals (backticks)try...except statementraise statementthrow statementlong data typexrange() global functionraw_input() and input() global functionsfunc_* function attributesxreadlines() I/O methodlambda functions with multiple parametersnext() iterator method__nonzero__ special class attributesys.maxintunicode() global functioncallable() global functionzip() global functionStandardError() exceptiontypes module constantsisinstance global function (3.1+)basestring datatypeitertools modulesys.exc_type, sys.exc_value, sys.exc_tracebackos.getcwdu() functionset() literalsbuffer() global functionfilter() global function
+map() global function
+reduce() global function (3.1+)
+apply() global function
+intern() global function
+exec statement
+execfile statement (3.1+)
+repr literals (backticks)
+try...except statement
+raise statement
+throw statement
+long data type
+xrange() global function
+raw_input() and input() global functions
+func_* function attributes
+xreadlines() I/O method
+lambda functions with multiple parameters
+next() iterator method
+__nonzero__ special class attribute
+sys.maxint
+unicode() global function
+callable() global function
+zip() global function
+StandardError() exception
+types module constants
+isinstance global function (3.1+)
+basestring datatype
+itertools module
+sys.exc_type, sys.exc_value, sys.exc_traceback
+os.getcwdu() function
+set() literals
+buffer() global function
+FIXME intro
+FIXME intro -
...
+... -
print statementIn Python 2, print was a statement -- whatever you wanted to print simply followed the print keyword. In Python 3, print() is a function -- whatever you want to print is passed to print() like any other function.
In Python 2, print was a statement -- whatever you wanted to print simply followed the print keyword. In Python 3, print() is a function -- whatever you want to print is passed to print() like any other function.
-
| Notes | @@ -163,21 +157,18 @@ for (var i = arTables.length - 1; i >= 0; i--) {
|---|
print() without any arguments.print() with one argumentprint() with two arguments.print statement with a comma, it would print the values separated by spaces, then print a trailing space, then stop without printing a carriage return. In Python 3, the way to do this is to pass end=' ' as a keyword argument to the print() function. The end argument defaults to '\n' (a carriage return), so overriding it will suppress the carriage return after printing the other arguments.sys.stderr -- by using the >>pipe_name syntax. In Python 3, the way to do this is to pass the pipe in the file keyword argument. The file argument defaults to sys.stdout (standard out), so overriding it will output to a different pipe instead.print() without any arguments.
+print() with one argument
+print() with two arguments.
+print statement with a comma, it would print the values separated by spaces, then print a trailing space, then stop without printing a carriage return. In Python 3, the way to do this is to pass end=' ' as a keyword argument to the print() function. The end argument defaults to '\n' (a carriage return), so overriding it will suppress the carriage return after printing the other arguments.
+sys.stderr -- by using the >>pipe_name syntax. In Python 3, the way to do this is to pass the pipe in the file keyword argument. The file argument defaults to sys.stdout (standard out), so overriding it will output to a different pipe instead.
Python 2 supported <> as a synonym for !=, the not-equals comparison operator. Python 3 supports the != operator, but not <>.
Python 2 supported <> as a synonym for !=, the not-equals comparison operator. Python 3 supports the != operator, but not <>.
-
| Notes | @@ -197,18 +188,15 @@ for (var i = arTables.length - 1; i >= 0; i--) {
|---|
has_key() dictionary methodIn Python 2, dictionaries had a has_key() method to test whether the dictionary had a certain key. In Python 3, this method no longer exists. Instead, you need to use the in operator.
In Python 2, dictionaries had a has_key() method to test whether the dictionary had a certain key. In Python 3, this method no longer exists. Instead, you need to use the in operator.
-
| Notes | @@ -243,21 +231,18 @@ for (var i = arTables.length - 1; i >= 0; i--) {
|---|
or operator takes precedence over the in operator, so there is no need for parentheses here.or takes precedence over in.in operator takes precedence over the + operator, so this form needs parentheses too.or operator takes precedence over the in operator, so there is no need for parentheses here.
+or takes precedence over in.
+in operator takes precedence over the + operator, so this form needs parentheses too.
+In Python 2, many dictionary methods returned lists. The most frequently used methods were keys(), items(), and values(). In Python 3, all of these methods return dynamic views. In some contexts, this is not a problem. If the method's return value is immediately passed to another function that iterates through the entire sequence, it makes no difference whether the actual type is a list or a view. In other contexts, it matters a great deal. If you were expecting a complete list with individually addressable elements, your code will choke, because views do not support indexing.
In Python 2, many dictionary methods returned lists. The most frequently used methods were keys(), items(), and values(). In Python 3, all of these methods return dynamic views. In some contexts, this is not a problem. If the method's return value is immediately passed to another function that iterates through the entire sequence, it makes no difference whether the actual type is a list or a view. In other contexts, it matters a great deal. If you were expecting a complete list with individually addressable elements, your code will choke, because views do not support indexing.
-
| Notes | @@ -292,28 +277,24 @@ for (var i = arTables.length - 1; i >= 0; i--) {
|---|
2to3 errs on the side of safety, converting the return value from keys() to a static list with the list() function. This will always work, but it will be less efficient than using a view. You should examine the converted code to see if a list is absolutely necessary, or if a view would do.items() method. 2to3 will do the same thing with the values() method.iterkeys() method anymore. Use keys(), and if necessary, convert the view to an iterator with the iter() function.2to3 recognizes when the iterkeys() method is used inside a list comprehension, and converts it to the keys() method (without wrapping it in an extra call to iter()). This works because views are iterable.2to3 recognizes that the keys() method is immediately passed to a function which iterates through an entire sequence, so there is no need to convert the return value to a list first. The min() function will happily iterate through the view instead. This applies to min(), max(), sum(), list(), tuple(), set(), sorted(), any(), and all().2to3 errs on the side of safety, converting the return value from keys() to a static list with the list() function. This will always work, but it will be less efficient than using a view. You should examine the converted code to see if a list is absolutely necessary, or if a view would do.
+items() method. 2to3 will do the same thing with the values() method.
+iterkeys() method anymore. Use keys(), and if necessary, convert the view to an iterator with the iter() function.
+2to3 recognizes when the iterkeys() method is used inside a list comprehension, and converts it to the keys() method (without wrapping it in an extra call to iter()). This works because views are iterable.
+2to3 recognizes that the keys() method is immediately passed to a function which iterates through an entire sequence, so there is no need to convert the return value to a list first. The min() function will happily iterate through the view instead. This applies to min(), max(), sum(), list(), tuple(), set(), sorted(), any(), and all().
Several modules in the Python Standard Library have been renamed. Several other modules which are related to each other have been combined or reorganized to make their association more logical.
+Several modules in the Python Standard Library have been renamed. Several other modules which are related to each other have been combined or reorganized to make their association more logical. -
FIXME: once the rest of the book is written, these should link back to the chapters and sections that explain these modules.
+FIXME: once the rest of the book is written, these should link back to the chapters and sections that explain these modules.
- In Python 3, several related HTTP modules have been combined into a single package, In Python 3, several related HTTP modules have been combined into a single package, Python 2 had a rat's nest of overlapping modules to parse, encode, and fetch URLs. In Python 3, these have all been refactored and combined in a single package, Python 2 had a rat's nest of overlapping modules to parse, encode, and fetch URLs. In Python 3, these have all been refactored and combined in a single package, All the various DBM clones are now in a single package, All the various DBM clones are now in a single package,
- XML-RPC is a lightweight method of performing remote RPC calls over HTTP. The XML-RPC client library and several XML-RPC server implementations are now combined in a single package, XML-RPC is a lightweight method of performing remote RPC calls over HTTP. The XML-RPC client library and several XML-RPC server implementations are now combined in a single package,
-
-http packagehttp.http.
-
Notes
@@ -345,20 +326,17 @@ import CGIHttpServer
-
-http.client module implements a low-level library that can request HTTP resources and interpret HTTP responses.http.cookies module provides a Pythonic interface to "cookies" that are sent in a Set-Cookie: HTTP header.http.cookiejar module manipulates the actual files on disk that popular web browsers use to store cookies.http.server module provides a basic HTTP server.http.client module implements a low-level library that can request HTTP resources and interpret HTTP responses.
+http.cookies module provides a Pythonic interface to "cookies" that are sent in a Set-Cookie: HTTP header.
+http.cookiejar module manipulates the actual files on disk that popular web browsers use to store cookies.
+http.server module provides a basic HTTP server.
-urllib packageurllib.urllib.
-
Notes
@@ -402,22 +380,19 @@ from urllib.error import HTTPError
-
-urllib module in Python 2 had a variety of functions, including urlopen() for fetching data and splittype(), splithost(), and splituser() for splitting a URL into its constituent parts. These functions have been reorganized more logically within the new urllib package. 2to3 will also change all calls to these functions so they use the new naming scheme.urllib2 module in Python 2 has been folded into into the urllib package in Python 3. All your urllib2 favorites -- the build_opener() method, Request objects, and HTTPBasicAuthHandler and friends -- are still available.urllib.parse module in Python 3 contains all the parsing functions from the old urlparse module in Python 2.urllib.robotparser module parses robots.txt files.FancyURLopener class, which handles HTTP redirects and other status codes, is still available in the new urllib.request module. The urlencode function has moved to urllib.parse.Request object is still available in urllib.request, but constants like HTTPError have been moved to urllib.error.urllib module in Python 2 had a variety of functions, including urlopen() for fetching data and splittype(), splithost(), and splituser() for splitting a URL into its constituent parts. These functions have been reorganized more logically within the new urllib package. 2to3 will also change all calls to these functions so they use the new naming scheme.
+urllib2 module in Python 2 has been folded into into the urllib package in Python 3. All your urllib2 favorites -- the build_opener() method, Request objects, and HTTPBasicAuthHandler and friends -- are still available.
+urllib.parse module in Python 3 contains all the parsing functions from the old urlparse module in Python 2.
+urllib.robotparser module parses robots.txt files.
+FancyURLopener class, which handles HTTP redirects and other status codes, is still available in the new urllib.request module. The urlencode function has moved to urllib.parse.
+Request object is still available in urllib.request, but constants like HTTPError have been moved to urllib.error.
-dbm packagedbm. If you need a specific variant like GNU DBM, you can import the appropriate module within the dbm package.dbm. If you need a specific variant like GNU DBM, you can import the appropriate module within the dbm package.
-
-
+
Notes
@@ -452,16 +427,13 @@ import whichdb
-xmlrpc packagexmlrpc.xmlrpc.
-
-
+
Notes
@@ -481,14 +453,11 @@ import SimpleXMLRPCServer
Other modules
-
+
Notes
@@ -549,27 +518,38 @@ except ImportError:
-
-cStringIO as StringIO, and if that failed, to import StringIO instead. Do not do this in Python 3; the io module does it for you. It will find the fastest implementation available and use it automatically.pickle module does it for you.builtins module contains the "global" functions, classes, and constants used throughout the Python language. Redefining a function in the builtins module will redefine the "global" function everywhere. That is exactly as powerful and scary as it sounds.copyreg module adds pickle support for custom types defined in C.queue module implements a multi-producer, multi-consumer queue.socketserver module provides generic base classes for implementing different kinds of socket servers.configparser module parses INI-style configuration files.reprlib module reimplements the built-in repr() function, but with limits on how many values are represented.subprocess module allows you to spawn processes, connect to their pipes, and obtain their return codes.cStringIO as StringIO, and if that failed, to import StringIO instead. Do not do this in Python 3; the io module does it for you. It will find the fastest implementation available and use it automatically.
+pickle module does it for you.
+builtins module contains the "global" functions, classes, and constants used throughout the Python language. Redefining a function in the builtins module will redefine the "global" function everywhere. That is exactly as powerful and scary as it sounds.
+copyreg module adds pickle support for custom types defined in C.
+queue module implements a multi-producer, multi-consumer queue.
+socketserver module provides generic base classes for implementing different kinds of socket servers.
+configparser module parses INI-style configuration files.
+reprlib module reimplements the built-in repr() function, but with limits on how many values are represented.
+subprocess module allows you to spawn processes, connect to their pipes, and obtain their return codes.
FIXME intro
+A package is a group of related modules that function as a single entity. In Python 2, when modules within a package need to reference each other, you use import foo or from foo import Bar. The Python 2 interpreter first searches within the current package to find foo.py, and then moves on to the other directories in the Python search path (sys.path). Python 3 works a bit differently. Instead of searching the current package, it goes directly to the Python search path. If you want one module within a package to import another module in the same package, you need to explicitly provide the relative path between the two modules.
-
Suppose you had this package, with multiple files in the same directory: + +
chardet/ +| ++--__init__.py +| ++--constants.py +| ++--mbcharsetprober.py +| ++--universaldetector.py+ +
Now suppose that universaldetector.py needs to import the entire constants.py file and one class from mbcharsetprober.py. How do you do it?
+
+
| Notes | @@ -578,23 +558,26 @@ except ImportError:||||
|---|---|---|---|---|
| ① | -FIXME |
-FIXME |
+import constants |
+from . import constants |
+
| ② | +from mbcharsetprober import MultiByteCharSetProber |
+from .mbcharsetprober import MultiByteCharsetProber |
from . import syntax. The period is actually a relative path from this file (universaldetector.py) to the file you want to import (constants.py). In this case, they are in the same directory, thus the single period. You can also import from the parent directory (from .. import anothermodule) or a subdirectory.
+mbcharsetprober.py is in the same directory as universaldetector.py, so the path is a single period. You can also import form the parent directory (from ..anothermodule import AnotherClass) or a subdirectory.
filter() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -629,21 +612,18 @@ except ImportError:
|---|
map() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -678,21 +658,18 @@ except ImportError:
|---|
reduce() global function (3.1+)FIXME intro
+FIXME intro -
+| Notes | @@ -708,17 +685,14 @@ reduce(a, b, c)
|---|
apply() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -748,20 +722,17 @@ reduce(a, b, c)
|---|
intern() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -776,17 +747,14 @@ reduce(a, b, c)
|---|
exec statementFIXME intro
+FIXME intro -
+| Notes | @@ -811,19 +779,16 @@ reduce(a, b, c)
|---|
execfile statement (3.1+)FIXME intro
+FIXME intro -
+| Notes | @@ -838,17 +803,14 @@ reduce(a, b, c)
|---|
repr literals (backticks)FIXME intro
+FIXME intro -
+| Notes | @@ -873,19 +835,16 @@ reduce(a, b, c)
|---|
try...except statementFIXME intro
+FIXME intro -
+| Notes | @@ -933,20 +892,17 @@ except:
|---|
raise statementFIXME intro
+FIXME intro -
+| Notes | @@ -971,19 +927,16 @@ except:
|---|
throw statementFIXME intro
+FIXME intro -
+| Notes | @@ -1008,19 +961,16 @@ except:
|---|
long data typeFIXME intro
+FIXME intro -
+| Notes | @@ -1055,21 +1005,18 @@ except:
|---|
xrange() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -1104,21 +1051,18 @@ except:
|---|
raw_input() and input() global functionsFIXME intro
+FIXME intro -
+| Notes | @@ -1148,20 +1092,17 @@ except:
|---|
func_* function attributesFIXME intro
+FIXME intro -
+| Notes | @@ -1206,23 +1147,20 @@ except:
|---|
xreadlines() I/O methodFIXME intro
+FIXME intro -
+| Notes | @@ -1242,18 +1180,15 @@ except:
|---|
lambda functions with multiple parametersFIXME intro
+FIXME intro -
+| Notes | @@ -1278,19 +1213,16 @@ except:
|---|
FIXME intro
+FIXME intro -
+| Notes | @@ -1315,19 +1247,16 @@ except:
|---|
next() iterator methodFIXME intro
+FIXME intro -
+| Notes | @@ -1372,21 +1301,18 @@ for an_iterator in a_sequence_of_iterators:
|---|
__nonzero__ special class attributeFIXME intro
+FIXME intro -
+| Notes | @@ -1412,18 +1338,15 @@ for an_iterator in a_sequence_of_iterators:
|---|
FIXME intro
+FIXME intro -
+| Notes | @@ -1443,18 +1366,15 @@ for an_iterator in a_sequence_of_iterators:
|---|
sys.maxintFIXME intro
+FIXME intro -
+| Notes | @@ -1476,18 +1396,15 @@ a_function(sys.maxsize)
|---|
unicode() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -1502,17 +1419,14 @@ a_function(sys.maxsize)
|---|
FIXME intro
+FIXME intro -
+| Notes | @@ -1532,18 +1446,15 @@ a_function(sys.maxsize)
|---|
callable() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -1558,17 +1469,14 @@ a_function(sys.maxsize)
|---|
zip() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -1588,18 +1496,15 @@ a_function(sys.maxsize)
|---|
StandardError() exceptionFIXME intro
+FIXME intro -
+| Notes | @@ -1619,18 +1524,15 @@ a_function(sys.maxsize)
|---|
types module constantsFIXME intro
+FIXME intro -
+| Notes | @@ -1670,22 +1572,19 @@ a_function(sys.maxsize)
|---|
isinstance global function (3.1+)FIXME intro
+FIXME intro -
+| Notes | @@ -1700,17 +1599,14 @@ a_function(sys.maxsize)
|---|
basestring datatypeFIXME intro
+FIXME intro -
+| Notes | @@ -1725,15 +1621,12 @@ a_function(sys.maxsize)
|---|
itertools moduleFIXME intro
+FIXME intro
sys.exc_type, sys.exc_value, sys.exc_tracebackFIXME intro
+FIXME intro -
+| Notes | @@ -1808,19 +1698,16 @@ a_function(sys.maxsize)
|---|
FIXME intro
+FIXME intro -
+| Notes | @@ -1835,17 +1722,14 @@ a_function(sys.maxsize)
|---|
os.getcwdu() functionFIXME intro
+FIXME intro -
+| Notes | @@ -1860,17 +1744,14 @@ a_function(sys.maxsize)
|---|
FIXME intro
+FIXME intro -
+| Notes | @@ -1894,18 +1775,15 @@ a_function(sys.maxsize)
|---|
set() literalsFIXME intro
+FIXME intro -
+| Notes | @@ -1930,19 +1808,16 @@ a_function(sys.maxsize)
|---|
buffer() global functionFIXME intro
+FIXME intro -
+| Notes | @@ -1957,17 +1832,14 @@ a_function(sys.maxsize)
|---|
FIXME intro
+FIXME intro -
+| Notes | @@ -1987,18 +1859,15 @@ a_function(sys.maxsize)
|---|
FIXME intro
+FIXME intro -
+| Notes | @@ -2033,16 +1902,14 @@ do_stuff(a_list)
|---|