mirror of
https://github.com/kennethreitz/dive-into-python3.git
synced 2026-06-05 23:10:17 +00:00
sick, can't sleep, may as well fiddle endlessly
This commit is contained in:
Mark Pilgrim
@@ -1,19 +1,21 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang=en>
|
||||
<head>
|
||||
<meta charset=utf-8>
|
||||
<title>Case study: porting chardet to Python 3 - Dive into Python 3</title>
|
||||
<!--[if IE]><script src=html5.js></script><![endif]-->
|
||||
<link rel="shortcut icon" href=data:image/ico,>
|
||||
<link rel=alternate type=application/atom+xml href=http://hg.diveintopython3.org/atom-log>
|
||||
<link rel=stylesheet type=text/css href=dip3.css>
|
||||
<style>
|
||||
body{counter-reset:h1 20}
|
||||
ins,del,mark{line-height:2.154;text-decoration:none;font-style:normal;display:inline-block;width:100%}
|
||||
ins{background:#9f9}
|
||||
del{background:#f87}
|
||||
mark{background:#ff8;font-weight:bold}
|
||||
</style>
|
||||
</head>
|
||||
<p class=skip><a href=#divingin>skip to main content</a>
|
||||
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8> <input name=q size=31> <input type=submit name=sa value=Search></div></form>
|
||||
<p class=nav>You are here: <a href=/>Home</a> <span>‣</span> <a href=table-of-contents.html#case-study-porting-chardet-to-python-3>Dive Into Python 3</a> <span>‣</span>
|
||||
<p class=s><a href=#divingin>skip to main content</a>
|
||||
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8> <input name=q size=31> <input type=submit name=sa value=Search></div></form>
|
||||
<p>You are here: <a href=index.html>Home</a> <span>‣</span> <a href=table-of-contents.html#case-study-porting-chardet-to-python-3>Dive Into Python 3</a> <span>‣</span>
|
||||
<h1>Case study: porting <code>chardet</code> to Python 3</h1>
|
||||
<blockquote class=q>
|
||||
<p><span>❝</span> Words, words. They’re all we have to go on. <span>❞</span><br>— <cite>Rosencrantz and Guildenstern are Dead</cite>
|
||||
@@ -49,7 +51,7 @@ body{counter-reset:h1 20}
|
||||
<li><a href=#summary>Summary</a>
|
||||
</ol>
|
||||
<h2 id=divingin>Diving in</h2>
|
||||
<p class=fancy>Unknown or incorrect character encoding is the #1 cause of gibberish text on the web, in your inbox, and indeed across every computer system ever written. In <a href=strings.html>Chapter 3</a>, I talked about the history of character encoding and the creation of Unicode, the “one encoding to rule them all.” I’d love it if I never had to see a gibberish character on a web page again, because all authoring systems stored accurate encoding information, all transfer protocols were Unicode-aware, and every system that handled text maintained perfect fidelity when converting between encodings.
|
||||
<p class=f>Unknown or incorrect character encoding is the #1 cause of gibberish text on the web, in your inbox, and indeed across every computer system ever written. In <a href=strings.html>Chapter 3</a>, I talked about the history of character encoding and the creation of Unicode, the “one encoding to rule them all.” I’d love it if I never had to see a gibberish character on a web page again, because all authoring systems stored accurate encoding information, all transfer protocols were Unicode-aware, and every system that handled text maintained perfect fidelity when converting between encodings.
|
||||
<p>I’d also like a pony.
|
||||
<p>A Unicode pony.
|
||||
<p>A Unipony, as it were.
|
||||
@@ -98,8 +100,8 @@ body{counter-reset:h1 20}
|
||||
<p>We’re going to migrate the <code>chardet</code> module from Python 2 to Python 3. Python 3 comes with a utility script called <code>2to3</code>, which takes your actual Python 2 source code as input and auto-converts as much as it can to Python 3. In some cases this is easy — a function was renamed or moved to a different modules — but in other cases it can get pretty complex. To get a sense of all that it <em>can</em> do, refer to the appendix, <a href=porting-code-to-python-3-with-2to3.html>Porting code to Python 3 with <code>2to3</code></a>. In this chapter, we’ll start by running <code>2to3</code> on the <code>chardet</code> package, but as you’ll see, there will still be a lot of work to do after the automated tools have performed their magic.
|
||||
<p>The main <code>chardet</code> package is split across several different files, all in the same directory. The <code>2to3</code> script makes it easy to convert multiple files at once: just pass a directory as a command line argument, and <code>2to3</code> will convert each of the files in turn.
|
||||
<p id=noscript>[The code examples will be easier to follow if you enable Javascript, but whatever.]
|
||||
<p class=skip><a href=#skip2to3output>skip over this</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python c:\Python30\Tools\Scripts\2to3.py -w chardet\</kbd>
|
||||
<p class=s><a href=#skip2to3output>skip over this</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python c:\Python30\Tools\Scripts\2to3.py -w chardet\</kbd>
|
||||
<samp>RefactoringTool: Skipping implicit fixer: buffer
|
||||
RefactoringTool: Skipping implicit fixer: idioms
|
||||
RefactoringTool: Skipping implicit fixer: set_literal
|
||||
@@ -566,8 +568,8 @@ RefactoringTool: chardet\sjisprober.py
|
||||
RefactoringTool: chardet\universaldetector.py
|
||||
RefactoringTool: chardet\utf8prober.py</samp></pre>
|
||||
<p id=skip2to3output>Now run the <code>2to3</code> script on the testing harness, <code>test.py</code>.
|
||||
<p class=skip><a href=#skip2to3outputtest>skip over this</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python c:\Python30\Tools\Scripts\2to3.py -w test.py</kbd>
|
||||
<p class=s><a href=#skip2to3outputtest>skip over this</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python c:\Python30\Tools\Scripts\2to3.py -w test.py</kbd>
|
||||
<samp>RefactoringTool: Skipping implicit fixer: buffer
|
||||
RefactoringTool: Skipping implicit fixer: idioms
|
||||
RefactoringTool: Skipping implicit fixer: set_literal
|
||||
@@ -602,8 +604,8 @@ RefactoringTool: test.py</samp></pre>
|
||||
<h2 id=manual>Fixing what <code>2to3</code> can’t</h2>
|
||||
<h3 id=falseisinvalidsyntax><code>False</code> is invalid syntax</h3>
|
||||
<p>Now for the real test: running the test harness against the test suite. Since the test suite is designed to cover all the possible code paths, it’s a good way to test our ported code to make sure there aren’t any bugs lurking anywhere.
|
||||
<p class=skip><a href=#skipinvalidsyntax>skip over this</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skipinvalidsyntax>skip over this</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "test.py", line 1, in <module>
|
||||
from chardet.universaldetector import UniversalDetector
|
||||
@@ -612,7 +614,7 @@ RefactoringTool: test.py</samp></pre>
|
||||
^
|
||||
SyntaxError: invalid syntax</samp></pre>
|
||||
<p id=skipinvalidsyntax>Hmm, a small snag. In Python 3, <code>False</code> is a reserved word, so you can’t use it as a variable name. Let’s look at <code>constants.py</code> to see where it’s defined. Here’s the original version from <code>constants.py</code>, before the <code>2to3</code> script changed it:
|
||||
<p class=skip><a href=#skipbuiltincode>skip over this</a>
|
||||
<p class=s><a href=#skipbuiltincode>skip over this</a>
|
||||
<pre><code>import __builtin__
|
||||
if not hasattr(__builtin__, 'False'):
|
||||
False = 0
|
||||
@@ -629,8 +631,8 @@ else:
|
||||
<p>Ah, wasn’t that satisfying? The code is shorter and more readable already.
|
||||
<h3 id=nomodulenamedconstants>No module named <code>constants</code></h3>
|
||||
<p>Time to run <code>test.py</code> again and see how far it gets.
|
||||
<p class=skip><a href=#skipnomodulenamedconstants>skip over this</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skipnomodulenamedconstants>skip over this</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "test.py", line 1, in <module>
|
||||
from chardet.universaldetector import UniversalDetector
|
||||
@@ -649,8 +651,8 @@ import sys</code></pre>
|
||||
<p>Onward!
|
||||
<h3 id=namefileisnotdefined>Name <var>'file'</var> is not defined</h3>
|
||||
<p>And here we go again, running <code>test.py</code> to try to execute our test cases…</p>
|
||||
<p class=skip><a href=#skipnamefileisnotdefined>skip over this</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skipnamefileisnotdefined>skip over this</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "test.py", line 9, in <module>
|
||||
@@ -662,8 +664,8 @@ NameError: name 'file' is not defined</samp></pre>
|
||||
<p>And that’s all I have to say about that.
|
||||
<h3 id=cantuseastringpattern>Can’t use a string pattern on a bytes-like object</h3>
|
||||
<p>Now things are starting to get interesting. And by “interesting,” I mean “confusing as all hell.”
|
||||
<p class=skip><a href=#skipcantuseastringpattern>skip over this</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skipcantuseastringpattern>skip over this</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "test.py", line 10, in <module>
|
||||
@@ -673,14 +675,14 @@ NameError: name 'file' is not defined</samp></pre>
|
||||
TypeError: can't use a string pattern on a bytes-like object</samp></pre>
|
||||
<p id=skipcantuseastringpattern>
|
||||
<p>To debug this, let’s see what <var>self._highBitDetector</var> is. It’s defined in the <var>__init__</var> method of the <var>UniversalDetector</var> class:
|
||||
<p class=skip><a href=#skiphighbitdetectorcode>skip over this</a>
|
||||
<p class=s><a href=#skiphighbitdetectorcode>skip over this</a>
|
||||
<pre><code>class UniversalDetector:
|
||||
def __init__(self):
|
||||
self._highBitDetector = re.compile(r'[\x80-\xFF]')</code></pre>
|
||||
<p id=skiphighbitdetectorcode>This pre-compiles a regular expression designed to find non-<abbr>ASCII</abbr> characters in the range 128–255 (0x80–0xFF). Wait, that’s not quite right; I need to be more precise with my terminology. This pattern is designed to find non-<abbr>ASCII</abbr> <em>bytes</em> in the range 128-255.
|
||||
<p>And therein lies the problem.
|
||||
<p>In Python 2, a string was an array of bytes whose character encoding was tracked separately. If you wanted Python 2 to keep track of the character encoding, you had to use a Unicode string (<code>u''</code>) instead. But in Python 3, a string is always what Python 2 called a Unicode string — that is, an array of Unicode characters (of possibly varying byte lengths). Since this regular expression is defined by a string pattern, it can only be used to search a string — again, an array of characters. But what we’re searching is not a string, it’s a byte array. Looking at the traceback, this error occurred in <code>universaldetector.py</code>:
|
||||
<p class=skip><a href=#skipfeedhighbitdetectorcode>skip over this</a>
|
||||
<p class=s><a href=#skipfeedhighbitdetectorcode>skip over this</a>
|
||||
<pre><code>def feed(self, aBuf):
|
||||
.
|
||||
.
|
||||
@@ -688,7 +690,7 @@ TypeError: can't use a string pattern on a bytes-like object</samp></pre>
|
||||
if self._mInputState == ePureAscii:
|
||||
if self._highBitDetector.search(aBuf):</code></pre>
|
||||
<p id=skipfeedhighbitdetectorcode>And what is <var>aBuf</var>? Let’s backtrack further to a place that calls <code>UniversalDetector.feed()</code>. One place that calls it is the test harness, <code>test.py</code>.
|
||||
<p class=skip><a href=#skiptestharnessfeedcode>skip over this</a>
|
||||
<p class=s><a href=#skiptestharnessfeedcode>skip over this</a>
|
||||
<pre><code>u = UniversalDetector()
|
||||
.
|
||||
.
|
||||
@@ -698,7 +700,7 @@ for line in open(f, 'rb'):
|
||||
<p id=skiptestharnessfeedcode>And here we find our answer: in the <code>UniversalDetector.feed()</code> method, <var>aBuf</var> is a line read from a file on disk. Look carefully at the parameters used to open the file: <code>'rb'</code>. <code>'r'</code> is for “read”; OK, big deal, we’re reading the file. Ah, but <code>'b'</code> is for “binary.” Without the <code>'b'</code> flag, this <code>for</code> loop would read the file, line by line, and convert each line into a string — an array of Unicode characters — according to the system default character encoding. (You could override the system encoding with another parameter to <var>open()</var>, but never mind that for now.) But with the <code>'b'</code> flag, this <code>for</code> loop reads the file, line by line, and stores each line exactly as it appears in the file, as an array of bytes. That byte array gets passed to <code>UniversalDetector.feed()</code>, and eventually gets passed to the pre-compiled regular expression, <var>self._highBitDetector</var>, to search for high-bit… characters. But we don’t have characters; we have bytes. Oops.
|
||||
<p>What we need this regular expression to search is not an array of characters, but an array of bytes.
|
||||
<p>Once you realize that, the solution is not difficult. Regular expressions defined with strings can search strings. Regular expressions defined with byte arrays can search byte arrays. To define a byte array pattern, we simply change the type of the argument we use to define the regular expression to a byte array. (There is one other case of this same problem, on the very next line.)
|
||||
<p class=skip><a href=#skip-cant-use-a-string-pattern-solution>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-cant-use-a-string-pattern-solution>skip over this code listing</a>
|
||||
<pre><code> class UniversalDetector:
|
||||
def __init__(self):
|
||||
<del>- self._highBitDetector = re.compile(b'[\x80-\xFF]')</del>
|
||||
@@ -709,7 +711,7 @@ for line in open(f, 'rb'):
|
||||
self._mCharSetProbers = []
|
||||
self.reset()</code></pre>
|
||||
<p id=skip-case-use-a-string-pattern-solution>Searching the entire codebase for other uses of the <code>re</code> module turns up two more instances, in <code>charsetprober.py</code>. Again, the code is defining regular expressions as strings but executing them on <var>aBuf</var>, which is a byte array. The solution is the same: define the regular expression patterns as byte arrays.
|
||||
<p class=skip><a href=#cantconvertbytesobject>skip over this code listing</a>
|
||||
<p class=s><a href=#cantconvertbytesobject>skip over this code listing</a>
|
||||
<pre><code> class CharSetProber:
|
||||
.
|
||||
.
|
||||
@@ -726,8 +728,8 @@ for line in open(f, 'rb'):
|
||||
|
||||
<h3 id=cantconvertbytesobject>Can't convert <code>'bytes'</code> object to <code>str</code> implicitly</h3>
|
||||
<p>Curiouser and curiouser…
|
||||
<p class=skip><a href=#skipcantconvertbytesobject>skip over this</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skipcantconvertbytesobject>skip over this</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "test.py", line 10, in <module>
|
||||
@@ -736,12 +738,12 @@ for line in open(f, 'rb'):
|
||||
elif (self._mInputState == ePureAscii) and self._escDetector.search(self._mLastChar + aBuf):
|
||||
TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
|
||||
<p id=skipcantconvertbytesobject>There's an unfortunate clash of coding style and Python interpreter here. The <code>TypeError</code> could be anywhere on that line, but the traceback doesn't tell you exactly where it is. It could be in the first conditional or the second, and the traceback would look the same. To narrow it down, you should split the line in half, like this:
|
||||
<p class=skip><a href=#skip-split-conditional>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-split-conditional>skip over this code listing</a>
|
||||
<pre><code>elif (self._mInputState == ePureAscii) and \
|
||||
self._escDetector.search(self._mLastChar + aBuf):</code></pre>
|
||||
<p id=skip-split-conditional>And re-run the test:</p>
|
||||
<p class=skip><a href=#skip-cant-convert-bytes-object-2>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skip-cant-convert-bytes-object-2>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "test.py", line 10, in <module>
|
||||
@@ -751,7 +753,7 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
|
||||
TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
|
||||
<p id=skip-over-cant-convert-bytes-object-2>Aha! The problem was not in the first conditional (<code>self._mInputState == ePureAscii</code>) but in the second one. So what could cause a <code>TypeError</code> there? Perhaps you're thinking that the <code>search()</code> method is expecting a value of a different type, but that wouldn't generate this traceback. Python functions can take any value; if you pass the right number of arguments, the function will execute. It may <em>crash</em> if you pass it a value of a different type than it's expecting, but if that happened, the traceback would point to somewhere inside the function. But this traceback says it never got as far as calling the <code>search()</code> method. So the problem must be in that <code>+</code> operation, as it's trying to construct the value that it will eventually pass to the <code>search()</code> method.
|
||||
<p>We know from <a href=#cantuseastringpattern>previous debugging</a> that <var>aBuf</var> is a byte array. So what is <code>self._mLastChar</code>? It's an instance variable, defined in the <code>reset()</code> method, which is actually called from the <code>__init__()</code> method.
|
||||
<p class=skip><a href=#skip-mlastchar-declaration>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-mlastchar-declaration>skip over this code listing</a>
|
||||
<pre><code>class UniversalDetector:
|
||||
def __init__(self):
|
||||
self._highBitDetector = re.compile(b'[\x80-\xFF]')
|
||||
@@ -769,7 +771,7 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
|
||||
<mark> self._mLastChar = ''</mark></code></pre>
|
||||
<p id=skip-mlastchar-declaration>And now we have our answer. Do you see it? <var>self._mLastChar</var> is a string, but <var>aBuf</var> is a byte array. And you can't concatenate a string to a byte array — not even a zero-length string.
|
||||
<p>So what is <var>self._mLastChar</var> anyway? The answer is in the <code>feed()</code> method, just a few lines down from where the trackback occurred.
|
||||
<p class=skip><a href=#skip-mlastchar-set>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-mlastchar-set>skip over this code listing</a>
|
||||
<pre><code>if self._mInputState == ePureAscii:
|
||||
if self._highBitDetector.search(aBuf):
|
||||
self._mInputState = eHighbyte
|
||||
@@ -779,7 +781,7 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
|
||||
|
||||
<mark>self._mLastChar = aBuf[-1]</mark></code></pre>
|
||||
<p>The calling function calls this <code>feed()</code> method over and over again with a few bytes at a time. The method processes the bytes it was given (passed in as <var>aBuf</var>), then stores the last byte in <var>self._mLastChar</var> in case it's needed during the next call. (In a multi-byte encoding, the <code>feed()</code> method might get called with half of a character, then called again with the other half.) But because <var>aBuf</var> is now a byte array instead of a string, <var>self._mLastChar</var> needs to be a byte array as well. Thus:
|
||||
<p class=skip><a href=#skip-mlastchar-solution>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-mlastchar-solution>skip over this code listing</a>
|
||||
<pre><code> def reset(self):
|
||||
.
|
||||
.
|
||||
@@ -787,7 +789,7 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
|
||||
<del>- self._mLastChar = ''</del>
|
||||
<ins>+ self._mLastChar = b''</ins></code></pre>
|
||||
<p id=skip-mlastchar-solution>Searching the entire codebase for <code>"mLastChar"</code> turns up a similar problem in <code>mbcharsetprober.py</code>, but instead of tracking the last character, it tracks the last <em>two</em> characters. The <code>MultiByteCharSetProber</code> class uses a list of 1-character strings to track the last two characters; in Python 3, it needs to use a list of integers.
|
||||
<p class=skip><a href=#skip-mbcharsetprober>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-mbcharsetprober>skip over this code listing</a>
|
||||
<pre><code>
|
||||
class MultiByteCharSetProber(CharSetProber):
|
||||
def __init__(self):
|
||||
@@ -807,8 +809,8 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
|
||||
<ins>+ self._mLastChar = [0, 0]</ins></code></pre>
|
||||
<h3 id=unsupportedoperandtypeforplus>Unsupported operand type(s) for +: <code>'int'</code> and <code>'bytes'</code></h3>
|
||||
<p>I have good news, and I have bad news. The good news is we're making progress…
|
||||
<p class=skip><a href=#skip-unsupported-operand-types>skip over this command listing</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skip-unsupported-operand-types>skip over this command listing</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "test.py", line 10, in <module>
|
||||
@@ -819,7 +821,7 @@ TypeError: unsupported operand type(s) for +: 'int' and 'bytes'</samp></pre>
|
||||
<p id=skip-unsupported-operand-types>…The bad news is it doesn't always feel like progress.
|
||||
<p>But this is progress! Really! Even though the traceback calls out the same line of code, it's a different error than it used to be. Progress! So what's the problem now? The last time I checked, this line of code didn't try to concatenate an <code>int</code> with a byte array (<code>bytes</code>). In fact, you just spent a lot of time <a href=#cantconvertbytesobject>ensuring that <var>self._mLastChar</var> was a byte array</a>. How did it turn into an <code>int</code>?
|
||||
<p>The answer lies not in the previous lines of code, but in the following lines.
|
||||
<p class=skip><a href=#skip-mlastchar-highlight>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-mlastchar-highlight>skip over this code listing</a>
|
||||
<pre><code>if self._mInputState == ePureAscii:
|
||||
if self._highBitDetector.search(aBuf):
|
||||
self._mInputState = eHighbyte
|
||||
@@ -829,24 +831,24 @@ TypeError: unsupported operand type(s) for +: 'int' and 'bytes'</samp></pre>
|
||||
|
||||
<mark>self._mLastChar = aBuf[-1]</mark></code></pre>
|
||||
<p id=skip-mlastchar-highlight>This error doesn't occur the first time the <code>feed()</code> method gets called; it occurs the <em>second time</em>, after <var>self._mLastChar</var> has been set to the last byte of <var>aBuf</var>. Well, what's the problem with that? Getting a single element from a byte array yields an integer, not a byte array. To see the difference, follow me to the interactive shell:
|
||||
<p class=skip><a href=#skip-mlastchar-interactive>skip over this interpreter listing</a>
|
||||
<p class=s><a href=#skip-mlastchar-interactive>skip over this interpreter listing</a>
|
||||
<pre class=screen>
|
||||
<a><samp class=prompt>>>> </samp><kbd>aBuf = b'\xEF\xBB\xBF'</kbd> <span>①</span></a>
|
||||
<samp class=prompt>>>> </samp><kbd>len(aBuf)</kbd>
|
||||
<a><samp class=p>>>> </samp><kbd>aBuf = b'\xEF\xBB\xBF'</kbd> <span>①</span></a>
|
||||
<samp class=p>>>> </samp><kbd>len(aBuf)</kbd>
|
||||
<samp>3</samp>
|
||||
<samp class=prompt>>>> </samp><kbd>mLastChar = aBuf[-1]</kbd>
|
||||
<a><samp class=prompt>>>> </samp><kbd>mLastChar</kbd> <span>②</span></a>
|
||||
<samp class=p>>>> </samp><kbd>mLastChar = aBuf[-1]</kbd>
|
||||
<a><samp class=p>>>> </samp><kbd>mLastChar</kbd> <span>②</span></a>
|
||||
<samp>191</samp>
|
||||
<a><samp class=prompt>>>> </samp><kbd>type(mLastChar)</kbd> <span>③</span></a>
|
||||
<a><samp class=p>>>> </samp><kbd>type(mLastChar)</kbd> <span>③</span></a>
|
||||
<samp><class 'int'></samp>
|
||||
<a><samp class=prompt>>>> </samp><kbd>mLastChar + aBuf</kbd> <span>④</span></a>
|
||||
<a><samp class=p>>>> </samp><kbd>mLastChar + aBuf</kbd> <span>④</span></a>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
File "<stdin>", line 1, in <module>
|
||||
TypeError: unsupported operand type(s) for +: 'int' and 'bytes'</samp>
|
||||
<a><samp class=prompt>>>> </samp><kbd>mLastChar = aBuf[-1:]</kbd> <span>⑤</span></a>
|
||||
<samp class=prompt>>>> </samp><kbd>mLastChar</kbd>
|
||||
<a><samp class=p>>>> </samp><kbd>mLastChar = aBuf[-1:]</kbd> <span>⑤</span></a>
|
||||
<samp class=p>>>> </samp><kbd>mLastChar</kbd>
|
||||
<samp>b'\xbf'</samp>
|
||||
<a><samp class=prompt>>>> </samp><kbd>mLastChar + aBuf</kbd> <span>⑥</span></a>
|
||||
<a><samp class=p>>>> </samp><kbd>mLastChar + aBuf</kbd> <span>⑥</span></a>
|
||||
<samp>b'\xbf\xef\xbb\xbf'</samp></pre>
|
||||
<ol id=skip-mlastchar-interactive>
|
||||
<li>Define a byte array of length 3.
|
||||
@@ -864,8 +866,8 @@ TypeError: unsupported operand type(s) for +: 'int' and 'bytes'</samp>
|
||||
<ins>+ self._mLastChar = aBuf[-1:]</ins></code></pre>
|
||||
<h3 id=ordexpectedstring><code>ord()</code> expected string of length 1, but <code>int</code> found</h3>
|
||||
<p>Tired yet? You're almost there…
|
||||
<p class=skip><a href=#skip-ord-expected-string>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skip-ord-expected-string>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml ascii with confidence 1.0
|
||||
tests\Big5\0804.blogspot.com.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
@@ -881,28 +883,28 @@ tests\Big5\0804.blogspot.com.xml</samp>
|
||||
byteCls = self._mModel['classTable'][ord(c)]
|
||||
TypeError: ord() expected string of length 1, but int found</samp></pre>
|
||||
<p id=skip-ord-expected-string>OK, so <var>c</var> is an <code>int</code>, but the <code>ord()</code> function was expecting a 1-character string. Fair enough. Where is <var>c</var> defined?
|
||||
<p class=skip><a href=#skip-next-state>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-next-state>skip over this code listing</a>
|
||||
<pre><code># codingstatemachine.py
|
||||
def next_state(self, c):
|
||||
# for each byte we get its class
|
||||
# if it is first byte, we also get byte length
|
||||
byteCls = self._mModel['classTable'][ord(c)]</code></pre>
|
||||
<p id=skip-next-state>That's no help; it's just passed into the function. Let's pop the stack.
|
||||
<p class=skip><a href=#skip-utf8prober-feed>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-utf8prober-feed>skip over this code listing</a>
|
||||
<pre><code># utf8prober.py
|
||||
def feed(self, aBuf):
|
||||
for c in aBuf:
|
||||
codingState = self._mCodingSM.next_state(c)</code></pre>
|
||||
<p id=skip-utf8prober-feed>And now we have the answer. Do you see it? In Python 2, <var>aBuf</var> was a string, so <var>c</var> was a 1-character string. (That's what you get when you iterate over a string — all the characters, one by one.) But now, <var>aBuf</var> is a byte array, so <var>c</var> is an <code>int</code>, not a 1-character string. In other words, there's no need to call the <code>ord()</code> function because <var>c</var> is already an <code>int</code>!
|
||||
<p>Thus:
|
||||
<p class=skip><a href=#skip-ordc-diff>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-ordc-diff>skip over this code listing</a>
|
||||
<pre><code> def next_state(self, c):
|
||||
# for each byte we get its class
|
||||
# if it is first byte, we also get byte length
|
||||
<del>- byteCls = self._mModel['classTable'][ord(c)]</del>
|
||||
<ins>+ byteCls = self._mModel['classTable'][c]</ins></code></pre>
|
||||
<p>Searching the entire codebase for instances of <code>"ord(c)"</code> uncovers similar problems in <code>sbcharsetprober.py</code>…
|
||||
<p class=skip><a href=#skip-sbcharsetprober-code>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-sbcharsetprober-code>skip over this code listing</a>
|
||||
<pre><code># sbcharsetprober.py
|
||||
def feed(self, aBuf):
|
||||
if not self._mModel['keepEnglishLetter']:
|
||||
@@ -913,14 +915,14 @@ def feed(self, aBuf):
|
||||
for c in aBuf:
|
||||
<mark> order = self._mModel['charToOrderMap'][ord(c)]</mark></code></pre>
|
||||
<p id=skip-sbcharsetprober-code>…and <code>latin1prober.py</code>…
|
||||
<p class=skip><a href=#skip-latin1prober-code-2>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-latin1prober-code-2>skip over this code listing</a>
|
||||
<pre><code># latin1prober.py
|
||||
def feed(self, aBuf):
|
||||
aBuf = self.filter_with_english_letters(aBuf)
|
||||
for c in aBuf:
|
||||
<mark> charClass = Latin1_CharToClass[ord(c)]</mark></code></pre>
|
||||
<p id=skip-sbcharsetprober-code-2><var>c</var> is iterating over <var>aBuf</var>, which means it is an integer, not a 1-character string. The solution is the same: change <code>ord(c)</code> to just plain <code>c</code>.
|
||||
<p class=skip><a href=#unorderabletypes>skip over this code listing</a>
|
||||
<p class=s><a href=#unorderabletypes>skip over this code listing</a>
|
||||
<pre><code> # sbcharsetprober.py
|
||||
def feed(self, aBuf):
|
||||
if not self._mModel['keepEnglishLetter']:
|
||||
@@ -941,8 +943,8 @@ def feed(self, aBuf):
|
||||
</code></pre>
|
||||
<h3 id=unorderabletypes>Unorderable types: <code>int()</code> >= <code>str()</code></h3>
|
||||
<p>Let's go again.
|
||||
<p class=skip><a href=#skip-unorderable-types-screen>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skip-unorderable-types-screen>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml ascii with confidence 1.0
|
||||
tests\Big5\0804.blogspot.com.xml</samp>
|
||||
<samp>Traceback (most recent call last):
|
||||
@@ -961,7 +963,7 @@ tests\Big5\0804.blogspot.com.xml</samp>
|
||||
TypeError: unorderable types: int() >= str()</samp></pre>
|
||||
<p id=skip-unorderable-types-screen>Did you notice? This time around, the code passed the first test case (<code>tests\ascii\howto.diveintomark.org.xml</code>). You're making real progress here.
|
||||
<p>So what's this all about? “Unorderable types”? Once again, the difference between byte arrays and strings is rearing its ugly head. Take a look at the code:
|
||||
<p class=skip><a href=#skip-unorderable-types-1>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-unorderable-types-1>skip over this code listing</a>
|
||||
<pre><code>class SJISContextAnalysis(JapaneseContextAnalysis):
|
||||
def get_order(self, aStr):
|
||||
if not aStr: return -1, 1
|
||||
@@ -972,7 +974,7 @@ TypeError: unorderable types: int() >= str()</samp></pre>
|
||||
else:
|
||||
charLen = 1</code></pre>
|
||||
<p id=skip-unorderable-types-1>And where does <var>aStr</var> come from? Let's pop the stack:
|
||||
<p class=skip><a href=#skip-unorderable-types-2>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-unorderable-types-2>skip over this code listing</a>
|
||||
<pre><code>def feed(self, aBuf, aLen):
|
||||
.
|
||||
.
|
||||
@@ -983,7 +985,7 @@ TypeError: unorderable types: int() >= str()</samp></pre>
|
||||
<p id=skip-unorderable-types-2>Oh look, it's our old friend, <var>aBuf</var>. As you might have guessed from every other issue we've encountered in this chapter, <var>aBuf</var> is a byte array. Here, the <code>feed()</code> method isn't just passing it on wholesale; it's slicing it. But as you saw <a href=#unsupportedoperandtypeforplus>earlier in this chapter</a>, slicing a byte array returns a byte array, so the <var>aStr</var> parameter that gets passed to the <code>get_order()</code> method is still a byte array.
|
||||
<p>And what is this code trying to do with <var>aStr</var>? It's taking the first element of the byte array and comparing it to a string of length 1. In Python 2, that worked, because <var>aStr</var> and <var>aBuf</var> were strings, and <var>aStr[0]</var> would be a string, and you can compare strings for inequality. But in Python 3, <var>aStr</var> and <var>aBuf</var> are byte arrays, <var>aStr[0]</var> is an integer, and you can't compare integers and strings for inequality without explicitly coercing one of them.
|
||||
<p>In this case, there's no need to make the code more complicated by adding an explicit coercion. <var>aStr[0]</var> yields an integer; the things you're comparing to are all constants. Let's change them from 1-character strings to integers.
|
||||
<p class=skip><a href=#skip-unorderable-types-3>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-unorderable-types-3>skip over this code listing</a>
|
||||
<pre><code> class SJISContextAnalysis(JapaneseContextAnalysis):
|
||||
def get_order(self, aStr):
|
||||
if not aStr: return -1, 1
|
||||
@@ -1037,8 +1039,8 @@ TypeError: unorderable types: int() >= str()</samp></pre>
|
||||
|
||||
return -1, charLen</code></pre>
|
||||
<p>Searching the entire codebase for occurrences of the <code>ord()</code> function uncovers the same problem in <code>chardistribution.py</code>:
|
||||
<p class=skip><a href=#skip-unorderable-types-4>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skip-unorderable-types-4>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml ascii with confidence 1.0
|
||||
tests\Big5\0804.blogspot.com.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
@@ -1056,7 +1058,7 @@ tests\Big5\0804.blogspot.com.xml</samp>
|
||||
if (aStr[0] >= '\x81') and (aStr[0] <= '\x9F'):
|
||||
TypeError: unorderable types: int() >= str()</samp></pre>
|
||||
<p id=skip-unorderable-types-4>The fix is the same:
|
||||
<p class=skip><a href=#reduceisnotdefined>skip over this code listing</a>
|
||||
<p class=s><a href=#reduceisnotdefined>skip over this code listing</a>
|
||||
<pre><code> class EUCTWDistributionAnalysis(CharDistributionAnalysis):
|
||||
def __init__(self):
|
||||
CharDistributionAnalysis.__init__(self)
|
||||
@@ -1163,8 +1165,8 @@ TypeError: unorderable types: int() >= str()</samp></pre>
|
||||
return -1</code></pre>
|
||||
<h3 id=reduceisnotdefined>Global name <code>'reduce'</code> is not defined</h3>
|
||||
<p>Once more into the breach…
|
||||
<p class=skip><a href=#skip-reduceisnotdefined-output>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skip-reduceisnotdefined-output>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml ascii with confidence 1.0
|
||||
tests\Big5\0804.blogspot.com.xml</samp>
|
||||
<samp class=traceback>Traceback (most recent call last):
|
||||
@@ -1177,14 +1179,14 @@ tests\Big5\0804.blogspot.com.xml</samp>
|
||||
NameError: global name 'reduce' is not defined</samp></pre>
|
||||
<p id=skip-reduceisnotdefined-output>According to the official <a href=http://docs.python.org/dev/3.0/whatsnew/3.0.html#builtins>What's New In Python 3.0</a> guide, the <code>reduce()</code> function has been moved out of the global namespace and into the <code>functools</code> module. Quoting the guide: "Use <code>functools.reduce()</code> if you really need it; however, 99 percent of the time an explicit <code>for</code> loop is more readable."
|
||||
<p>OK then, let's refactor it to use a <code>for</code> loop.
|
||||
<p class=skip><a href=#skip-reduce-code>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-reduce-code>skip over this code listing</a>
|
||||
<pre><code>def get_confidence(self):
|
||||
if self.get_state() == constants.eNotMe:
|
||||
return 0.01
|
||||
|
||||
<mark> total = reduce(operator.add, self._mFreqCounter)</mark></code></pre>
|
||||
<p>The <code>reduce()</code> function takes two arguments — a function and a list (strictly speaking, any iterable object will do) — and applies the function cumulatively to each item of the list. In other words, this is a fancy and roundabout way of adding up all the items in a list and returning the result. It looks much more readable as a <code>for</code> loop.
|
||||
<p class=skip><a href=#skip-reduce-refactoring>skip over this code listing</a>
|
||||
<p class=s><a href=#skip-reduce-refactoring>skip over this code listing</a>
|
||||
<pre><code> def get_confidence(self):
|
||||
if self.get_state() == constants.eNotMe:
|
||||
return 0.01
|
||||
@@ -1194,8 +1196,8 @@ NameError: global name 'reduce' is not defined</samp></pre>
|
||||
<ins>+ for frequency in self._mFreqCounter:</ins>
|
||||
<ins>+ total += frequency</ins></code></pre>
|
||||
<p id=skip-reduce-refactoring>I CAN HAZ TESTZ?
|
||||
<p class=skip><a href=#skip-final-output>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=prompt>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<p class=s><a href=#skip-final-output>skip over this command output listing</a>
|
||||
<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
|
||||
<samp>tests\ascii\howto.diveintomark.org.xml ascii with confidence 1.0
|
||||
tests\Big5\0804.blogspot.com.xml Big5 with confidence 0.99
|
||||
tests\Big5\blog.worren.net.xml Big5 with confidence 0.99
|
||||
@@ -1239,6 +1241,6 @@ tests\EUC-JP\arclamp.jp.xml EUC-JP with confide
|
||||
<li><em>You</em> need to understand your program. Thoroughly. Preferably because you wrote it, but at the very least, you need to be comfortable with all its quirks and musty corners. The bugs are everywhere.
|
||||
<li>Test cases are essential. Don't port anything without them. Don't even try. The <em>only</em> reason I have any confidence at all that <code>chardet</code> works in Python 3 is because I had a test suite that exercised every line of code in the entire library. I <em>never</em> would have found half of these problems with manual spot-checking.
|
||||
</ol>
|
||||
<p class=c>© 2001–4, 2009 <span>ℳ</span>ark Pilgrim • <a href=about.html>open standards • open content • open source</a>
|
||||
<p class=c>© 2001–4, 2009 <span>ℳ</span>ark Pilgrim • <a href=about.html>open standards • open content • open source</a>
|
||||
<script src=jquery.js></script>
|
||||
<script src=dip3.js></script>
|
||||
|
||||
Reference in New Issue
Block a user