class=nd fiddling

advanced-iterators.html

@@ -150,7 +150,7 @@ if __name__ == '__main__':
 
 <p>The alphametics solver uses this technique to get a list of all the unique characters in the puzzle.
 
-<pre><code class=pp>unique_characters = {c for c in ''.join(words)}</code></pre>
+<pre class=nd><code class=pp>unique_characters = {c for c in ''.join(words)}</code></pre>
 
 <p>This list is later used to assign digits to characters as the solver iterates through the possible solutions.
 
@@ -173,11 +173,11 @@ AssertionError</samp></pre>
 
 <p>Therefore, this line of code:
 
-<pre><code class=pp>assert len(unique_characters) <= 10</code></pre>
+<pre class=nd><code class=pp>assert len(unique_characters) <= 10</code></pre>
 
 <p>&hellip;is equivalent to&hellip;
 
-<pre><code class=pp>if len(unique_characters) > 10:
+<pre class=nd><code class=pp>if len(unique_characters) > 10:
     raise AssertionError</code></pre>
 
 <p>But a bit easier to read and write.
@@ -210,7 +210,7 @@ AssertionError</samp></pre>
 
 <p>Here&#8217;s another way to accomplish the same thing, using a <a href=generators.html>generator function</a>:
 
-<pre><code class=pp>def ord_map(a_string):
+<pre class=nd><code class=pp>def ord_map(a_string):
     for c in a_string:
         yield ord(c)
 
@@ -398,7 +398,7 @@ Wesley</samp></pre>
 
 <p id=guess>The alphametics solver uses this technique to create a dictionary that maps letters in the puzzle to digits in the solution, for each possible solution.
 
-<pre><code class=pp>characters = tuple(ord(c) for c in sorted_characters)
+<pre class=nd><code class=pp>characters = tuple(ord(c) for c in sorted_characters)
 digits = tuple(ord(c) for c in '0123456789')
 ...
 for guess in itertools.permutations(digits, len(characters)):
@@ -454,7 +454,7 @@ for guess in itertools.permutations(digits, len(characters)):
 
 <p>This is the final piece of the puzzle (or rather, the final piece of the puzzle solver). After all that fancy string manipulation, we&#8217;re left with a string like <code>'9567 + 1085 == 10652'</code>. But that&#8217;s a string, and what good is a string? Enter <code>eval()</code>, the universal Python evaluation tool.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>eval('1 + 1 == 2')</kbd>
 <samp class=pp>True</samp>
 <samp class=p>>>> </samp><kbd class=pp>eval('1 + 1 == 3')</kbd>
@@ -464,7 +464,7 @@ for guess in itertools.permutations(digits, len(characters)):
 
 <p>But wait, there&#8217;s more! The <code>eval()</code> function isn&#8217;t limited to boolean expressions. It can handle <em>any</em> Python expression and returns <em>any</em> datatype.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>eval('"A" + "B"')</kbd>
 <samp class=pp>'AB'</samp>
 <samp class=p>>>> </samp><kbd class=pp>eval('"MARK".translate({65: 79})')</kbd>

case-study-porting-chardet-to-python-3.html

@@ -545,7 +545,7 @@ RefactoringTool: chardet\sjisprober.py
 RefactoringTool: chardet\universaldetector.py
 RefactoringTool: chardet\utf8prober.py</samp></pre>
 <p>Now run the <code>2to3</code> script on the testing harness, <code>test.py</code>.
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python c:\Python30\Tools\Scripts\2to3.py -w test.py</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python c:\Python30\Tools\Scripts\2to3.py -w test.py</kbd>
 <samp>RefactoringTool: Skipping implicit fixer: buffer
 RefactoringTool: Skipping implicit fixer: idioms
 RefactoringTool: Skipping implicit fixer: set_literal
@@ -583,7 +583,7 @@ RefactoringTool: test.py</samp></pre>
 <h3 id=falseisinvalidsyntax><code>False</code> is invalid syntax</h3>
 <aside>You do have tests, right?</aside>
 <p>Now for the real test: running the test harness against the test suite. Since the test suite is designed to cover all the possible code paths, it&#8217;s a good way to test our ported code to make sure there aren&#8217;t any bugs lurking anywhere.
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp class=traceback>Traceback (most recent call last):
   File "test.py", line 1, in &lt;module>
     from chardet.universaldetector import UniversalDetector
@@ -592,7 +592,7 @@ RefactoringTool: test.py</samp></pre>
                               ^
 SyntaxError: invalid syntax</samp></pre>
 <p>Hmm, a small snag. In Python 3, <code>False</code> is a reserved word, so you can&#8217;t use it as a variable name. Let&#8217;s look at <code>constants.py</code> to see where it&#8217;s defined. Here&#8217;s the original version from <code>constants.py</code>, before the <code>2to3</code> script changed it:
-<pre><code class=pp>import __builtin__
+<pre class=nd><code class=pp>import __builtin__
 if not hasattr(__builtin__, 'False'):
     False = 0
     True = 1
@@ -602,13 +602,13 @@ else:
 <p>This piece of code is designed to allow this library to run under older versions of Python 2. Prior to Python 2.3 [FIXME-LINK], Python had no built-in <code>bool</code> type. This code detects the absence of the built-in constants <code>True</code> and <code>False</code>, and defines them if necessary.
 <p>However, Python 3 will always have a <code>bool</code> type, so this entire code snippet is unnecessary. The simplest solution is to replace all instances of <code>constants.True</code> and <code>constants.False</code> with <code>True</code> and <code>False</code>, respectively, then delete this dead code from <code>constants.py</code>.
 <p>So this line in <code>universaldetector.py</code>:
-<pre><code class=pp>self.done = constants.False</code></pre>
+<pre class=nd><code class=pp>self.done = constants.False</code></pre>
 <p>Becomes
-<pre><code class=pp>self.done = False</code></pre>
+<pre class=nd><code class=pp>self.done = False</code></pre>
 <p>Ah, wasn&#8217;t that satisfying?  The code is shorter and more readable already.
 <h3 id=nomodulenamedconstants>No module named <code>constants</code></h3>
 <p>Time to run <code>test.py</code> again and see how far it gets.
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp class=traceback>Traceback (most recent call last):
   File "test.py", line 1, in &lt;module>
     from chardet.universaldetector import UniversalDetector
@@ -616,12 +616,12 @@ else:
     import constants, sys
 ImportError: No module named constants</samp></pre>
 <p>What&#8217;s that you say?  No module named <code>constants</code>?  Of course there&#8217;s a module named <code>constants</code>. &hellip;Oh wait, no there isn&#8217;t. Remember when the <code>2to3</code> script fixed up all those import statements?  This library has a lot of relative imports&nbsp;&mdash;&nbsp;that is, modules that import other modules within the library. In Python 3, all import statements are absolute by default [FIXME-LINK PEP 0328]. To do relative imports, you need to do something like this instead:
-<pre><code class=pp>from . import constants</code></pre>
+<pre class=nd><code class=pp>from . import constants</code></pre>
 <p>But wait. Wasn&#8217;t the <code>2to3</code> script supposed to take care of these for you?  Well, it did, but this particular import statement combines two different types of imports into one line: a relative import of the <code>constants</code> module within the library, and an absolute import of the <code>sys</code> module that is pre-installed in the Python standard library. In Python 2, you could combine these into one import statement. In Python 3, you can&#8217;t, and the <code>2to3</code> script is not smart enough to split the import statement into two.
 <p>The solution is to split the import statement manually. So this two-in-one import:
-<pre><code class=pp>import constants, sys</code></pre>
+<pre class=nd><code class=pp>import constants, sys</code></pre>
 <p>Needs to become two separate imports:
-<pre><code class=pp>from . import constants
+<pre class=nd><code class=pp>from . import constants
 import sys</code></pre>
 <p>There are variations of this problem scattered throughout the <code>chardet</code> library. In some places it&#8217;s &#8220;<code>import constants, sys</code>&#8221;; in other places, it&#8217;s &#8220;<code>import constants, re</code>&#8221;. The fix is the same: manually split the import statement into two lines, one for the relative import, the other for the absolute import.
 <p>FIXME-xref to as-yet-unwritten PEP 8 style section (which says you should put all imports on their own line)
@@ -629,7 +629,7 @@ import sys</code></pre>
 <h3 id=namefileisnotdefined>Name <var>'file'</var> is not defined</h3>
 <aside>open() is the new file(). PapayaWhip is the new black.</aside>
 <p>And here we go again, running <code>test.py</code> to try to execute our test cases&hellip;
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
   File "test.py", line 9, in &lt;module>
@@ -637,11 +637,11 @@ import sys</code></pre>
 NameError: name 'file' is not defined</samp></pre>
 <p>This one surprised me, because I&#8217;ve been using this idiom as long as I can remember. In Python 2, the global <code>file()</code> function was an alias for the <code>open()</code> function, which was the standard way of opening files for reading. In Python 3, the entire system for reading and writing files has been refactored into the <code>io</code> module. [FIXME-LINK PEP 3116]  I&#8217;ll cover the new I/O module in more detail in Chapter FIXME, but for now, the important bit is that the global <code>file()</code> function no longer exists. However, the <code>open()</code> function does still exist. (Technically, it&#8217;s an alias for <var>io.open()</var>, but never mind that right now.)
 <p>Thus, the simplest solution to the problem of the missing <code>file()</code> is to call the <code>open()</code> function instead:
-<pre><code class=pp>for line in open(f, 'rb'):</code></pre>
+<pre class=nd><code class=pp>for line in open(f, 'rb'):</code></pre>
 <p>And that&#8217;s all I have to say about that.
 <h3 id=cantuseastringpattern>Can&#8217;t use a string pattern on a bytes-like object</h3>
 <p>Now things are starting to get interesting. And by &#8220;interesting,&#8221; I mean &#8220;confusing as all hell.&#8221;
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
   File "test.py", line 10, in &lt;module>
@@ -650,20 +650,20 @@ NameError: name 'file' is not defined</samp></pre>
     if self._highBitDetector.search(aBuf):
 TypeError: can't use a string pattern on a bytes-like object</samp></pre>
 <p>To debug this, let&#8217;s see what <var>self._highBitDetector</var> is. It&#8217;s defined in the <var>__init__</var> method of the <var>UniversalDetector</var> class:
-<pre><code class=pp>class UniversalDetector:
+<pre class=nd><code class=pp>class UniversalDetector:
     def __init__(self):
         self._highBitDetector = re.compile(r'[\x80-\xFF]')</code></pre>
 <p>This pre-compiles a regular expression designed to find non-<abbr>ASCII</abbr> characters in the range 128&ndash;255 (0x80&ndash;0xFF). Wait, that&#8217;s not quite right; I need to be more precise with my terminology. This pattern is designed to find non-<abbr>ASCII</abbr> <em>bytes</em> in the range 128-255.
 <p>And therein lies the problem.
 <p>In Python 2, a string was an array of bytes whose character encoding was tracked separately. If you wanted Python 2 to keep track of the character encoding, you had to use a Unicode string (<code>u''</code>) instead. But in Python 3, a string is always what Python 2 called a Unicode string&nbsp;&mdash;&nbsp;that is, an array of Unicode characters (of possibly varying byte lengths). Since this regular expression is defined by a string pattern, it can only be used to search a string&nbsp;&mdash;&nbsp;again, an array of characters. But what we&#8217;re searching is not a string, it&#8217;s a byte array. Looking at the traceback, this error occurred in <code>universaldetector.py</code>:
-<pre><code class=pp>def feed(self, aBuf):
+<pre class=nd><code class=pp>def feed(self, aBuf):
     .
     .
     .
     if self._mInputState == ePureAscii:
         if self._highBitDetector.search(aBuf):</code></pre>
 <p>And what is <var>aBuf</var>?  Let&#8217;s backtrack further to a place that calls <code>UniversalDetector.feed()</code>. One place that calls it is the test harness, <code>test.py</code>.
-<pre><code class=pp>u = UniversalDetector()
+<pre class=nd><code class=pp>u = UniversalDetector()
 .
 .
 .
@@ -673,7 +673,7 @@ for line in open(f, 'rb'):
 <p>And here we find our answer: in the <code>UniversalDetector.feed()</code> method, <var>aBuf</var> is a line read from a file on disk. Look carefully at the parameters used to open the file: <code>'rb'</code>. <code>'r'</code> is for &#8220;read&#8221;; OK, big deal, we&#8217;re reading the file. Ah, but <code>'b'</code> is for &#8220;binary.&#8221;  Without the <code>'b'</code> flag, this <code>for</code> loop would read the file, line by line, and convert each line into a string&nbsp;&mdash;&nbsp;an array of Unicode characters&nbsp;&mdash;&nbsp;according to the system default character encoding. (You could override the system encoding with another parameter to the <code>open()</code> function, but never mind that for now.)  But with the <code>'b'</code> flag, this <code>for</code> loop reads the file, line by line, and stores each line exactly as it appears in the file, as an array of bytes. That byte array gets passed to <code>UniversalDetector.feed()</code>, and eventually gets passed to the pre-compiled regular expression, <var>self._highBitDetector</var>, to search for high-bit&hellip; characters. But we don&#8217;t have characters; we have bytes. Oops.
 <p>What we need this regular expression to search is not an array of characters, but an array of bytes.
 <p>Once you realize that, the solution is not difficult. Regular expressions defined with strings can search strings. Regular expressions defined with byte arrays can search byte arrays. To define a byte array pattern, we simply change the type of the argument we use to define the regular expression to a byte array. (There is one other case of this same problem, on the very next line.)
-<pre><code class=pp>  class UniversalDetector:
+<pre class=nd><code class=pp>  class UniversalDetector:
       def __init__(self):
 <del>-         self._highBitDetector = re.compile(r'[\x80-\xFF]')</del>
 <del>-         self._escDetector = re.compile(r'(\033|~{)')</del>
@@ -683,7 +683,7 @@ for line in open(f, 'rb'):
           self._mCharSetProbers = []
           self.reset()</code></pre>
 <p>Searching the entire codebase for other uses of the <code>re</code> module turns up two more instances, in <code>charsetprober.py</code>. Again, the code is defining regular expressions as strings but executing them on <var>aBuf</var>, which is a byte array. The solution is the same: define the regular expression patterns as byte arrays.
-<pre><code class=pp>  class CharSetProber:
+<pre class=nd><code class=pp>  class CharSetProber:
       .
       .
       .
@@ -699,7 +699,7 @@ for line in open(f, 'rb'):
         
 <h3 id=cantconvertbytesobject>Can't convert <code>'bytes'</code> object to <code>str</code> implicitly</h3>
 <p>Curiouser and curiouser&hellip;
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
   File "test.py", line 10, in &lt;module>
@@ -708,10 +708,10 @@ for line in open(f, 'rb'):
     elif (self._mInputState == ePureAscii) and self._escDetector.search(self._mLastChar + aBuf):
 TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
 <p>There&#8217;s an unfortunate clash of coding style and Python interpreter here. The <code>TypeError</code> could be anywhere on that line, but the traceback doesn&#8217;t tell you exactly where it is. It could be in the first conditional or the second, and the traceback would look the same. To narrow it down, you should split the line in half, like this:
-<pre><code class=pp>elif (self._mInputState == ePureAscii) and \
+<pre class=nd><code class=pp>elif (self._mInputState == ePureAscii) and \
     self._escDetector.search(self._mLastChar + aBuf):</code></pre>
 <p>And re-run the test:
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
   File "test.py", line 10, in &lt;module>
@@ -721,7 +721,7 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
 TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
 <p>Aha! The problem was not in the first conditional (<code>self._mInputState == ePureAscii</code>) but in the second one. So what could cause a <code>TypeError</code> there? Perhaps you&#8217;re thinking that the <code>search()</code> method is expecting a value of a different type, but that wouldn&#8217;t generate this traceback. Python functions can take any value; if you pass the right number of arguments, the function will execute. It may <em>crash</em> if you pass it a value of a different type than it&#8217;s expecting, but if that happened, the traceback would point to somewhere inside the function. But this traceback says it never got as far as calling the <code>search()</code> method. So the problem must be in that <code>+</code> operation, as it&#8217;s trying to construct the value that it will eventually pass to the <code>search()</code> method.
 <p>We know from <a href=#cantuseastringpattern>previous debugging</a> that <var>aBuf</var> is a byte array. So what is <code>self._mLastChar</code>? It&#8217;s an instance variable, defined in the <code>reset()</code> method, which is actually called from the <code>__init__()</code> method.
-<pre><code class=pp>class UniversalDetector:
+<pre class=nd><code class=pp>class UniversalDetector:
     def __init__(self):
         self._highBitDetector = re.compile(b'[\x80-\xFF]')
         self._escDetector = re.compile(b'(\033|~{)')
@@ -738,7 +738,7 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
 <mark>        self._mLastChar = ''</mark></code></pre>
 <p>And now we have our answer. Do you see it? <var>self._mLastChar</var> is a string, but <var>aBuf</var> is a byte array. And you can&#8217;t concatenate a string to a byte array&nbsp;&mdash;&nbsp;not even a zero-length string.
 <p>So what is <var>self._mLastChar</var> anyway? The answer is in the <code>feed()</code> method, just a few lines down from where the trackback occurred.
-<pre><code class=pp>if self._mInputState == ePureAscii:
+<pre class=nd><code class=pp>if self._mInputState == ePureAscii:
     if self._highBitDetector.search(aBuf):
         self._mInputState = eHighbyte
     elif (self._mInputState == ePureAscii) and \
@@ -747,14 +747,14 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
 
 <mark>self._mLastChar = aBuf[-1]</mark></code></pre>
 <p>The calling function calls this <code>feed()</code> method over and over again with a few bytes at a time. The method processes the bytes it was given (passed in as <var>aBuf</var>), then stores the last byte in <var>self._mLastChar</var> in case it&#8217;s needed during the next call. (In a multi-byte encoding, the <code>feed()</code> method might get called with half of a character, then called again with the other half.)  But because <var>aBuf</var> is now a byte array instead of a string, <var>self._mLastChar</var> needs to be a byte array as well. Thus:
-<pre><code class=pp>  def reset(self):
+<pre class=nd><code class=pp>  def reset(self):
       .
       .
       .
 <del>-     self._mLastChar = ''</del>
 <ins>+     self._mLastChar = b''</ins></code></pre>
 <p>Searching the entire codebase for &#8220;<code>mLastChar</code>&#8221; turns up a similar problem in <code>mbcharsetprober.py</code>, but instead of tracking the last character, it tracks the last <em>two</em> characters. The <code>MultiByteCharSetProber</code> class uses a list of 1-character strings to track the last two characters; in Python 3, it needs to use a list of integers.
-<pre><code class=pp>  class MultiByteCharSetProber(CharSetProber):
+<pre class=nd><code class=pp>  class MultiByteCharSetProber(CharSetProber):
       def __init__(self):
           CharSetProber.__init__(self)
           self._mDistributionAnalyzer = None
@@ -772,7 +772,7 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp></pre>
 <ins>+         self._mLastChar = [0, 0]</ins></code></pre>
 <h3 id=unsupportedoperandtypeforplus>Unsupported operand type(s) for +: <code>'int'</code> and <code>'bytes'</code></h3>
 <p>I have good news, and I have bad news. The good news is we&#8217;re making progress&hellip;
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
   File "test.py", line 10, in &lt;module>
@@ -783,7 +783,7 @@ TypeError: unsupported operand type(s) for +: 'int' and 'bytes'</samp></pre>
 <p>&hellip;The bad news is it doesn&#8217;t always feel like progress.
 <p>But this is progress! Really! Even though the traceback calls out the same line of code, it&#8217;s a different error than it used to be. Progress! So what&#8217;s the problem now? The last time I checked, this line of code didn&#8217;t try to concatenate an <code>int</code> with a byte array (<code>bytes</code>). In fact, you just spent a lot of time <a href=#cantconvertbytesobject>ensuring that <var>self._mLastChar</var> was a byte array</a>. How did it turn into an <code>int</code>?
 <p>The answer lies not in the previous lines of code, but in the following lines.
-<pre><code class=pp>if self._mInputState == ePureAscii:
+<pre class=nd><code class=pp>if self._mInputState == ePureAscii:
     if self._highBitDetector.search(aBuf):
         self._mInputState = eHighbyte
     elif (self._mInputState == ePureAscii) and \
@@ -820,14 +820,14 @@ TypeError: unsupported operand type(s) for +: 'int' and 'bytes'</samp>
 <li>Concatenating a byte array of length 1 with a byte array of length 3 returns a new byte array of length 4.
 </ol>
 <p>So, to ensure that the <code>feed()</code> method in <code>universaldetector.py</code> continues to work no matter how often it&#8217;s called, you need to <a href=#cantconvertbytesobject>initialize <var>self._mLastChar</var> as a 0-length byte array</a>, then <em>make sure it stays a byte array</em>.
-<pre><code class=pp>              self._escDetector.search(self._mLastChar + aBuf):
+<pre class=nd><code class=pp>              self._escDetector.search(self._mLastChar + aBuf):
           self._mInputState = eEscAscii
 
 <del>- self._mLastChar = aBuf[-1]</del>
 <ins>+ self._mLastChar = aBuf[-1:]</ins></code></pre>
 <h3 id=ordexpectedstring><code>ord()</code> expected string of length 1, but <code>int</code> found</h3>
 <p>Tired yet? You&#8217;re almost there&hellip;
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml                       ascii with confidence 1.0
 tests\Big5\0804.blogspot.com.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
@@ -843,25 +843,25 @@ tests\Big5\0804.blogspot.com.xml</samp>
     byteCls = self._mModel['classTable'][ord(c)]
 TypeError: ord() expected string of length 1, but int found</samp></pre>
 <p>OK, so <var>c</var> is an <code>int</code>, but the <code>ord()</code> function was expecting a 1-character string. Fair enough. Where is <var>c</var> defined?
-<pre><code class=pp># codingstatemachine.py
+<pre class=nd><code class=pp># codingstatemachine.py
 def next_state(self, c):
     # for each byte we get its class
     # if it is first byte, we also get byte length
     byteCls = self._mModel['classTable'][ord(c)]</code></pre>
 <p>That&#8217;s no help; it&#8217;s just passed into the function. Let&#8217;s pop the stack.
-<pre><code class=pp># utf8prober.py
+<pre class=nd><code class=pp># utf8prober.py
 def feed(self, aBuf):
     for c in aBuf:
         codingState = self._mCodingSM.next_state(c)</code></pre>
 <p>And now we have the answer. Do you see it? In Python 2, <var>aBuf</var> was a string, so <var>c</var> was a 1-character string. (That&#8217;s what you get when you iterate over a string&nbsp;&mdash;&nbsp;all the characters, one by one.) But now, <var>aBuf</var> is a byte array, so <var>c</var> is an <code>int</code>, not a 1-character string. In other words, there&#8217;s no need to call the <code>ord()</code> function because <var>c</var> is already an <code>int</code>!
 <p>Thus:
-<pre><code class=pp>  def next_state(self, c):
+<pre class=nd><code class=pp>  def next_state(self, c):
       # for each byte we get its class
       # if it is first byte, we also get byte length
 <del>-     byteCls = self._mModel['classTable'][ord(c)]</del>
 <ins>+     byteCls = self._mModel['classTable'][c]</ins></code></pre>
 <p>Searching the entire codebase for instances of &#8220;<code>ord(c)</code>&#8221; uncovers similar problems in <code>sbcharsetprober.py</code>&hellip;
-<pre><code class=pp># sbcharsetprober.py
+<pre class=nd><code class=pp># sbcharsetprober.py
 def feed(self, aBuf):
     if not self._mModel['keepEnglishLetter']:
         aBuf = self.filter_without_english_letters(aBuf)
@@ -871,13 +871,13 @@ def feed(self, aBuf):
     for c in aBuf:
 <mark>        order = self._mModel['charToOrderMap'][ord(c)]</mark></code></pre>
 <p>&hellip;and <code>latin1prober.py</code>&hellip;
-<pre><code class=pp># latin1prober.py
+<pre class=nd><code class=pp># latin1prober.py
 def feed(self, aBuf):
     aBuf = self.filter_with_english_letters(aBuf)
     for c in aBuf:
 <mark>        charClass = Latin1_CharToClass[ord(c)]</mark></code></pre>
 <p><var>c</var> is iterating over <var>aBuf</var>, which means it is an integer, not a 1-character string.  The solution is the same: change <code>ord(c)</code> to just plain <code>c</code>.
-<pre><code class=pp>  # sbcharsetprober.py
+<pre class=nd><code class=pp>  # sbcharsetprober.py
   def feed(self, aBuf):
       if not self._mModel['keepEnglishLetter']:
           aBuf = self.filter_without_english_letters(aBuf)
@@ -897,7 +897,7 @@ def feed(self, aBuf):
 </code></pre>
 <h3 id=unorderabletypes>Unorderable types: <code>int()</code> >= <code>str()</code></h3>
 <p>Let&#8217;s go again.
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml                       ascii with confidence 1.0
 tests\Big5\0804.blogspot.com.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
@@ -916,7 +916,7 @@ tests\Big5\0804.blogspot.com.xml</samp>
 TypeError: unorderable types: int() >= str()</samp></pre>
 <p>Did you notice? This time around, the code passed the first test case (<code>tests\ascii\howto.diveintomark.org.xml</code>). You&#8217;re making real progress here.
 <p>So what&#8217;s this all about? &#8220;Unorderable types&#8221;? Once again, the difference between byte arrays and strings is rearing its ugly head. Take a look at the code:
-<pre><code class=pp>class SJISContextAnalysis(JapaneseContextAnalysis):
+<pre class=nd><code class=pp>class SJISContextAnalysis(JapaneseContextAnalysis):
     def get_order(self, aStr):
         if not aStr: return -1, 1
         # find out current char's byte length
@@ -926,7 +926,7 @@ TypeError: unorderable types: int() >= str()</samp></pre>
         else:
             charLen = 1</code></pre>
 <p>And where does <var>aStr</var> come from? Let&#8217;s pop the stack:
-<pre><code class=pp>def feed(self, aBuf, aLen):
+<pre class=nd><code class=pp>def feed(self, aBuf, aLen):
     .
     .
     .
@@ -936,7 +936,7 @@ TypeError: unorderable types: int() >= str()</samp></pre>
 <p>Oh look, it&#8217;s our old friend, <var>aBuf</var>. As you might have guessed from every other issue we&#8217;ve encountered in this chapter, <var>aBuf</var> is a byte array. Here, the <code>feed()</code> method isn&#8217;t just passing it on wholesale; it&#8217;s slicing it. But as you saw <a href=#unsupportedoperandtypeforplus>earlier in this chapter</a>, slicing a byte array returns a byte array, so the <var>aStr</var> parameter that gets passed to the <code>get_order()</code> method is still a byte array.
 <p>And what is this code trying to do with <var>aStr</var>? It&#8217;s taking the first element of the byte array and comparing it to a string of length 1. In Python 2, that worked, because <var>aStr</var> and <var>aBuf</var> were strings, and <var>aStr[0]</var> would be a string, and you can compare strings for inequality. But in Python 3, <var>aStr</var> and <var>aBuf</var> are byte arrays, <var>aStr[0]</var> is an integer, and you can&#8217;t compare integers and strings for inequality without explicitly coercing one of them.
 <p>In this case, there&#8217;s no need to make the code more complicated by adding an explicit coercion. <var>aStr[0]</var> yields an integer; the things you&#8217;re comparing to are all constants. Let&#8217;s change them from 1-character strings to integers.
-<pre><code class=pp>  class SJISContextAnalysis(JapaneseContextAnalysis):
+<pre class=nd><code class=pp>  class SJISContextAnalysis(JapaneseContextAnalysis):
       def get_order(self, aStr):
           if not aStr: return -1, 1
           # find out current char's byte length
@@ -989,7 +989,7 @@ TypeError: unorderable types: int() >= str()</samp></pre>
 
         return -1, charLen</code></pre>
 <p>Searching the entire codebase for occurrences of the <code>ord()</code> function uncovers the same problem in <code>chardistribution.py</code>:
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml                       ascii with confidence 1.0
 tests\Big5\0804.blogspot.com.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
@@ -1007,7 +1007,7 @@ tests\Big5\0804.blogspot.com.xml</samp>
     if (aStr[0] >= '\x81') and (aStr[0] &lt;= '\x9F'):
 TypeError: unorderable types: int() >= str()</samp></pre>
 <p>The fix is the same:
-<pre><code class=pp>  class EUCTWDistributionAnalysis(CharDistributionAnalysis):
+<pre class=nd><code class=pp>  class EUCTWDistributionAnalysis(CharDistributionAnalysis):
       def __init__(self):
           CharDistributionAnalysis.__init__(self)
           self._mCharToFreqOrder = EUCTWCharToFreqOrder
@@ -1113,7 +1113,7 @@ TypeError: unorderable types: int() >= str()</samp></pre>
               return -1</code></pre>
 <h3 id=reduceisnotdefined>Global name <code>'reduce'</code> is not defined</h3>
 <p>Once more into the breach&hellip;
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml                       ascii with confidence 1.0
 tests\Big5\0804.blogspot.com.xml</samp>
 <samp class=traceback>Traceback (most recent call last):
@@ -1125,25 +1125,25 @@ tests\Big5\0804.blogspot.com.xml</samp>
     total = reduce(operator.add, self._mFreqCounter)
 NameError: global name 'reduce' is not defined</samp></pre>
 <p>According to the official <a href=http://docs.python.org/3.0/whatsnew/3.0.html#builtins>What&#8217;s New In Python 3.0</a> guide, the <code>reduce()</code> function has been moved out of the global namespace and into the <code>functools</code> module. Quoting the guide: &#8220;Use <code>functools.reduce()</code> if you really need it; however, 99 percent of the time an explicit <code>for</code> loop is more readable.&#8221; You can read more about the decision from Guido van Rossum&#8217;s weblog: <a href='http://www.artima.com/weblogs/viewpost.jsp?thread=98196'>The fate of reduce() in Python 3000</a>.
-<pre><code class=pp>def get_confidence(self):
+<pre class=nd><code class=pp>def get_confidence(self):
     if self.get_state() == constants.eNotMe:
         return 0.01
   
 <mark>    total = reduce(operator.add, self._mFreqCounter)</mark></code></pre>
 <p>The <code>reduce()</code> function takes two arguments&nbsp;&mdash;&nbsp;a function and a list (strictly speaking, any iterable object will do)&nbsp;&mdash;&nbsp;and applies the function cumulatively to each item of the list. In other words, this is a fancy and roundabout way of adding up all the items in a list and returning the result.
 <p>This monstrosity was so common that Python added a global <code>sum()</code> function.
-<pre><code class=pp>  def get_confidence(self):
+<pre class=nd><code class=pp>  def get_confidence(self):
       if self.get_state() == constants.eNotMe:
           return 0.01
   
 <del>-     total = reduce(operator.add, self._mFreqCounter)</del>
 <ins>+     total = sum(self._mFreqCounter)</ins></code></pre>
 <p>Since you&#8217;re no longer using the <code>operator</code> module, you can remove that <code>import</code> from the top of the file as well.
-<pre><code class=pp>  from .charsetprober import CharSetProber
+<pre class=nd><code class=pp>  from .charsetprober import CharSetProber
   from . import constants
 <del>- import operator</del></code></pre>
 <p>I CAN HAZ TESTZ?
-<pre class=screen><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
+<pre class='nd screen'><samp class=p>C:\home\chardet> </samp><kbd>python test.py tests\*\*</kbd>
 <samp>tests\ascii\howto.diveintomark.org.xml                       ascii with confidence 1.0
 tests\Big5\0804.blogspot.com.xml                             Big5 with confidence 0.99
 tests\Big5\blog.worren.net.xml                               Big5 with confidence 0.99

generators.html

@@ -174,7 +174,7 @@ def plural(noun):
 
 <p>If this additional level of abstraction is confusing, try unrolling the function to see the equivalence. The entire <code>for</code> loop is equivalent to the following:
 
-<pre><code class=pp>
+<pre class=nd><code class=pp>
 def plural(noun):
     if match_sxz(noun):
         return apply_sxz(noun)
@@ -256,7 +256,7 @@ def build_match_and_apply_functions(pattern, search, replace):
 <p>First, let&#8217;s create a text file that contains the rules you want. No fancy data structures, just whitespace-delimited strings in three columns. Let&#8217;s call it <code>plural4-rules.txt</code>.
 
 <p class=d>[<a href=examples/plural4-rules.txt>download <code>plural4-rules.txt</code></a>]
-<pre><code class=pp>[sxz]$               $    es
+<pre class=nd><code class=pp>[sxz]$               $    es
 [^aeioudgkprt]h$     $    es
 [^aeiou]y$          y$    ies
 $                    $    s</code></pre>
@@ -295,7 +295,7 @@ rules = []
 <p>Wouldn&#8217;t it be grand to have a generic <code>plural()</code> function that parses the rules file? Get rules, check for a match, apply appropriate transformation, go to next rule. That&#8217;s all the <code>plural()</code> function has to do, and that&#8217;s all the <code>plural()</code> function should do.
 
 <p class=d>[<a href=examples/plural5.py>download <code>plural5.py</code></a>]
-<pre><code class=pp>def rules():
+<pre class=nd><code class=pp>def rules():
     with open('plural5-rules.txt') as pattern_file:
         for line in pattern_file:
             pattern, search, replace = line.split(None, 3)

http-web-services.html

@@ -58,7 +58,7 @@ mark{display:inline}
 
 <p>Here&#8217;s a concrete example of how caching works. You visit <a href=http://diveintomark.org/><code>diveintomark.org</code></a> in your browser. That page includes a background image, <a href=http://wearehugh.com/m.jpg><code>wearehugh.com/m.jpg</code></a>. When your browser downloads that image, the server includes the following <abbr>HTTP</abbr> headers:
 
-<pre><code>HTTP/1.1 200 OK
+<pre class=nd><code>HTTP/1.1 200 OK
 Date: Sun, 31 May 2009 17:14:04 GMT
 Server: Apache
 Last-Modified: Fri, 22 Aug 2008 04:28:16 GMT
@@ -86,7 +86,7 @@ Content-Type: image/jpeg</code></pre>
 
 <p><abbr>HTTP</abbr> has a solution to this, too. When you request data for the first time, the server can send back a <code>Last-Modified</code> header. This is exactly what it sounds like: the date that the data was changed. That background image referenced from <code>diveintomark.org</code> included a <code>Last-Modified</code> header.
 
-<pre><code>HTTP/1.1 200 OK
+<pre class=nd><code>HTTP/1.1 200 OK
 Date: Sun, 31 May 2009 17:14:04 GMT
 Server: Apache
 <mark>Last-Modified: Fri, 22 Aug 2008 04:28:16 GMT</mark>
@@ -101,7 +101,7 @@ Content-Type: image/jpeg
 
 <p>When you request the same data a second (or third or fourth) time, you can send an <code>If-Modified-Since</code> header with your request, with the date you got back from the server last time. If the data hasn&#8217;t changed since then, the server sends back a special <abbr>HTTP</abbr> <code>304</code> status code, which means &#8220;this data hasn&#8217;t changed since the last time you asked for it.&#8221; You can test this on the command line, using <a href=http://curl.haxx.se/>curl</a>:
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>curl -I <mark>-H "If-Modified-Since: Fri, 22 Aug 2008 04:28:16 GMT"</mark> http://wearehugh.com/m.jpg</kbd>
 <samp>HTTP/1.1 304 Not Modified
 Date: Sun, 31 May 2009 18:04:39 GMT
@@ -119,7 +119,7 @@ Cache-Control: max-age=31536000, public</samp></pre>
 
 <p>ETags are an alternate way to accomplish the same thing as the <a href=#last-modified>last-modified checking</a>. With Etags, the server sends a hash code in an <code>ETag</code> header along with the data you requested. (Exactly how this hash is determined is entirely up to the server. The only requirement is that it changes when the data changes.) That background image referenced from <code>diveintomark.org</code> had an <code>ETag</code> header.
 
-<pre><code>HTTP/1.1 200 OK
+<pre class=nd><code>HTTP/1.1 200 OK
 Date: Sun, 31 May 2009 17:14:04 GMT
 Server: Apache
 Last-Modified: Fri, 22 Aug 2008 04:28:16 GMT
@@ -136,7 +136,7 @@ The second time you request the same data, you include the ETag hash in an <code
 
 <p>Again with the <kbd>curl</kbd>:
 
-<pre class=screen>
+<pre class='nd screen'>
 <a><samp class=p>you@localhost:~$ </samp><kbd>curl -I <mark>-H "If-None-Match: \"3075-ddc8d800\""</mark> http://wearehugh.com/m.jpg</kbd>  <span class=u>&#x2460;</span></a>
 <samp>HTTP/1.1 304 Not Modified
 Date: Sun, 31 May 2009 18:04:39 GMT
@@ -176,7 +176,7 @@ Cache-Control: max-age=31536000, public</samp></pre>
 <h2 id=dont-try-this-at-home>How Not To Fetch Data Over HTTP</h2>
 
 <p>Let&#8217;s say you want to download a resource over <abbr>HTTP</abbr>, such as <a href=xml.html>an Atom feed</a>. Being a feed, you&#8217;re not just going to download it once; you&#8217;re going to download it over and over again. (Most feed readers will check for changes once an hour.) Let&#8217;s do it the quick-and-dirty way first, and then see how you can do better.
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>import urllib.request</kbd>
 <a><samp class=p>>>> </samp><kbd class=pp>data = urllib.request.urlopen('http://diveintopython3.org/examples/feed.xml').read()</kbd>  <span class=u>&#x2460;</span></a>
 <samp class=p>>>> </samp><kbd class=pp>print(data)</kbd>
@@ -255,7 +255,7 @@ Content-Type: application/xml</samp>
 
 <p>But wait, it gets worse! To see just how inefficient this code is, let&#8217;s request the same feed a second time.
 
-<pre class=screen>
+<pre class='nd screen'>
 # continued from the <a href=#whats-on-the-wire>previous example</a>
 <samp class=p>>>> </samp><kbd class=pp>response2 = urlopen('http://diveintopython3.org/examples/feed.xml')</kbd>
 <samp>send: b'GET /examples/feed.xml HTTP/1.1

installing-python.html

@@ -36,7 +36,7 @@ h2,.i>li{clear:both}
 
 <p>Once you&#8217;re at a command line prompt, just type <kbd>python3</kbd> (all lowercase, no spaces) and see what happens. On my home Linux system, Python 3 is already installed, and this command gets me into the <i>Python <dfn>interactive shell</dfn></i>.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>mark@atlantis:~$ </samp><kbd>python3</kbd>
 <samp>Python 3.0.1+ (r301:69556, Apr 15 2009, 17:25:52)
 [GCC 4.3.3] on linux2
@@ -47,7 +47,7 @@ Type "help", "copyright", "credits" or "license" for more information.
 
 <p>My <a href=http://cornerhost.com/>web hosting provider</a> also runs Linux and provides command-line access, but my server does not have Python 3 installed. (Boo!)
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>mark@manganese:~$ </samp><kbd>python3</kbd>
 <samp>bash: python3: command not found</samp></pre>
 
@@ -274,7 +274,7 @@ Type "help", "copyright", "credits" or "license" for more information.
 
 <p>First things first. The Python Shell itself is an amazing interactive playground. Throughout this book, you&#8217;ll see examples like this:
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>1 + 1</kbd>
 <samp class=pp>2</samp></pre>
 
@@ -286,14 +286,14 @@ Type "help", "copyright", "credits" or "license" for more information.
 
 <p>Let&#8217;s try another one.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>print('Hello world!')</kbd>
 <samp>Hello world!</samp>
 </pre>
 
 <p>Pretty simple, no? But there&#8217;s lots more you can do in the Python shell. If you ever get stuck&nbsp;&mdash;&nbsp;you can&#8217;t remember a command, or you can&#8217;t remember the proper arguments to pass a certain function&nbsp;&mdash;&nbsp;you can get interactive help in the Python Shell. Just type <kbd>help</kbd> and press <kbd>ENTER</kbd>.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd>help</kbd>
 <samp>Type help() for interactive help, or help(object) for help about object.</samp></pre>
 
@@ -301,7 +301,7 @@ Type "help", "copyright", "credits" or "license" for more information.
 
 <p>To enter the interactive help mode, type <kbd>help()</kbd> and press <kbd>ENTER</kbd>.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>help()</kbd>
 <samp>Welcome to Python 3.0!  This is the online help utility.
 

iterators.html

@@ -45,7 +45,7 @@ body{counter-reset:h1 6}
 
 <p>Let&#8217;s take that one line at a time.
 
-<pre><code class=pp>class Fib:</code></pre>
+<pre class=nd><code class=pp>class Fib:</code></pre>
 
 <p><code>class</code>? What&#8217;s a class?
 
@@ -143,7 +143,7 @@ body{counter-reset:h1 6}
 
 <p>Instance variables are specific to one instance of a class. For example, if you create two <code>Fib</code> instances with different maximum values, they will each remember their own values.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>import fibonacci2</kbd>
 <samp class=p>>>> </samp><kbd class=pp>fib1 = fibonacci2.Fib(100)</kbd>
 <samp class=p>>>> </samp><kbd class=pp>fib2 = fibonacci2.Fib(200)</kbd>
@@ -189,7 +189,7 @@ All three of these class methods, <code>__init__</code>, <code>__iter__</code>,
 
 <p>Thoroughly confused yet? Excellent. Let&#8217;s see how to call this iterator:
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>from fibonacci2 import Fib</kbd>
 <samp class=p>>>> </samp><kbd class=pp>for n in Fib(1000):</kbd>
 <samp class=p>... </samp><kbd class=pp>    print(n, end=' ')</kbd>

native-datatypes.html

@@ -39,10 +39,10 @@ body{counter-reset:h1 2}
 <aside>You can use virtually any expression in a boolean context.</aside>
 <p>Booleans are either true or false. Python has two constants, cleverly <code>True</code> and <code>False</code>, which can be used to assign boolean values directly. Expressions can also evaluate to a boolean value. In certain places (like <code>if</code> statements), Python expects an expression to evaluate to a boolean value. These places are called <i>boolean contexts</i>. You can use virtually any expression in a boolean context, and Python will try to determine its truth value. Different datatypes have different rules about which values are true or false in a boolean context. (This will make more sense once you see some concrete examples later in this chapter.)
 <p>For example, take this snippet from <a href=your-first-python-program.html#divingin><code>humansize.py</code></a>:
-<pre><code class=pp>if size &lt; 0:
+<pre class=nd><code class=pp>if size &lt; 0:
     raise ValueError('number must be non-negative')</code></pre>
 <p><var>size</var> is an integer, <code>0</code> is an integer, and <code>&lt;</code> is a numerical operator. The result of the expression <code>size &lt; 0</code> is always a boolean. You can test this yourself in the Python interactive shell:
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>size = 1</kbd>
 <samp class=p>>>> </samp><kbd class=pp>size &lt; 0</kbd>
 <samp class=pp>False</samp>
@@ -53,7 +53,7 @@ body{counter-reset:h1 2}
 <samp class=p>>>> </samp><kbd class=pp>size &lt; 0</kbd>
 <samp class=pp>True</samp></pre>
 <p>Due to some legacy issues left over from Python 2, booleans can be treated as numbers. <code>True</code> is <code>1</code>; <code>False</code> is <code>0</code>.
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>True + True</kbd>
 <samp class=pp>2</samp>
 <samp class=p>>>> </samp><kbd class=pp>True + False</kbd>
@@ -741,7 +741,7 @@ KeyError: 'db.diveintopython3.org'</samp></pre>
 <h3 id=mixed-value-dictionaries>Mixed-Value Dictionaries</h3>
 <p>Dictionaries aren&#8217;t just for strings. Dictionary values can be any datatype, including integers, booleans, arbitrary objects, or even other dictionaries. And within a single dictionary, the values don&#8217;t all need to be the same type; you can mix and match as needed. Dictionary keys are more restricted, but they can be strings, integers, and a few other types. You can also mix and match key datatypes within a dictionary.
 <p>In fact, you&#8217;ve already seen a dictionary with non-string keys and values, in <a href=your-first-python-program.html#divingin>your first Python program</a>.
-<pre><code class=pp>SUFFIXES = {1000: ['KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'],
+<pre class=nd><code class=pp>SUFFIXES = {1000: ['KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'],
             1024: ['KiB', 'MiB', 'GiB', 'TiB', 'PiB', 'EiB', 'ZiB', 'YiB']}</code></pre>
 <p>Let's tear that apart in the interactive shell.
 <pre class=screen>
@@ -787,7 +787,7 @@ KeyError: 'db.diveintopython3.org'</samp></pre>
 <h2 id=none><code>None</code></h2>
 <p><code>None</code> is a special constant in Python. It is a null value. <code>None</code> is not the same as <code>False</code>. <code>None</code> is not <code>0</code>. <code>None</code> is not an empty string. Comparing <code>None</code> to anything other than <code>None</code> will always return <code>False</code>.
 <p><code>None</code> is the only null value. It has its own datatype (<code>NoneType</code>). You can assign <code>None</code> to any variable, but you can not create other <code>NoneType</code> objects. All variables whose value is <code>None</code> are equal to each other.
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>type(None)</kbd>
 <samp class=pp>&lt;class 'NoneType'></samp>
 <samp class=p>>>> </samp><kbd class=pp>None == False</kbd>
@@ -807,7 +807,7 @@ KeyError: 'db.diveintopython3.org'</samp></pre>
 </pre>
 <h3 id=none-in-a-boolean-context><code>None</code> In A Boolean Context</h3>
 <p>In <a href=#booleans>a boolean context</a>, <code>None</code> is false and <code>not None</code> is true.
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>def is_it_true(anything):</kbd>
 <samp class=p>... </samp><kbd class=pp>  if anything:</kbd>
 <samp class=p>... </samp><kbd class=pp>    print('yes, it's true')</kbd>

refactoring.html

@@ -43,7 +43,7 @@ body{counter-reset:h1 10}
 </ol>
 
 <p>Since your code has a bug, and you now have a test case that tests this bug, the test case will fail:
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>python3 romantest8.py -v</kbd>
 <samp>from_roman should fail with blank string ... FAIL
 from_roman should fail with malformed antecedents ... ok
@@ -264,7 +264,7 @@ def from_roman(s):
 
 <p>You may be skeptical that these two small changes are all that you need. Hey, don&#8217;t take my word for it; see for yourself.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>python3 romantest9.py -v</kbd>
 <samp>from_roman should fail with blank string ... ok
 from_roman should fail with malformed antecedents ... ok
@@ -364,7 +364,7 @@ build_lookup_tables()</code></pre>
 
 <p>Let&#8217;s break that down into digestable pieces. Arguably, the most important line is the last one:
 
-<pre><code class=pp>build_lookup_tables()</code></pre>
+<pre class=nd><code class=pp>build_lookup_tables()</code></pre>
 
 <p>You will note that is a function call, but there&#8217;s no <code>if</code> statement around it. This is not an <code>if __name__ == '__main__'</code> block; it gets called <em>when the module is imported</em>. (It is important to understand that modules are only imported once, then cached. If you import an already-imported module, it does nothing. So this code will only get called the first time you import this module.)
 

regular-expressions.html

@@ -225,7 +225,7 @@ body{counter-reset:h1 4}
 </ol>
 <aside>(A|B) matches either pattern A or pattern B.</aside>
 <p>The expression for the ones place follows the same pattern. I&#8217;ll spare you the details and show you the end result.
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>pattern = '^M?M?M?(CM|CD|D?C?C?C?)(XC|XL|L?X?X?X?)(IX|IV|V?I?I?I?)$'</kbd>
 </pre><p>So what does that look like using this alternate <code>{n,m}</code> syntax?  This example shows the new syntax.
 <pre class=screen>

strings.html

@@ -180,7 +180,7 @@ def approximate_size(size, a_kilobyte_is_1024_bytes=True):
 
 <p>Just to blow your mind, here&#8217;s an example that combines all of the above:
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>import humansize</kbd>
 <samp class=p>>>> </samp><kbd class=pp>import sys</kbd>
 <samp class=p>>>> </samp><kbd class=pp>'1MB = 1000{0.modules[humansize].SUFFIXES[1000][0]}'.format(sys)</kbd>
@@ -201,7 +201,7 @@ def approximate_size(size, a_kilobyte_is_1024_bytes=True):
 
 <p>But wait! There&#8217;s more! Let&#8217;s take another look at that strange line of code from <code>humansize.py</code>:
 
-<pre><code class=pp>if size &lt; multiple:
+<pre class=nd><code class=pp>if size &lt; multiple:
     return '{0:.1f} {1}'.format(size, suffix)</code></pre>
 
 <p><code>{1}</code> is replaced with the second argument passed to the <code>format()</code> method, which is <var>suffix</var>. But what is <code>{0:.1f}</code>? It&#8217;s two things: <code>{0}</code>, which you recognize, and <code>:.1f</code>, which you don&#8217;t. The second half (including and after the colon) defines the <i>format specifier</i>, which further refines how the replaced variable should be formatted.
@@ -212,7 +212,7 @@ def approximate_size(size, a_kilobyte_is_1024_bytes=True):
 
 <p>Within a replacement field, a colon (<code>:</code>) marks the start of the format specifier. The format specifier &#8220;<code>.1</code>&#8221; means &#8220;round to the nearest tenth&#8221; (<i>i.e.</i> display only one digit after the decimal point). The format specifier &#8220;<code>f</code>&#8221; means &#8220;fixed-point number&#8221; (as opposed to exponential notation or some other decimal representation). Thus, given a <var>size</var> of <code>698.25</code> and <var>suffix</var> of <code>'GB'</code>, the formatted string would be <code>'698.3 GB'</code>, because <code>698.25</code> gets rounded to one decimal place, then the suffix is appended after the number.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>'{0:.1f} {1}'.format(698.25, 'GB')</kbd>
 <samp class=pp>'698.3 GB'</samp></pre>
 
@@ -414,11 +414,11 @@ TypeError: Can't convert 'bytes' object to str implicitly</samp>
 
 <p>If you would like to use a different encoding within your Python code, you can put an encoding declaration on the first line of each file. This declaration defines a <code>.py</code> file to be windows-1252:
 
-<pre><code class=pp># -*- coding: windows-1252 -*-</code></pre>
+<pre class=nd><code class=pp># -*- coding: windows-1252 -*-</code></pre>
 
 <p>Technically, the character encoding override can also be on the second line, if the first line is a <abbr>UNIX</abbr>-like hash-bang command.
 
-<pre><code class=pp>#!/usr/bin/python3
+<pre class=nd><code class=pp>#!/usr/bin/python3
 # -*- coding: windows-1252 -*-</code></pre>
 
 <p>For more information, consult <a href=http://www.python.org/dev/peps/pep-0263/><abbr>PEP</abbr> 263: Defining Python Source Code Encodings</a>.

table-of-contents.html

@@ -68,16 +68,14 @@ ul li ol{margin:0;padding:0 0 0 2.5em}
   <li><a href=native-datatypes.html#searchinglists>Searching for values in a list</a>
   <li><a href=native-datatypes.html#lists-in-a-boolean-context>Lists in a boolean context</a>
   </ol>
-<!--
 <li><a href=native-datatypes.html#sets>Sets</a>
-    <ol>
-    <li>Creating a new set
-    <li>Modifying a set
-    <li>Deleting items from a set
-    <li>Common operations on sets (union, intersection, and difference)
-    <li>Frozen sets
-    </ol>
--->
+  <ol>
+  <li><a href=native-datatypes.html#creating-a-set>Creating A Set</a>
+  <li><a href=native-datatypes.html#modifying-sets>Modifying A Set</a>
+  <li><a href=native-datatypes.html#removing-from-sets>Removing Items From A Set</a>
+  <li><a href=native-datatypes.html#common-set-operations>Common Set Operations</a>
+  <li><a href=native-datatypes.html#sets-in-a-boolean-context>Sets In A Boolean Context</a>
+  </ol>
 <li><a href=native-datatypes.html#dictionaries>Dictionaries</a>
   <ol>
   <li><a href=native-datatypes.html#creating-dictionaries>Creating a dictionary</a>
@@ -92,21 +90,23 @@ ul li ol{margin:0;padding:0 0 0 2.5em}
 <li><a href=native-datatypes.html#furtherreading>Further reading</a>
 </ol>
 <li id=strings><a href=strings.html>Strings</a>
-<ol>
-<li><a href=strings.html#divingin>Diving in</a>
-<li><a href=strings.html#one-ring-to-rule-them-all>Unicode</a>
   <ol>
-  <li>How strings are stored in memory
-  <li>Converting between different character encodings
-  <li><a href=strings.html#py-encoding>Specifying character encoding in <code>.py</code> files</a>
+  <li><a href=strings.html#boring-stuff>Some Boring Stuff You Need To Understand Before You Can Dive In</a>
+  <li><a href=strings.html#one-ring-to-rule-them-all>Unicode</a>
+  <li><a href=strings.html#divingin>Diving In</a>
+  <li><a href=strings.html#formatting-strings>Formatting Strings</a>
+    <ol>
+    <li><a href=strings.html#compound-field-names>Compound Field Names</a>
+    <li><a href=strings.html#format-specifiers>Format Specifiers</a>
+    </ol>
+  <li><a href=strings.html#common-string-methods>Other Common String Methods</a>
+    <ol>
+    <li><a href=strings.html#slicingstrings>Slicing A String</a>
+    </ol>
+  <li><a href=strings.html#byte-arrays>Strings vs. Bytes</a>
+  <li><a href=strings.html#py-encoding>Postscript: Character Encoding Of Python Source Code</a>
+  <li><a href=strings.html#furtherreading>Further Reading</a>
   </ol>
-<li>Strings in Python 3
-<li>Common string operations
-<li>Formatting strings
-<li><a href=strings.html#string-module>The <code>string</code> module</a>
-<li><a href=strings.html#byte-arrays>Strings vs. bytes</a>
-<li><a href=strings.html#furtherreading>Further reading</a>
-</ol>
 <li id=regular-expressions><a href=regular-expressions.html>Regular expressions</a>
   <ol>
   <li><a href=regular-expressions.html#divingin>Diving in</a>
@@ -361,22 +361,4 @@ ul li ol{margin:0;padding:0 0 0 2.5em}
   <li><a href=special-method-names.html#esoterica>Really Esoteric Stuff</a>
   </ol>
 </ul>
-<p>Orphans (not sure where these belong yet):
-<ul>
-  <li>Tuples
-  <li>List comprehensions
-  <li>Set comprehensions
-  <li>Dictionary comprehensions
-  <li>Views (several dictionary methods return them, they're dynamic, update when the dictionary changes, etc.)
-  <li>Function annotations
-  <li>PEP 8 style conventions
-  <li>Decorators
-    <ol>
-    <li><a href=http://docs.python.org/3.1/whatsnew/3.1.html><code>@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")</code></a>
-    </ol>
-  <li>Importing modules
-    <ol>
-    <li>...mention why from module import * is only allowed at module level
-    </ol>
-</ul>
 <p class=c>&copy; 2001&ndash;9 <a href=about.html>Mark Pilgrim</a>

unit-testing.html

@@ -195,13 +195,13 @@ def to_roman(n):
 <li>Here&#8217;s where the rich data structure of <var>roman_numeral_map</var> pays off, because you don&#8217;t need any special logic to handle the subtraction rule. To convert to Roman numerals, simply iterate through <var>roman_numeral_map</var> looking for the largest integer value less than or equal to the input. Once found, add the Roman numeral representation to the end of the output, subtract the corresponding integer value from the input, lather, rinse, repeat.
 </ol>
 <p>If you&#8217;re still not clear how the <code>to_roman()</code> function works, add a <code>print()</code> call to the end of the <code>while</code> loop:
-<pre><code class=pp>
+<pre class=nd><code class=pp>
 while n >= integer:
     result += numeral
     n -= integer
     print('subtracting {0} from input, adding {1} to output'.format(integer, numeral))</code></pre>
 <p>With the debug <code>print()</code> statements, the output looks like this:
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>import roman1</kbd>
 <samp class=p>>>> </samp><kbd class=pp>roman1.to_roman(1424)</kbd>
 <samp>subtracting 1000 from input, adding M to output
@@ -211,7 +211,7 @@ subtracting 10 from input, adding X to output
 subtracting 4 from input, adding IV to output
 'MCDXXIV'</samp></pre>
 <p>So the <code>to_roman()</code> function appears to work, at least in this manual spot check. But will it pass the test case you wrote?
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>python3 romantest1.py -v</kbd>
 <samp>to_roman should give known result with known input ... ok
 
@@ -343,7 +343,7 @@ OK</samp></pre>
 
 <p>Along with testing numbers that are too large, you need to test numbers that are too small. As <a href=#divingin>we noted in our functional requirements</a>, Roman numerals cannot express <code>0</code> or negative numbers.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>import roman2</kbd>
 <samp class=p>>>> </samp><kbd class=pp>roman2.to_roman(0)</kbd>
 <samp class=pp>''</samp>
@@ -373,7 +373,7 @@ OK</samp></pre>
 
 <p>Now check that the tests fail:
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>python3 romantest3.py -v</kbd>
 <samp>to_roman should give known result with known input ... ok
 to_roman should fail with negative input ... FAIL
@@ -422,7 +422,7 @@ FAILED (failures=2)</samp></pre>
 
 <p>I could show you a whole series of unrelated examples to show that the multiple-comparisons-at-once shortcut works, but instead I&#8217;ll just run the unit tests and prove it.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>python3 romantest3.py -v</kbd>
 <samp>to_roman should give known result with known input ... ok
 to_roman should fail with negative input ... ok
@@ -453,13 +453,13 @@ OK</samp></pre>
 
 <p>Testing for non-integers is not difficult. First, define a <code>NonIntegerError</code> exception.
 
-<pre><code class=pp># roman4.py
+<pre class=nd><code class=pp># roman4.py
 class OutOfRangeError(ValueError): pass
 <mark>class NotIntegerError(ValueError): pass</mark></code></pre>
 
 <p>Next, write a test case that checks for the <code>NonIntegerError</code> exception.
 
-<pre><code class=pp>class ToRomanBadInput(unittest.TestCase):
+<pre class=nd><code class=pp>class ToRomanBadInput(unittest.TestCase):
     .
     .
     .
@@ -469,7 +469,7 @@ class OutOfRangeError(ValueError): pass
 
 <p>Now check that the test fails properly.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>python3 romantest4.py -v</kbd>
 <samp>to_roman should give known result with known input ... ok
 to_roman should fail with negative input ... ok
@@ -512,7 +512,7 @@ FAILED (failures=1)</samp></pre>
 
 <p>Finally, check that the code does indeed make the test pass.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>you@localhost:~$ </samp><kbd>python3 romantest4.py -v</kbd>
 <samp>to_roman should give known result with known input ... ok
 to_roman should fail with negative input ... ok

xml.html

@@ -443,7 +443,7 @@ StopIteration</samp></pre>
 
 <p>For large <abbr>XML</abbr> documents, <code>lxml</code> is significantly faster than the built-in ElementTree libary. If you&#8217;re only using the ElementTree <abbr>API</abbr> and want to use the fastest available implementation, you can try to import <code>lxml</code> and fall back to the built-in ElementTree.
 
-<pre><code class=pp>try:
+<pre class=nd><code class=pp>try:
     from lxml import etree
 except ImportError:
     import xml.etree.ElementTree as etree</code></pre>
@@ -582,7 +582,7 @@ except ImportError:
 
 <p>That&#8217;s an error, because the <code>&amp;hellip;</code> entity is not defined in <abbr>XML</abbr>. (It is defined in <abbr>HTML</abbr>.) If you try to parse this broken feed with the default settings, <code>lxml</code> will choke on the undefined entity.
 
-<pre class=screen>
+<pre class='nd screen'>
 <samp class=p>>>> </samp><kbd class=pp>import lxml.etree</kbd>
 <samp class=p>>>> </samp><kbd class=pp>tree = lxml.etree.parse('examples/feed-broken.xml')</kbd>
 <samp class=traceback>Traceback (most recent call last):

your-first-python-program.html

@@ -137,7 +137,7 @@ SyntaxError: non-keyword arg after keyword arg</samp></pre>
 <p>I won&#8217;t bore you with a long finger-wagging speech about the importance of documenting your code. Just know that code is written once but read many times, and the most important audience for your code is yourself, six months after writing it (<i>i.e.</i> after you&#8217;ve forgotten everything but need to fix something). Python makes it easy to write readable code, so take advantage of it. You&#8217;ll thank me in six months.
 <h3 id=docstrings>Documentation Strings</h3>
 <p>You can document a Python function by giving it a documentation string (<code>docstring</code> for short). In this program, the <code>approximate_size()</code> function has a <code>docstring</code>:
-<pre><code class=pp>def approximate_size(size, a_kilobyte_is_1024_bytes=True):
+<pre class=nd><code class=pp>def approximate_size(size, a_kilobyte_is_1024_bytes=True):
     '''Convert a file size to human-readable form.
 
     Keyword arguments: