kennethreitz/dive-into-python3
1
0
Fork 0
mirror of https://github.com/kennethreitz/dive-into-python3.git synced 2026-06-05 15:00:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

line ending fiddling, some build fiddling on new laptop

Browse Source
This commit is contained in:
Mark Pilgrim
2010-04-26 22:04:13 -04:00
parent 66e867b82f
commit 72ba72a07b
17 changed files with 4805 additions and 4811 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.htaccess
+3 -3
View File
@@ -1,3 +1,3 @@
FileETag MTime Size
SetEnv dont-vary
FileETag MTime Size
SetEnv dont-vary
advanced-iterators.html
+647 -647
View File
File diff suppressed because it is too large Load Diff
colophon.html
+87 -87
View File
@@ -1,87 +1,87 @@
<!DOCTYPE html>
<meta charset=utf-8>
<meta name=robots content=noindex>
<title>Colophon - Dive Into Python 3</title>
<link rel=stylesheet href=dip3.css>
<style>
h1:before,h2:before{content:''}
.ss{float:right;margin:0 0 1.75em 1.75em}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html>Dive Into Python 3</a> <span class=u>&#8227;</span>
<h1>Colophon</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> <i lang=fr>Je n&#8217;ai fait celle-ci plus longue que parce que je n&#8217;ai pas eu le loisir de la faire plus courte.</i><br>(I would have written a shorter letter, but I did not have the time.) <span class=u>&#x275E;</span><br>&mdash; <a href=http://en.wikiquote.org/wiki/Blaise_Pascal>Blaise Pascal</a>
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>This book, like all books, was a labor of love. Oh sure, I got paid the medium-sized bucks for it, but nobody writes technical books for the money. And since this book is available on the web as well as on paper, I spent a lot of time fiddling with webby stuff when I should have been writing.
<p class='ss nm'><img src=i/openclipart.org_media_files_johnny_automatic_5261.png width=314 height=273 alt='[typewriter]'>
<p>The online edition loads as efficiently as possible. Efficiency never happens by accident; I spent many hours making it so. Perhaps too many hours. Yes, almost certainly too many hours. Never underestimate the depths to which a procrastinating writer will sink.
<p>I won&#8217;t bore you with all the details. Wait, yes&nbsp;&mdash;&nbsp;I will bore you with all the details. But here&#8217;s the short version.
<ol>
<li>HTML is minimized, then served <a href=http://httpd.apache.org/docs/trunk/mod/mod_deflate.html>compressed</a>.
<li>Scripts and stylesheets are minimized by <a href=http://developer.yahoo.com/yui/compressor/>YUI Compressor</a> (and also served compressed).
<li>Scripts are combined to reduce HTTP requests.
<li>Stylesheets are combined and inlined to reduce HTTP requests.
<li>Unused CSS selectors and properties are <a href=http://hg.diveintopython3.org/file/default/util/lesscss.py>removed on a page-by-page basis</a> with a little help from <a href=http://pyquery.org/>pyquery</a>.
<li>HTTP caching and other server-side options are optimized based on advice from <a href=http://developer.yahoo.com/yslow/>YSlow</a> and <a href=http://code.google.com/speed/page-speed/>Page Speed</a>.
<li>Pages use <a href=http://www.alanwood.net/unicode/unicode_samples.html>Unicode characters</a> in place of images wherever possible.
<li>Images are optimized with <a href=http://optipng.sourceforge.net/>OptiPNG</a>.
<li>The entire book was <a href=http://diveintomark.org/archives/2009/03/27/dive-into-history-2009-edition>lovingly hand-authored in HTML 5</a> to avoid markup cruft.
</ol>
<p class=a>&#x2042;
<h2 id=typography>Typography</h2>
<p>vertical rhythm, best available ampersand, curly quotes/apostrophes, other stuff from webtypography.net
<p class=a>&#x2042;
<h2 id=graphics>Graphics</h2>
<p>Unicode, callouts, font-family issues on Windows
<p class=a>&#x2042;
<h2 id=performance>Performance</h2>
<p>"Dive Into History 2009 edition", minimizing CSS + JS + HTML, inline CSS, optimizing images
<p class=a>&#x2042;
<h2 id=fun>Fun stuff</h2>
<p>Quotes, constrained writing(?), PapayaWhip
<p class=a>&#x2042;
<h2 id=furtherreading>Further Reading</h2>
<ul>
<li><a href=http://webtypography.net/toc/>The Elements of Typographic Style Applied to the Web</a>
<li><a href=http://www.alistapart.com/articles/settingtypeontheweb>Setting Type on the Web to a Baseline Grid</a>
<li><a href=http://24ways.org/2006/compose-to-a-vertical-rhythm>Compose to a Vertical Rhythm</a>
<li><a href=http://simplebits.com/notebook/2008/08/14/ampersands.html>Use the Best Available Ampersand</a>
<li><a href=http://alanwood.net/unicode/>Unicode Support in HTML, Fonts, and Web Browsers</a>
<li><a href=http://developer.yahoo.com/yslow/>YSlow</a> for <a href=http://getfirebug.com/>Firebug</a>
<li><a href=http://developer.yahoo.com/performance/rules.html>Best Practices for Speeding Up Your Web Site</a>
<li><a href=http://stevesouders.com/hpws/rules.php>14 Rules for Faster-Loading Web Sites</a>
<li><a href=http://developer.yahoo.com/yui/compressor/>YUI Compressor</a>
<li><a href=http://code.google.com/speed/page-speed/>Google Page Speed</a>
<li><a href=http://code.google.com/speed/page-speed/docs/using.html>Using Google Page Speed</a>
<li><a href=http://optipng.sourceforge.net/>OptiPNG</a>
</ul>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/dip3.js></script>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<!DOCTYPE html>
<meta charset=utf-8>
<meta name=robots content=noindex>
<title>Colophon - Dive Into Python 3</title>
<link rel=stylesheet href=dip3.css>
<style>
h1:before,h2:before{content:''}
.ss{float:right;margin:0 0 1.75em 1.75em}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html>Dive Into Python 3</a> <span class=u>&#8227;</span>
<h1>Colophon</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> <i lang=fr>Je n&#8217;ai fait celle-ci plus longue que parce que je n&#8217;ai pas eu le loisir de la faire plus courte.</i><br>(I would have written a shorter letter, but I did not have the time.) <span class=u>&#x275E;</span><br>&mdash; <a href=http://en.wikiquote.org/wiki/Blaise_Pascal>Blaise Pascal</a>
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>This book, like all books, was a labor of love. Oh sure, I got paid the medium-sized bucks for it, but nobody writes technical books for the money. And since this book is available on the web as well as on paper, I spent a lot of time fiddling with webby stuff when I should have been writing.
<p class='ss nm'><img src=i/openclipart.org_media_files_johnny_automatic_5261.png width=314 height=273 alt='[typewriter]'>
<p>The online edition loads as efficiently as possible. Efficiency never happens by accident; I spent many hours making it so. Perhaps too many hours. Yes, almost certainly too many hours. Never underestimate the depths to which a procrastinating writer will sink.
<p>I won&#8217;t bore you with all the details. Wait, yes&nbsp;&mdash;&nbsp;I will bore you with all the details. But here&#8217;s the short version.
<ol>
<li>HTML is minimized, then served <a href=http://httpd.apache.org/docs/trunk/mod/mod_deflate.html>compressed</a>.
<li>Scripts and stylesheets are minimized by <a href=http://developer.yahoo.com/yui/compressor/>YUI Compressor</a> (and also served compressed).
<li>Scripts are combined to reduce HTTP requests.
<li>Stylesheets are combined and inlined to reduce HTTP requests.
<li>Unused CSS selectors and properties are <a href=http://hg.diveintopython3.org/file/default/util/lesscss.py>removed on a page-by-page basis</a> with a little help from <a href=http://pyquery.org/>pyquery</a>.
<li>HTTP caching and other server-side options are optimized based on advice from <a href=http://developer.yahoo.com/yslow/>YSlow</a> and <a href=http://code.google.com/speed/page-speed/>Page Speed</a>.
<li>Pages use <a href=http://www.alanwood.net/unicode/unicode_samples.html>Unicode characters</a> in place of images wherever possible.
<li>Images are optimized with <a href=http://optipng.sourceforge.net/>OptiPNG</a>.
<li>The entire book was <a href=http://diveintomark.org/archives/2009/03/27/dive-into-history-2009-edition>lovingly hand-authored in HTML 5</a> to avoid markup cruft.
</ol>
<p class=a>&#x2042;
<h2 id=typography>Typography</h2>
<p>vertical rhythm, best available ampersand, curly quotes/apostrophes, other stuff from webtypography.net
<p class=a>&#x2042;
<h2 id=graphics>Graphics</h2>
<p>Unicode, callouts, font-family issues on Windows
<p class=a>&#x2042;
<h2 id=performance>Performance</h2>
<p>"Dive Into History 2009 edition", minimizing CSS + JS + HTML, inline CSS, optimizing images
<p class=a>&#x2042;
<h2 id=fun>Fun stuff</h2>
<p>Quotes, constrained writing(?), PapayaWhip
<p class=a>&#x2042;
<h2 id=furtherreading>Further Reading</h2>
<ul>
<li><a href=http://webtypography.net/toc/>The Elements of Typographic Style Applied to the Web</a>
<li><a href=http://www.alistapart.com/articles/settingtypeontheweb>Setting Type on the Web to a Baseline Grid</a>
<li><a href=http://24ways.org/2006/compose-to-a-vertical-rhythm>Compose to a Vertical Rhythm</a>
<li><a href=http://simplebits.com/notebook/2008/08/14/ampersands.html>Use the Best Available Ampersand</a>
<li><a href=http://alanwood.net/unicode/>Unicode Support in HTML, Fonts, and Web Browsers</a>
<li><a href=http://developer.yahoo.com/yslow/>YSlow</a> for <a href=http://getfirebug.com/>Firebug</a>
<li><a href=http://developer.yahoo.com/performance/rules.html>Best Practices for Speeding Up Your Web Site</a>
<li><a href=http://stevesouders.com/hpws/rules.php>14 Rules for Faster-Loading Web Sites</a>
<li><a href=http://developer.yahoo.com/yui/compressor/>YUI Compressor</a>
<li><a href=http://code.google.com/speed/page-speed/>Google Page Speed</a>
<li><a href=http://code.google.com/speed/page-speed/docs/using.html>Using Google Page Speed</a>
<li><a href=http://optipng.sourceforge.net/>OptiPNG</a>
</ul>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/dip3.js></script>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
files.html
+607 -607
View File
File diff suppressed because it is too large Load Diff
generators.html
+418 -418
View File
@@ -1,418 +1,418 @@
<!DOCTYPE html>
<meta charset=utf-8>
<title>Closures &amp; Generators - Dive Into Python 3</title>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<link rel=stylesheet href=dip3.css>
<style>
body{counter-reset:h1 6}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#generators>Dive Into Python 3</a> <span class=u>&#8227;</span>
<p id=level>Difficulty level: <span class=u title=intermediate>&#x2666;&#x2666;&#x2666;&#x2662;&#x2662;</span>
<h1>Closures <i class=baa>&amp;</i> Generators</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> My spelling is Wobbly. It&#8217;s good spelling but it Wobbles, and the letters get in the wrong places. <span class=u>&#x275E;</span><br>&mdash; Winnie-the-Pooh
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>Having grown up the son of a librarian and an English major, I have always been fascinated by languages. Not programming languages. Well yes, programming languages, but also natural languages. Take English. English is a schizophrenic language that borrows words from German, French, Spanish, and Latin (to name a few). Actually, &#8220;borrows&#8221; is the wrong word; &#8220;pillages&#8221; is more like it. Or perhaps &#8220;assimilates&#8221;&nbsp;&mdash;&nbsp;like the Borg. Yes, I like that.
<p class=c><code>We are the Borg. Your linguistic and etymological distinctiveness will be added to our own. Resistance is futile.</code>
<p>In this chapter, you&#8217;re going to learn about plural nouns. Also, functions that return other functions, advanced regular expressions, and generators. But first, let&#8217;s talk about how to make plural nouns. (If you haven&#8217;t read <a href=regular-expressions.html>the chapter on regular expressions</a>, now would be a good time. This chapter assumes you understand the basics of regular expressions, and it quickly descends into more advanced uses.)
<p>If you grew up in an English-speaking country or learned English in a formal school setting, you&#8217;re probably familiar with the basic rules:
<ul>
<li>If a word ends in S, X, or Z, add ES. <i>Bass</i> becomes <i>basses</i>, <i>fax</i> becomes <i>faxes</i>, and <i>waltz</i> becomes <i>waltzes</i>.
<li>If a word ends in a noisy H, add ES; if it ends in a silent H, just add S. What&#8217;s a noisy H? One that gets combined with other letters to make a sound that you can hear. So <i>coach</i> becomes <i>coaches</i> and <i>rash</i> becomes <i>rashes</i>, because you can hear the CH and SH sounds when you say them. But <i>cheetah</i> becomes <i>cheetahs</i>, because the H is silent.
<li>If a word ends in Y that sounds like I, change the Y to IES; if the Y is combined with a vowel to sound like something else, just add S. So <i>vacancy</i> becomes <i>vacancies</i>, but <i>day</i> becomes <i>days</i>.
<li>If all else fails, just add S and hope for the best.
</ul>
<p>(I know, there are a lot of exceptions. <i>Man</i> becomes <i>men</i> and <i>woman</i> becomes <i>women</i>, but <i>human</i> becomes <i>humans</i>. <i>Mouse</i> becomes <i>mice</i> and <i>louse</i> becomes <i>lice</i>, but <i>house</i> becomes <i>houses</i>. <i>Knife</i> becomes <i>knives</i> and <i>wife</i> becomes <i>wives</i>, but <i>lowlife</i> becomes <i>lowlifes</i>. And don&#8217;t even get me started on words that are their own plural, like <i>sheep</i>, <i>deer</i>, and <i>haiku</i>.)
<p>Other languages, of course, are completely different.
<p>Let&#8217;s design a Python library that automatically pluralizes English nouns. We&#8217;ll start with just these four rules, but keep in mind that you&#8217;ll inevitably need to add more.
<p class=a>&#x2042;
<h2 id=i-know>I Know, Let&#8217;s Use Regular Expressions!</h2>
<p>So you&#8217;re looking at words, which, at least in English, means you&#8217;re looking at strings of characters. You have rules that say you need to find different combinations of characters, then do different things to them. This sounds like a job for regular expressions!
<p class=d>[<a href=examples/plural1.py>download <code>plural1.py</code></a>]
<pre class=pp><code>import re
def plural(noun):
<a> if re.search('[sxz]$', noun): <span class=u>&#x2460;</span></a>
<a> return re.sub('$', 'es', noun) <span class=u>&#x2461;</span></a>
elif re.search('[^aeioudgkprt]h$', noun):
return re.sub('$', 'es', noun)
elif re.search('[^aeiou]y$', noun):
return re.sub('y$', 'ies', noun)
else:
return noun + 's'</code></pre>
<ol>
<li>This is a regular expression, but it uses a syntax you didn&#8217;t see in <a href=regular-expressions.html><i>Regular Expressions</i></a>. The square brackets mean &#8220;match exactly one of these characters.&#8221; So <code>[sxz]</code> means &#8220;<code>s</code>, or <code>x</code>, or <code>z</code>&#8221;, but only one of them. The <code>$</code> should be familiar; it matches the end of string. Combined, this regular expression tests whether <var>noun</var> ends with <code>s</code>, <code>x</code>, or <code>z</code>.
<li>This <code>re.sub()</code> function performs regular expression-based string substitutions.
</ol>
<p>Let&#8217;s look at regular expression substitutions in more detail.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import re</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[abc]', 'Mark')</kbd> <span class=u>&#x2460;</span></a>
&lt;_sre.SRE_Match object at 0x001C1FA8>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('[abc]', 'o', 'Mark')</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>'Mork'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('[abc]', 'o', 'rock')</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>'rook'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('[abc]', 'o', 'caps')</kbd> <span class=u>&#x2463;</span></a>
<samp class=pp>'oops'</samp></pre>
<ol>
<li>Does the string <code>Mark</code> contain <code>a</code>, <code>b</code>, or <code>c</code>? Yes, it contains <code>a</code>.
<li>OK, now find <code>a</code>, <code>b</code>, or <code>c</code>, and replace it with <code>o</code>. <code>Mark</code> becomes <code>Mork</code>.
<li>The same function turns <code>rock</code> into <code>rook</code>.
<li>You might think this would turn <code>caps</code> into <code>oaps</code>, but it doesn&#8217;t. <code>re.sub</code> replaces <em>all</em> of the matches, not just the first one. So this regular expression turns <code>caps</code> into <code>oops</code>, because both the <code>c</code> and the <code>a</code> get turned into <code>o</code>.
</ol>
<p>And now, back to the <code>plural()</code> function&hellip;
<pre class=pp><code>def plural(noun):
if re.search('[sxz]$', noun):
<a> return re.sub('$', 'es', noun) <span class=u>&#x2460;</span></a>
<a> elif re.search('[^aeioudgkprt]h$', noun): <span class=u>&#x2461;</span></a>
return re.sub('$', 'es', noun)
<a> elif re.search('[^aeiou]y$', noun): <span class=u>&#x2462;</span></a>
return re.sub('y$', 'ies', noun)
else:
return noun + 's'</code></pre>
<ol>
<li>Here, you&#8217;re replacing the end of the string (matched by <code>$</code>) with the string <code>es</code>. In other words, adding <code>es</code> to the string. You could accomplish the same thing with string concatenation, for example <code>noun + 'es'</code>, but I chose to use regular expressions for each rule, for reasons that will become clear later in the chapter.
<li>Look closely, this is another new variation. The <code>^</code> as the first character inside the square brackets means something special: negation. <code>[^abc]</code> means &#8220;any single character <em>except</em> <code>a</code>, <code>b</code>, or <code>c</code>&#8221;. So <code>[^aeioudgkprt]</code> means any character except <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, <code>u</code>, <code>d</code>, <code>g</code>, <code>k</code>, <code>p</code>, <code>r</code>, or <code>t</code>. Then that character needs to be followed by <code>h</code>, followed by end of string. You&#8217;re looking for words that end in H where the H can be heard.
<li>Same pattern here: match words that end in Y, where the character before the Y is <em>not</em> <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, or <code>u</code>. You&#8217;re looking for words that end in Y that sounds like I.
</ol>
<p>Let&#8217;s look at negation regular expressions in more detail.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import re</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'vacancy')</kbd> <span class=u>&#x2460;</span></a>
&lt;_sre.SRE_Match object at 0x001C1FA8>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'boy')</kbd> <span class=u>&#x2461;</span></a>
<samp class=p>>>> </samp>
<samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'day')</kbd>
<samp class=p>>>> </samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'pita')</kbd> <span class=u>&#x2462;</span></a>
<samp class=p>>>> </samp></pre>
<ol>
<li><code>vacancy</code> matches this regular expression, because it ends in <code>cy</code>, and <code>c</code> is not <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, or <code>u</code>.
<li><code>boy</code> does not match, because it ends in <code>oy</code>, and you specifically said that the character before the <code>y</code> could not be <code>o</code>. <code>day</code> does not match, because it ends in <code>ay</code>.
<li><code>pita</code> does not match, because it does not end in <code>y</code>.
</ol>
<pre class=screen>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('y$', 'ies', 'vacancy')</kbd> <span class=u>&#x2460;</span></a>
<samp class=pp>'vacancies'</samp>
<samp class=p>>>> </samp><kbd class=pp>re.sub('y$', 'ies', 'agency')</kbd>
<samp class=pp>'agencies'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('([^aeiou])y$', r'\1ies', 'vacancy')</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>'vacancies'</samp></pre>
<ol>
<li>This regular expression turns <code>vacancy</code> into <code>vacancies</code> and <code>agency</code> into <code>agencies</code>, which is what you wanted. Note that it would also turn <code>boy</code> into <code>boies</code>, but that will never happen in the function because you did that <code>re.search</code> first to find out whether you should do this <code>re.sub</code>.
<li>Just in passing, I want to point out that it is possible to combine these two regular expressions (one to find out if the rule applies, and another to actually apply it) into a single regular expression. Here&#8217;s what that would look like. Most of it should look familiar: you&#8217;re using a remembered group, which you learned in <a href=regular-expressions.html#phonenumbers>Case study: Parsing Phone Numbers</a>. The group is used to remember the character before the letter <code>y</code>. Then in the substitution string, you use a new syntax, <code>\1</code>, which means &#8220;hey, that first group you remembered? put it right here.&#8221; In this case, you remember the <code>c</code> before the <code>y</code>; when you do the substitution, you substitute <code>c</code> in place of <code>c</code>, and <code>ies</code> in place of <code>y</code>. (If you have more than one remembered group, you can use <code>\2</code> and <code>\3</code> and so on.)
</ol>
<p>Regular expression substitutions are extremely powerful, and the <code>\1</code> syntax makes them even more powerful. But combining the entire operation into one regular expression is also much harder to read, and it doesn&#8217;t directly map to the way you first described the pluralizing rules. You originally laid out rules like &#8220;if the word ends in S, X, or Z, then add ES&#8221;. If you look at this function, you have two lines of code that say &#8220;if the word ends in S, X, or Z, then add ES&#8221;. It doesn&#8217;t get much more direct than that.
<p class=a>&#x2042;
<h2 id=a-list-of-functions>A List Of Functions</h2>
<p>Now you&#8217;re going to add a level of abstraction. You started by defining a list of rules: if this, do that, otherwise go to the next rule. Let&#8217;s temporarily complicate part of the program so you can simplify another part.
<p class=d>[<a href=examples/plural2.py>download <code>plural2.py</code></a>]
<pre class=pp><code>import re
def match_sxz(noun):
return re.search('[sxz]$', noun)
def apply_sxz(noun):
return re.sub('$', 'es', noun)
def match_h(noun):
return re.search('[^aeioudgkprt]h$', noun)
def apply_h(noun):
return re.sub('$', 'es', noun)
<a>def match_y(noun): <span class=u>&#x2460;</span></a>
return re.search('[^aeiou]y$', noun)
<a>def apply_y(noun): <span class=u>&#x2461;</span></a>
return re.sub('y$', 'ies', noun)
def match_default(noun):
return True
def apply_default(noun):
return noun + 's'
<a>rules = ((match_sxz, apply_sxz), <span class=u>&#x2462;</span></a>
(match_h, apply_h),
(match_y, apply_y),
(match_default, apply_default)
)
def plural(noun):
<a> for matches_rule, apply_rule in rules: <span class=u>&#x2463;</span></a>
if matches_rule(noun):
return apply_rule(noun)</code></pre>
<ol>
<li>Now, each match rule is its own function which returns the results of calling the <code>re.search()</code> function.
<li>Each apply rule is also its own function which calls the <code>re.sub()</code> function to apply the appropriate pluralization rule.
<li>Instead of having one function (<code>plural()</code>) with multiple rules, you have the <code>rules</code> data structure, which is a sequence of pairs of functions.
<li>Since the rules have been broken out into a separate data structure, the new <code>plural()</code> function can be reduced to a few lines of code. Using a <code>for</code> loop, you can pull out the match and apply rules two at a time (one match, one apply) from the <var>rules</var> structure. On the first iteration of the <code>for</code> loop, <var>matches_rule</var> will get <code>match_sxz</code>, and <var>apply_rule</var> will get <code>apply_sxz</code>. On the second iteration (assuming you get that far), <var>matches_rule</var> will be assigned <code>match_h</code>, and <var>apply_rule</var> will be assigned <code>apply_h</code>. The function is guaranteed to return something eventually, because the final match rule (<code>match_default</code>) simply returns <code>True</code>, meaning the corresponding apply rule (<code>apply_default</code>) will always be applied.
</ol>
<aside>The &#8220;rules&#8221; variable is a sequence of pairs of functions.</aside>
<p>The reason this technique works is that <a href=your-first-python-program.html#everythingisanobject>everything in Python is an object</a>, including functions. The <var>rules</var> data structure contains functions&nbsp;&mdash;&nbsp;not names of functions, but actual function objects. When they get assigned in the <code>for</code> loop, then <var>matches_rule</var> and <var>apply_rule</var> are actual functions that you can call. On the first iteration of the <code>for</code> loop, this is equivalent to calling <code>matches_sxz(noun)</code>, and if it returns a match, calling <code>apply_sxz(noun)</code>.
<p>If this additional level of abstraction is confusing, try unrolling the function to see the equivalence. The entire <code>for</code> loop is equivalent to the following:
<pre class='nd pp'><code>
def plural(noun):
if match_sxz(noun):
return apply_sxz(noun)
if match_h(noun):
return apply_h(noun)
if match_y(noun):
return apply_y(noun)
if match_default(noun):
return apply_default(noun)</code></pre>
<p>The benefit here is that the <code>plural()</code> function is now simplified. It takes a sequence of rules, defined elsewhere, and iterates through them in a generic fashion.
<ol>
<li>Get a match rule
<li>Does it match? Then call the apply rule and return the result.
<li>No match? Go to step 1.
</ol>
<p>The rules could be defined anywhere, in any way. The <code>plural()</code> function doesn&#8217;t care.
<p>Now, was adding this level of abstraction worth it? Well, not yet. Let&#8217;s consider what it would take to add a new rule to the function. In the first example, it would require adding an <code>if</code> statement to the <code>plural()</code> function. In this second example, it would require adding two functions, <code>match_foo()</code> and <code>apply_foo()</code>, and then updating the <var>rules</var> sequence to specify where in the order the new match and apply functions should be called relative to the other rules.
<p>But this is really just a stepping stone to the next section. Let&#8217;s move on&hellip;
<p class=a>&#x2042;
<h2 id=a-list-of-patterns>A List Of Patterns</h2>
<p>Defining separate named functions for each match and apply rule isn&#8217;t really necessary. You never call them directly; you add them to the <var>rules</var> sequence and call them through there. Furthermore, each function follows one of two patterns. All the match functions call <code>re.search()</code>, and all the apply functions call <code>re.sub()</code>. Let&#8217;s factor out the patterns so that defining new rules can be easier.
<p class=d>[<a href=examples/plural3.py>download <code>plural3.py</code></a>]
<pre class=pp><code>import re
def build_match_and_apply_functions(pattern, search, replace):
<a> def matches_rule(word): <span class=u>&#x2460;</span></a>
return re.search(pattern, word)
<a> def apply_rule(word): <span class=u>&#x2461;</span></a>
return re.sub(search, replace, word)
<a> return (matches_rule, apply_rule) <span class=u>&#x2462;</span></a></code></pre>
<ol>
<li><code>build_match_and_apply_functions()</code> is a function that builds other functions dynamically. It takes <var>pattern</var>, <var>search</var> and <var>replace</var>, then defines a <code>matches_rule()</code> function which calls <code>re.search()</code> with the <var>pattern</var> that was passed to the <code>build_match_and_apply_functions()</code> function, and the <var>word</var> that was passed to the <code>matches_rule()</code> function you&#8217;re building. Whoa.
<li>Building the apply function works the same way. The apply function is a function that takes one parameter, and calls <code>re.sub()</code> with the <var>search</var> and <var>replace</var> parameters that were passed to the <code>build_match_and_apply_functions()</code> function, and the <var>word</var> that was passed to the <code>apply_rule()</code> function you&#8217;re building. This technique of using the values of outside parameters within a dynamic function is called <em>closures</em>. You&#8217;re essentially defining constants within the apply function you&#8217;re building: it takes one parameter (<var>word</var>), but it then acts on that plus two other values (<var>search</var> and <var>replace</var>) which were set when you defined the apply function.
<li>Finally, the <code>build_match_and_apply_functions()</code> function returns a tuple of two values: the two functions you just created. The constants you defined within those functions (<var>pattern</var> within the <code>matches_rule()</code> function, and <var>search</var> and <var>replace</var> within the <code>apply_rule()</code> function) stay with those functions, even after you return from <code>build_match_and_apply_functions()</code>. That&#8217;s insanely cool.
</ol>
<p>If this is incredibly confusing (and it should be, this is weird stuff), it may become clearer when you see how to use it.
<pre class=pp><code><a>patterns = \ <span class=u>&#x2460;</span></a>
(
('[sxz]$', '$', 'es'),
('[^aeioudgkprt]h$', '$', 'es'),
('(qu|[^aeiou])y$', 'y$', 'ies'),
<a> ('$', '$', 's') <span class=u>&#x2461;</span></a>
)
<a>rules = [build_match_and_apply_functions(pattern, search, replace) <span class=u>&#x2462;</span></a>
for (pattern, search, replace) in patterns]</code></pre>
<ol>
<li>Our pluralization &#8220;rules&#8221; are now defined as a tuple of tuples of <em>strings</em> (not functions). The first string in each group is the regular expression pattern that you would use in <code>re.search()</code> to see if this rule matches. The second and third strings in each group are the search and replace expressions you would use in <code>re.sub()</code> to actually apply the rule to turn a noun into its plural.
<li>There&#8217;s a slight change here, in the fallback rule. In the previous example, the <code>match_default()</code> function simply returned <code>True</code>, meaning that if none of the more specific rules matched, the code would simply add an <code>s</code> to the end of the given word. This example does something functionally equivalent. The final regular expression asks whether the word has an end (<code>$</code> matches the end of a string). Of course, every string has an end, even an empty string, so this expression always matches. Thus, it serves the same purpose as the <code>match_default()</code> function that always returned <code>True</code>: it ensures that if no more specific rule matches, the code adds an <code>s</code> to the end of the given word.
<li>This line is magic. It takes the sequence of strings in <var>patterns</var> and turns them into a sequence of functions. How? By &#8220;mapping&#8221; the strings to the <code>build_match_and_apply_functions()</code> function. That is, it takes each triplet of strings and calls the <code>build_match_and_apply_functions()</code> function with those three strings as arguments. The <code>build_match_and_apply_functions()</code> function returns a tuple of two functions. This means that <var>rules</var> ends up being functionally equivalent to the previous example: a list of tuples, where each tuple is a pair of functions. The first function is the match function that calls <code>re.search()</code>, and the second function is the apply function that calls <code>re.sub()</code>.
</ol>
<p>Rounding out this version of the script is the main entry point, the <code>plural()</code> function.
<pre class=pp><code>def plural(noun):
<a> for matches_rule, apply_rule in rules: <span class=u>&#x2460;</span></a>
if matches_rule(noun):
return apply_rule(noun)</code></pre>
<ol>
<li>Since the <var>rules</var> list is the same as the previous example (really, it is), it should come as no surprise that the <code>plural()</code> function hasn&#8217;t changed at all. It&#8217;s completely generic; it takes a list of rule functions and calls them in order. It doesn&#8217;t care how the rules are defined. In the previous example, they were defined as separate named functions. Now they are built dynamically by mapping the output of the <code>build_match_and_apply_functions()</code> function onto a list of raw strings. It doesn&#8217;t matter; the <code>plural()</code> function still works the same way.
</ol>
<p class=a>&#x2042;
<h2 id=a-file-of-patterns>A File Of Patterns</h2>
<p>You&#8217;ve factored out all the duplicate code and added enough abstractions so that the pluralization rules are defined in a list of strings. The next logical step is to take these strings and put them in a separate file, where they can be maintained separately from the code that uses them.
<p>First, let&#8217;s create a text file that contains the rules you want. No fancy data structures, just whitespace-delimited strings in three columns. Let&#8217;s call it <code>plural4-rules.txt</code>.
<p class=d>[<a href=examples/plural4-rules.txt>download <code>plural4-rules.txt</code></a>]
<pre class='nd pp'><code>[sxz]$ $ es
[^aeioudgkprt]h$ $ es
[^aeiou]y$ y$ ies
$ $ s</code></pre>
<p>Now let&#8217;s see how you can use this rules file.
<p class=d>[<a href=examples/plural4.py>download <code>plural4.py</code></a>]
<pre class=pp><code>import re
<a>def build_match_and_apply_functions(pattern, search, replace): <span class=u>&#x2460;</span></a>
def matches_rule(word):
return re.search(pattern, word)
def apply_rule(word):
return re.sub(search, replace, word)
return (matches_rule, apply_rule)
rules = []
<a>with open('plural4-rules.txt', encoding='utf-8') as pattern_file: <span class=u>&#x2461;</span></a>
<a> for line in pattern_file: <span class=u>&#x2462;</span></a>
<a> pattern, search, replace = line.split(None, 3) <span class=u>&#x2463;</span></a>
<a> rules.append(build_match_and_apply_functions( <span class=u>&#x2464;</span></a>
pattern, search, replace))</code></pre>
<ol>
<li>The <code>build_match_and_apply_functions()</code> function has not changed. You&#8217;re still using closures to build two functions dynamically that use variables defined in the outer function.
<li>The global <code>open()</code> function opens a file and returns a file object. In this case, the file we&#8217;re opening contains the pattern strings for pluralizing nouns. The <code>with</code> statement creates what&#8217;s called a <i>context</i>: when the <code>with</code> block ends, Python will automatically close the file, even if an exception is raised inside the <code>with</code> block. You&#8217;ll learn more about <code>with</code> blocks and file objects in the <a href=files.html>Files</a> chapter.
<li>The <code>for line in &lt;fileobject></code> idiom reads data from the open file, one line at a time, and assigns the text to the <var>line</var> variable. You&#8217;ll learn more about reading from files in the <a href=files.html>Files</a> chapter.
<li>Each line in the file really has three values, but they&#8217;re separated by whitespace (tabs or spaces, it makes no difference). To split it out, use the <code>split()</code> string method. The first argument to the <code>split()</code> method is <code>None</code>, which means &#8220;split on any whitespace (tabs or spaces, it makes no difference).&#8221; The second argument is <code>3</code>, which means &#8220;split on whitespace 3 times, then leave the rest of the line alone.&#8221; A line like <code>[sxz]$ $ es</code> will be broken up into the list <code>['[sxz]$', '$', 'es']</code>, which means that <var>pattern</var> will get <code>'[sxz]$'</code>, <var>search</var> will get <code>'$'</code>, and <var>replace</var> will get <code>'es'</code>. That&#8217;s a lot of power in one little line of code.
<li>Finally, you pass <code>pattern</code>, <code>search</code>, and <code>replace</code> to the <code>build_match_and_apply_functions()</code> function, which returns a tuple of functions. You append this tuple to the <var>rules</var> list, and <var>rules</var> ends up storing the list of match and apply functions that the <code>plural()</code> function expects.
</ol>
<p>The improvement here is that you&#8217;ve completely separated the pluralization rules into an external file, so it can be maintained separately from the code that uses it. Code is code, data is data, and life is good.
<p class=a>&#x2042;
<h2 id=generators>Generators</h2>
<p>Wouldn&#8217;t it be grand to have a generic <code>plural()</code> function that parses the rules file? Get rules, check for a match, apply appropriate transformation, go to next rule. That&#8217;s all the <code>plural()</code> function has to do, and that&#8217;s all the <code>plural()</code> function should do.
<p class=d>[<a href=examples/plural5.py>download <code>plural5.py</code></a>]
<pre class='nd pp'><code>def rules(rules_filename):
with open(rules_filename, encoding='utf-8') as pattern_file:
for line in pattern_file:
pattern, search, replace = line.split(None, 3)
yield build_match_and_apply_functions(pattern, search, replace)
def plural(noun, rules_filename='plural5-rules.txt'):
for matches_rule, apply_rule in rules(rules_filename):
if matches_rule(noun):
return apply_rule(noun)
raise ValueError('no matching rule for {0}'.format(noun))</code></pre>
<p>How the heck does <em>that</em> work? Let&#8217;s look at an interactive example first.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>def make_counter(x):</kbd>
<samp class=p>... </samp><kbd class=pp> print('entering make_counter')</kbd>
<samp class=p>... </samp><kbd class=pp> while True:</kbd>
<a><samp class=p>... </samp><kbd class=pp> yield x</kbd> <span class=u>&#x2460;</span></a>
<samp class=p>... </samp><kbd class=pp> print('incrementing x')</kbd>
<samp class=p>... </samp><kbd class=pp> x = x + 1</kbd>
<samp class=p>... </samp>
<a><samp class=p>>>> </samp><kbd class=pp>counter = make_counter(2)</kbd> <span class=u>&#x2461;</span></a>
<a><samp class=p>>>> </samp><kbd class=pp>counter</kbd> <span class=u>&#x2462;</span></a>
&lt;generator object at 0x001C9C10>
<a><samp class=p>>>> </samp><kbd class=pp>next(counter)</kbd> <span class=u>&#x2463;</span></a>
<samp>entering make_counter
2</samp>
<a><samp class=p>>>> </samp><kbd class=pp>next(counter)</kbd> <span class=u>&#x2464;</span></a>
<samp>incrementing x
3</samp>
<a><samp class=p>>>> </samp><kbd class=pp>next(counter)</kbd> <span class=u>&#x2465;</span></a>
<samp>incrementing x
4</samp></pre>
<ol>
<li>The presence of the <code>yield</code> keyword in <code>make_counter</code> means that this is not a normal function. It is a special kind of function which generates values one at a time. You can think of it as a resumable function. Calling it will return a <i>generator</i> that can be used to generate successive values of <var>x</var>.
<li>To create an instance of the <code>make_counter</code> generator, just call it like any other function. Note that this does not actually execute the function code. You can tell this because the first line of the <code>make_counter()</code> function calls <code>print()</code>, but nothing has been printed yet.
<li>The <code>make_counter()</code> function returns a generator object.
<li>The <code>next()</code> function takes a generator object and returns its next value. The first time you call <code>next()</code> with the <var>counter</var> generator, it executes the code in <code>make_counter()</code> up to the first <code>yield</code> statement, then returns the value that was yielded. In this case, that will be <code>2</code>, because you originally created the generator by calling <code>make_counter(2)</code>.
<li>Repeatedly calling <code>next()</code> with the same generator object resumes exactly where it left off and continues until it hits the next <code>yield</code> statement. All variables, local state, <i class=baa>&amp;</i>c. are saved on <code>yield</code> and restored on <code>next()</code>. The next line of code waiting to be executed calls <code>print()</code>, which prints <samp>incrementing x</samp>. After that, the statement <code>x = x + 1</code>. Then it loops through the <code>while</code> loop again, and the first thing it hits is the statement <code>yield x</code>, which saves the state of everything and returns the current value of <var>x</var> (now <code>3</code>).
<li>The second time you call <code>next(counter)</code>, you do all the same things again, but this time <var>x</var> is now <code>4</code>.
</ol>
<p>Since <code>make_counter</code> sets up an infinite loop, you could theoretically do this forever, and it would just keep incrementing <var>x</var> and spitting out values. But let&#8217;s look at more productive uses of generators instead.
<h3 id=a-fibonacci-generator>A Fibonacci Generator</h3>
<aside>&#8220;yield&#8221; pauses a function. &#8220;next()&#8221; resumes where it left off.</aside>
<p class=d>[<a href=examples/fibonacci.py>download <code>fibonacci.py</code></a>]
<pre class=pp><code>def fib(max):
<a> a, b = 0, 1 <span class=u>&#x2460;</span></a>
while a &lt; max:
<a> yield a <span class=u>&#x2461;</span></a>
<a> a, b = b, a + b <span class=u>&#x2462;</span></a></code></pre>
<ol>
<li>The Fibonacci sequence is a sequence of numbers where each number is the sum of the two numbers before it. It starts with 0 and <code>1</code>, goes up slowly at first, then more and more rapidly. To start the sequence, you need two variables: <var>a</var> starts at 0, and <var>b</var> starts at <code>1</code>.
<li><var>a</var> is the current number in the sequence, so yield it.
<li><var>b</var> is the next number in the sequence, so assign that to <var>a</var>, but also calculate the next value (<code>a + b</code>) and assign that to <var>b</var> for later use. Note that this happens in parallel; if <var>a</var> is <code>3</code> and <var>b</var> is <code>5</code>, then <code>a, b = b, a + b</code> will set <var>a</var> to <code>5</code> (the previous value of <var>b</var>) and <var>b</var> to <code>8</code> (the sum of the previous values of <var>a</var> and <var>b</var>).
</ol>
<p>So you have a function that spits out successive Fibonacci numbers. Sure, you could do that with recursion, but this way is easier to read. Also, it works well with <code>for</code> loops.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>from fibonacci import fib</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>for n in fib(1000):</kbd> <span class=u>&#x2460;</span></a>
<a><samp class=p>... </samp><kbd class=pp> print(n, end=' ')</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987</samp>
<a><samp class=p>>>> </samp><kbd class=pp>list(fib(1000))</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987]</samp></pre>
<ol>
<li>You can use a generator like <code>fib()</code> in a <code>for</code> loop directly. The <code>for</code> loop will automatically call the <code>next()</code> function to get values from the <code>fib()</code> generator and assign them to the <code>for</code> loop index variable (<var>n</var>).
<li>Each time through the <code>for</code> loop, <var>n</var> gets a new value from the <code>yield</code> statement in <code>fib()</code>, and all you have to do is print it out. Once <code>fib()</code> runs out of numbers (<var>a</var> becomes bigger than <var>max</var>, which in this case is <code>1000</code>), then the <code>for</code> loop exits gracefully.
<li>This is a useful idiom: pass a generator to the <code>list()</code> function, and it will iterate through the entire generator (just like the <code>for</code> loop in the previous example) and return a list of all the values.
</ol>
<h3 id=a-plural-rule-generator>A Plural Rule Generator</h3>
<p>Let&#8217;s go back to <code>plural5.py</code> and see how this version of the <code>plural()</code> function works.
<pre class=pp><code>def rules(rules_filename):
with open(rules_filename, encoding='utf-8') as pattern_file:
for line in pattern_file:
<a> pattern, search, replace = line.split(None, 3) <span class=u>&#x2460;</span></a>
<a> yield build_match_and_apply_functions(pattern, search, replace) <span class=u>&#x2461;</span></a>
def plural(noun, rules_filename='plural5-rules.txt'):
<a> for matches_rule, apply_rule in rules(rules_filename): <span class=u>&#x2462;</span></a>
if matches_rule(noun):
return apply_rule(noun)
raise ValueError('no matching rule for {0}'.format(noun))</code></pre>
<ol>
<li>No magic here. Remember that the lines of the rules file have three values separated by whitespace, so you use <code>line.split(None, 3)</code> to get the three &#8220;columns&#8221; and assign them to three local variables.
<li><em>And then you yield.</em> What do you yield? Two functions, built dynamically with your old friend, <code>build_match_and_apply_functions()</code>, which is identical to the previous examples. In other words, <code>rules()</code> is a generator that spits out match and apply functions <em>on demand</em>.
<li>Since <code>rules()</code> is a generator, you can use it directly in a <code>for</code> loop. The first time through the <code>for</code> loop, you will call the <code>rules()</code> function, which will open the pattern file, read the first line, dynamically build a match function and an apply function from the patterns on that line, and yield the dynamically built functions. The second time through the <code>for</code> loop, you will pick up exactly where you left off in <code>rules()</code> (which was in the middle of the <code>for line in pattern_file</code> loop). The first thing it will do is read the next line of the file (which is still open), dynamically build another match and apply function based on the patterns on that line in the file, and yield the two functions.
</ol>
<p>What have you gained over stage 4? Startup time. In stage 4, when you imported the <code>plural4</code> module, it read the entire patterns file and built a list of all the possible rules, before you could even think about calling the <code>plural()</code> function. With generators, you can do everything lazily: you read the first rule and create functions and try them, and if that works you don&#8217;t ever read the rest of the file or create any other functions.
<p>What have you lost? Performance! Every time you call the <code>plural()</code> function, the <code>rules()</code> generator starts over from the beginning&nbsp;&mdash;&nbsp;which means re-opening the patterns file and reading from the beginning, one line at a time.
<p>What if you could have the best of both worlds: minimal startup cost (don&#8217;t execute any code on <code>import</code>), <em>and</em> maximum performance (don&#8217;t build the same functions over and over again). Oh, and you still want to keep the rules in a separate file (because code is code and data is data), just as long as you never have to read the same line twice.
<p>To do that, you&#8217;ll need to build your own iterator. But before you do <em>that</em>, you need to learn about Python classes.
<p class=a>&#x2042;
<h2 id=furtherreading>Further Reading</h2>
<ul>
<li><a href=http://www.python.org/dev/peps/pep-0255/>PEP 255: Simple Generators</a>
<li><a href=http://effbot.org/zone/python-with-statement.htm>Understanding Python&#8217;s &#8220;with&#8221; statement</a>
<li><a href=http://ynniv.com/blog/2007/08/closures-in-python.html>Closures in Python</a>
<li><a href=http://en.wikipedia.org/wiki/Fibonacci_number>Fibonacci numbers</a>
<li><a href=http://www2.gsu.edu/~wwwesl/egw/crump.htm>English Irregular Plural Nouns</a>
</ul>
<p class=v><a href=regular-expressions.html rel=prev title='back to &#8220;Regular Expressions&#8221;'><span class=u>&#x261C;</span></a> <a href=iterators.html rel=next title='onward to &#8220;Classes &amp; Iterators&#8221;'><span class=u>&#x261E;</span></a>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/prettify.js></script>
<script src=j/dip3.js></script>
<!DOCTYPE html>
<meta charset=utf-8>
<title>Closures &amp; Generators - Dive Into Python 3</title>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<link rel=stylesheet href=dip3.css>
<style>
body{counter-reset:h1 6}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#generators>Dive Into Python 3</a> <span class=u>&#8227;</span>
<p id=level>Difficulty level: <span class=u title=intermediate>&#x2666;&#x2666;&#x2666;&#x2662;&#x2662;</span>
<h1>Closures <i class=baa>&amp;</i> Generators</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> My spelling is Wobbly. It&#8217;s good spelling but it Wobbles, and the letters get in the wrong places. <span class=u>&#x275E;</span><br>&mdash; Winnie-the-Pooh
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>Having grown up the son of a librarian and an English major, I have always been fascinated by languages. Not programming languages. Well yes, programming languages, but also natural languages. Take English. English is a schizophrenic language that borrows words from German, French, Spanish, and Latin (to name a few). Actually, &#8220;borrows&#8221; is the wrong word; &#8220;pillages&#8221; is more like it. Or perhaps &#8220;assimilates&#8221;&nbsp;&mdash;&nbsp;like the Borg. Yes, I like that.
<p class=c><code>We are the Borg. Your linguistic and etymological distinctiveness will be added to our own. Resistance is futile.</code>
<p>In this chapter, you&#8217;re going to learn about plural nouns. Also, functions that return other functions, advanced regular expressions, and generators. But first, let&#8217;s talk about how to make plural nouns. (If you haven&#8217;t read <a href=regular-expressions.html>the chapter on regular expressions</a>, now would be a good time. This chapter assumes you understand the basics of regular expressions, and it quickly descends into more advanced uses.)
<p>If you grew up in an English-speaking country or learned English in a formal school setting, you&#8217;re probably familiar with the basic rules:
<ul>
<li>If a word ends in S, X, or Z, add ES. <i>Bass</i> becomes <i>basses</i>, <i>fax</i> becomes <i>faxes</i>, and <i>waltz</i> becomes <i>waltzes</i>.
<li>If a word ends in a noisy H, add ES; if it ends in a silent H, just add S. What&#8217;s a noisy H? One that gets combined with other letters to make a sound that you can hear. So <i>coach</i> becomes <i>coaches</i> and <i>rash</i> becomes <i>rashes</i>, because you can hear the CH and SH sounds when you say them. But <i>cheetah</i> becomes <i>cheetahs</i>, because the H is silent.
<li>If a word ends in Y that sounds like I, change the Y to IES; if the Y is combined with a vowel to sound like something else, just add S. So <i>vacancy</i> becomes <i>vacancies</i>, but <i>day</i> becomes <i>days</i>.
<li>If all else fails, just add S and hope for the best.
</ul>
<p>(I know, there are a lot of exceptions. <i>Man</i> becomes <i>men</i> and <i>woman</i> becomes <i>women</i>, but <i>human</i> becomes <i>humans</i>. <i>Mouse</i> becomes <i>mice</i> and <i>louse</i> becomes <i>lice</i>, but <i>house</i> becomes <i>houses</i>. <i>Knife</i> becomes <i>knives</i> and <i>wife</i> becomes <i>wives</i>, but <i>lowlife</i> becomes <i>lowlifes</i>. And don&#8217;t even get me started on words that are their own plural, like <i>sheep</i>, <i>deer</i>, and <i>haiku</i>.)
<p>Other languages, of course, are completely different.
<p>Let&#8217;s design a Python library that automatically pluralizes English nouns. We&#8217;ll start with just these four rules, but keep in mind that you&#8217;ll inevitably need to add more.
<p class=a>&#x2042;
<h2 id=i-know>I Know, Let&#8217;s Use Regular Expressions!</h2>
<p>So you&#8217;re looking at words, which, at least in English, means you&#8217;re looking at strings of characters. You have rules that say you need to find different combinations of characters, then do different things to them. This sounds like a job for regular expressions!
<p class=d>[<a href=examples/plural1.py>download <code>plural1.py</code></a>]
<pre class=pp><code>import re
def plural(noun):
<a> if re.search('[sxz]$', noun): <span class=u>&#x2460;</span></a>
<a> return re.sub('$', 'es', noun) <span class=u>&#x2461;</span></a>
elif re.search('[^aeioudgkprt]h$', noun):
return re.sub('$', 'es', noun)
elif re.search('[^aeiou]y$', noun):
return re.sub('y$', 'ies', noun)
else:
return noun + 's'</code></pre>
<ol>
<li>This is a regular expression, but it uses a syntax you didn&#8217;t see in <a href=regular-expressions.html><i>Regular Expressions</i></a>. The square brackets mean &#8220;match exactly one of these characters.&#8221; So <code>[sxz]</code> means &#8220;<code>s</code>, or <code>x</code>, or <code>z</code>&#8221;, but only one of them. The <code>$</code> should be familiar; it matches the end of string. Combined, this regular expression tests whether <var>noun</var> ends with <code>s</code>, <code>x</code>, or <code>z</code>.
<li>This <code>re.sub()</code> function performs regular expression-based string substitutions.
</ol>
<p>Let&#8217;s look at regular expression substitutions in more detail.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import re</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[abc]', 'Mark')</kbd> <span class=u>&#x2460;</span></a>
&lt;_sre.SRE_Match object at 0x001C1FA8>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('[abc]', 'o', 'Mark')</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>'Mork'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('[abc]', 'o', 'rock')</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>'rook'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('[abc]', 'o', 'caps')</kbd> <span class=u>&#x2463;</span></a>
<samp class=pp>'oops'</samp></pre>
<ol>
<li>Does the string <code>Mark</code> contain <code>a</code>, <code>b</code>, or <code>c</code>? Yes, it contains <code>a</code>.
<li>OK, now find <code>a</code>, <code>b</code>, or <code>c</code>, and replace it with <code>o</code>. <code>Mark</code> becomes <code>Mork</code>.
<li>The same function turns <code>rock</code> into <code>rook</code>.
<li>You might think this would turn <code>caps</code> into <code>oaps</code>, but it doesn&#8217;t. <code>re.sub</code> replaces <em>all</em> of the matches, not just the first one. So this regular expression turns <code>caps</code> into <code>oops</code>, because both the <code>c</code> and the <code>a</code> get turned into <code>o</code>.
</ol>
<p>And now, back to the <code>plural()</code> function&hellip;
<pre class=pp><code>def plural(noun):
if re.search('[sxz]$', noun):
<a> return re.sub('$', 'es', noun) <span class=u>&#x2460;</span></a>
<a> elif re.search('[^aeioudgkprt]h$', noun): <span class=u>&#x2461;</span></a>
return re.sub('$', 'es', noun)
<a> elif re.search('[^aeiou]y$', noun): <span class=u>&#x2462;</span></a>
return re.sub('y$', 'ies', noun)
else:
return noun + 's'</code></pre>
<ol>
<li>Here, you&#8217;re replacing the end of the string (matched by <code>$</code>) with the string <code>es</code>. In other words, adding <code>es</code> to the string. You could accomplish the same thing with string concatenation, for example <code>noun + 'es'</code>, but I chose to use regular expressions for each rule, for reasons that will become clear later in the chapter.
<li>Look closely, this is another new variation. The <code>^</code> as the first character inside the square brackets means something special: negation. <code>[^abc]</code> means &#8220;any single character <em>except</em> <code>a</code>, <code>b</code>, or <code>c</code>&#8221;. So <code>[^aeioudgkprt]</code> means any character except <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, <code>u</code>, <code>d</code>, <code>g</code>, <code>k</code>, <code>p</code>, <code>r</code>, or <code>t</code>. Then that character needs to be followed by <code>h</code>, followed by end of string. You&#8217;re looking for words that end in H where the H can be heard.
<li>Same pattern here: match words that end in Y, where the character before the Y is <em>not</em> <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, or <code>u</code>. You&#8217;re looking for words that end in Y that sounds like I.
</ol>
<p>Let&#8217;s look at negation regular expressions in more detail.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import re</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'vacancy')</kbd> <span class=u>&#x2460;</span></a>
&lt;_sre.SRE_Match object at 0x001C1FA8>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'boy')</kbd> <span class=u>&#x2461;</span></a>
<samp class=p>>>> </samp>
<samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'day')</kbd>
<samp class=p>>>> </samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.search('[^aeiou]y$', 'pita')</kbd> <span class=u>&#x2462;</span></a>
<samp class=p>>>> </samp></pre>
<ol>
<li><code>vacancy</code> matches this regular expression, because it ends in <code>cy</code>, and <code>c</code> is not <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, or <code>u</code>.
<li><code>boy</code> does not match, because it ends in <code>oy</code>, and you specifically said that the character before the <code>y</code> could not be <code>o</code>. <code>day</code> does not match, because it ends in <code>ay</code>.
<li><code>pita</code> does not match, because it does not end in <code>y</code>.
</ol>
<pre class=screen>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('y$', 'ies', 'vacancy')</kbd> <span class=u>&#x2460;</span></a>
<samp class=pp>'vacancies'</samp>
<samp class=p>>>> </samp><kbd class=pp>re.sub('y$', 'ies', 'agency')</kbd>
<samp class=pp>'agencies'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>re.sub('([^aeiou])y$', r'\1ies', 'vacancy')</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>'vacancies'</samp></pre>
<ol>
<li>This regular expression turns <code>vacancy</code> into <code>vacancies</code> and <code>agency</code> into <code>agencies</code>, which is what you wanted. Note that it would also turn <code>boy</code> into <code>boies</code>, but that will never happen in the function because you did that <code>re.search</code> first to find out whether you should do this <code>re.sub</code>.
<li>Just in passing, I want to point out that it is possible to combine these two regular expressions (one to find out if the rule applies, and another to actually apply it) into a single regular expression. Here&#8217;s what that would look like. Most of it should look familiar: you&#8217;re using a remembered group, which you learned in <a href=regular-expressions.html#phonenumbers>Case study: Parsing Phone Numbers</a>. The group is used to remember the character before the letter <code>y</code>. Then in the substitution string, you use a new syntax, <code>\1</code>, which means &#8220;hey, that first group you remembered? put it right here.&#8221; In this case, you remember the <code>c</code> before the <code>y</code>; when you do the substitution, you substitute <code>c</code> in place of <code>c</code>, and <code>ies</code> in place of <code>y</code>. (If you have more than one remembered group, you can use <code>\2</code> and <code>\3</code> and so on.)
</ol>
<p>Regular expression substitutions are extremely powerful, and the <code>\1</code> syntax makes them even more powerful. But combining the entire operation into one regular expression is also much harder to read, and it doesn&#8217;t directly map to the way you first described the pluralizing rules. You originally laid out rules like &#8220;if the word ends in S, X, or Z, then add ES&#8221;. If you look at this function, you have two lines of code that say &#8220;if the word ends in S, X, or Z, then add ES&#8221;. It doesn&#8217;t get much more direct than that.
<p class=a>&#x2042;
<h2 id=a-list-of-functions>A List Of Functions</h2>
<p>Now you&#8217;re going to add a level of abstraction. You started by defining a list of rules: if this, do that, otherwise go to the next rule. Let&#8217;s temporarily complicate part of the program so you can simplify another part.
<p class=d>[<a href=examples/plural2.py>download <code>plural2.py</code></a>]
<pre class=pp><code>import re
def match_sxz(noun):
return re.search('[sxz]$', noun)
def apply_sxz(noun):
return re.sub('$', 'es', noun)
def match_h(noun):
return re.search('[^aeioudgkprt]h$', noun)
def apply_h(noun):
return re.sub('$', 'es', noun)
<a>def match_y(noun): <span class=u>&#x2460;</span></a>
return re.search('[^aeiou]y$', noun)
<a>def apply_y(noun): <span class=u>&#x2461;</span></a>
return re.sub('y$', 'ies', noun)
def match_default(noun):
return True
def apply_default(noun):
return noun + 's'
<a>rules = ((match_sxz, apply_sxz), <span class=u>&#x2462;</span></a>
(match_h, apply_h),
(match_y, apply_y),
(match_default, apply_default)
)
def plural(noun):
<a> for matches_rule, apply_rule in rules: <span class=u>&#x2463;</span></a>
if matches_rule(noun):
return apply_rule(noun)</code></pre>
<ol>
<li>Now, each match rule is its own function which returns the results of calling the <code>re.search()</code> function.
<li>Each apply rule is also its own function which calls the <code>re.sub()</code> function to apply the appropriate pluralization rule.
<li>Instead of having one function (<code>plural()</code>) with multiple rules, you have the <code>rules</code> data structure, which is a sequence of pairs of functions.
<li>Since the rules have been broken out into a separate data structure, the new <code>plural()</code> function can be reduced to a few lines of code. Using a <code>for</code> loop, you can pull out the match and apply rules two at a time (one match, one apply) from the <var>rules</var> structure. On the first iteration of the <code>for</code> loop, <var>matches_rule</var> will get <code>match_sxz</code>, and <var>apply_rule</var> will get <code>apply_sxz</code>. On the second iteration (assuming you get that far), <var>matches_rule</var> will be assigned <code>match_h</code>, and <var>apply_rule</var> will be assigned <code>apply_h</code>. The function is guaranteed to return something eventually, because the final match rule (<code>match_default</code>) simply returns <code>True</code>, meaning the corresponding apply rule (<code>apply_default</code>) will always be applied.
</ol>
<aside>The &#8220;rules&#8221; variable is a sequence of pairs of functions.</aside>
<p>The reason this technique works is that <a href=your-first-python-program.html#everythingisanobject>everything in Python is an object</a>, including functions. The <var>rules</var> data structure contains functions&nbsp;&mdash;&nbsp;not names of functions, but actual function objects. When they get assigned in the <code>for</code> loop, then <var>matches_rule</var> and <var>apply_rule</var> are actual functions that you can call. On the first iteration of the <code>for</code> loop, this is equivalent to calling <code>matches_sxz(noun)</code>, and if it returns a match, calling <code>apply_sxz(noun)</code>.
<p>If this additional level of abstraction is confusing, try unrolling the function to see the equivalence. The entire <code>for</code> loop is equivalent to the following:
<pre class='nd pp'><code>
def plural(noun):
if match_sxz(noun):
return apply_sxz(noun)
if match_h(noun):
return apply_h(noun)
if match_y(noun):
return apply_y(noun)
if match_default(noun):
return apply_default(noun)</code></pre>
<p>The benefit here is that the <code>plural()</code> function is now simplified. It takes a sequence of rules, defined elsewhere, and iterates through them in a generic fashion.
<ol>
<li>Get a match rule
<li>Does it match? Then call the apply rule and return the result.
<li>No match? Go to step 1.
</ol>
<p>The rules could be defined anywhere, in any way. The <code>plural()</code> function doesn&#8217;t care.
<p>Now, was adding this level of abstraction worth it? Well, not yet. Let&#8217;s consider what it would take to add a new rule to the function. In the first example, it would require adding an <code>if</code> statement to the <code>plural()</code> function. In this second example, it would require adding two functions, <code>match_foo()</code> and <code>apply_foo()</code>, and then updating the <var>rules</var> sequence to specify where in the order the new match and apply functions should be called relative to the other rules.
<p>But this is really just a stepping stone to the next section. Let&#8217;s move on&hellip;
<p class=a>&#x2042;
<h2 id=a-list-of-patterns>A List Of Patterns</h2>
<p>Defining separate named functions for each match and apply rule isn&#8217;t really necessary. You never call them directly; you add them to the <var>rules</var> sequence and call them through there. Furthermore, each function follows one of two patterns. All the match functions call <code>re.search()</code>, and all the apply functions call <code>re.sub()</code>. Let&#8217;s factor out the patterns so that defining new rules can be easier.
<p class=d>[<a href=examples/plural3.py>download <code>plural3.py</code></a>]
<pre class=pp><code>import re
def build_match_and_apply_functions(pattern, search, replace):
<a> def matches_rule(word): <span class=u>&#x2460;</span></a>
return re.search(pattern, word)
<a> def apply_rule(word): <span class=u>&#x2461;</span></a>
return re.sub(search, replace, word)
<a> return (matches_rule, apply_rule) <span class=u>&#x2462;</span></a></code></pre>
<ol>
<li><code>build_match_and_apply_functions()</code> is a function that builds other functions dynamically. It takes <var>pattern</var>, <var>search</var> and <var>replace</var>, then defines a <code>matches_rule()</code> function which calls <code>re.search()</code> with the <var>pattern</var> that was passed to the <code>build_match_and_apply_functions()</code> function, and the <var>word</var> that was passed to the <code>matches_rule()</code> function you&#8217;re building. Whoa.
<li>Building the apply function works the same way. The apply function is a function that takes one parameter, and calls <code>re.sub()</code> with the <var>search</var> and <var>replace</var> parameters that were passed to the <code>build_match_and_apply_functions()</code> function, and the <var>word</var> that was passed to the <code>apply_rule()</code> function you&#8217;re building. This technique of using the values of outside parameters within a dynamic function is called <em>closures</em>. You&#8217;re essentially defining constants within the apply function you&#8217;re building: it takes one parameter (<var>word</var>), but it then acts on that plus two other values (<var>search</var> and <var>replace</var>) which were set when you defined the apply function.
<li>Finally, the <code>build_match_and_apply_functions()</code> function returns a tuple of two values: the two functions you just created. The constants you defined within those functions (<var>pattern</var> within the <code>matches_rule()</code> function, and <var>search</var> and <var>replace</var> within the <code>apply_rule()</code> function) stay with those functions, even after you return from <code>build_match_and_apply_functions()</code>. That&#8217;s insanely cool.
</ol>
<p>If this is incredibly confusing (and it should be, this is weird stuff), it may become clearer when you see how to use it.
<pre class=pp><code><a>patterns = \ <span class=u>&#x2460;</span></a>
(
('[sxz]$', '$', 'es'),
('[^aeioudgkprt]h$', '$', 'es'),
('(qu|[^aeiou])y$', 'y$', 'ies'),
<a> ('$', '$', 's') <span class=u>&#x2461;</span></a>
)
<a>rules = [build_match_and_apply_functions(pattern, search, replace) <span class=u>&#x2462;</span></a>
for (pattern, search, replace) in patterns]</code></pre>
<ol>
<li>Our pluralization &#8220;rules&#8221; are now defined as a tuple of tuples of <em>strings</em> (not functions). The first string in each group is the regular expression pattern that you would use in <code>re.search()</code> to see if this rule matches. The second and third strings in each group are the search and replace expressions you would use in <code>re.sub()</code> to actually apply the rule to turn a noun into its plural.
<li>There&#8217;s a slight change here, in the fallback rule. In the previous example, the <code>match_default()</code> function simply returned <code>True</code>, meaning that if none of the more specific rules matched, the code would simply add an <code>s</code> to the end of the given word. This example does something functionally equivalent. The final regular expression asks whether the word has an end (<code>$</code> matches the end of a string). Of course, every string has an end, even an empty string, so this expression always matches. Thus, it serves the same purpose as the <code>match_default()</code> function that always returned <code>True</code>: it ensures that if no more specific rule matches, the code adds an <code>s</code> to the end of the given word.
<li>This line is magic. It takes the sequence of strings in <var>patterns</var> and turns them into a sequence of functions. How? By &#8220;mapping&#8221; the strings to the <code>build_match_and_apply_functions()</code> function. That is, it takes each triplet of strings and calls the <code>build_match_and_apply_functions()</code> function with those three strings as arguments. The <code>build_match_and_apply_functions()</code> function returns a tuple of two functions. This means that <var>rules</var> ends up being functionally equivalent to the previous example: a list of tuples, where each tuple is a pair of functions. The first function is the match function that calls <code>re.search()</code>, and the second function is the apply function that calls <code>re.sub()</code>.
</ol>
<p>Rounding out this version of the script is the main entry point, the <code>plural()</code> function.
<pre class=pp><code>def plural(noun):
<a> for matches_rule, apply_rule in rules: <span class=u>&#x2460;</span></a>
if matches_rule(noun):
return apply_rule(noun)</code></pre>
<ol>
<li>Since the <var>rules</var> list is the same as the previous example (really, it is), it should come as no surprise that the <code>plural()</code> function hasn&#8217;t changed at all. It&#8217;s completely generic; it takes a list of rule functions and calls them in order. It doesn&#8217;t care how the rules are defined. In the previous example, they were defined as separate named functions. Now they are built dynamically by mapping the output of the <code>build_match_and_apply_functions()</code> function onto a list of raw strings. It doesn&#8217;t matter; the <code>plural()</code> function still works the same way.
</ol>
<p class=a>&#x2042;
<h2 id=a-file-of-patterns>A File Of Patterns</h2>
<p>You&#8217;ve factored out all the duplicate code and added enough abstractions so that the pluralization rules are defined in a list of strings. The next logical step is to take these strings and put them in a separate file, where they can be maintained separately from the code that uses them.
<p>First, let&#8217;s create a text file that contains the rules you want. No fancy data structures, just whitespace-delimited strings in three columns. Let&#8217;s call it <code>plural4-rules.txt</code>.
<p class=d>[<a href=examples/plural4-rules.txt>download <code>plural4-rules.txt</code></a>]
<pre class='nd pp'><code>[sxz]$ $ es
[^aeioudgkprt]h$ $ es
[^aeiou]y$ y$ ies
$ $ s</code></pre>
<p>Now let&#8217;s see how you can use this rules file.
<p class=d>[<a href=examples/plural4.py>download <code>plural4.py</code></a>]
<pre class=pp><code>import re
<a>def build_match_and_apply_functions(pattern, search, replace): <span class=u>&#x2460;</span></a>
def matches_rule(word):
return re.search(pattern, word)
def apply_rule(word):
return re.sub(search, replace, word)
return (matches_rule, apply_rule)
rules = []
<a>with open('plural4-rules.txt', encoding='utf-8') as pattern_file: <span class=u>&#x2461;</span></a>
<a> for line in pattern_file: <span class=u>&#x2462;</span></a>
<a> pattern, search, replace = line.split(None, 3) <span class=u>&#x2463;</span></a>
<a> rules.append(build_match_and_apply_functions( <span class=u>&#x2464;</span></a>
pattern, search, replace))</code></pre>
<ol>
<li>The <code>build_match_and_apply_functions()</code> function has not changed. You&#8217;re still using closures to build two functions dynamically that use variables defined in the outer function.
<li>The global <code>open()</code> function opens a file and returns a file object. In this case, the file we&#8217;re opening contains the pattern strings for pluralizing nouns. The <code>with</code> statement creates what&#8217;s called a <i>context</i>: when the <code>with</code> block ends, Python will automatically close the file, even if an exception is raised inside the <code>with</code> block. You&#8217;ll learn more about <code>with</code> blocks and file objects in the <a href=files.html>Files</a> chapter.
<li>The <code>for line in &lt;fileobject></code> idiom reads data from the open file, one line at a time, and assigns the text to the <var>line</var> variable. You&#8217;ll learn more about reading from files in the <a href=files.html>Files</a> chapter.
<li>Each line in the file really has three values, but they&#8217;re separated by whitespace (tabs or spaces, it makes no difference). To split it out, use the <code>split()</code> string method. The first argument to the <code>split()</code> method is <code>None</code>, which means &#8220;split on any whitespace (tabs or spaces, it makes no difference).&#8221; The second argument is <code>3</code>, which means &#8220;split on whitespace 3 times, then leave the rest of the line alone.&#8221; A line like <code>[sxz]$ $ es</code> will be broken up into the list <code>['[sxz]$', '$', 'es']</code>, which means that <var>pattern</var> will get <code>'[sxz]$'</code>, <var>search</var> will get <code>'$'</code>, and <var>replace</var> will get <code>'es'</code>. That&#8217;s a lot of power in one little line of code.
<li>Finally, you pass <code>pattern</code>, <code>search</code>, and <code>replace</code> to the <code>build_match_and_apply_functions()</code> function, which returns a tuple of functions. You append this tuple to the <var>rules</var> list, and <var>rules</var> ends up storing the list of match and apply functions that the <code>plural()</code> function expects.
</ol>
<p>The improvement here is that you&#8217;ve completely separated the pluralization rules into an external file, so it can be maintained separately from the code that uses it. Code is code, data is data, and life is good.
<p class=a>&#x2042;
<h2 id=generators>Generators</h2>
<p>Wouldn&#8217;t it be grand to have a generic <code>plural()</code> function that parses the rules file? Get rules, check for a match, apply appropriate transformation, go to next rule. That&#8217;s all the <code>plural()</code> function has to do, and that&#8217;s all the <code>plural()</code> function should do.
<p class=d>[<a href=examples/plural5.py>download <code>plural5.py</code></a>]
<pre class='nd pp'><code>def rules(rules_filename):
with open(rules_filename, encoding='utf-8') as pattern_file:
for line in pattern_file:
pattern, search, replace = line.split(None, 3)
yield build_match_and_apply_functions(pattern, search, replace)
def plural(noun, rules_filename='plural5-rules.txt'):
for matches_rule, apply_rule in rules(rules_filename):
if matches_rule(noun):
return apply_rule(noun)
raise ValueError('no matching rule for {0}'.format(noun))</code></pre>
<p>How the heck does <em>that</em> work? Let&#8217;s look at an interactive example first.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>def make_counter(x):</kbd>
<samp class=p>... </samp><kbd class=pp> print('entering make_counter')</kbd>
<samp class=p>... </samp><kbd class=pp> while True:</kbd>
<a><samp class=p>... </samp><kbd class=pp> yield x</kbd> <span class=u>&#x2460;</span></a>
<samp class=p>... </samp><kbd class=pp> print('incrementing x')</kbd>
<samp class=p>... </samp><kbd class=pp> x = x + 1</kbd>
<samp class=p>... </samp>
<a><samp class=p>>>> </samp><kbd class=pp>counter = make_counter(2)</kbd> <span class=u>&#x2461;</span></a>
<a><samp class=p>>>> </samp><kbd class=pp>counter</kbd> <span class=u>&#x2462;</span></a>
&lt;generator object at 0x001C9C10>
<a><samp class=p>>>> </samp><kbd class=pp>next(counter)</kbd> <span class=u>&#x2463;</span></a>
<samp>entering make_counter
2</samp>
<a><samp class=p>>>> </samp><kbd class=pp>next(counter)</kbd> <span class=u>&#x2464;</span></a>
<samp>incrementing x
3</samp>
<a><samp class=p>>>> </samp><kbd class=pp>next(counter)</kbd> <span class=u>&#x2465;</span></a>
<samp>incrementing x
4</samp></pre>
<ol>
<li>The presence of the <code>yield</code> keyword in <code>make_counter</code> means that this is not a normal function. It is a special kind of function which generates values one at a time. You can think of it as a resumable function. Calling it will return a <i>generator</i> that can be used to generate successive values of <var>x</var>.
<li>To create an instance of the <code>make_counter</code> generator, just call it like any other function. Note that this does not actually execute the function code. You can tell this because the first line of the <code>make_counter()</code> function calls <code>print()</code>, but nothing has been printed yet.
<li>The <code>make_counter()</code> function returns a generator object.
<li>The <code>next()</code> function takes a generator object and returns its next value. The first time you call <code>next()</code> with the <var>counter</var> generator, it executes the code in <code>make_counter()</code> up to the first <code>yield</code> statement, then returns the value that was yielded. In this case, that will be <code>2</code>, because you originally created the generator by calling <code>make_counter(2)</code>.
<li>Repeatedly calling <code>next()</code> with the same generator object resumes exactly where it left off and continues until it hits the next <code>yield</code> statement. All variables, local state, <i class=baa>&amp;</i>c. are saved on <code>yield</code> and restored on <code>next()</code>. The next line of code waiting to be executed calls <code>print()</code>, which prints <samp>incrementing x</samp>. After that, the statement <code>x = x + 1</code>. Then it loops through the <code>while</code> loop again, and the first thing it hits is the statement <code>yield x</code>, which saves the state of everything and returns the current value of <var>x</var> (now <code>3</code>).
<li>The second time you call <code>next(counter)</code>, you do all the same things again, but this time <var>x</var> is now <code>4</code>.
</ol>
<p>Since <code>make_counter</code> sets up an infinite loop, you could theoretically do this forever, and it would just keep incrementing <var>x</var> and spitting out values. But let&#8217;s look at more productive uses of generators instead.
<h3 id=a-fibonacci-generator>A Fibonacci Generator</h3>
<aside>&#8220;yield&#8221; pauses a function. &#8220;next()&#8221; resumes where it left off.</aside>
<p class=d>[<a href=examples/fibonacci.py>download <code>fibonacci.py</code></a>]
<pre class=pp><code>def fib(max):
<a> a, b = 0, 1 <span class=u>&#x2460;</span></a>
while a &lt; max:
<a> yield a <span class=u>&#x2461;</span></a>
<a> a, b = b, a + b <span class=u>&#x2462;</span></a></code></pre>
<ol>
<li>The Fibonacci sequence is a sequence of numbers where each number is the sum of the two numbers before it. It starts with 0 and <code>1</code>, goes up slowly at first, then more and more rapidly. To start the sequence, you need two variables: <var>a</var> starts at 0, and <var>b</var> starts at <code>1</code>.
<li><var>a</var> is the current number in the sequence, so yield it.
<li><var>b</var> is the next number in the sequence, so assign that to <var>a</var>, but also calculate the next value (<code>a + b</code>) and assign that to <var>b</var> for later use. Note that this happens in parallel; if <var>a</var> is <code>3</code> and <var>b</var> is <code>5</code>, then <code>a, b = b, a + b</code> will set <var>a</var> to <code>5</code> (the previous value of <var>b</var>) and <var>b</var> to <code>8</code> (the sum of the previous values of <var>a</var> and <var>b</var>).
</ol>
<p>So you have a function that spits out successive Fibonacci numbers. Sure, you could do that with recursion, but this way is easier to read. Also, it works well with <code>for</code> loops.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>from fibonacci import fib</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>for n in fib(1000):</kbd> <span class=u>&#x2460;</span></a>
<a><samp class=p>... </samp><kbd class=pp> print(n, end=' ')</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987</samp>
<a><samp class=p>>>> </samp><kbd class=pp>list(fib(1000))</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987]</samp></pre>
<ol>
<li>You can use a generator like <code>fib()</code> in a <code>for</code> loop directly. The <code>for</code> loop will automatically call the <code>next()</code> function to get values from the <code>fib()</code> generator and assign them to the <code>for</code> loop index variable (<var>n</var>).
<li>Each time through the <code>for</code> loop, <var>n</var> gets a new value from the <code>yield</code> statement in <code>fib()</code>, and all you have to do is print it out. Once <code>fib()</code> runs out of numbers (<var>a</var> becomes bigger than <var>max</var>, which in this case is <code>1000</code>), then the <code>for</code> loop exits gracefully.
<li>This is a useful idiom: pass a generator to the <code>list()</code> function, and it will iterate through the entire generator (just like the <code>for</code> loop in the previous example) and return a list of all the values.
</ol>
<h3 id=a-plural-rule-generator>A Plural Rule Generator</h3>
<p>Let&#8217;s go back to <code>plural5.py</code> and see how this version of the <code>plural()</code> function works.
<pre class=pp><code>def rules(rules_filename):
with open(rules_filename, encoding='utf-8') as pattern_file:
for line in pattern_file:
<a> pattern, search, replace = line.split(None, 3) <span class=u>&#x2460;</span></a>
<a> yield build_match_and_apply_functions(pattern, search, replace) <span class=u>&#x2461;</span></a>
def plural(noun, rules_filename='plural5-rules.txt'):
<a> for matches_rule, apply_rule in rules(rules_filename): <span class=u>&#x2462;</span></a>
if matches_rule(noun):
return apply_rule(noun)
raise ValueError('no matching rule for {0}'.format(noun))</code></pre>
<ol>
<li>No magic here. Remember that the lines of the rules file have three values separated by whitespace, so you use <code>line.split(None, 3)</code> to get the three &#8220;columns&#8221; and assign them to three local variables.
<li><em>And then you yield.</em> What do you yield? Two functions, built dynamically with your old friend, <code>build_match_and_apply_functions()</code>, which is identical to the previous examples. In other words, <code>rules()</code> is a generator that spits out match and apply functions <em>on demand</em>.
<li>Since <code>rules()</code> is a generator, you can use it directly in a <code>for</code> loop. The first time through the <code>for</code> loop, you will call the <code>rules()</code> function, which will open the pattern file, read the first line, dynamically build a match function and an apply function from the patterns on that line, and yield the dynamically built functions. The second time through the <code>for</code> loop, you will pick up exactly where you left off in <code>rules()</code> (which was in the middle of the <code>for line in pattern_file</code> loop). The first thing it will do is read the next line of the file (which is still open), dynamically build another match and apply function based on the patterns on that line in the file, and yield the two functions.
</ol>
<p>What have you gained over stage 4? Startup time. In stage 4, when you imported the <code>plural4</code> module, it read the entire patterns file and built a list of all the possible rules, before you could even think about calling the <code>plural()</code> function. With generators, you can do everything lazily: you read the first rule and create functions and try them, and if that works you don&#8217;t ever read the rest of the file or create any other functions.
<p>What have you lost? Performance! Every time you call the <code>plural()</code> function, the <code>rules()</code> generator starts over from the beginning&nbsp;&mdash;&nbsp;which means re-opening the patterns file and reading from the beginning, one line at a time.
<p>What if you could have the best of both worlds: minimal startup cost (don&#8217;t execute any code on <code>import</code>), <em>and</em> maximum performance (don&#8217;t build the same functions over and over again). Oh, and you still want to keep the rules in a separate file (because code is code and data is data), just as long as you never have to read the same line twice.
<p>To do that, you&#8217;ll need to build your own iterator. But before you do <em>that</em>, you need to learn about Python classes.
<p class=a>&#x2042;
<h2 id=furtherreading>Further Reading</h2>
<ul>
<li><a href=http://www.python.org/dev/peps/pep-0255/>PEP 255: Simple Generators</a>
<li><a href=http://effbot.org/zone/python-with-statement.htm>Understanding Python&#8217;s &#8220;with&#8221; statement</a>
<li><a href=http://ynniv.com/blog/2007/08/closures-in-python.html>Closures in Python</a>
<li><a href=http://en.wikipedia.org/wiki/Fibonacci_number>Fibonacci numbers</a>
<li><a href=http://www2.gsu.edu/~wwwesl/egw/crump.htm>English Irregular Plural Nouns</a>
</ul>
<p class=v><a href=regular-expressions.html rel=prev title='back to &#8220;Regular Expressions&#8221;'><span class=u>&#x261C;</span></a> <a href=iterators.html rel=next title='onward to &#8220;Classes &amp; Iterators&#8221;'><span class=u>&#x261E;</span></a>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/prettify.js></script>
<script src=j/dip3.js></script>
http-web-services.html
+1003 -1003
View File
File diff suppressed because it is too large Load Diff
installing-python.html
+364 -364
View File
@@ -1,364 +1,364 @@
<!DOCTYPE html>
<meta charset=utf-8>
<title>Installing Python - Dive Into Python 3</title>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<link rel=stylesheet href=dip3.css>
<style>
body{counter-reset:h1 0}
.i{list-style:none;margin:0;padding:0}
#which{padding-top:1.75em}
h2,.i>li{clear:both}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#installing-python>Dive Into Python 3</a> <span class=u>&#8227;</span>
<p id=level>Difficulty level: <span class=u title=novice>&#x2666;&#x2662;&#x2662;&#x2662;&#x2662;</span>
<h1>Installing Python</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> <i lang=la>Tempora mutantur nos et mutamur in illis.</i> (Times change, and we change with them.) <span class=u>&#x275E;</span><br>&mdash; ancient Roman proverb
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>Before you can start programming in Python 3, you need to install it. Or do you?
<h2 id=which>Which Python Is Right For You?</h2>
<p>If you're using an account on a hosted server, your <abbr>ISP</abbr> may have already installed Python 3. If you&#8217;re running Linux at home, you may already have Python 3, too. Most popular GNU/Linux distributions come with Python 2 in the default installation; a small but growing number of distributions also include Python 3. Mac OS X includes a command-line version of Python 2, but as of this writing it does not include Python 3. Microsoft Windows does not come with any version of Python. But don&#8217;t despair! You can point-and-click your way through installing Python, regardless of what operating system you have.
<p>The easiest way to check for Python 3 on your Linux or Mac OS X system is to get to a command line. On Linux, look in your <b><code>Applications</code></b> menu for a program called <b><code>Terminal</code></b>. (It may be in a submenu like <b><code>Accessories</code></b> or <b><code>System</code></b>.) On Mac OS X, there is an application called <b><code>Terminal.app</code></b> in your <code>/Application/Utilities/</code> folder.
<p>Once you&#8217;re at a command line prompt, just type <kbd>python3</kbd> (all lowercase, no spaces) and see what happens. On my home Linux system, Python 3 is already installed, and this command gets me into the <i>Python <dfn>interactive shell</dfn></i>.
<pre class='nd screen'>
<samp class=p>mark@atlantis:~$ </samp><kbd>python3</kbd>
<samp>Python 3.0.1+ (r301:69556, Apr 15 2009, 17:25:52)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>></samp></pre>
<p>(Type <kbd>exit()</kbd> and press <kbd>ENTER</kbd> to exit the Python interactive shell.)
<p>My <a href=http://cornerhost.com/>web hosting provider</a> also runs Linux and provides command-line access, but my server does not have Python 3 installed. (Boo!)
<pre class='nd screen'>
<samp class=p>mark@manganese:~$ </samp><kbd>python3</kbd>
<samp>bash: python3: command not found</samp></pre>
<p>So back to the question that started this section, &#8220;Which Python is right for you?&#8221; Whichever one runs on the computer you already have.
<p>[Read on for Windows instructions, or skip to <a href=#macosx>Installing on Mac OS X</a>, <a href=#ubuntu>Installing on Ubuntu Linux</a>, or <a href=#other>Installing on Other Platforms</a>.]
<p class=a>&#x2042;
<h2 id=windows>Installing on Microsoft Windows</h2>
<p>Windows comes in two architectures these days: 32-bit and 64-bit. Of course, there are lots of different <i>versions</i> of Windows&nbsp;&mdash;&nbsp;XP, Vista, Windows 7&nbsp;&mdash;&nbsp;but Python runs on all of them. The more important distinction is 32-bit v. 64-bit. If you have no idea what architecture you&#8217;re running, it&#8217;s probably 32-bit.
<p>Visit <a href=http://python.org/download/><code>python.org/download/</code></a> and download the appropriate Python 3 Windows installer for your architecture. Your choices will look something like this:
<ul>
<li><b>Python 3.1 Windows installer</b> (Windows binary&nbsp;&mdash;&nbsp;does not include source)
<li><b>Python 3.1 Windows AMD64 installer</b> (Windows AMD64 binary&nbsp;&mdash;&nbsp;does not include source)
</ul>
<p>I don&#8217;t want to include direct download links here, because minor updates of Python happen all the time and I don&#8217;t want to be responsible for you missing important updates. You should always install the most recent version of Python 3.x unless you have some esoteric reason not to.
<ol class=i>
<li>
<p class='ss nm'><img src=i/win-install-0-security-warning.png width=409 height=309 alt='[Windows dialog: open file security warning]'>
<p>Once your download is complete, double-click the <code>.msi</code> file. Windows will pop up a security alert, since you&#8217;re about to be running executable code. The official Python installer is digitally signed by the <a href=http://www.python.org/psf/>Python Software Foundation</a>, the non-profit corporation that oversees Python development. Don&#8217;t accept imitations!
<p>Click the <code>Run</code> button to launch the Python 3 installer.
<li>
<p class='ss nm'><img src=i/win-install-1-all-users-or-just-me.png width=499 height=432 alt='[Python installer: select whether to install Python 3.1 for all users of this computer]'>
<p>The first question the installer will ask you is whether you want to install Python 3 for all users or just for you. The default choice is &#8220;install for all users,&#8221; which is the best choice unless you have a good reason to choose otherwise. (One possible reason why you would want to &#8220;install just for me&#8221; is that you are installing Python on your company&#8217;s computer and you don&#8217;t have administrative rights on your Windows account. But then, why are you installing Python without permission from your company&#8217;s Windows administrator? Don&#8217;t get me in trouble here!)
<p>Click the <code>Next</code> button to accept your choice of installation type.
<li>
<p class='ss nm'><img src=i/win-install-2-destination-directory.png width=499 height=432 alt='[Python installer: select destination directory]'>
<p>Next, the installer will prompt you to choose a destination directory. The default for all versions of Python 3.1.x is <code>C:\Python31\</code>, which should work well for most users unless you have a specific reason to change it. If you maintain a separate drive letter for installing applications, you can browse to it using the embedded controls, or simply type the pathname in the box below. You are not limited to installing Python on the <code>C:</code> drive; you can install it on any drive, in any folder.
<p>Click the <code>Next</code> button to accept your choice of destination directory.
<li>
<p class='ss nm'><img src=i/win-install-3-customize.png width=499 height=432 alt='[Python installer: customize Python 3.1]'>
<p>The next page looks complicated, but it&#8217;s not really. Like many installers, you have the option not to install every single component of Python 3. If disk space is especially tight, you can exclude certain components.
<ul>
<li><b>Register Extensions</b> allows you to double-click Python scripts (<code>.py</code> files) and run them. Recommended but not required. (This option doesn&#8217;t require any disk space, so there is little point in excluding it.)
<li><b>Tcl/Tk</b> is the graphics library used by the Python Shell, which you will use throughout this book. I strongly recommend keeping this option.
<li><b>Documentation</b> installs a help file that contains much of the information on <a href=http://docs.python.org/><code>docs.python.org</code></a>. Recommended if you are on dialup or have limited Internet access.
<li><b>Utility Scripts</b> includes the <code>2to3.py</code> script which you&#8217;ll learn about <a href=case-study-porting-chardet-to-python-3.html>later in this book</a>. Required if you want to learn about migrating existing Python 2 code to Python 3. If you have no existing Python 2 code, you can skip this option.
<li><b>Test Suite</b> is a collection of scripts used to test the Python interpreter itself. We will not use it in this book, nor have I ever used it in the course of programming in Python. Completely optional.
</ul>
<li>
<p class='ss nm'><img src=i/win-install-3a-disk-usage.png width=499 height=432 alt='[Python installer: disk space requirements]'>
<p>If you&#8217;re unsure how much disk space you have, click the <code>Disk Usage</code> button. The installer will list your drive letters, compute how much space is available on each drive, and calculate how much would be left after installation.
<p>Click the <code>OK</code> button to return to the &#8220;Customizing Python&#8221; page.
<li>
<p class='ss nm'><img src=i/win-install-3b-test-suite.png width=499 height=432 alt='[Python installer: removing Test Suite option will save 7908KB on your hard drive]'>
<p>If you decide to exclude an option, select the drop-down button before the option and select &#8220;Entire feature will be unavailable.&#8221; For example, excluding the test suite will save you a whopping 7908<abbr>KB</abbr> of disk space.
<p>Click the <code>Next</code> button to accept your choice of options.
<li>
<p class='ss nm'><img src=i/win-install-4-copying.png width=499 height=432 alt='[Python installer: progress meter]'>
<p>The installer will copy all the necessary files to your chosen destination directory. (This happens so quickly, I had to try it three times to even get a screenshot of it!)
<li>
<p class='ss nm'><img src=i/win-install-5-finish.png width=499 height=432 alt='[Python installer: installation completed. Special Windows thanks to Mark Hammond, without whose years of freely shared Windows expertise, Python for Windows would still be Python for DOS.]'>
<p>Click the <code>Finish</code> button to exit the installer.
<li>
<p class='ss nm'><img src=i/win-interactive-shell.png width=677 height=715 alt='[Windows Python Shell, a graphical interactive shell for Python]'>
<p>In your <code>Start</code> menu, there should be a new item called <code>Python 3.1</code>. Within that, there is a program called <abbr>IDLE</abbr>. Select this item to run the interactive Python Shell.
</ol>
<p>[Skip to <a href=#idle>using the Python Shell</a>]
<p class=a>&#x2042;
<h2 id=macosx>Installing on Mac OS X</h2>
<p>All modern Macintosh computers use the Intel chip (like most Windows PCs). Older Macs used PowerPC chips. You don&#8217;t need to understand the difference, because there&#8217;s just one Mac Python installer for all Macs.
<p>Visit <a href=http://python.org/download/><code>python.org/download/</code></a> and download the Mac installer. It will be called something like <b>Python 3.1 Mac Installer Disk Image</b>, although the version number may vary. Be sure to download version 3.x, not 2.x.
<ol class=i>
<li>
<p class='ss nm'><img src=i/mac-install-0-dmg-contents.png width=752 height=438 alt='[contents of Python installer disk image]'>
<p>Your browser should automatically mount the disk image and open a Finder window to show you the contents. (If this doesn&#8217;t happen, you&#8217;ll need to find the disk image in your downloads folder and double-click to mount it. It will be named something like <code>python-3.1.dmg</code>.) The disk image contains a number of text files (<code>Build.txt</code>, <code>License.txt</code>, <code>ReadMe.txt</code>), and the actual installer package, <code>Python.mpkg</code>.
<p>Double-click the <code>Python.mpkg</code> installer package to launch the Mac Python installer.
<li>
<p class='ss nm'><img src=i/mac-install-1-welcome.png width=622 height=442 alt='[Python installer: welcome screen]'>
<p>The first page of the installer gives a brief description of Python itself, then refers you to the <code>ReadMe.txt</code> file (which you didn&#8217;t read, did you?) for more details.
<p>Click the <code>Continue</code> button to move along.
<li>
<p class='ss nm'><img src=i/mac-install-2-information.png width=622 height=442 alt='[Python installer: information about supported architectures, disk space, and acceptable destination folders]'>
<p>The next page actually contains some important information: Python requires Mac OS X 10.3 or later. If you are still running Mac OS X 10.2, you should really upgrade. Apple no longer provides security updates for your operating system, and your computer is probably at risk if you ever go online. Also, you can&#8217;t run Python 3.
<p>Click the <code>Continue</code> button to advance.
<li>
<p class='ss nm'><img src=i/mac-install-3-license.png width=622 height=442 alt='[Python installer: software license agreement]'>
<p>Like all good installers, the Python installer displays the software license agreement. Python is open source, and its license is <a href=http://opensource.org/licenses/>approved by the Open Source Initiative</a>. Python has had a number of owners and sponsors throughout its history, each of which has left its mark on the software license. But the end result is this: Python is open source, and you may use it on any platform, for any purpose, without fee or obligation of reciprocity.
<p>Click the <code>Continue</code> button once again.
<li>
<p class='ss nm'><img src=i/mac-install-4-license-dialog.png width=622 height=442 alt='[Python installer: dialog to accept license agreement]'>
<p>Due to quirks in the standard Apple installer framework, you must &#8220;agree&#8221; to the software license in order to complete the installation. Since Python is open source, you are really &#8220;agreeing&#8221; that the license is granting you additional rights, rather than taking them away.
<p>Click the <code>Agree</code> button to continue.
<li>
<p class='ss nm'><img src=i/mac-install-5-standard-install.png width=622 height=442 alt='[Python installer: standard install screen]'>
<p>The next screen allows you to change your install location. You <strong>must</strong> install Python on your boot drive, but due to limitations of the installer, it does not enforce this. In truth, I have never had the need to change the install location.
<p>From this screen, you can also customize the installation to exclude certain features. If you want to do this, click the <code>Customize</code> button; otherwise click the <code>Install</code> button.
<li>
<p class='ss nm'><img src=i/mac-install-6-custom-install.png width=622 height=442 alt='[Python installer: custom install screen]'>
<p>If you choose a Custom Install, the installer will present you with the following list of features:
<ul>
<li><b>Python Framework</b>. This is the guts of Python, and is both selected and disabled because it must be installed.
<li><b>GUI Applications</b> includes IDLE, the graphical Python Shell which you will use throughout this book. I strongly recommend keeping this option selected.
<li><b>UNIX command-line tools</b> includes the command-line <code>python3</code> application. I strongly recommend keeping this option, too.
<li><b>Python Documentation</b> contains much of the information on <a href=http://docs.python.org/><code>docs.python.org</code></a>. Recommended if you are on dialup or have limited Internet access.
<li><b>Shell profile updater</b> controls whether to update your shell profile (used in <code>Terminal.app</code>) to ensure that this version of Python is on the search path of your shell. You probably don&#8217;t need to change this.
<li><b>Fix system Python</b> should not be changed. (It tells your Mac to use Python 3 as the default Python for all scripts, including built-in system scripts from Apple. This would be very bad, since most of those scripts are written for Python 2, and they would fail to run properly under Python 3.)
</ul>
<p>Click the <code>Install</code> button to continue.
<li>
<p class='ss nm'><img src=i/mac-install-7-admin-password.png width=622 height=457 alt='[Python installer: dialog to enter administrative password]'>
<p>Because it installs system-wide frameworks and binaries in <code>/usr/local/bin/</code>, the installer will ask you for an administrative password. There is no way to install Mac Python without administrator privileges.
<p>Click the <code>OK</code> button to begin the installation.
<li>
<p class='ss nm'><img src=i/mac-install-8-progress.png width=622 height=442 alt='[Python installer: progress meter]'>
<p>The installer will display a progress meter while it installs the features you&#8217;ve selected.
<li>
<p class='ss nm'><img src=i/mac-install-9-succeeded.png width=622 height=442 alt='[Python installer: install succeeded]'>
<p>Assuming all went well, the installer will give you a big green checkmark to tell you that the installation completed successfully.
<p>Click the <code>Close</code> button to exit the installer.
<li>
<p class='ss nm'><img src=i/mac-install-10-application-folder.png width=488 height=482 alt='[contents of /Applications/Python 3.1/ folder]'>
<p>Assuming you didn&#8217;t change the install location, you can find the newly installed files in the <code>Python 3.1</code> folder within your <code>/Applications</code> folder. The most important piece is <abbr>IDLE</abbr>, the graphical Python Shell.
<p>Double-click <abbr>IDLE</abbr> to launch the Python Shell.
<li>
<p class='ss nm'><img src=i/mac-interactive-shell.png width=522 height=538 alt='[Mac Python Shell, a graphical interactive shell for Python]'>
<p>The Python Shell is where you will spend most of your time exploring Python. Examples throughout this book will assume that you can find your way into the Python Shell.
</ol>
<p>[Skip to <a href=#idle>using the Python Shell</a>]
<p class=a>&#x2042;
<h2 id=ubuntu>Installing on Ubuntu Linux</h2>
<p>Modern Linux distributions are backed by vast repositories of precompiled applications, ready to install. The exact details vary by distribution. In Ubuntu Linux, the easiest way to install Python 3 is through the <code>Add/Remove</code> application in your <code>Applications</code> menu.
<ol class=i>
<li>
<p class='ss nm'><img src=i/ubu-install-0-add-remove-programs.png width=920 height=473 alt='[Add/Remove: Canonical-maintained applications]'>
<p>When you first launch the <code>Add/Remove</code> application, it will show you a list of preselected applications in different categories. Some are already installed; most are not. Because the repository contains over 10,000 applications, there are different filters you can apply to see small parts of the repository. The default filter is &#8220;Canonical-maintained applications,&#8221; which is a small subset of the total number of applications that are officially supported by Canonical, the company that creates and maintains Ubuntu Linux.
<li>
<p class='ss nm'><img src=i/ubu-install-1-all-open-source-applications.png width=920 height=473 alt='[Add/Remove: all open source applications]'>
<p>Python 3 is not maintained by Canonical, so the first step is to drop down this filter menu and select &#8220;All Open Source applications.&#8221;
<li>
<p class='ss nm'><img src=i/ubu-install-2-search-python-3.png width=920 height=473 alt='[Add/Remove: search for Python 3]'>
<p>Once you&#8217;ve widened the filter to include all open source applications, use the Search box immediately after the filter menu to search for <kbd>Python 3</kbd>.
<li>
<p class='ss nm'><img src=i/ubu-install-3-select-python-3.png width=920 height=473 alt='[Add/Remove: select Python 3.0 package]'>
<p>Now the list of applications narrows to just those matching <kbd>Python 3</kbd>. You&#8217;re going to check two packages. The first is <code>Python (v3.0)</code>. This contains the Python interpreter itself.
<li>
<p class='ss nm'><img src=i/ubu-install-4-select-idle.png width=920 height=473 alt='[Add/Remove: select IDLE for Python 3.0 package]'>
<p>The second package you want is immediately above: <code>IDLE (using Python-3.0)</code>. This is a graphical Python Shell that you will use throughout this book.
<p>After you&#8217;ve checked those two packages, click the <code>Apply Changes</code> button to continue.
<li>
<p class='ss nm'><img src=i/ubu-install-5-apply-changes.png width=635 height=364 alt='[Add/Remove: apply changes]'>
<p>The package manager will ask you to confirm that you want to add both <code>IDLE (using Python-3.0)</code> and <code>Python (v3.0)</code>.
<p>Click the <code>Apply</code> button to continue.
<li>
<p class='ss nm'><img src=i/ubu-install-6-download-progress.png width=287 height=211 alt='[Add/Remove: download progress meter]'>
<p>The package manager will show you a progress meter while it downloads the necessary packages from Canonical&#8217;s Internet repository.
<li>
<p class='ss nm'><img src=i/ubu-install-7-install-progress.png width=486 height=258 alt='[Add/Remove: installation progress meter]'>
<p>Once the packages are downloaded, the package manager will automatically begin installing them.
<li>
<p class='ss nm'><img src=i/ubu-install-8-success.png width=591 height=296 alt='[Add/Remove: new applications have been installed]'>
<p>If all went well, the package manager will confirm that both packages were successfully installed. From here, you can double-click <abbr>IDLE</abbr> to launch the Python Shell, or click the <code>Close</code> button to exit the package manager.
<p>You can always relaunch the Python Shell by going to your <code>Applications</code> menu, then the <code>Programming</code> submenu, and selecting <abbr>IDLE</abbr>.
<li>
<p class='ss nm'><img src=i/ubu-interactive-shell.png width=679 height=687 alt='[Linux Python Shell, a graphical interactive shell for Python]'>
<p>The Python Shell is where you will spend most of your time exploring Python. Examples throughout this book will assume that you can find your way into the Python Shell.
</ol>
<p>[Skip to <a href=#idle>using the Python Shell</a>]
<p class=a>&#x2042;
<h2 id=other>Installing on Other Platforms</h2>
<p>Python 3 is available on a number of different platforms. In particular, it is available in virtually every Linux, <abbr>BSD</abbr>, and Solaris-based distribution. For example, RedHat Linux uses the <code>yum</code> package manager; FreeBSD has its <a href=http://www.freebsd.org/ports/>ports and packages collection</a>; Solaris has <code>pkgadd</code> and friends. A quick web search for <code>Python 3</code> + <i>your operating system</i> will tell you whether a Python 3 package is available, and how to install it.
<p class=a>&#x2042;
<h2 id=idle>Using The Python Shell</h2>
<p>The Python Shell is where you can explore Python syntax, get interactive help on commands, and debug short programs. The graphical Python Shell (named <abbr>IDLE</abbr>) also contains a decent text editor that supports Python syntax coloring and integrates with the Python Shell. If you don&#8217;t already have a favorite text editor, you should give <abbr>IDLE</abbr> a try.
<p>First things first. The Python Shell itself is an amazing interactive playground. Throughout this book, you&#8217;ll see examples like this:
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>1 + 1</kbd>
<samp class=pp>2</samp></pre>
<p>The three angle brackets, <samp class=p>>>></samp>, denote the Python Shell prompt. Don&#8217;t type that part. That&#8217;s just to let you know that this example is meant to be followed in the Python Shell.
<p><kbd class=pp>1 + 1</kbd> is the part you type. You can type any valid Python expression or command in the Python Shell. Don&#8217;t be shy; it won&#8217;t bite! The worst that will happen is you&#8217;ll get an error message. Commands get executed immediately (once you press <kbd>ENTER</kbd>); expressions get evaluated immediately, and the Python Shell prints out the result.
<p><samp class=pp>2</samp> is the result of evaluating this expression. As it happens, <kbd class=pp>1 + 1</kbd> is a valid Python expression. The result, of course, is <samp class=pp>2</samp>.
<p>Let&#8217;s try another one.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>print('Hello world!')</kbd>
<samp>Hello world!</samp>
</pre>
<p>Pretty simple, no? But there&#8217;s lots more you can do in the Python shell. If you ever get stuck&nbsp;&mdash;&nbsp;you can&#8217;t remember a command, or you can&#8217;t remember the proper arguments to pass a certain function&nbsp;&mdash;&nbsp;you can get interactive help in the Python Shell. Just type <kbd>help</kbd> and press <kbd>ENTER</kbd>.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd>help</kbd>
<samp>Type help() for interactive help, or help(object) for help about object.</samp></pre>
<p>There are two modes of help. You can get help about a single object, which just prints out the documentation and returns you to the Python Shell prompt. You can also enter <i>help mode</i>, where instead of evaluating Python expressions, you just type keywords or command names and it will print out whatever it knows about that command.
<p>To enter the interactive help mode, type <kbd>help()</kbd> and press <kbd>ENTER</kbd>.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>help()</kbd>
<samp>Welcome to Python 3.0! This is the online help utility.
If this is your first time using Python, you should definitely check out
the tutorial on the Internet at http://docs.python.org/tutorial/.
Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules. To quit this help utility and
return to the interpreter, just type "quit".
To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics". Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".
</samp>
<samp class=p>help> </samp></pre>
<p>Note how the prompt changes from <samp class=p>>>></samp> to <samp class=p>help></samp>. This reminds you that you&#8217;re in the interactive help mode. Now you can enter any keyword, command, module name, function name&nbsp;&mdash;&nbsp;pretty much anything Python understands&nbsp;&mdash;&nbsp;and read documentation on it.
<pre class=screen>
<a><samp class=p>help> </samp><kbd class=pp>print</kbd> <span class=u>&#x2460;</span></a>
<samp>Help on built-in function print in module builtins:
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout)
Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file: a file-like object (stream); defaults to the current sys.stdout.
sep: string inserted between values, default a space.
end: string appended after the last value, default a newline.
</samp>
<a><samp class=p>help> </samp><kbd class=pp>PapayaWhip</kbd> <span class=u>&#x2461;</span></a>
<samp>no Python documentation found for 'PapayaWhip'
</samp>
<a><samp class=p>help> </samp><kbd class=pp>quit</kbd> <span class=u>&#x2462;</span></a>
<samp>
You are now leaving help and returning to the Python interpreter.
If you want to ask for help on a particular object directly from the
interpreter, you can type "help(object)". Executing "help('string')"
has the same effect as typing a particular string at the help> prompt.</samp>
<a><samp class=p>>>> </samp> <span class=u>&#x2463;</span></a></pre>
<ol>
<li>To get documentation on the <code>print()</code> function, just type <kbd>print</kbd> and press <kbd>ENTER</kbd>. The interactive help mode will display something akin to a man page: the function name, a brief synopsis, the function&#8217;s arguments and their default values, and so on. If the documentation seems opaque to you, don&#8217;t panic. You&#8217;ll learn more about all these concepts in the next few chapters.
<li>Of course, the interactive help mode doesn&#8217;t know everything. If you type something that isn&#8217;t a Python command, module, function, or other built-in keyword, the interactive help mode will just shrug its virtual shoulders.
<li>To quit the interactive help mode, type <kbd>quit</kbd> and press <kbd>ENTER</kbd>.
<li>The prompt changes back to <samp class=p>>>></samp> to signal that you&#8217;ve left the interactive help mode and returned to the Python Shell.
</ol>
<p><abbr>IDLE</abbr>, the graphical Python Shell, also includes a Python-aware text editor.
<p class=a>&#x2042;
<h2 id=editors>Python Editors and IDEs</h2>
<p><abbr>IDLE</abbr> is not the only game in town when it comes to writing programs in Python. While it&#8217;s useful to get started with learning the language itself, many developers prefer other text editors or Integrated Development Environments (<abbr>IDE</abbr>s). I won&#8217;t cover them here, but the Python community maintains <a href=http://wiki.python.org/moin/PythonEditors>a list of Python-aware editors</a> that covers a wide range of supported platforms and software licenses.
<p>You might also want to check out the <a href=http://wiki.python.org/moin/IntegratedDevelopmentEnvironments>list of Python-aware <abbr>IDE</abbr>s</a>, although few of them support Python 3 yet. One that does is <a href=http://pydev.sourceforge.net/>PyDev</a>, a plugin for <a href=http://eclipse.org/>Eclipse</a> that turns Eclipse into a full-fledged Python <abbr>IDE</abbr>. Both Eclipse and PyDev are cross-platform and open source.
<p>On the commercial front, there is ActiveState&#8217;s <a href=http://www.activestate.com/komodo/>Komodo <abbr>IDE</abbr></a>. It has per-user licensing, but students can get a discount, and a free time-limited trial version is available.
<p>I&#8217;ve been programming in Python for nine years, and I edit my Python programs in <a href=http://www.gnu.org/software/emacs/>GNU Emacs</a> and debug them in the command-line Python Shell. There&#8217;s no right or wrong way to develop in Python. Find a way that works for you!
<p class=v><a href=whats-new.html rel=prev title='back to &#8220;What&#8217;s New In Dive Into Python 3&#8221;'><span class=u>&#x261C;</span></a> <a href=your-first-python-program.html rel=next title='onward to &#8220;Your First Python Program&#8221;'><span class=u>&#x261E;</span></a>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/prettify.js></script>
<script src=j/dip3.js></script>
<!DOCTYPE html>
<meta charset=utf-8>
<title>Installing Python - Dive Into Python 3</title>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<link rel=stylesheet href=dip3.css>
<style>
body{counter-reset:h1 0}
.i{list-style:none;margin:0;padding:0}
#which{padding-top:1.75em}
h2,.i>li{clear:both}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#installing-python>Dive Into Python 3</a> <span class=u>&#8227;</span>
<p id=level>Difficulty level: <span class=u title=novice>&#x2666;&#x2662;&#x2662;&#x2662;&#x2662;</span>
<h1>Installing Python</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> <i lang=la>Tempora mutantur nos et mutamur in illis.</i> (Times change, and we change with them.) <span class=u>&#x275E;</span><br>&mdash; ancient Roman proverb
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>Before you can start programming in Python 3, you need to install it. Or do you?
<h2 id=which>Which Python Is Right For You?</h2>
<p>If you're using an account on a hosted server, your <abbr>ISP</abbr> may have already installed Python 3. If you&#8217;re running Linux at home, you may already have Python 3, too. Most popular GNU/Linux distributions come with Python 2 in the default installation; a small but growing number of distributions also include Python 3. Mac OS X includes a command-line version of Python 2, but as of this writing it does not include Python 3. Microsoft Windows does not come with any version of Python. But don&#8217;t despair! You can point-and-click your way through installing Python, regardless of what operating system you have.
<p>The easiest way to check for Python 3 on your Linux or Mac OS X system is to get to a command line. On Linux, look in your <b><code>Applications</code></b> menu for a program called <b><code>Terminal</code></b>. (It may be in a submenu like <b><code>Accessories</code></b> or <b><code>System</code></b>.) On Mac OS X, there is an application called <b><code>Terminal.app</code></b> in your <code>/Application/Utilities/</code> folder.
<p>Once you&#8217;re at a command line prompt, just type <kbd>python3</kbd> (all lowercase, no spaces) and see what happens. On my home Linux system, Python 3 is already installed, and this command gets me into the <i>Python <dfn>interactive shell</dfn></i>.
<pre class='nd screen'>
<samp class=p>mark@atlantis:~$ </samp><kbd>python3</kbd>
<samp>Python 3.0.1+ (r301:69556, Apr 15 2009, 17:25:52)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>></samp></pre>
<p>(Type <kbd>exit()</kbd> and press <kbd>ENTER</kbd> to exit the Python interactive shell.)
<p>My <a href=http://cornerhost.com/>web hosting provider</a> also runs Linux and provides command-line access, but my server does not have Python 3 installed. (Boo!)
<pre class='nd screen'>
<samp class=p>mark@manganese:~$ </samp><kbd>python3</kbd>
<samp>bash: python3: command not found</samp></pre>
<p>So back to the question that started this section, &#8220;Which Python is right for you?&#8221; Whichever one runs on the computer you already have.
<p>[Read on for Windows instructions, or skip to <a href=#macosx>Installing on Mac OS X</a>, <a href=#ubuntu>Installing on Ubuntu Linux</a>, or <a href=#other>Installing on Other Platforms</a>.]
<p class=a>&#x2042;
<h2 id=windows>Installing on Microsoft Windows</h2>
<p>Windows comes in two architectures these days: 32-bit and 64-bit. Of course, there are lots of different <i>versions</i> of Windows&nbsp;&mdash;&nbsp;XP, Vista, Windows 7&nbsp;&mdash;&nbsp;but Python runs on all of them. The more important distinction is 32-bit v. 64-bit. If you have no idea what architecture you&#8217;re running, it&#8217;s probably 32-bit.
<p>Visit <a href=http://python.org/download/><code>python.org/download/</code></a> and download the appropriate Python 3 Windows installer for your architecture. Your choices will look something like this:
<ul>
<li><b>Python 3.1 Windows installer</b> (Windows binary&nbsp;&mdash;&nbsp;does not include source)
<li><b>Python 3.1 Windows AMD64 installer</b> (Windows AMD64 binary&nbsp;&mdash;&nbsp;does not include source)
</ul>
<p>I don&#8217;t want to include direct download links here, because minor updates of Python happen all the time and I don&#8217;t want to be responsible for you missing important updates. You should always install the most recent version of Python 3.x unless you have some esoteric reason not to.
<ol class=i>
<li>
<p class='ss nm'><img src=i/win-install-0-security-warning.png width=409 height=309 alt='[Windows dialog: open file security warning]'>
<p>Once your download is complete, double-click the <code>.msi</code> file. Windows will pop up a security alert, since you&#8217;re about to be running executable code. The official Python installer is digitally signed by the <a href=http://www.python.org/psf/>Python Software Foundation</a>, the non-profit corporation that oversees Python development. Don&#8217;t accept imitations!
<p>Click the <code>Run</code> button to launch the Python 3 installer.
<li>
<p class='ss nm'><img src=i/win-install-1-all-users-or-just-me.png width=499 height=432 alt='[Python installer: select whether to install Python 3.1 for all users of this computer]'>
<p>The first question the installer will ask you is whether you want to install Python 3 for all users or just for you. The default choice is &#8220;install for all users,&#8221; which is the best choice unless you have a good reason to choose otherwise. (One possible reason why you would want to &#8220;install just for me&#8221; is that you are installing Python on your company&#8217;s computer and you don&#8217;t have administrative rights on your Windows account. But then, why are you installing Python without permission from your company&#8217;s Windows administrator? Don&#8217;t get me in trouble here!)
<p>Click the <code>Next</code> button to accept your choice of installation type.
<li>
<p class='ss nm'><img src=i/win-install-2-destination-directory.png width=499 height=432 alt='[Python installer: select destination directory]'>
<p>Next, the installer will prompt you to choose a destination directory. The default for all versions of Python 3.1.x is <code>C:\Python31\</code>, which should work well for most users unless you have a specific reason to change it. If you maintain a separate drive letter for installing applications, you can browse to it using the embedded controls, or simply type the pathname in the box below. You are not limited to installing Python on the <code>C:</code> drive; you can install it on any drive, in any folder.
<p>Click the <code>Next</code> button to accept your choice of destination directory.
<li>
<p class='ss nm'><img src=i/win-install-3-customize.png width=499 height=432 alt='[Python installer: customize Python 3.1]'>
<p>The next page looks complicated, but it&#8217;s not really. Like many installers, you have the option not to install every single component of Python 3. If disk space is especially tight, you can exclude certain components.
<ul>
<li><b>Register Extensions</b> allows you to double-click Python scripts (<code>.py</code> files) and run them. Recommended but not required. (This option doesn&#8217;t require any disk space, so there is little point in excluding it.)
<li><b>Tcl/Tk</b> is the graphics library used by the Python Shell, which you will use throughout this book. I strongly recommend keeping this option.
<li><b>Documentation</b> installs a help file that contains much of the information on <a href=http://docs.python.org/><code>docs.python.org</code></a>. Recommended if you are on dialup or have limited Internet access.
<li><b>Utility Scripts</b> includes the <code>2to3.py</code> script which you&#8217;ll learn about <a href=case-study-porting-chardet-to-python-3.html>later in this book</a>. Required if you want to learn about migrating existing Python 2 code to Python 3. If you have no existing Python 2 code, you can skip this option.
<li><b>Test Suite</b> is a collection of scripts used to test the Python interpreter itself. We will not use it in this book, nor have I ever used it in the course of programming in Python. Completely optional.
</ul>
<li>
<p class='ss nm'><img src=i/win-install-3a-disk-usage.png width=499 height=432 alt='[Python installer: disk space requirements]'>
<p>If you&#8217;re unsure how much disk space you have, click the <code>Disk Usage</code> button. The installer will list your drive letters, compute how much space is available on each drive, and calculate how much would be left after installation.
<p>Click the <code>OK</code> button to return to the &#8220;Customizing Python&#8221; page.
<li>
<p class='ss nm'><img src=i/win-install-3b-test-suite.png width=499 height=432 alt='[Python installer: removing Test Suite option will save 7908KB on your hard drive]'>
<p>If you decide to exclude an option, select the drop-down button before the option and select &#8220;Entire feature will be unavailable.&#8221; For example, excluding the test suite will save you a whopping 7908<abbr>KB</abbr> of disk space.
<p>Click the <code>Next</code> button to accept your choice of options.
<li>
<p class='ss nm'><img src=i/win-install-4-copying.png width=499 height=432 alt='[Python installer: progress meter]'>
<p>The installer will copy all the necessary files to your chosen destination directory. (This happens so quickly, I had to try it three times to even get a screenshot of it!)
<li>
<p class='ss nm'><img src=i/win-install-5-finish.png width=499 height=432 alt='[Python installer: installation completed. Special Windows thanks to Mark Hammond, without whose years of freely shared Windows expertise, Python for Windows would still be Python for DOS.]'>
<p>Click the <code>Finish</code> button to exit the installer.
<li>
<p class='ss nm'><img src=i/win-interactive-shell.png width=677 height=715 alt='[Windows Python Shell, a graphical interactive shell for Python]'>
<p>In your <code>Start</code> menu, there should be a new item called <code>Python 3.1</code>. Within that, there is a program called <abbr>IDLE</abbr>. Select this item to run the interactive Python Shell.
</ol>
<p>[Skip to <a href=#idle>using the Python Shell</a>]
<p class=a>&#x2042;
<h2 id=macosx>Installing on Mac OS X</h2>
<p>All modern Macintosh computers use the Intel chip (like most Windows PCs). Older Macs used PowerPC chips. You don&#8217;t need to understand the difference, because there&#8217;s just one Mac Python installer for all Macs.
<p>Visit <a href=http://python.org/download/><code>python.org/download/</code></a> and download the Mac installer. It will be called something like <b>Python 3.1 Mac Installer Disk Image</b>, although the version number may vary. Be sure to download version 3.x, not 2.x.
<ol class=i>
<li>
<p class='ss nm'><img src=i/mac-install-0-dmg-contents.png width=752 height=438 alt='[contents of Python installer disk image]'>
<p>Your browser should automatically mount the disk image and open a Finder window to show you the contents. (If this doesn&#8217;t happen, you&#8217;ll need to find the disk image in your downloads folder and double-click to mount it. It will be named something like <code>python-3.1.dmg</code>.) The disk image contains a number of text files (<code>Build.txt</code>, <code>License.txt</code>, <code>ReadMe.txt</code>), and the actual installer package, <code>Python.mpkg</code>.
<p>Double-click the <code>Python.mpkg</code> installer package to launch the Mac Python installer.
<li>
<p class='ss nm'><img src=i/mac-install-1-welcome.png width=622 height=442 alt='[Python installer: welcome screen]'>
<p>The first page of the installer gives a brief description of Python itself, then refers you to the <code>ReadMe.txt</code> file (which you didn&#8217;t read, did you?) for more details.
<p>Click the <code>Continue</code> button to move along.
<li>
<p class='ss nm'><img src=i/mac-install-2-information.png width=622 height=442 alt='[Python installer: information about supported architectures, disk space, and acceptable destination folders]'>
<p>The next page actually contains some important information: Python requires Mac OS X 10.3 or later. If you are still running Mac OS X 10.2, you should really upgrade. Apple no longer provides security updates for your operating system, and your computer is probably at risk if you ever go online. Also, you can&#8217;t run Python 3.
<p>Click the <code>Continue</code> button to advance.
<li>
<p class='ss nm'><img src=i/mac-install-3-license.png width=622 height=442 alt='[Python installer: software license agreement]'>
<p>Like all good installers, the Python installer displays the software license agreement. Python is open source, and its license is <a href=http://opensource.org/licenses/>approved by the Open Source Initiative</a>. Python has had a number of owners and sponsors throughout its history, each of which has left its mark on the software license. But the end result is this: Python is open source, and you may use it on any platform, for any purpose, without fee or obligation of reciprocity.
<p>Click the <code>Continue</code> button once again.
<li>
<p class='ss nm'><img src=i/mac-install-4-license-dialog.png width=622 height=442 alt='[Python installer: dialog to accept license agreement]'>
<p>Due to quirks in the standard Apple installer framework, you must &#8220;agree&#8221; to the software license in order to complete the installation. Since Python is open source, you are really &#8220;agreeing&#8221; that the license is granting you additional rights, rather than taking them away.
<p>Click the <code>Agree</code> button to continue.
<li>
<p class='ss nm'><img src=i/mac-install-5-standard-install.png width=622 height=442 alt='[Python installer: standard install screen]'>
<p>The next screen allows you to change your install location. You <strong>must</strong> install Python on your boot drive, but due to limitations of the installer, it does not enforce this. In truth, I have never had the need to change the install location.
<p>From this screen, you can also customize the installation to exclude certain features. If you want to do this, click the <code>Customize</code> button; otherwise click the <code>Install</code> button.
<li>
<p class='ss nm'><img src=i/mac-install-6-custom-install.png width=622 height=442 alt='[Python installer: custom install screen]'>
<p>If you choose a Custom Install, the installer will present you with the following list of features:
<ul>
<li><b>Python Framework</b>. This is the guts of Python, and is both selected and disabled because it must be installed.
<li><b>GUI Applications</b> includes IDLE, the graphical Python Shell which you will use throughout this book. I strongly recommend keeping this option selected.
<li><b>UNIX command-line tools</b> includes the command-line <code>python3</code> application. I strongly recommend keeping this option, too.
<li><b>Python Documentation</b> contains much of the information on <a href=http://docs.python.org/><code>docs.python.org</code></a>. Recommended if you are on dialup or have limited Internet access.
<li><b>Shell profile updater</b> controls whether to update your shell profile (used in <code>Terminal.app</code>) to ensure that this version of Python is on the search path of your shell. You probably don&#8217;t need to change this.
<li><b>Fix system Python</b> should not be changed. (It tells your Mac to use Python 3 as the default Python for all scripts, including built-in system scripts from Apple. This would be very bad, since most of those scripts are written for Python 2, and they would fail to run properly under Python 3.)
</ul>
<p>Click the <code>Install</code> button to continue.
<li>
<p class='ss nm'><img src=i/mac-install-7-admin-password.png width=622 height=457 alt='[Python installer: dialog to enter administrative password]'>
<p>Because it installs system-wide frameworks and binaries in <code>/usr/local/bin/</code>, the installer will ask you for an administrative password. There is no way to install Mac Python without administrator privileges.
<p>Click the <code>OK</code> button to begin the installation.
<li>
<p class='ss nm'><img src=i/mac-install-8-progress.png width=622 height=442 alt='[Python installer: progress meter]'>
<p>The installer will display a progress meter while it installs the features you&#8217;ve selected.
<li>
<p class='ss nm'><img src=i/mac-install-9-succeeded.png width=622 height=442 alt='[Python installer: install succeeded]'>
<p>Assuming all went well, the installer will give you a big green checkmark to tell you that the installation completed successfully.
<p>Click the <code>Close</code> button to exit the installer.
<li>
<p class='ss nm'><img src=i/mac-install-10-application-folder.png width=488 height=482 alt='[contents of /Applications/Python 3.1/ folder]'>
<p>Assuming you didn&#8217;t change the install location, you can find the newly installed files in the <code>Python 3.1</code> folder within your <code>/Applications</code> folder. The most important piece is <abbr>IDLE</abbr>, the graphical Python Shell.
<p>Double-click <abbr>IDLE</abbr> to launch the Python Shell.
<li>
<p class='ss nm'><img src=i/mac-interactive-shell.png width=522 height=538 alt='[Mac Python Shell, a graphical interactive shell for Python]'>
<p>The Python Shell is where you will spend most of your time exploring Python. Examples throughout this book will assume that you can find your way into the Python Shell.
</ol>
<p>[Skip to <a href=#idle>using the Python Shell</a>]
<p class=a>&#x2042;
<h2 id=ubuntu>Installing on Ubuntu Linux</h2>
<p>Modern Linux distributions are backed by vast repositories of precompiled applications, ready to install. The exact details vary by distribution. In Ubuntu Linux, the easiest way to install Python 3 is through the <code>Add/Remove</code> application in your <code>Applications</code> menu.
<ol class=i>
<li>
<p class='ss nm'><img src=i/ubu-install-0-add-remove-programs.png width=920 height=473 alt='[Add/Remove: Canonical-maintained applications]'>
<p>When you first launch the <code>Add/Remove</code> application, it will show you a list of preselected applications in different categories. Some are already installed; most are not. Because the repository contains over 10,000 applications, there are different filters you can apply to see small parts of the repository. The default filter is &#8220;Canonical-maintained applications,&#8221; which is a small subset of the total number of applications that are officially supported by Canonical, the company that creates and maintains Ubuntu Linux.
<li>
<p class='ss nm'><img src=i/ubu-install-1-all-open-source-applications.png width=920 height=473 alt='[Add/Remove: all open source applications]'>
<p>Python 3 is not maintained by Canonical, so the first step is to drop down this filter menu and select &#8220;All Open Source applications.&#8221;
<li>
<p class='ss nm'><img src=i/ubu-install-2-search-python-3.png width=920 height=473 alt='[Add/Remove: search for Python 3]'>
<p>Once you&#8217;ve widened the filter to include all open source applications, use the Search box immediately after the filter menu to search for <kbd>Python 3</kbd>.
<li>
<p class='ss nm'><img src=i/ubu-install-3-select-python-3.png width=920 height=473 alt='[Add/Remove: select Python 3.0 package]'>
<p>Now the list of applications narrows to just those matching <kbd>Python 3</kbd>. You&#8217;re going to check two packages. The first is <code>Python (v3.0)</code>. This contains the Python interpreter itself.
<li>
<p class='ss nm'><img src=i/ubu-install-4-select-idle.png width=920 height=473 alt='[Add/Remove: select IDLE for Python 3.0 package]'>
<p>The second package you want is immediately above: <code>IDLE (using Python-3.0)</code>. This is a graphical Python Shell that you will use throughout this book.
<p>After you&#8217;ve checked those two packages, click the <code>Apply Changes</code> button to continue.
<li>
<p class='ss nm'><img src=i/ubu-install-5-apply-changes.png width=635 height=364 alt='[Add/Remove: apply changes]'>
<p>The package manager will ask you to confirm that you want to add both <code>IDLE (using Python-3.0)</code> and <code>Python (v3.0)</code>.
<p>Click the <code>Apply</code> button to continue.
<li>
<p class='ss nm'><img src=i/ubu-install-6-download-progress.png width=287 height=211 alt='[Add/Remove: download progress meter]'>
<p>The package manager will show you a progress meter while it downloads the necessary packages from Canonical&#8217;s Internet repository.
<li>
<p class='ss nm'><img src=i/ubu-install-7-install-progress.png width=486 height=258 alt='[Add/Remove: installation progress meter]'>
<p>Once the packages are downloaded, the package manager will automatically begin installing them.
<li>
<p class='ss nm'><img src=i/ubu-install-8-success.png width=591 height=296 alt='[Add/Remove: new applications have been installed]'>
<p>If all went well, the package manager will confirm that both packages were successfully installed. From here, you can double-click <abbr>IDLE</abbr> to launch the Python Shell, or click the <code>Close</code> button to exit the package manager.
<p>You can always relaunch the Python Shell by going to your <code>Applications</code> menu, then the <code>Programming</code> submenu, and selecting <abbr>IDLE</abbr>.
<li>
<p class='ss nm'><img src=i/ubu-interactive-shell.png width=679 height=687 alt='[Linux Python Shell, a graphical interactive shell for Python]'>
<p>The Python Shell is where you will spend most of your time exploring Python. Examples throughout this book will assume that you can find your way into the Python Shell.
</ol>
<p>[Skip to <a href=#idle>using the Python Shell</a>]
<p class=a>&#x2042;
<h2 id=other>Installing on Other Platforms</h2>
<p>Python 3 is available on a number of different platforms. In particular, it is available in virtually every Linux, <abbr>BSD</abbr>, and Solaris-based distribution. For example, RedHat Linux uses the <code>yum</code> package manager; FreeBSD has its <a href=http://www.freebsd.org/ports/>ports and packages collection</a>; Solaris has <code>pkgadd</code> and friends. A quick web search for <code>Python 3</code> + <i>your operating system</i> will tell you whether a Python 3 package is available, and how to install it.
<p class=a>&#x2042;
<h2 id=idle>Using The Python Shell</h2>
<p>The Python Shell is where you can explore Python syntax, get interactive help on commands, and debug short programs. The graphical Python Shell (named <abbr>IDLE</abbr>) also contains a decent text editor that supports Python syntax coloring and integrates with the Python Shell. If you don&#8217;t already have a favorite text editor, you should give <abbr>IDLE</abbr> a try.
<p>First things first. The Python Shell itself is an amazing interactive playground. Throughout this book, you&#8217;ll see examples like this:
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>1 + 1</kbd>
<samp class=pp>2</samp></pre>
<p>The three angle brackets, <samp class=p>>>></samp>, denote the Python Shell prompt. Don&#8217;t type that part. That&#8217;s just to let you know that this example is meant to be followed in the Python Shell.
<p><kbd class=pp>1 + 1</kbd> is the part you type. You can type any valid Python expression or command in the Python Shell. Don&#8217;t be shy; it won&#8217;t bite! The worst that will happen is you&#8217;ll get an error message. Commands get executed immediately (once you press <kbd>ENTER</kbd>); expressions get evaluated immediately, and the Python Shell prints out the result.
<p><samp class=pp>2</samp> is the result of evaluating this expression. As it happens, <kbd class=pp>1 + 1</kbd> is a valid Python expression. The result, of course, is <samp class=pp>2</samp>.
<p>Let&#8217;s try another one.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>print('Hello world!')</kbd>
<samp>Hello world!</samp>
</pre>
<p>Pretty simple, no? But there&#8217;s lots more you can do in the Python shell. If you ever get stuck&nbsp;&mdash;&nbsp;you can&#8217;t remember a command, or you can&#8217;t remember the proper arguments to pass a certain function&nbsp;&mdash;&nbsp;you can get interactive help in the Python Shell. Just type <kbd>help</kbd> and press <kbd>ENTER</kbd>.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd>help</kbd>
<samp>Type help() for interactive help, or help(object) for help about object.</samp></pre>
<p>There are two modes of help. You can get help about a single object, which just prints out the documentation and returns you to the Python Shell prompt. You can also enter <i>help mode</i>, where instead of evaluating Python expressions, you just type keywords or command names and it will print out whatever it knows about that command.
<p>To enter the interactive help mode, type <kbd>help()</kbd> and press <kbd>ENTER</kbd>.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>help()</kbd>
<samp>Welcome to Python 3.0! This is the online help utility.
If this is your first time using Python, you should definitely check out
the tutorial on the Internet at http://docs.python.org/tutorial/.
Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules. To quit this help utility and
return to the interpreter, just type "quit".
To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics". Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".
</samp>
<samp class=p>help> </samp></pre>
<p>Note how the prompt changes from <samp class=p>>>></samp> to <samp class=p>help></samp>. This reminds you that you&#8217;re in the interactive help mode. Now you can enter any keyword, command, module name, function name&nbsp;&mdash;&nbsp;pretty much anything Python understands&nbsp;&mdash;&nbsp;and read documentation on it.
<pre class=screen>
<a><samp class=p>help> </samp><kbd class=pp>print</kbd> <span class=u>&#x2460;</span></a>
<samp>Help on built-in function print in module builtins:
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout)
Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file: a file-like object (stream); defaults to the current sys.stdout.
sep: string inserted between values, default a space.
end: string appended after the last value, default a newline.
</samp>
<a><samp class=p>help> </samp><kbd class=pp>PapayaWhip</kbd> <span class=u>&#x2461;</span></a>
<samp>no Python documentation found for 'PapayaWhip'
</samp>
<a><samp class=p>help> </samp><kbd class=pp>quit</kbd> <span class=u>&#x2462;</span></a>
<samp>
You are now leaving help and returning to the Python interpreter.
If you want to ask for help on a particular object directly from the
interpreter, you can type "help(object)". Executing "help('string')"
has the same effect as typing a particular string at the help> prompt.</samp>
<a><samp class=p>>>> </samp> <span class=u>&#x2463;</span></a></pre>
<ol>
<li>To get documentation on the <code>print()</code> function, just type <kbd>print</kbd> and press <kbd>ENTER</kbd>. The interactive help mode will display something akin to a man page: the function name, a brief synopsis, the function&#8217;s arguments and their default values, and so on. If the documentation seems opaque to you, don&#8217;t panic. You&#8217;ll learn more about all these concepts in the next few chapters.
<li>Of course, the interactive help mode doesn&#8217;t know everything. If you type something that isn&#8217;t a Python command, module, function, or other built-in keyword, the interactive help mode will just shrug its virtual shoulders.
<li>To quit the interactive help mode, type <kbd>quit</kbd> and press <kbd>ENTER</kbd>.
<li>The prompt changes back to <samp class=p>>>></samp> to signal that you&#8217;ve left the interactive help mode and returned to the Python Shell.
</ol>
<p><abbr>IDLE</abbr>, the graphical Python Shell, also includes a Python-aware text editor.
<p class=a>&#x2042;
<h2 id=editors>Python Editors and IDEs</h2>
<p><abbr>IDLE</abbr> is not the only game in town when it comes to writing programs in Python. While it&#8217;s useful to get started with learning the language itself, many developers prefer other text editors or Integrated Development Environments (<abbr>IDE</abbr>s). I won&#8217;t cover them here, but the Python community maintains <a href=http://wiki.python.org/moin/PythonEditors>a list of Python-aware editors</a> that covers a wide range of supported platforms and software licenses.
<p>You might also want to check out the <a href=http://wiki.python.org/moin/IntegratedDevelopmentEnvironments>list of Python-aware <abbr>IDE</abbr>s</a>, although few of them support Python 3 yet. One that does is <a href=http://pydev.sourceforge.net/>PyDev</a>, a plugin for <a href=http://eclipse.org/>Eclipse</a> that turns Eclipse into a full-fledged Python <abbr>IDE</abbr>. Both Eclipse and PyDev are cross-platform and open source.
<p>On the commercial front, there is ActiveState&#8217;s <a href=http://www.activestate.com/komodo/>Komodo <abbr>IDE</abbr></a>. It has per-user licensing, but students can get a discount, and a free time-limited trial version is available.
<p>I&#8217;ve been programming in Python for nine years, and I edit my Python programs in <a href=http://www.gnu.org/software/emacs/>GNU Emacs</a> and debug them in the command-line Python Shell. There&#8217;s no right or wrong way to develop in Python. Find a way that works for you!
<p class=v><a href=whats-new.html rel=prev title='back to &#8220;What&#8217;s New In Dive Into Python 3&#8221;'><span class=u>&#x261C;</span></a> <a href=your-first-python-program.html rel=next title='onward to &#8220;Your First Python Program&#8221;'><span class=u>&#x261E;</span></a>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/prettify.js></script>
<script src=j/dip3.js></script>
iterators.html
+394 -394
View File
@@ -1,394 +1,394 @@
<!DOCTYPE html>
<meta charset=utf-8>
<title>Classes &amp; Iterators - Dive Into Python 3</title>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<link rel=stylesheet href=dip3.css>
<style>
body{counter-reset:h1 7}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#iterators>Dive Into Python 3</a> <span class=u>&#8227;</span>
<p id=level>Difficulty level: <span class=u title=intermediate>&#x2666;&#x2666;&#x2666;&#x2662;&#x2662;</span>
<h1>Classes <i class=baa>&amp;</i> Iterators</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> East is East, and West is West, and never the twain shall meet. <span class=u>&#x275E;</span><br>&mdash; <a href=http://en.wikiquote.org/wiki/Rudyard_Kipling>Rudyard Kipling</a>
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>Iterators are the &#8220;secret sauce&#8221; of Python 3. They&#8217;re everywhere, underlying everything, always just out of sight. <a href=comprehensions.html>Comprehensions</a> are just a simple form of <i>iterators</i>. Generators are just a simple form of <i>iterators</i>. A function that <code>yield</code>s values is a nice, compact way of building an iterator without building an iterator. Let me show you what I mean by that.
<p>Remember <a href=generators.html#a-fibonacci-generator>the Fibonacci generator</a>? Here it is as a built-from-scratch iterator:
<p class=d>[<a href=examples/fibonacci2.py>download <code>fibonacci2.py</code></a>]
<pre class=pp><code>class Fib:
'''iterator that yields numbers in the Fibonacci sequence'''
def __init__(self, max):
self.max = max
def __iter__(self):
self.a = 0
self.b = 1
return self
def __next__(self):
fib = self.a
if fib > self.max:
raise StopIteration
self.a, self.b = self.b, self.a + self.b
return fib</code></pre>
<p>Let&#8217;s take that one line at a time.
<pre class='nd pp'><code>class Fib:</code></pre>
<p><code>class</code>? What&#8217;s a class?
<p class=a>&#x2042;
<h2 id=defining-classes>Defining Classes</h2>
<p>Python is fully object-oriented: you can define your own classes, inherit from your own or built-in classes, and instantiate the classes you&#8217;ve defined.
<p>Defining a class in Python is simple. As with functions, there is no separate interface definition. Just define the class and start coding. A Python class starts with the reserved word <code>class</code>, followed by the class name. Technically, that&#8217;s all that&#8217;s required, since a class doesn&#8217;t need to inherit from any other class.
<pre class=pp><code><a>class PapayaWhip: <span class=u>&#x2460;</span></a>
<a> pass <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li>The name of this class is <code>PapayaWhip</code>, and it doesn&#8217;t inherit from any other class. Class names are usually capitalized, <code>EachWordLikeThis</code>, but this is only a convention, not a requirement.
<li>You probably guessed this, but everything in a class is indented, just like the code within a function, <code>if</code> statement, <code>for</code> loop, or any other block of code. The first line not indented is outside the class.
</ol>
<p>This <code>PapayaWhip</code> class doesn&#8217;t define any methods or attributes, but syntactically, there needs to be something in the definition, thus the <code>pass</code> statement. This is a Python reserved word that just means &#8220;move along, nothing to see here&#8221;. It&#8217;s a statement that does nothing, and it&#8217;s a good placeholder when you&#8217;re stubbing out functions or classes.
<blockquote class='note compare java'>
<p><span class=u>&#x261E;</span>The <code>pass</code> statement in Python is like a empty set of curly braces (<code>{}</code>) in Java or C.
</blockquote>
<p>Many classes are inherited from other classes, but this one is not. Many classes define methods, but this one does not. There is nothing that a Python class absolutely must have, other than a name. In particular, C++ programmers may find it odd that Python classes don&#8217;t have explicit constructors and destructors. Although it&#8217;s not required, Python classes <em>can</em> have something similar to a constructor: the <code>__init__()</code> method.
<h3 id=init-method>The <code>__init__()</code> Method</h3>
<p>This example shows the initialization of the <code>Fib</code> class using the <code>__init__</code> method.
<pre class=pp><code>class Fib:
<a> '''iterator that yields numbers in the Fibonacci sequence''' <span class=u>&#x2460;</span></a>
<a> def __init__(self, max): <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li>Classes can (and should) have <code>docstring</code>s too, just like modules and functions.
<li>The <code>__init__()</code> method is called immediately after an instance of the class is created. It would be tempting&nbsp;&mdash;&nbsp;but technically incorrect&nbsp;&mdash;&nbsp;to call this the &#8220;constructor&#8221; of the class. It&#8217;s tempting, because it looks like a C++ constructor (by convention, the <code>__init__()</code> method is the first method defined for the class), acts like one (it&#8217;s the first piece of code executed in a newly created instance of the class), and even sounds like one. Incorrect, because the object has already been constructed by the time the <code>__init__()</code> method is called, and you already have a valid reference to the new instance of the class.
</ol>
<p>The first argument of every class method, including the <code>__init__()</code> method, is always a reference to the current instance of the class. By convention, this argument is named <var>self</var>. This argument fills the role of the reserved word <code>this</code> in <abbr>C++</abbr> or Java, but <var>self</var> is not a reserved word in Python, merely a naming convention. Nonetheless, please don&#8217;t call it anything but <var>self</var>; this is a very strong convention.
<p>In the <code>__init__()</code> method, <var>self</var> refers to the newly created object; in other class methods, it refers to the instance whose method was called. Although you need to specify <var>self</var> explicitly when defining the method, you do <em>not</em> specify it when calling the method; Python will add it for you automatically.
<p class=a>&#x2042;
<h2 id=instantiating-classes>Instantiating Classes</h2>
<p>Instantiating classes in Python is straightforward. To instantiate a class, simply call the class as if it were a function, passing the arguments that the <code>__init__()</code> method requires. The return value will be the newly created object.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import fibonacci2</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>fib = fibonacci2.Fib(100)</kbd> <span class=u>&#x2460;</span></a>
<a><samp class=p>>>> </samp><kbd class=pp>fib</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>&lt;fibonacci2.Fib object at 0x00DB8810></samp>
<a><samp class=p>>>> </samp><kbd class=pp>fib.__class__</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>&lt;class 'fibonacci2.Fib'></samp>
<a><samp class=p>>>> </samp><kbd class=pp>fib.__doc__</kbd> <span class=u>&#x2463;</span></a>
<samp class=pp>'iterator that yields numbers in the Fibonacci sequence'</samp></pre>
<ol>
<li>You are creating an instance of the <code>Fib</code> class (defined in the <code>fibonacci2</code> module) and assigning the newly created instance to the variable <var>fib</var>. You are passing one parameter, <code>100</code>, which will end up as the <var>max</var> argument in <code>Fib</code>&#8217;s <code>__init__()</code> method.
<li><var>fib</var> is now an instance of the <code>Fib</code> class.
<li>Every class instance has a built-in attribute, <code>__class__</code>, which is the object&#8217;s class. Java programmers may be familiar with the <code>Class</code> class, which contains methods like <code>getName()</code> and <code>getSuperclass()</code> to get metadata information about an object. In Python, this kind of metadata is available through attributes, but the idea is the same.
<li>You can access the instance&#8217;s <code>docstring</code> just as with a function or a module. All instances of a class share the same <code>docstring</code>.
</ol>
<blockquote class='note compare java'>
<p><span class=u>&#x261E;</span>In Python, simply call a class as if it were a function to create a new instance of the class. There is no explicit <code>new</code> operator like there is in <abbr>C++</abbr> or Java.
</blockquote>
<p class=a>&#x2042;
<h2 id=instance-variables>Instance Variables</h2>
<p>On to the next line:
<pre class=pp><code>class Fib:
def __init__(self, max):
<a> self.max = max <span class=u>&#x2460;</span></a></code></pre>
<ol>
<li>What is <var>self.max</var>? It&#8217;s an instance variable. It is completely separate from <var>max</var>, which was passed into the <code>__init__()</code> method as an argument. <var>self.max</var> is &#8220;global&#8221; to the instance. That means that you can access it from other methods.
</ol>
<pre class=pp><code>class Fib:
def __init__(self, max):
<a> self.max = max <span class=u>&#x2460;</span></a>
.
.
.
def __next__(self):
fib = self.a
<a> if fib > self.max: <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li><var>self.max</var> is defined in the <code>__init__()</code> method&hellip;
<li>&hellip;and referenced in the <code>__next__()</code> method.
</ol>
<p>Instance variables are specific to one instance of a class. For example, if you create two <code>Fib</code> instances with different maximum values, they will each remember their own values.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>import fibonacci2</kbd>
<samp class=p>>>> </samp><kbd class=pp>fib1 = fibonacci2.Fib(100)</kbd>
<samp class=p>>>> </samp><kbd class=pp>fib2 = fibonacci2.Fib(200)</kbd>
<samp class=p>>>> </samp><kbd class=pp>fib1.max</kbd>
<samp class=pp>100</samp>
<samp class=p>>>> </samp><kbd class=pp>fib2.max</kbd>
<samp class=pp>200</samp></pre>
<p class=a>&#x2042;
<h2 id=a-fibonacci-iterator>A Fibonacci Iterator</h2>
<p><em>Now</em> you&#8217;re ready to learn how to build an iterator. An iterator is just a class that defines an <code>__iter__()</code> method.
<aside class=ots>
All three of these class methods, <code>__init__</code>, <code>__iter__</code>, and <code>__next__</code>, begin and end with a pair of underscore (<code>_</code>) characters. Why is that? There&#8217;s nothing magical about it, but it usually indicates that these are &#8220;<dfn>special methods</dfn>.&#8221; The only thing &#8220;special&#8221; about special methods is that they aren&#8217;t called directly; Python calls them when you use some other syntax on the class or an instance of the class. <a href=special-method-names.html>More about special methods</a>.
</aside>
<p class=d>[<a href=examples/fibonacci2.py>download <code>fibonacci2.py</code></a>]
<pre class=pp><code><a>class Fib: <span class=u>&#x2460;</span></a>
<a> def __init__(self, max): <span class=u>&#x2461;</span></a>
self.max = max
<a> def __iter__(self): <span class=u>&#x2462;</span></a>
self.a = 0
self.b = 1
return self
<a> def __next__(self): <span class=u>&#x2463;</span></a>
fib = self.a
if fib > self.max:
<a> raise StopIteration <span class=u>&#x2464;</span></a>
self.a, self.b = self.b, self.a + self.b
<a> return fib <span class=u>&#x2465;</span></a></code></pre>
<ol>
<li>To build an iterator from scratch, <code>fib</code> needs to be a class, not a function.
<li>&#8220;Calling&#8221; <code>Fib(max)</code> is really creating an instance of this class and calling its <code>__init__()</code> method with <var>max</var>. The <code>__init__()</code> method saves the maximum value as an instance variable so other methods can refer to it later.
<li>The <code>__iter__()</code> method is called whenever someone calls <code>iter(fib)</code>. (As you&#8217;ll see in a minute, a <code>for</code> loop will call this automatically, but you can also call it yourself manually.) After performing beginning-of-iteration initialization (in this case, resetting <code>self.a</code> and <code>self.b</code>, our two counters), the <code>__iter__()</code> method can return any object that implements a <code>__next__()</code> method. In this case (and in most cases), <code>__iter__()</code> simply returns <var>self</var>, since this class implements its own <code>__next__()</code> method.
<li>The <code>__next__()</code> method is called whenever someone calls <code>next()</code> on an iterator of an instance of a class. That will make more sense in a minute.
<li>When the <code>__next__()</code> method raises a <code>StopIteration</code> exception, this signals to the caller that the iteration is exhausted. Unlike most exceptions, this is not an error; it&#8217;s a normal condition that just means that the iterator has no more values to generate. If the caller is a <code>for</code> loop, it will notice this <code>StopIteration</code> exception and gracefully exit the loop. (In other words, it will swallow the exception.) This little bit of magic is actually the key to using iterators in <code>for</code> loops.
<li>To spit out the next value, an iterator&#8217;s <code>__next__()</code> method simply <code>return</code>s the value. Do not use <code>yield</code> here; that&#8217;s a bit of syntactic sugar that only applies when you&#8217;re using generators. Here you&#8217;re creating your own iterator from scratch; use <code>return</code> instead.
</ol>
<p>Thoroughly confused yet? Excellent. Let&#8217;s see how to call this iterator:
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>from fibonacci2 import Fib</kbd>
<samp class=p>>>> </samp><kbd class=pp>for n in Fib(1000):</kbd>
<samp class=p>... </samp><kbd class=pp> print(n, end=' ')</kbd>
<samp class=pp>0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987</samp></pre>
<p>Why, it&#8217;s exactly the same! Byte for byte identical to how you called <a href=generators.html#a-fibonacci-generator>Fibonacci-as-a-generator</a> (modulo one capital letter). But how?
<p>There&#8217;s a bit of magic involved in <code>for</code> loops. Here&#8217;s what happens:
<ul>
<li>The <code>for</code> loop calls <code>Fib(1000)</code>, as shown. This returns an instance of the <code>Fib</code> class. Call this <var>fib_inst</var>.
<li>Secretly, and quite cleverly, the <code>for</code> loop calls <code>iter(fib_inst)</code>, which returns an iterator object. Call this <var>fib_iter</var>. In this case, <var>fib_iter</var> == <var>fib_inst</var>, because the <code>__iter__()</code> method returns <var>self</var>, but the <code>for</code> loop doesn&#8217;t know (or care) about that.
<li>To &#8220;loop through&#8221; the iterator, the <code>for</code> loop calls <code>next(fib_iter)</code>, which calls the <code>__next__()</code> method on the <code>fib_iter</code> object, which does the next-Fibonacci-number calculations and returns a value. The <code>for</code> loop takes this value and assigns it to <var>n</var>, then executes the body of the <code>for</code> loop for that value of <var>n</var>.
<li>How does the <code>for</code> loop know when to stop? I&#8217;m glad you asked! When <code>next(fib_iter)</code> raises a <code>StopIteration</code> exception, the <code>for</code> loop will swallow the exception and gracefully exit. (Any other exception will pass through and be raised as usual.) And where have you seen a <code>StopIteration</code> exception? In the <code>__next__()</code> method, of course!
</ul>
<p class=a>&#x2042;
<h2 id=a-plural-rule-iterator>A Plural Rule Iterator</h2>
<aside>iter(f) calls f.__iter__<br>next(f) calls f.__next__</aside>
<p>Now it&#8217;s time for the finale. Let&#8217;s rewrite the <a href=generators.html>plural rules generator</a> as an iterator.
<p class=d>[<a href=examples/plural6.py>download <code>plural6.py</code></a>]
<pre class=pp><code>class LazyRules:
rules_filename = 'plural6-rules.txt'
def __init__(self):
self.pattern_file = open(self.rules_filename, encoding='utf-8')
self.cache = []
def __iter__(self):
self.cache_index = 0
return self
def __next__(self):
self.cache_index += 1
if len(self.cache) >= self.cache_index:
return self.cache[self.cache_index - 1]
if self.pattern_file.closed:
raise StopIteration
line = self.pattern_file.readline()
if not line:
self.pattern_file.close()
raise StopIteration
pattern, search, replace = line.split(None, 3)
funcs = build_match_and_apply_functions(
pattern, search, replace)
self.cache.append(funcs)
return funcs
rules = LazyRules()</code></pre>
<p>So this is a class that implements <code>__iter__()</code> and <code>__next__()</code>, so it can be used as an iterator. Then, you instantiate the class and assign it to <var>rules</var>. This happens just once, on import.
<p>Let&#8217;s take the class one bite at a time.
<pre class=pp><code>class LazyRules:
rules_filename = 'plural6-rules.txt'
def __init__(self):
<a> self.pattern_file = open(self.rules_filename, encoding='utf-8') <span class=u>&#x2460;</span></a>
<a> self.cache = [] <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li>When we instantiate the <code>LazyRules</code> class, open the pattern file but don&#8217;t read anything from it. (That comes later.)
<li>After opening the patterns file, initialize the cache. You&#8217;ll use this cache later (in the <code>__next__()</code> method) as you read lines from the pattern file.
</ol>
<p>Before we continue, let&#8217;s take a closer look at <var>rules_filename</var>. It&#8217;s not defined within the <code>__iter__()</code> method. In fact, it&#8217;s not defined within <em>any</em> method. It&#8217;s defined at the class level. It&#8217;s a <i>class variable</i>, and although you can access it just like an instance variable (<var>self.rules_filename</var>), it is shared across all instances of the <code>LazyRules</code> class.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import plural6</kbd>
<samp class=p>>>> </samp><kbd class=pp>r1 = plural6.LazyRules()</kbd>
<samp class=p>>>> </samp><kbd class=pp>r2 = plural6.LazyRules()</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>r1.rules_filename</kbd> <span class=u>&#x2460;</span></a>
<samp class=pp>'plural6-rules.txt'</samp>
<samp class=p>>>> </samp><kbd class=pp>r2.rules_filename</kbd>
<samp class=pp>'plural6-rules.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.rules_filename = 'r2-override.txt'</kbd> <span class=u>&#x2461;</span></a>
<samp class=p>>>> </samp><kbd class=pp>r2.rules_filename</kbd>
<samp class=pp>'r2-override.txt'</samp>
<samp class=p>>>> </samp><kbd class=pp>r1.rules_filename</kbd>
<samp class=pp>'plural6-rules.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.__class__.rules_filename</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>'plural6-rules.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.__class__.rules_filename = 'papayawhip.txt'</kbd> <span class=u>&#x2463;</span></a>
<samp class=p>>>> </samp><kbd class=pp>r1.rules_filename</kbd>
<samp class=pp>'papayawhip.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.rules_filename</kbd> <span class=u>&#x2464;</span></a>
<samp class=pp>'r2-overridetxt'</samp></pre>
<ol>
<li>Each instance of the class inherits the <var>rules_filename</var> attribute with the value defined by the class.
<li>Changing the attribute&#8217;s value in one instance does not affect other instances&hellip;
<li>&hellip;nor does it change the class attribute. You can access the class attribute (as opposed to an individual instance&#8217;s attribute) by using the special <code>__class__</code> attribute to access the class itself.
<li>If you change the class attribute, all instances that are still inheriting that value (like <var>r1</var> here) will be affected.
<li>Instances that have overridden that attribute (like <var>r2</var> here) will not be affected.
</ol>
<p>And now back to our show.
<pre class=pp><code><a> def __iter__(self): <span class=u>&#x2460;</span></a>
self.cache_index = 0
<a> return self <span class=u>&#x2461;</span></a>
</code></pre>
<ol>
<li>The <code>__iter__()</code> method will be called every time someone&nbsp;&mdash;&nbsp;say, a <code>for</code> loop&nbsp;&mdash;&nbsp;calls <code>iter(rules)</code>.
<li>The one thing that every <code>__iter__()</code> method must do is return an iterator. In this case, it returns <var>self</var>, which signals that this class defines a <code>__next__()</code> method which will take care of returning values throughout the iteration.
</ol>
<pre class=pp><code><a> def __next__(self): <span class=u>&#x2460;</span></a>
.
.
.
pattern, search, replace = line.split(None, 3)
<a> funcs = build_match_and_apply_functions( <span class=u>&#x2461;</span></a>
pattern, search, replace)
<a> self.cache.append(funcs) <span class=u>&#x2462;</span></a>
return funcs</code></pre>
<ol>
<li>The <code>__next__()</code> method gets called whenever someone&nbsp;&mdash;&nbsp;say, a <code>for</code> loop&nbsp;&mdash;&nbsp;calls <code>next(rules)</code>. This method will only make sense if we start at the end and work backwards. So let&#8217;s do that.
<li>The last part of this function should look familiar, at least. The <code>build_match_and_apply_functions()</code> function hasn&#8217;t changed; it&#8217;s the same as it ever was.
<li>The only difference is that, before returning the match and apply functions (which are stored in the tuple <var>funcs</var>), we&#8217;re going to save them in <code>self.cache</code>.
</ol>
<p>Moving backwards&hellip;
<pre class=pp><code> def __next__(self):
.
.
.
<a> line = self.pattern_file.readline() <span class=u>&#x2460;</span></a>
<a> if not line: <span class=u>&#x2461;</span></a>
self.pattern_file.close()
<a> raise StopIteration <span class=u>&#x2462;</span></a>
.
.
.</code></pre>
<ol>
<li>A bit of advanced file trickery here. The <code>readline()</code> method (note: singular, not the plural <code>readlines()</code>) reads exactly one line from an open file. Specifically, the next line. (<em>File objects are iterators too! It&#8217;s iterators all the way down&hellip;</em>)
<li>If there was a line for <code>readline()</code> to read, <var>line</var> will not be an empty string. Even if the file contained a blank line, <var>line</var> would end up as the one-character string <code>'\n'</code> (a carriage return). If <var>line</var> is really an empty string, that means there are no more lines to read from the file.
<li>When we reach the end of the file, we should close the file and raise the magic <code>StopIteration</code> exception. Remember, we got to this point because we needed a match and apply function for the next rule. The next rule comes from the next line of the file&hellip; but there is no next line! Therefore, we have no value to return. The iteration is over. (<span class=u>&#x266B;</span> The party&#8217;s over&hellip; <span class=u>&#x266B;</span>)
</ol>
<p>Moving backwards all the way to the start of the <code>__next__()</code> method&hellip;
<pre class=pp><code> def __next__(self):
self.cache_index += 1
if len(self.cache) >= self.cache_index:
<a> return self.cache[self.cache_index - 1] <span class=u>&#x2460;</span></a>
if self.pattern_file.closed:
<a> raise StopIteration <span class=u>&#x2461;</span></a>
.
.
.</code></pre>
<ol>
<li><code>self.cache</code> will be a list of the functions we need to match and apply individual rules. (At least <em>that</em> should sound familiar!) <code>self.cache_index</code> keeps track of which cached item we should return next. If we haven&#8217;t exhausted the cache yet (<i>i.e.</i> if the length of <code>self.cache</code> is greater than <code>self.cache_index</code>), then we have a cache hit! Hooray! We can return the match and apply functions from the cache instead of building them from scratch.
<li>On the other hand, if we don&#8217;t get a hit from the cache, <em>and</em> the file object has been closed (which could happen, further down the method, as you saw in the previous code snippet), then there&#8217;s nothing more we can do. If the file is closed, it means we&#8217;ve exhausted it&nbsp;&mdash;&nbsp;we&#8217;ve already read through every line from the pattern file, and we&#8217;ve already built and cached the match and apply functions for each pattern. The file is exhausted; the cache is exhausted; I&#8217;m exhausted. Wait, what? Hang in there, we&#8217;re almost done.
</ol>
<p>Putting it all together, here&#8217;s what happens when:
<ul>
<li>When the module is imported, it creates a single instance of the <code>LazyRules</code> class, called <var>rules</var>, which opens the pattern file but does not read from it.
<li>When asked for the first match and apply function, it checks its cache but finds the cache is empty. So it reads a single line from the pattern file, builds the match and apply functions from those patterns, and caches them.
<li>Let&#8217;s say, for the sake of argument, that the very first rule matched. If so, no further match and apply functions are built, and no further lines are read from the pattern file.
<li>Furthermore, for the sake of argument, suppose that the caller calls the <code>plural()</code> function <em>again</em> to pluralize a different word. The <code>for</code> loop in the <code>plural()</code> function will call <code>iter(rules)</code>, which will reset the cache index but will not reset the open file object.
<li>The first time through, the <code>for</code> loop will ask for a value from <var>rules</var>, which will invoke its <code>__next__()</code> method. This time, however, the cache is primed with a single pair of match and apply functions, corresponding to the patterns in the first line of the pattern file. Since they were built and cached in the course of pluralizing the previous word, they&#8217;re retrieved from the cache. The cache index increments, and the open file is never touched.
<li>Let&#8217;s say, for the sake of argument, that the first rule does <em>not</em> match this time around. So the <code>for</code> loop comes around again and asks for another value from <var>rules</var>. This invokes the <code>__next__()</code> method a second time. This time, the cache is exhausted&nbsp;&mdash;&nbsp;it only contained one item, and we&#8217;re asking for a second&nbsp;&mdash;&nbsp;so the <code>__next__()</code> method continues. It reads another line from the open file, builds match and apply functions out of the patterns, and caches them.
<li>This read-build-and-cache process will continue as long as the rules being read from the pattern file don&#8217;t match the word we&#8217;re trying to pluralize. If we do find a matching rule before the end of the file, we simply use it and stop, with the file still open. The file pointer will stay wherever we stopped reading, waiting for the next <code>readline()</code> command. In the meantime, the cache now has more items in it, and if we start all over again trying to pluralize a new word, each of those items in the cache will be tried before reading the next line from the pattern file.
</ul>
<p>We have achieved pluralization nirvana.
<ol>
<li><strong>Minimal startup cost.</strong> The only thing that happens on <code>import</code> is instantiating a single class and opening a file (but not reading from it).
<li><strong>Maximum performance.</strong> The previous example would read through the file and build functions dynamically every time you wanted to pluralize a word. This version will cache functions as soon as they&#8217;re built, and in the worst case, it will only read through the pattern file once, no matter how many words you pluralize.
<li><strong>Separation of code and data.</strong> All the patterns are stored in a separate file. Code is code, and data is data, and never the twain shall meet.
</ol>
<blockquote class=note>
<p><span class=u>&#x261E;</span>Is this really nirvana? Well, yes and no. Here&#8217;s something to consider with the <code>LazyRules</code> example: the pattern file is opened (during <code>__init__()</code>), and it remains open until the final rule is reached. Python will eventually close the file when it exits, or after the last instantiation of the <code>LazyRules</code> class is destroyed, but still, that could be a <em>long</em> time. If this class is part of a long-running Python process, the Python interpreter may never exit, and the <code>LazyRules</code> object may never get destroyed.
<p>There are ways around this. Instead of opening the file during <code>__init__()</code> and leaving it open while you read rules one line at a time, you could open the file, read all the rules, and immediately close the file. Or you could open the file, read one rule, save the file position with the <a href=files.html#read><code>tell()</code> method</a>, close the file, and later re-open it and use the <a href=files.html#read><code>seek()</code> method</a> to continue reading where you left off. Or you could not worry about it and just leave the file open, like this example code does. Programming is design, and design is all about trade-offs and constraints. Leaving a file open too long might be a problem; making your code more complicated might be a problem. Which one is the bigger problem depends on your development team, your application, and your runtime environment.
</blockquote>
<p class=a>&#x2042;
<h2 id=furtherreading>Further Reading</h2>
<ul>
<li><a href=http://docs.python.org/3.1/library/stdtypes.html#iterator-types>Iterator types</a>
<li><a href=http://www.python.org/dev/peps/pep-0234/>PEP 234: Iterators</a>
<li><a href=http://www.python.org/dev/peps/pep-0255/>PEP 255: Simple Generators</a>
<li><a href=http://www.dabeaz.com/generators/>Generator Tricks for Systems Programmers</a>
</ul>
<p class=v><a href=generators.html rel=prev title='back to &#8220;Closures &amp; Generators&#8221;'><span class=u>&#x261C;</span></a> <a href=advanced-iterators.html rel=next title='onward to &#8220;Advanced Iterators&#8221;'><span class=u>&#x261E;</span></a>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/prettify.js></script>
<script src=j/dip3.js></script>
<!DOCTYPE html>
<meta charset=utf-8>
<title>Classes &amp; Iterators - Dive Into Python 3</title>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<link rel=stylesheet href=dip3.css>
<style>
body{counter-reset:h1 7}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#iterators>Dive Into Python 3</a> <span class=u>&#8227;</span>
<p id=level>Difficulty level: <span class=u title=intermediate>&#x2666;&#x2666;&#x2666;&#x2662;&#x2662;</span>
<h1>Classes <i class=baa>&amp;</i> Iterators</h1>
<blockquote class=q>
<p><span class=u>&#x275D;</span> East is East, and West is West, and never the twain shall meet. <span class=u>&#x275E;</span><br>&mdash; <a href=http://en.wikiquote.org/wiki/Rudyard_Kipling>Rudyard Kipling</a>
</blockquote>
<p id=toc>&nbsp;
<h2 id=divingin>Diving In</h2>
<p class=f>Iterators are the &#8220;secret sauce&#8221; of Python 3. They&#8217;re everywhere, underlying everything, always just out of sight. <a href=comprehensions.html>Comprehensions</a> are just a simple form of <i>iterators</i>. Generators are just a simple form of <i>iterators</i>. A function that <code>yield</code>s values is a nice, compact way of building an iterator without building an iterator. Let me show you what I mean by that.
<p>Remember <a href=generators.html#a-fibonacci-generator>the Fibonacci generator</a>? Here it is as a built-from-scratch iterator:
<p class=d>[<a href=examples/fibonacci2.py>download <code>fibonacci2.py</code></a>]
<pre class=pp><code>class Fib:
'''iterator that yields numbers in the Fibonacci sequence'''
def __init__(self, max):
self.max = max
def __iter__(self):
self.a = 0
self.b = 1
return self
def __next__(self):
fib = self.a
if fib > self.max:
raise StopIteration
self.a, self.b = self.b, self.a + self.b
return fib</code></pre>
<p>Let&#8217;s take that one line at a time.
<pre class='nd pp'><code>class Fib:</code></pre>
<p><code>class</code>? What&#8217;s a class?
<p class=a>&#x2042;
<h2 id=defining-classes>Defining Classes</h2>
<p>Python is fully object-oriented: you can define your own classes, inherit from your own or built-in classes, and instantiate the classes you&#8217;ve defined.
<p>Defining a class in Python is simple. As with functions, there is no separate interface definition. Just define the class and start coding. A Python class starts with the reserved word <code>class</code>, followed by the class name. Technically, that&#8217;s all that&#8217;s required, since a class doesn&#8217;t need to inherit from any other class.
<pre class=pp><code><a>class PapayaWhip: <span class=u>&#x2460;</span></a>
<a> pass <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li>The name of this class is <code>PapayaWhip</code>, and it doesn&#8217;t inherit from any other class. Class names are usually capitalized, <code>EachWordLikeThis</code>, but this is only a convention, not a requirement.
<li>You probably guessed this, but everything in a class is indented, just like the code within a function, <code>if</code> statement, <code>for</code> loop, or any other block of code. The first line not indented is outside the class.
</ol>
<p>This <code>PapayaWhip</code> class doesn&#8217;t define any methods or attributes, but syntactically, there needs to be something in the definition, thus the <code>pass</code> statement. This is a Python reserved word that just means &#8220;move along, nothing to see here&#8221;. It&#8217;s a statement that does nothing, and it&#8217;s a good placeholder when you&#8217;re stubbing out functions or classes.
<blockquote class='note compare java'>
<p><span class=u>&#x261E;</span>The <code>pass</code> statement in Python is like a empty set of curly braces (<code>{}</code>) in Java or C.
</blockquote>
<p>Many classes are inherited from other classes, but this one is not. Many classes define methods, but this one does not. There is nothing that a Python class absolutely must have, other than a name. In particular, C++ programmers may find it odd that Python classes don&#8217;t have explicit constructors and destructors. Although it&#8217;s not required, Python classes <em>can</em> have something similar to a constructor: the <code>__init__()</code> method.
<h3 id=init-method>The <code>__init__()</code> Method</h3>
<p>This example shows the initialization of the <code>Fib</code> class using the <code>__init__</code> method.
<pre class=pp><code>class Fib:
<a> '''iterator that yields numbers in the Fibonacci sequence''' <span class=u>&#x2460;</span></a>
<a> def __init__(self, max): <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li>Classes can (and should) have <code>docstring</code>s too, just like modules and functions.
<li>The <code>__init__()</code> method is called immediately after an instance of the class is created. It would be tempting&nbsp;&mdash;&nbsp;but technically incorrect&nbsp;&mdash;&nbsp;to call this the &#8220;constructor&#8221; of the class. It&#8217;s tempting, because it looks like a C++ constructor (by convention, the <code>__init__()</code> method is the first method defined for the class), acts like one (it&#8217;s the first piece of code executed in a newly created instance of the class), and even sounds like one. Incorrect, because the object has already been constructed by the time the <code>__init__()</code> method is called, and you already have a valid reference to the new instance of the class.
</ol>
<p>The first argument of every class method, including the <code>__init__()</code> method, is always a reference to the current instance of the class. By convention, this argument is named <var>self</var>. This argument fills the role of the reserved word <code>this</code> in <abbr>C++</abbr> or Java, but <var>self</var> is not a reserved word in Python, merely a naming convention. Nonetheless, please don&#8217;t call it anything but <var>self</var>; this is a very strong convention.
<p>In the <code>__init__()</code> method, <var>self</var> refers to the newly created object; in other class methods, it refers to the instance whose method was called. Although you need to specify <var>self</var> explicitly when defining the method, you do <em>not</em> specify it when calling the method; Python will add it for you automatically.
<p class=a>&#x2042;
<h2 id=instantiating-classes>Instantiating Classes</h2>
<p>Instantiating classes in Python is straightforward. To instantiate a class, simply call the class as if it were a function, passing the arguments that the <code>__init__()</code> method requires. The return value will be the newly created object.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import fibonacci2</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>fib = fibonacci2.Fib(100)</kbd> <span class=u>&#x2460;</span></a>
<a><samp class=p>>>> </samp><kbd class=pp>fib</kbd> <span class=u>&#x2461;</span></a>
<samp class=pp>&lt;fibonacci2.Fib object at 0x00DB8810></samp>
<a><samp class=p>>>> </samp><kbd class=pp>fib.__class__</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>&lt;class 'fibonacci2.Fib'></samp>
<a><samp class=p>>>> </samp><kbd class=pp>fib.__doc__</kbd> <span class=u>&#x2463;</span></a>
<samp class=pp>'iterator that yields numbers in the Fibonacci sequence'</samp></pre>
<ol>
<li>You are creating an instance of the <code>Fib</code> class (defined in the <code>fibonacci2</code> module) and assigning the newly created instance to the variable <var>fib</var>. You are passing one parameter, <code>100</code>, which will end up as the <var>max</var> argument in <code>Fib</code>&#8217;s <code>__init__()</code> method.
<li><var>fib</var> is now an instance of the <code>Fib</code> class.
<li>Every class instance has a built-in attribute, <code>__class__</code>, which is the object&#8217;s class. Java programmers may be familiar with the <code>Class</code> class, which contains methods like <code>getName()</code> and <code>getSuperclass()</code> to get metadata information about an object. In Python, this kind of metadata is available through attributes, but the idea is the same.
<li>You can access the instance&#8217;s <code>docstring</code> just as with a function or a module. All instances of a class share the same <code>docstring</code>.
</ol>
<blockquote class='note compare java'>
<p><span class=u>&#x261E;</span>In Python, simply call a class as if it were a function to create a new instance of the class. There is no explicit <code>new</code> operator like there is in <abbr>C++</abbr> or Java.
</blockquote>
<p class=a>&#x2042;
<h2 id=instance-variables>Instance Variables</h2>
<p>On to the next line:
<pre class=pp><code>class Fib:
def __init__(self, max):
<a> self.max = max <span class=u>&#x2460;</span></a></code></pre>
<ol>
<li>What is <var>self.max</var>? It&#8217;s an instance variable. It is completely separate from <var>max</var>, which was passed into the <code>__init__()</code> method as an argument. <var>self.max</var> is &#8220;global&#8221; to the instance. That means that you can access it from other methods.
</ol>
<pre class=pp><code>class Fib:
def __init__(self, max):
<a> self.max = max <span class=u>&#x2460;</span></a>
.
.
.
def __next__(self):
fib = self.a
<a> if fib > self.max: <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li><var>self.max</var> is defined in the <code>__init__()</code> method&hellip;
<li>&hellip;and referenced in the <code>__next__()</code> method.
</ol>
<p>Instance variables are specific to one instance of a class. For example, if you create two <code>Fib</code> instances with different maximum values, they will each remember their own values.
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>import fibonacci2</kbd>
<samp class=p>>>> </samp><kbd class=pp>fib1 = fibonacci2.Fib(100)</kbd>
<samp class=p>>>> </samp><kbd class=pp>fib2 = fibonacci2.Fib(200)</kbd>
<samp class=p>>>> </samp><kbd class=pp>fib1.max</kbd>
<samp class=pp>100</samp>
<samp class=p>>>> </samp><kbd class=pp>fib2.max</kbd>
<samp class=pp>200</samp></pre>
<p class=a>&#x2042;
<h2 id=a-fibonacci-iterator>A Fibonacci Iterator</h2>
<p><em>Now</em> you&#8217;re ready to learn how to build an iterator. An iterator is just a class that defines an <code>__iter__()</code> method.
<aside class=ots>
All three of these class methods, <code>__init__</code>, <code>__iter__</code>, and <code>__next__</code>, begin and end with a pair of underscore (<code>_</code>) characters. Why is that? There&#8217;s nothing magical about it, but it usually indicates that these are &#8220;<dfn>special methods</dfn>.&#8221; The only thing &#8220;special&#8221; about special methods is that they aren&#8217;t called directly; Python calls them when you use some other syntax on the class or an instance of the class. <a href=special-method-names.html>More about special methods</a>.
</aside>
<p class=d>[<a href=examples/fibonacci2.py>download <code>fibonacci2.py</code></a>]
<pre class=pp><code><a>class Fib: <span class=u>&#x2460;</span></a>
<a> def __init__(self, max): <span class=u>&#x2461;</span></a>
self.max = max
<a> def __iter__(self): <span class=u>&#x2462;</span></a>
self.a = 0
self.b = 1
return self
<a> def __next__(self): <span class=u>&#x2463;</span></a>
fib = self.a
if fib > self.max:
<a> raise StopIteration <span class=u>&#x2464;</span></a>
self.a, self.b = self.b, self.a + self.b
<a> return fib <span class=u>&#x2465;</span></a></code></pre>
<ol>
<li>To build an iterator from scratch, <code>fib</code> needs to be a class, not a function.
<li>&#8220;Calling&#8221; <code>Fib(max)</code> is really creating an instance of this class and calling its <code>__init__()</code> method with <var>max</var>. The <code>__init__()</code> method saves the maximum value as an instance variable so other methods can refer to it later.
<li>The <code>__iter__()</code> method is called whenever someone calls <code>iter(fib)</code>. (As you&#8217;ll see in a minute, a <code>for</code> loop will call this automatically, but you can also call it yourself manually.) After performing beginning-of-iteration initialization (in this case, resetting <code>self.a</code> and <code>self.b</code>, our two counters), the <code>__iter__()</code> method can return any object that implements a <code>__next__()</code> method. In this case (and in most cases), <code>__iter__()</code> simply returns <var>self</var>, since this class implements its own <code>__next__()</code> method.
<li>The <code>__next__()</code> method is called whenever someone calls <code>next()</code> on an iterator of an instance of a class. That will make more sense in a minute.
<li>When the <code>__next__()</code> method raises a <code>StopIteration</code> exception, this signals to the caller that the iteration is exhausted. Unlike most exceptions, this is not an error; it&#8217;s a normal condition that just means that the iterator has no more values to generate. If the caller is a <code>for</code> loop, it will notice this <code>StopIteration</code> exception and gracefully exit the loop. (In other words, it will swallow the exception.) This little bit of magic is actually the key to using iterators in <code>for</code> loops.
<li>To spit out the next value, an iterator&#8217;s <code>__next__()</code> method simply <code>return</code>s the value. Do not use <code>yield</code> here; that&#8217;s a bit of syntactic sugar that only applies when you&#8217;re using generators. Here you&#8217;re creating your own iterator from scratch; use <code>return</code> instead.
</ol>
<p>Thoroughly confused yet? Excellent. Let&#8217;s see how to call this iterator:
<pre class='nd screen'>
<samp class=p>>>> </samp><kbd class=pp>from fibonacci2 import Fib</kbd>
<samp class=p>>>> </samp><kbd class=pp>for n in Fib(1000):</kbd>
<samp class=p>... </samp><kbd class=pp> print(n, end=' ')</kbd>
<samp class=pp>0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987</samp></pre>
<p>Why, it&#8217;s exactly the same! Byte for byte identical to how you called <a href=generators.html#a-fibonacci-generator>Fibonacci-as-a-generator</a> (modulo one capital letter). But how?
<p>There&#8217;s a bit of magic involved in <code>for</code> loops. Here&#8217;s what happens:
<ul>
<li>The <code>for</code> loop calls <code>Fib(1000)</code>, as shown. This returns an instance of the <code>Fib</code> class. Call this <var>fib_inst</var>.
<li>Secretly, and quite cleverly, the <code>for</code> loop calls <code>iter(fib_inst)</code>, which returns an iterator object. Call this <var>fib_iter</var>. In this case, <var>fib_iter</var> == <var>fib_inst</var>, because the <code>__iter__()</code> method returns <var>self</var>, but the <code>for</code> loop doesn&#8217;t know (or care) about that.
<li>To &#8220;loop through&#8221; the iterator, the <code>for</code> loop calls <code>next(fib_iter)</code>, which calls the <code>__next__()</code> method on the <code>fib_iter</code> object, which does the next-Fibonacci-number calculations and returns a value. The <code>for</code> loop takes this value and assigns it to <var>n</var>, then executes the body of the <code>for</code> loop for that value of <var>n</var>.
<li>How does the <code>for</code> loop know when to stop? I&#8217;m glad you asked! When <code>next(fib_iter)</code> raises a <code>StopIteration</code> exception, the <code>for</code> loop will swallow the exception and gracefully exit. (Any other exception will pass through and be raised as usual.) And where have you seen a <code>StopIteration</code> exception? In the <code>__next__()</code> method, of course!
</ul>
<p class=a>&#x2042;
<h2 id=a-plural-rule-iterator>A Plural Rule Iterator</h2>
<aside>iter(f) calls f.__iter__<br>next(f) calls f.__next__</aside>
<p>Now it&#8217;s time for the finale. Let&#8217;s rewrite the <a href=generators.html>plural rules generator</a> as an iterator.
<p class=d>[<a href=examples/plural6.py>download <code>plural6.py</code></a>]
<pre class=pp><code>class LazyRules:
rules_filename = 'plural6-rules.txt'
def __init__(self):
self.pattern_file = open(self.rules_filename, encoding='utf-8')
self.cache = []
def __iter__(self):
self.cache_index = 0
return self
def __next__(self):
self.cache_index += 1
if len(self.cache) >= self.cache_index:
return self.cache[self.cache_index - 1]
if self.pattern_file.closed:
raise StopIteration
line = self.pattern_file.readline()
if not line:
self.pattern_file.close()
raise StopIteration
pattern, search, replace = line.split(None, 3)
funcs = build_match_and_apply_functions(
pattern, search, replace)
self.cache.append(funcs)
return funcs
rules = LazyRules()</code></pre>
<p>So this is a class that implements <code>__iter__()</code> and <code>__next__()</code>, so it can be used as an iterator. Then, you instantiate the class and assign it to <var>rules</var>. This happens just once, on import.
<p>Let&#8217;s take the class one bite at a time.
<pre class=pp><code>class LazyRules:
rules_filename = 'plural6-rules.txt'
def __init__(self):
<a> self.pattern_file = open(self.rules_filename, encoding='utf-8') <span class=u>&#x2460;</span></a>
<a> self.cache = [] <span class=u>&#x2461;</span></a></code></pre>
<ol>
<li>When we instantiate the <code>LazyRules</code> class, open the pattern file but don&#8217;t read anything from it. (That comes later.)
<li>After opening the patterns file, initialize the cache. You&#8217;ll use this cache later (in the <code>__next__()</code> method) as you read lines from the pattern file.
</ol>
<p>Before we continue, let&#8217;s take a closer look at <var>rules_filename</var>. It&#8217;s not defined within the <code>__iter__()</code> method. In fact, it&#8217;s not defined within <em>any</em> method. It&#8217;s defined at the class level. It&#8217;s a <i>class variable</i>, and although you can access it just like an instance variable (<var>self.rules_filename</var>), it is shared across all instances of the <code>LazyRules</code> class.
<pre class=screen>
<samp class=p>>>> </samp><kbd class=pp>import plural6</kbd>
<samp class=p>>>> </samp><kbd class=pp>r1 = plural6.LazyRules()</kbd>
<samp class=p>>>> </samp><kbd class=pp>r2 = plural6.LazyRules()</kbd>
<a><samp class=p>>>> </samp><kbd class=pp>r1.rules_filename</kbd> <span class=u>&#x2460;</span></a>
<samp class=pp>'plural6-rules.txt'</samp>
<samp class=p>>>> </samp><kbd class=pp>r2.rules_filename</kbd>
<samp class=pp>'plural6-rules.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.rules_filename = 'r2-override.txt'</kbd> <span class=u>&#x2461;</span></a>
<samp class=p>>>> </samp><kbd class=pp>r2.rules_filename</kbd>
<samp class=pp>'r2-override.txt'</samp>
<samp class=p>>>> </samp><kbd class=pp>r1.rules_filename</kbd>
<samp class=pp>'plural6-rules.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.__class__.rules_filename</kbd> <span class=u>&#x2462;</span></a>
<samp class=pp>'plural6-rules.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.__class__.rules_filename = 'papayawhip.txt'</kbd> <span class=u>&#x2463;</span></a>
<samp class=p>>>> </samp><kbd class=pp>r1.rules_filename</kbd>
<samp class=pp>'papayawhip.txt'</samp>
<a><samp class=p>>>> </samp><kbd class=pp>r2.rules_filename</kbd> <span class=u>&#x2464;</span></a>
<samp class=pp>'r2-overridetxt'</samp></pre>
<ol>
<li>Each instance of the class inherits the <var>rules_filename</var> attribute with the value defined by the class.
<li>Changing the attribute&#8217;s value in one instance does not affect other instances&hellip;
<li>&hellip;nor does it change the class attribute. You can access the class attribute (as opposed to an individual instance&#8217;s attribute) by using the special <code>__class__</code> attribute to access the class itself.
<li>If you change the class attribute, all instances that are still inheriting that value (like <var>r1</var> here) will be affected.
<li>Instances that have overridden that attribute (like <var>r2</var> here) will not be affected.
</ol>
<p>And now back to our show.
<pre class=pp><code><a> def __iter__(self): <span class=u>&#x2460;</span></a>
self.cache_index = 0
<a> return self <span class=u>&#x2461;</span></a>
</code></pre>
<ol>
<li>The <code>__iter__()</code> method will be called every time someone&nbsp;&mdash;&nbsp;say, a <code>for</code> loop&nbsp;&mdash;&nbsp;calls <code>iter(rules)</code>.
<li>The one thing that every <code>__iter__()</code> method must do is return an iterator. In this case, it returns <var>self</var>, which signals that this class defines a <code>__next__()</code> method which will take care of returning values throughout the iteration.
</ol>
<pre class=pp><code><a> def __next__(self): <span class=u>&#x2460;</span></a>
.
.
.
pattern, search, replace = line.split(None, 3)
<a> funcs = build_match_and_apply_functions( <span class=u>&#x2461;</span></a>
pattern, search, replace)
<a> self.cache.append(funcs) <span class=u>&#x2462;</span></a>
return funcs</code></pre>
<ol>
<li>The <code>__next__()</code> method gets called whenever someone&nbsp;&mdash;&nbsp;say, a <code>for</code> loop&nbsp;&mdash;&nbsp;calls <code>next(rules)</code>. This method will only make sense if we start at the end and work backwards. So let&#8217;s do that.
<li>The last part of this function should look familiar, at least. The <code>build_match_and_apply_functions()</code> function hasn&#8217;t changed; it&#8217;s the same as it ever was.
<li>The only difference is that, before returning the match and apply functions (which are stored in the tuple <var>funcs</var>), we&#8217;re going to save them in <code>self.cache</code>.
</ol>
<p>Moving backwards&hellip;
<pre class=pp><code> def __next__(self):
.
.
.
<a> line = self.pattern_file.readline() <span class=u>&#x2460;</span></a>
<a> if not line: <span class=u>&#x2461;</span></a>
self.pattern_file.close()
<a> raise StopIteration <span class=u>&#x2462;</span></a>
.
.
.</code></pre>
<ol>
<li>A bit of advanced file trickery here. The <code>readline()</code> method (note: singular, not the plural <code>readlines()</code>) reads exactly one line from an open file. Specifically, the next line. (<em>File objects are iterators too! It&#8217;s iterators all the way down&hellip;</em>)
<li>If there was a line for <code>readline()</code> to read, <var>line</var> will not be an empty string. Even if the file contained a blank line, <var>line</var> would end up as the one-character string <code>'\n'</code> (a carriage return). If <var>line</var> is really an empty string, that means there are no more lines to read from the file.
<li>When we reach the end of the file, we should close the file and raise the magic <code>StopIteration</code> exception. Remember, we got to this point because we needed a match and apply function for the next rule. The next rule comes from the next line of the file&hellip; but there is no next line! Therefore, we have no value to return. The iteration is over. (<span class=u>&#x266B;</span> The party&#8217;s over&hellip; <span class=u>&#x266B;</span>)
</ol>
<p>Moving backwards all the way to the start of the <code>__next__()</code> method&hellip;
<pre class=pp><code> def __next__(self):
self.cache_index += 1
if len(self.cache) >= self.cache_index:
<a> return self.cache[self.cache_index - 1] <span class=u>&#x2460;</span></a>
if self.pattern_file.closed:
<a> raise StopIteration <span class=u>&#x2461;</span></a>
.
.
.</code></pre>
<ol>
<li><code>self.cache</code> will be a list of the functions we need to match and apply individual rules. (At least <em>that</em> should sound familiar!) <code>self.cache_index</code> keeps track of which cached item we should return next. If we haven&#8217;t exhausted the cache yet (<i>i.e.</i> if the length of <code>self.cache</code> is greater than <code>self.cache_index</code>), then we have a cache hit! Hooray! We can return the match and apply functions from the cache instead of building them from scratch.
<li>On the other hand, if we don&#8217;t get a hit from the cache, <em>and</em> the file object has been closed (which could happen, further down the method, as you saw in the previous code snippet), then there&#8217;s nothing more we can do. If the file is closed, it means we&#8217;ve exhausted it&nbsp;&mdash;&nbsp;we&#8217;ve already read through every line from the pattern file, and we&#8217;ve already built and cached the match and apply functions for each pattern. The file is exhausted; the cache is exhausted; I&#8217;m exhausted. Wait, what? Hang in there, we&#8217;re almost done.
</ol>
<p>Putting it all together, here&#8217;s what happens when:
<ul>
<li>When the module is imported, it creates a single instance of the <code>LazyRules</code> class, called <var>rules</var>, which opens the pattern file but does not read from it.
<li>When asked for the first match and apply function, it checks its cache but finds the cache is empty. So it reads a single line from the pattern file, builds the match and apply functions from those patterns, and caches them.
<li>Let&#8217;s say, for the sake of argument, that the very first rule matched. If so, no further match and apply functions are built, and no further lines are read from the pattern file.
<li>Furthermore, for the sake of argument, suppose that the caller calls the <code>plural()</code> function <em>again</em> to pluralize a different word. The <code>for</code> loop in the <code>plural()</code> function will call <code>iter(rules)</code>, which will reset the cache index but will not reset the open file object.
<li>The first time through, the <code>for</code> loop will ask for a value from <var>rules</var>, which will invoke its <code>__next__()</code> method. This time, however, the cache is primed with a single pair of match and apply functions, corresponding to the patterns in the first line of the pattern file. Since they were built and cached in the course of pluralizing the previous word, they&#8217;re retrieved from the cache. The cache index increments, and the open file is never touched.
<li>Let&#8217;s say, for the sake of argument, that the first rule does <em>not</em> match this time around. So the <code>for</code> loop comes around again and asks for another value from <var>rules</var>. This invokes the <code>__next__()</code> method a second time. This time, the cache is exhausted&nbsp;&mdash;&nbsp;it only contained one item, and we&#8217;re asking for a second&nbsp;&mdash;&nbsp;so the <code>__next__()</code> method continues. It reads another line from the open file, builds match and apply functions out of the patterns, and caches them.
<li>This read-build-and-cache process will continue as long as the rules being read from the pattern file don&#8217;t match the word we&#8217;re trying to pluralize. If we do find a matching rule before the end of the file, we simply use it and stop, with the file still open. The file pointer will stay wherever we stopped reading, waiting for the next <code>readline()</code> command. In the meantime, the cache now has more items in it, and if we start all over again trying to pluralize a new word, each of those items in the cache will be tried before reading the next line from the pattern file.
</ul>
<p>We have achieved pluralization nirvana.
<ol>
<li><strong>Minimal startup cost.</strong> The only thing that happens on <code>import</code> is instantiating a single class and opening a file (but not reading from it).
<li><strong>Maximum performance.</strong> The previous example would read through the file and build functions dynamically every time you wanted to pluralize a word. This version will cache functions as soon as they&#8217;re built, and in the worst case, it will only read through the pattern file once, no matter how many words you pluralize.
<li><strong>Separation of code and data.</strong> All the patterns are stored in a separate file. Code is code, and data is data, and never the twain shall meet.
</ol>
<blockquote class=note>
<p><span class=u>&#x261E;</span>Is this really nirvana? Well, yes and no. Here&#8217;s something to consider with the <code>LazyRules</code> example: the pattern file is opened (during <code>__init__()</code>), and it remains open until the final rule is reached. Python will eventually close the file when it exits, or after the last instantiation of the <code>LazyRules</code> class is destroyed, but still, that could be a <em>long</em> time. If this class is part of a long-running Python process, the Python interpreter may never exit, and the <code>LazyRules</code> object may never get destroyed.
<p>There are ways around this. Instead of opening the file during <code>__init__()</code> and leaving it open while you read rules one line at a time, you could open the file, read all the rules, and immediately close the file. Or you could open the file, read one rule, save the file position with the <a href=files.html#read><code>tell()</code> method</a>, close the file, and later re-open it and use the <a href=files.html#read><code>seek()</code> method</a> to continue reading where you left off. Or you could not worry about it and just leave the file open, like this example code does. Programming is design, and design is all about trade-offs and constraints. Leaving a file open too long might be a problem; making your code more complicated might be a problem. Which one is the bigger problem depends on your development team, your application, and your runtime environment.
</blockquote>
<p class=a>&#x2042;
<h2 id=furtherreading>Further Reading</h2>
<ul>
<li><a href=http://docs.python.org/3.1/library/stdtypes.html#iterator-types>Iterator types</a>
<li><a href=http://www.python.org/dev/peps/pep-0234/>PEP 234: Iterators</a>
<li><a href=http://www.python.org/dev/peps/pep-0255/>PEP 255: Simple Generators</a>
<li><a href=http://www.dabeaz.com/generators/>Generator Tricks for Systems Programmers</a>
</ul>
<p class=v><a href=generators.html rel=prev title='back to &#8220;Closures &amp; Generators&#8221;'><span class=u>&#x261C;</span></a> <a href=advanced-iterators.html rel=next title='onward to &#8220;Advanced Iterators&#8221;'><span class=u>&#x261E;</span></a>
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<script src=j/jquery.js></script>
<script src=j/prettify.js></script>
<script src=j/dip3.js></script>
j/.htaccess
+4 -4
View File
@@ -1,4 +1,4 @@
FileETag MTime Size
ExpiresActive On
ExpiresDefault "access plus 1 year"
FileETag MTime Size
ExpiresActive On
ExpiresDefault "access plus 1 year"
j/html5.js
+3 -1
View File
@@ -1 +1,3 @@
(function(){var e="abbr,article,aside,audio,bb,canvas,datagrid,datalist,details,dialog,figure,footer,header,mark,menu,meter,nav,output,progress,section,time,video".split(','),i=e.length;while(i--){document.createElement(e[i])}})()
/*@cc_on@if(@_jscript_version<9)(function(p,e){function q(a,b){if(g[a])g[a].styleSheet.cssText+=b;else{var c=r[l],d=e[j]("style");d.media=a;c.insertBefore(d,c[l]);g[a]=d;q(a,b)}}function s(a,b){for(var c=new RegExp("\\b("+m+")\\b(?!.*[;}])","gi"),d=function(k){return".iepp_"+k},h=-1;++h<a.length;){b=a[h].media||b;s(a[h].imports,b);q(b,a[h].cssText.replace(c,d))}}function t(){for(var a,b=e.getElementsByTagName("*"),c,d,h=new RegExp("^"+m+"$","i"),k=-1;++k<b.length;)if((a=b[k])&&(d=a.nodeName.match(h))){c=new RegExp("^\\s*<"+d+"(.*)\\/"+
d+">\\s*$","i");i.innerHTML=a.outerHTML.replace(/\r|\n/g," ").replace(c,a.currentStyle.display=="block"?"<div$1/div>":"<span$1/span>");c=i.childNodes[0];c.className+=" iepp_"+d;c=f[f.length]=[a,c];a.parentNode.replaceChild(c[1],c[0])}s(e.styleSheets,"all")}function u(){for(var a=-1,b;++a<f.length;)f[a][1].parentNode.replaceChild(f[a][0],f[a][1]);for(b in g)r[l].removeChild(g[b]);g={};f=[]}for(var m="abbr article aside audio canvas command datalist details figure figcaption footer header hgroup mark meter nav output progress section summary time video".replace(/ /g,
"|"),n=m.split("|"),r=e.documentElement,i=e.createDocumentFragment(),g={},f=[],o=-1,l="firstChild",j="createElement";++o<n.length;){e[j](n[o]);i[j](n[o])}i=i.appendChild(e[j]("div"));p.attachEvent("onbeforeprint",t);p.attachEvent("onafterprint",u)})(this,document)@end@*/
layout.css
+74 -74
View File
@@ -1,74 +1,74 @@
/*
"Dive Into Python 3" layout stylesheet
Copyright (c) 2009, Mark Pilgrim, All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS'
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
*/
@page {
size: US-Letter;
margin: 1.75in;
padding: 0;
@bottom-center {
font: 12pt/1.75 serif;
content: counter(page);
}
}
body, .w a {
font: 10pt/1.3 serif;
}
pre, kbd, samp, code, var, .b {
font: 8pt/1.3 monospace;
}
span {
font-size: 10pt;
}
.baa {
font-size: 11pt;
}
.q span {
font-size: 13pt;
}
.f:first-letter {
color: #888;
font: normal 48pt/0.68 serif;
}
p, ul, ol {
margin: 0;
font-size: 11pt;
}
p + p {
text-indent: 1em;
}
h1 {
page-break-before: always;
prince-bookmark-level: 1;
}
h2 {
prince-bookmark-level: 2;
}
h3 {
prince-bookmark-level: 3;
}
/*
"Dive Into Python 3" layout stylesheet
Copyright (c) 2009, Mark Pilgrim, All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS'
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
*/
@page {
size: US-Letter;
margin: 1.75in;
padding: 0;
@bottom-center {
font: 12pt/1.75 serif;
content: counter(page);
}
}
body, .w a {
font: 10pt/1.3 serif;
}
pre, kbd, samp, code, var, .b {
font: 8pt/1.3 monospace;
}
span {
font-size: 10pt;
}
.baa {
font-size: 11pt;
}
.q span {
font-size: 13pt;
}
.f:first-letter {
color: #888;
font: normal 48pt/0.68 serif;
}
p, ul, ol {
margin: 0;
font-size: 11pt;
}
p + p {
text-indent: 1em;
}
h1 {
page-break-before: always;
prince-bookmark-level: 1;
}
h2 {
prince-bookmark-level: 2;
}
h3 {
prince-bookmark-level: 3;
}
mobile.css
+112 -112
View File
@@ -1,112 +1,112 @@
/*
"Dive Into Python 3" mobile stylesheet for iPhone, Android, and other
small-screen devices
Copyright (c) 2009, Mark Pilgrim, All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS'
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
Acknowledgements & Inspirations
"Return of the Mobile Style Sheet" ....................... http://www.alistapart.com/articles/returnofthemobilestylesheet
"Optimizing Web Content Using Conditional CSS" ........... http://developer.apple.com/safari/library/documentation/AppleApplications/Reference/SafariWebContent/OptimizingforSafarioniPhone/chapter_3_section_2.html
*/
/* typography */
body, .c, span, pre span, .c, .note, p, ul, ol {
font:normal 12px/18px sans-serif;
}
pre, kbd, samp, code, var {
font:normal 12px/18px monospace;
}
.baa {
font:normal 14px/18px serif;
}
abbr {
font-variant:normal;
text-transform:none;
letter-spacing:0;
}
.c, .note, p, ul, ol, h2, h3 {
margin:1.75em 0;
}
/* basics */
html {
color:#000;
}
body {
margin:4px 2px 0 2px;
}
/* links */
a {
text-decoration:underline;
border-bottom:0;
}
pre a {
text-decoration:none;
}
/* headers and pullquotes */
h1, h2, h3, pre {
padding:0;
border:0;
letter-spacing:0;
}
h1 {
margin:0;
}
h1, h1 code {
font:normal 18px/18px serif;
}
h2, h2 code {
font:normal 16px/18px serif;
}
h3, h3 code {
font:normal 14px/18px serif;
}
h1:before {
content:"";
}
/* overrides */
.nm, .w, aside, form, form+p, .note span, .q span, .a {
display:none;
}
dd {
margin:0 0 0 1.75em;
}
.nav span {
font-size:200%;
}
.xxxl {
font-size: xx-large;
line-height: 0.875;
}
/*
"Dive Into Python 3" mobile stylesheet for iPhone, Android, and other
small-screen devices
Copyright (c) 2009, Mark Pilgrim, All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS'
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
Acknowledgements & Inspirations
"Return of the Mobile Style Sheet" ....................... http://www.alistapart.com/articles/returnofthemobilestylesheet
"Optimizing Web Content Using Conditional CSS" ........... http://developer.apple.com/safari/library/documentation/AppleApplications/Reference/SafariWebContent/OptimizingforSafarioniPhone/chapter_3_section_2.html
*/
/* typography */
body, .c, span, pre span, .c, .note, p, ul, ol {
font:normal 12px/18px sans-serif;
}
pre, kbd, samp, code, var {
font:normal 12px/18px monospace;
}
.baa {
font:normal 14px/18px serif;
}
abbr {
font-variant:normal;
text-transform:none;
letter-spacing:0;
}
.c, .note, p, ul, ol, h2, h3 {
margin:1.75em 0;
}
/* basics */
html {
color:#000;
}
body {
margin:4px 2px 0 2px;
}
/* links */
a {
text-decoration:underline;
border-bottom:0;
}
pre a {
text-decoration:none;
}
/* headers and pullquotes */
h1, h2, h3, pre {
padding:0;
border:0;
letter-spacing:0;
}
h1 {
margin:0;
}
h1, h1 code {
font:normal 18px/18px serif;
}
h2, h2 code {
font:normal 16px/18px serif;
}
h3, h3 code {
font:normal 14px/18px serif;
}
h1:before {
content:"";
}
/* overrides */
.nm, .w, aside, form, form+p, .note span, .q span, .a {
display:none;
}
dd {
margin:0 0 0 1.75em;
}
.nav span {
font-size:200%;
}
.xxxl {
font-size: xx-large;
line-height: 0.875;
}
packaging.html
+577 -577
View File
File diff suppressed because it is too large Load Diff
prince.css
+59 -59
View File
@@ -1,59 +1,59 @@
/*
"Dive Into Python 3" Prince stylesheet
Copyright (c) 2009, Mark Pilgrim, All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS'
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
*/
/* some Prince-specific rules to generate a nicer PDF */
/* see http://www.princexml.com/ */
@page {
size: US-Letter;
margin: 30pt;
padding: 0;
@bottom-center {
font: 12pt/1.75 'Gill Sans', 'Gill Sans MT', Helvetica, Corbel, 'Nimbus Sans L', sans-serif;
content: counter(page);
}
}
pre {
page-break-inside: avoid;
}
h1 {
page-break-before: always;
prince-bookmark-level: 1;
}
h2 {
prince-bookmark-level: 2;
}
h3 {
prince-bookmark-level: 3;
}
ul, ol {
margin: 1.75em 20pt;
}
abbr {
text-decoration: none;
}
/*
"Dive Into Python 3" Prince stylesheet
Copyright (c) 2009, Mark Pilgrim, All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS'
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
*/
/* some Prince-specific rules to generate a nicer PDF */
/* see http://www.princexml.com/ */
@page {
size: US-Letter;
margin: 30pt;
padding: 0;
@bottom-center {
font: 12pt/1.75 'Gill Sans', 'Gill Sans MT', Helvetica, Corbel, 'Nimbus Sans L', sans-serif;
content: counter(page);
}
}
pre {
page-break-inside: avoid;
}
h1 {
page-break-before: always;
prince-bookmark-level: 1;
}
h2 {
prince-bookmark-level: 2;
}
h3 {
prince-bookmark-level: 3;
}
ul, ol {
margin: 1.75em 20pt;
}
abbr {
text-decoration: none;
}
publish
+4 -7
View File
@@ -2,7 +2,6 @@
die () {
echo "$1" >/dev/stderr
[ -n "$(which Snarl_CMD 2>/dev/null)" ] && Snarl_CMD snShowMessage 10 "Dive Into Python 3" "$1." "C:\Users\pilgrim\site-lisp\todochiku-icons\alert.png"
exit 1
}
@@ -119,9 +118,9 @@ java -jar util/yuicompressor-2.4.2.jar build/dip3.css > build/$revision.css && \
echo "inlining CSS, minimizing URLs, adding evil tracking code"
ga=`cat j/ga.js`
for f in build/*.html; do
css=`python2.6 util/lesscss.py "$f" "build/$revision.css"` || die "Failed to remove unused CSS"
mobilecss=`python2.6 util/lesscss.py "$f" "build/m-$revision.css"` || die "Failed to remove unused CSS"
printcss=`python2.6 util/lesscss.py "$f" "build/p-$revision.css"` || die "Failed to remove unused CSS"
css=`python2.5 util/lesscss.py "$f" "build/$revision.css"` || die "Failed to remove unused CSS"
mobilecss=`python2.5 util/lesscss.py "$f" "build/m-$revision.css"` || die "Failed to remove unused CSS"
printcss=`python2.5 util/lesscss.py "$f" "build/p-$revision.css"` || die "Failed to remove unused CSS"
sed -i -e "s|<link rel=stylesheet href=dip3.css>|<style>${css}</style>|g" -e "s|<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>|<style>@media screen and (max-device-width:480px){${mobilecss}}</style>|g" -e "s|<link rel=stylesheet media=print href=print.css>|<style>@media print{${printcss}}</style>|g" -e "s|</style><style>||g" -e "s|href=index.html|href=/|g" -e "s|</style>|</style>${ga}|g" "$f" || die "Failed to inline CSS"
done
@@ -130,7 +129,7 @@ chmod 755 build/examples build/j build/i build/d && \
chmod 644 build/*.html build/*.css build/*.txt build/*.zip build/examples/* build/examples/.htaccess build/j/* build/j/.htaccess build/i/* build/i/.htaccess build/d/.htaccess build/.htaccess || die "Failed to reset file permissions"
# ship it!
#die "Aborting without publishing"
die "Aborting without publishing"
echo -n "publishing"
rsync -essh -a build/d/.htaccess build/*.zip diveintomark.org:~/web/diveintopython3.org/d/ && \
echo -n "." && \
@@ -140,5 +139,3 @@ rsync -essh -a build/d/.htaccess build/*.zip diveintomark.org:~/web/diveintopyth
echo -n "." && \
rsync -essh -a build/examples build/*.txt build/*.html build/.htaccess diveintomark.org:~/web/diveintopython3.org/ && \
echo "." || die "Failed to publish to remote server"
[ -n "$(which Snarl_CMD 2>/dev/null)" ] && Snarl_CMD snShowMessage 10 "Dive Into Python 3" "Published." "C:\Users\pilgrim\site-lisp\todochiku-icons\clean.png"
table-of-contents.html
+446 -446
View File
@@ -1,446 +1,446 @@
<!DOCTYPE html>
<meta charset=utf-8>
<title>Table of contents - Dive Into Python 3</title>
<link rel=stylesheet href=dip3.css>
<style>
h1:before{content:''}
ol,ul{font-weight:bold}
li ol{font-weight:normal}
#porting-code-to-python-3-with-2to3,#special-method-names,#where-to-go-from-here{list-style:none;margin:0 0 0 -2em}
#porting-code-to-python-3-with-2to3 > ol,#special-method-names > ol,#where-to-go-from-here > ol{margin:0;padding:0 0 0 4.5em}
#porting-code-to-python-3-with-2to3:before{content:'A. \00a0 \00a0'}
#special-method-names:before{content:'B. \00a0 \00a0'}
#where-to-go-from-here:before{content:'C. \00a0 \00a0'}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8><input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> Dive Into Python 3 <span class=u>&#8227;</span>
<h1>Table of Contents</h1>
<!-- toc -->
<ol start=-1>
<li id=whats-new><a href=whats-new.html>What&#8217;s New in &#8220;Dive Into Python 3&#8221;</a>
<ol>
<li><a href=whats-new.html#divingin><i>a.k.a.</i> &#8220;the minus level&#8221;</a>
</ol>
<li id=installing-python><a href=installing-python.html>Installing Python</a>
<ol>
<li><a href=installing-python.html#divingin>Diving In</a>
<li><a href=installing-python.html#which>Which Python Is Right For You?</a>
<li><a href=installing-python.html#windows>Installing on Microsoft Windows</a>
<li><a href=installing-python.html#macosx>Installing on Mac OS X</a>
<li><a href=installing-python.html#ubuntu>Installing on Ubuntu Linux</a>
<li><a href=installing-python.html#other>Installing on Other Platforms</a>
<li><a href=installing-python.html#idle>Using The Python Shell</a>
<li><a href=installing-python.html#editors>Python Editors and IDEs</a>
</ol>
<li id=your-first-python-program><a href=your-first-python-program.html>Your First Python Program</a>
<ol>
<li><a href=your-first-python-program.html#divingin>Diving In</a>
<li><a href=your-first-python-program.html#declaringfunctions>Declaring Functions</a>
<ol>
<li><a href=your-first-python-program.html#optional-arguments>Optional and Named Arguments</a>
</ol>
<li><a href=your-first-python-program.html#readability>Writing Readable Code</a>
<ol>
<li><a href=your-first-python-program.html#docstrings>Documentation Strings</a>
</ol>
<li><a href=your-first-python-program.html#importsearchpath>The <code>import</code> Search Path</a>
<li><a href=your-first-python-program.html#everythingisanobject>Everything Is An Object</a>
<ol>
<li><a href=your-first-python-program.html#whatsanobject>What&#8217;s An Object?</a>
</ol>
<li><a href=your-first-python-program.html#indentingcode>Indenting Code</a>
<li><a href=your-first-python-program.html#exceptions>Exceptions</a>
<ol>
<li><a href=your-first-python-program.html#importerror>Catching Import Errors</a>
</ol>
<li><a href=your-first-python-program.html#nameerror>Unbound Variables</a>
<li><a href=your-first-python-program.html#case>Everything is Case-Sensitive</a>
<li><a href=your-first-python-program.html#runningscripts>Running Scripts</a>
<li><a href=your-first-python-program.html#furtherreading>Further Reading</a>
</ol>
<li id=native-datatypes><a href=native-datatypes.html>Native Datatypes</a>
<ol>
<li><a href=native-datatypes.html#divingin>Diving In</a>
<li><a href=native-datatypes.html#booleans>Booleans</a>
<li><a href=native-datatypes.html#numbers>Numbers</a>
<ol>
<li><a href=native-datatypes.html#number-coercion>Coercing Integers To Floats And Vice-Versa</a>
<li><a href=native-datatypes.html#common-numerical-operations>Common Numerical Operations</a>
<li><a href=native-datatypes.html#fractions>Fractions</a>
<li><a href=native-datatypes.html#trig>Trigonometry</a>
<li><a href=native-datatypes.html#numbers-in-a-boolean-context>Numbers In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#lists>Lists</a>
<ol>
<li><a href=native-datatypes.html#creatinglists>Creating A List</a>
<li><a href=native-datatypes.html#slicinglists>Slicing A List</a>
<li><a href=native-datatypes.html#extendinglists>Adding Items To A List</a>
<li><a href=native-datatypes.html#searchinglists>Searching For Values In A List</a>
<li><a href=native-datatypes.html#removingfromlists>Removing Items From A List</a>
<li><a href=native-datatypes.html#popgoestheweasel>Removing Items From A List: Bonus Round</a>
<li><a href=native-datatypes.html#lists-in-a-boolean-context>Lists In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#tuples>Tuples</a>
<ol>
<li><a href=native-datatypes.html#tuples-in-a-boolean-context>Tuples In A Boolean Context</a>
<li><a href=native-datatypes.html#multivar>Assigning Multiple Values At Once</a>
</ol>
<li><a href=native-datatypes.html#sets>Sets</a>
<ol>
<li><a href=native-datatypes.html#creating-a-set>Creating A Set</a>
<li><a href=native-datatypes.html#modifying-sets>Modifying A Set</a>
<li><a href=native-datatypes.html#removing-from-sets>Removing Items From A Set</a>
<li><a href=native-datatypes.html#common-set-operations>Common Set Operations</a>
<li><a href=native-datatypes.html#sets-in-a-boolean-context>Sets In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#dictionaries>Dictionaries</a>
<ol>
<li><a href=native-datatypes.html#creating-dictionaries>Creating A Dictionary</a>
<li><a href=native-datatypes.html#modifying-dictionaries>Modifying A Dictionary</a>
<li><a href=native-datatypes.html#mixed-value-dictionaries>Mixed-Value Dictionaries</a>
<li><a href=native-datatypes.html#dictionaries-in-a-boolean-context>Dictionaries In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#none><code>None</code></a>
<ol>
<li><a href=native-datatypes.html#none-in-a-boolean-context><code>None</code> In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#furtherreading>Further Reading</a>
</ol>
<li id=comprehensions><a href=comprehensions.html>Comprehensions</a>
<ol>
<li><a href=comprehensions.html#divingin>Diving In</a>
<li><a href=comprehensions.html#os>Working With Files And Directories</a>
<ol>
<li><a href=comprehensions.html#getcwd>The Current Working Directory</a>
<li><a href=comprehensions.html#ospath>Working With Filenames and Directory Names</a>
<li><a href=comprehensions.html#glob>Listing Directories</a>
<li><a href=comprehensions.html#osstat>Getting File Metadata</a>
<li><a href=comprehensions.html#abspath>Constructing Absolute Pathnames</a>
</ol>
<li><a href=comprehensions.html#listcomprehension>List Comprehensions</a>
<li><a href=comprehensions.html#dictionarycomprehension>Dictionary Comprehensions</a>
<ol>
<li><a href=comprehensions.html#stupiddicttricks>Other Fun Stuff To Do With Dictionary Comprehensions</a>
</ol>
<li><a href=comprehensions.html#setcomprehension>Set Comprehensions</a>
<li><a href=comprehensions.html#furtherreading>Further Reading</a>
</ol>
<li id=strings><a href=strings.html>Strings</a>
<ol>
<li><a href=strings.html#boring-stuff>Some Boring Stuff You Need To Understand Before You Can Dive In</a>
<li><a href=strings.html#one-ring-to-rule-them-all>Unicode</a>
<li><a href=strings.html#divingin>Diving In</a>
<li><a href=strings.html#formatting-strings>Formatting Strings</a>
<ol>
<li><a href=strings.html#compound-field-names>Compound Field Names</a>
<li><a href=strings.html#format-specifiers>Format Specifiers</a>
</ol>
<li><a href=strings.html#common-string-methods>Other Common String Methods</a>
<ol>
<li><a href=strings.html#slicingstrings>Slicing A String</a>
</ol>
<li><a href=strings.html#byte-arrays>Strings vs. Bytes</a>
<li><a href=strings.html#py-encoding>Postscript: Character Encoding Of Python Source Code</a>
<li><a href=strings.html#furtherreading>Further Reading</a>
</ol>
<li id=regular-expressions><a href=regular-expressions.html>Regular Expressions</a>
<ol>
<li><a href=regular-expressions.html#divingin>Diving In</a>
<li><a href=regular-expressions.html#streetaddresses>Case Study: Street Addresses</a>
<li><a href=regular-expressions.html#romannumerals>Case Study: Roman Numerals</a>
<ol>
<li><a href=regular-expressions.html#thousands>Checking For Thousands</a>
<li><a href=regular-expressions.html#hundreds>Checking For Hundreds</a>
</ol>
<li><a href=regular-expressions.html#nmsyntax>Using The <code>{n,m}</code> Syntax</a>
<ol>
<li><a href=regular-expressions.html#tensandones>Checking For Tens And Ones</a>
</ol>
<li><a href=regular-expressions.html#verbosere>Verbose Regular Expressions</a>
<li><a href=regular-expressions.html#phonenumbers>Case study: Parsing Phone Numbers</a>
<li><a href=regular-expressions.html#summary>Summary</a>
</ol>
<li id=generators><a href=generators.html>Closures <i class=baa>&amp;</i> Generators</a>
<ol>
<li><a href=generators.html#divingin>Diving In</a>
<li><a href=generators.html#i-know>I Know, Let&#8217;s Use Regular Expressions!</a>
<li><a href=generators.html#a-list-of-functions>A List Of Functions</a>
<li><a href=generators.html#a-list-of-patterns>A List Of Patterns</a>
<li><a href=generators.html#a-file-of-patterns>A File Of Patterns</a>
<li><a href=generators.html#generators>Generators</a>
<ol>
<li><a href=generators.html#a-fibonacci-generator>A Fibonacci Generator</a>
<li><a href=generators.html#a-plural-rule-generator>A Plural Rule Generator</a>
</ol>
<li><a href=generators.html#furtherreading>Further Reading</a>
</ol>
<li id=iterators><a href=iterators.html>Classes <i class=baa>&amp;</i> Iterators</a>
<ol>
<li><a href=iterators.html#divingin>Diving In</a>
<li><a href=iterators.html#defining-classes>Defining Classes</a>
<ol>
<li><a href=iterators.html#init-method>The <code>__init__()</code> Method</a>
</ol>
<li><a href=iterators.html#instantiating-classes>Instantiating Classes</a>
<li><a href=iterators.html#instance-variables>Instance Variables</a>
<li><a href=iterators.html#a-fibonacci-iterator>A Fibonacci Iterator</a>
<li><a href=iterators.html#a-plural-rule-iterator>A Plural Rule Iterator</a>
<li><a href=iterators.html#furtherreading>Further Reading</a>
</ol>
<li id=advanced-iterators><a href=advanced-iterators.html>Advanced Iterators</a>
<ol>
<li><a href=advanced-iterators.html#divingin>Diving In</a>
<li><a href=advanced-iterators.html#re-findall>Finding all occurrences of a pattern</a>
<li><a href=advanced-iterators.html#unique-items>Finding the unique items in a sequence</a>
<li><a href=advanced-iterators.html#assert>Making assertions</a>
<li><a href=advanced-iterators.html#generator-expressions>Generator expressions</a>
<li><a href=advanced-iterators.html#permutations>Calculating Permutations&hellip; The Lazy Way!</a>
<li><a href=advanced-iterators.html#more-itertools>Other Fun Stuff in the <code>itertools</code> Module</a>
<li><a href=advanced-iterators.html#string-translate>A New Kind Of String Manipulation</a>
<li><a href=advanced-iterators.html#eval>Evaluating Arbitrary Strings As Python Expressions</a>
<li><a href=advanced-iterators.html#alphametics-finale>Putting It All Together</a>
<li><a href=advanced-iterators.html#furtherreading>Further Reading</a>
</ol>
<li id=unit-testing><a href=unit-testing.html>Unit Testing</a>
<ol>
<li><a href=unit-testing.html#divingin>(Not) Diving In</a>
<li><a href=unit-testing.html#romantest1>A Single Question</a>
<li><a href=unit-testing.html#romantest2>&#8220;Halt And Catch Fire&#8221;</a>
<li><a href=unit-testing.html#romantest3>More Halting, More Fire</a>
<li><a href=unit-testing.html#romantest4>And One More Thing&hellip;</a>
<li><a href=unit-testing.html#romantest5>A Pleasing Symmetry</a>
<li><a href=unit-testing.html#romantest6>More Bad Input</a>
</ol>
<li id=refactoring><a href=refactoring.html>Refactoring</a>
<ol>
<li><a href=refactoring.html#divingin>Diving In</a>
<li><a href=refactoring.html#changing-requirements>Handling Changing Requirements</a>
<li><a href=refactoring.html#refactoring>Refactoring</a>
<li><a href=refactoring.html#summary>Summary</a>
</ol>
<li id=files><a href=files.html>Files</a>
<ol>
<li><a href=files.html#divingin>Diving In</a>
<li><a href=files.html#reading>Reading From Text Files</a>
<ol>
<li><a href=files.html#encoding>Character Encoding Rears Its Ugly Head</a>
<li><a href=files.html#file-objects>Stream Objects</a>
<li><a href=files.html#read>Reading Data From A Text File</a>
<li><a href=files.html#close>Closing Files</a>
<li><a href=files.html#with>Closing Files Automatically</a>
<li><a href=files.html#for>Reading Data One Line At A Time</a>
</ol>
<li><a href=files.html#writing>Writing to Text Files</a>
<ol>
<li><a href=files.html#encoding-again>Character Encoding Again</a>
</ol>
<li><a href=files.html#binary>Binary Files</a>
<li><a href=files.html#file-like-objects>Stream Objects From Non-File Sources</a>
<ol>
<li><a href=files.html#gzip>Handling Compressed Files</a>
</ol>
<li><a href=files.html#stdio>Standard Input, Output, and Error</a>
<ol>
<li><a href=files.html#redirect>Redirecting Standard Output</a>
</ol>
<li><a href=files.html#furtherreading>Further Reading</a>
</ol>
<li id=xml><a href=xml.html>XML</a>
<ol>
<li><a href=xml.html#divingin>Diving In</a>
<li><a href=xml.html#xml-intro>A 5-Minute Crash Course in XML</a>
<li><a href=xml.html#xml-structure>The Structure Of An Atom Feed</a>
<li><a href=xml.html#xml-parse>Parsing XML</a>
<ol>
<li><a href=xml.html#xml-elements>Elements Are Lists</a>
<li><a href=xml.html#xml-attributes>Attributes Are Dictonaries</a>
</ol>
<li><a href=xml.html#xml-find>Searching For Nodes Within An XML Document</a>
<li><a href=xml.html#xml-lxml>Going Further With lxml</a>
<li><a href=xml.html#xml-generate>Generating XML</a>
<li><a href=xml.html#xml-custom-parser>Parsing Broken XML</a>
<li><a href=xml.html#furtherreading>Further Reading</a>
</ol>
<li id=serializing><a href=serializing.html>Serializing Python Objects</a>
<ol>
<li><a href=serializing.html#divingin>Diving In</a>
<ol>
<li><a href=serializing.html#administrivia>A Quick Note About The Examples in This Chapter</a>
</ol>
<li><a href=serializing.html#dump>Saving Data to a Pickle File</a>
<li><a href=serializing.html#load>Loading Data from a Pickle File</a>
<li><a href=serializing.html#dumps>Pickling Without a File</a>
<li><a href=serializing.html#protocol-versions>Bytes and Strings Rear Their Ugly Heads Again</a>
<li><a href=serializing.html#debugging>Debugging Pickle Files</a>
<li><a href=serializing.html#json>Serializing Python Objects to be Read by Other Languages</a>
<li><a href=serializing.html#json-dump>Saving Data to a <abbr>JSON</abbr> File</a>
<li><a href=serializing.html#json-types>Mapping of Python Datatypes to <abbr>JSON</abbr></a>
<li><a href=serializing.html#json-unknown-types>Serializing Datatypes Unsupported by <abbr>JSON</abbr></a>
<li><a href=serializing.html#json-load>Loading Data from a <abbr>JSON</abbr> File</a>
<li><a href=serializing.html#furtherreading>Further Reading</a>
</ol>
<li id=http-web-services><a href=http-web-services.html>HTTP Web Services</a>
<ol>
<li><a href=http-web-services.html#divingin>Diving In</a>
<li><a href=http-web-services.html#http-features>Features of HTTP</a>
<ol>
<li><a href=http-web-services.html#caching>Caching</a>
<li><a href=http-web-services.html#last-modified>Last-Modified Checking</a>
<li><a href=http-web-services.html#etags>ETag Checking</a>
<li><a href=http-web-services.html#compression>Compression</a>
<li><a href=http-web-services.html#redirects>Redirects</a>
</ol>
<li><a href=http-web-services.html#dont-try-this-at-home>How Not To Fetch Data Over HTTP</a>
<li><a href=http-web-services.html#whats-on-the-wire>What&#8217;s On The Wire?</a>
<li><a href=http-web-services.html#introducing-httplib2>Introducing <code>httplib2</code></a>
<ol>
<li><a href=http-web-services.html#why-bytes>A Short Digression To Explain Why <code>httplib2</code> Returns Bytes Instead of Strings</a>
<li><a href=http-web-services.html#httplib2-caching>How <code>httplib2</code> Handles Caching</a>
<li><a href=http-web-services.html#httplib2-etags>How <code>httplib2</code> Handles <code>Last-Modified</code> and <code>ETag</code> Headers</a>
<li><a href=http-web-services.html#httplib2-compression>How <code>http2lib</code> Handles Compression</a>
<li><a href=http-web-services.html#httplib2-redirects>How <code>httplib2</code> Handles Redirects</a>
</ol>
<li><a href=http-web-services.html#beyond-get>Beyond HTTP GET</a>
<li><a href=http-web-services.html#beyond-post>Beyond HTTP POST</a>
<li><a href=http-web-services.html#furtherreading>Further Reading</a>
</ol>
<li id=case-study-porting-chardet-to-python-3><a href=case-study-porting-chardet-to-python-3.html>Case Study: Porting <code>chardet</code> to Python 3</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#divingin>Diving In</a>
<li><a href=case-study-porting-chardet-to-python-3.html#faq.what>What is Character Encoding Auto-Detection?</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#faq.impossible>Isn&#8217;t That Impossible?</a>
<li><a href=case-study-porting-chardet-to-python-3.html#faq.who>Does Such An Algorithm Exist?</a>
</ol>
<li><a href=case-study-porting-chardet-to-python-3.html#divingin2>Introducing The <code>chardet</code> Module</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#how.bom><abbr>UTF-n</abbr> With A <abbr>BOM</abbr></a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.esc>Escaped Encodings</a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.mb>Multi-Byte Encodings</a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.sb>Single-Byte Encodings</a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.windows1252><code>windows-1252</code></a>
</ol>
<li><a href=case-study-porting-chardet-to-python-3.html#running2to3>Running <code>2to3</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#multifile-modules>A Short Digression Into Multi-File Modules</a>
<li><a href=case-study-porting-chardet-to-python-3.html#manual>Fixing What <code>2to3</code> Can&#8217;t</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#falseisinvalidsyntax><code>False</code> is invalid syntax</a>
<li><a href=case-study-porting-chardet-to-python-3.html#nomodulenamedconstants>No module named <code>constants</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#namefileisnotdefined>Name <var>'file'</var> is not defined</a>
<li><a href=case-study-porting-chardet-to-python-3.html#cantuseastringpattern>Can&#8217;t use a string pattern on a bytes-like object</a>
<li><a href=case-study-porting-chardet-to-python-3.html#cantconvertbytesobject>Can't convert <code>'bytes'</code> object to <code>str</code> implicitly</a>
<li><a href=case-study-porting-chardet-to-python-3.html#unsupportedoperandtypeforplus>Unsupported operand type(s) for +: <code>'int'</code> and <code>'bytes'</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#ordexpectedstring><code>ord()</code> expected string of length 1, but <code>int</code> found</a>
<li><a href=case-study-porting-chardet-to-python-3.html#unorderabletypes>Unorderable types: <code>int()</code> >= <code>str()</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#reduceisnotdefined>Global name <code>'reduce'</code> is not defined</a>
</ol>
<li><a href=case-study-porting-chardet-to-python-3.html#summary>Summary</a>
</ol>
<li id=packaging><a href=packaging.html>Packaging Python Libraries</a>
<ol>
<li><a href=packaging.html#divingin>Diving In</a>
<li><a href=packaging.html#cantdo>Things Distutils Can&#8217;t Do For You</a>
<li><a href=packaging.html#structure>Directory Structure</a>
<li><a href=packaging.html#setuppy>Writing Your Setup Script</a>
<li><a href=packaging.html#trove>Classifying Your Package</a>
<ol>
<li><a href=packaging.html#trove-examples>Examples of Good Package Classifiers</a>
</ol>
<li><a href=packaging.html#manifest>Specifying Additional Files With A Manifest</a>
<li><a href=packaging.html#check>Checking Your Setup Script for Errors</a>
<li><a href=packaging.html#sdist>Creating a Source Distribution</a>
<li><a href=packaging.html#bdist>Creating a Graphical Installer</a>
<ol>
<li><a href=packaging.html#linux>Building Installable Packages for Other Operating Systems</a>
</ol>
<li><a href=packaging.html#pypi>Adding Your Software to The Python Package Index</a>
<li><a href=packaging.html#future>The Many Possible Futures of Python Packaging</a>
<li><a href=packaging.html#furtherreading>Further Reading</a>
</ol>
<li id=porting-code-to-python-3-with-2to3><a href=porting-code-to-python-3-with-2to3.html>Porting Code to Python 3 with <code>2to3</code></a>
<ol>
<li><a href=porting-code-to-python-3-with-2to3.html#divingin>Diving In</a>
<li><a href=porting-code-to-python-3-with-2to3.html#print><code>print</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#unicodeliteral>Unicode string literals</a>
<li><a href=porting-code-to-python-3-with-2to3.html#unicode><code>unicode()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#long><code>long</code> data type</a>
<li><a href=porting-code-to-python-3-with-2to3.html#ne>&lt;> comparison</a>
<li><a href=porting-code-to-python-3-with-2to3.html#has_key><code>has_key()</code> dictionary method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#dict>Dictionary methods that return lists</a>
<li><a href=porting-code-to-python-3-with-2to3.html#imports>Modules that have been renamed or reorganized</a>
<ol>
<li><a href=porting-code-to-python-3-with-2to3.html#http><code>http</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#urllib><code>urllib</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#dbm><code>dbm</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#xmlrpc><code>xmlrpc</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#othermodules>Other modules</a>
</ol>
<li><a href=porting-code-to-python-3-with-2to3.html#import>Relative imports within a package</a>
<li><a href=porting-code-to-python-3-with-2to3.html#next><code>next()</code> iterator method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#filter><code>filter()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#map><code>map()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#reduce><code>reduce()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#apply><code>apply()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#intern><code>intern()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#exec><code>exec</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#execfile><code>execfile</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#repr><code>repr</code> literals (backticks)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#except><code>try...except</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#raise><code>raise</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#throw><code>throw</code> method on generators</a>
<li><a href=porting-code-to-python-3-with-2to3.html#xrange><code>xrange()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#raw_input><code>raw_input()</code> and <code>input()</code> global functions</a>
<li><a href=porting-code-to-python-3-with-2to3.html#funcattrs><code>func_*</code> function attributes</a>
<li><a href=porting-code-to-python-3-with-2to3.html#xreadlines><code>xreadlines()</code> I/O method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#tuple_params><code>lambda</code> functions that take a tuple instead of multiple parameters</a>
<li><a href=porting-code-to-python-3-with-2to3.html#methodattrs>Special method attributes</a>
<li><a href=porting-code-to-python-3-with-2to3.html#nonzero><code>__nonzero__</code> special method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#numliterals>Octal literals</a>
<li><a href=porting-code-to-python-3-with-2to3.html#renames><code>sys.maxint</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#callable><code>callable()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#zip><code>zip()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#standarderror><code>StandardError</code> exception</a>
<li><a href=porting-code-to-python-3-with-2to3.html#types><code>types</code> module constants</a>
<li><a href=porting-code-to-python-3-with-2to3.html#isinstance><code>isinstance()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#basestring><code>basestring</code> datatype</a>
<li><a href=porting-code-to-python-3-with-2to3.html#itertools><code>itertools</code> module</a>
<li><a href=porting-code-to-python-3-with-2to3.html#sys_exc><code>sys.exc_type</code>, <code>sys.exc_value</code>, <code>sys.exc_traceback</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#paren>List comprehensions over tuples</a>
<li><a href=porting-code-to-python-3-with-2to3.html#getcwdu><code>os.getcwdu()</code> function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#metaclass>Metaclasses</a>
<li><a href=porting-code-to-python-3-with-2to3.html#nitpick>Matters of style</a>
<ol>
<li><a href=porting-code-to-python-3-with-2to3.html#set_literal><code>set()</code> literals (explicit)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#buffer><code>buffer()</code> global function (explicit)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#wscomma>Whitespace around commas (explicit)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#idioms>Common idioms (explicit)</a>
</ol>
</ol>
<li id=special-method-names><a href=special-method-names.html>Special Method Names</a>
<ol>
<li><a href=special-method-names.html#divingin>Diving In</a>
<li><a href=special-method-names.html#basics>Basics</a>
<li><a href=special-method-names.html#acts-like-iterator>Classes That Act Like Iterators</a>
<li><a href=special-method-names.html#computed-attributes>Computed Attributes</a>
<li><a href=special-method-names.html#acts-like-function>Classes That Act Like Functions</a>
<li><a href=special-method-names.html#acts-like-list>Classes That Act Like Sequences</a>
<li><a href=special-method-names.html#acts-like-dict>Classes That Act Like Dictionaries</a>
<li><a href=special-method-names.html#acts-like-number>Classes That Act Like Numbers</a>
<li><a href=special-method-names.html#rich-comparisons>Classes That Can Be Compared</a>
<li><a href=special-method-names.html#pickle>Classes That Can Be Serialized</a>
<li><a href=special-method-names.html#context-managers>Classes That Can Be Used in a <code>with</code> Block</a>
<li><a href=special-method-names.html#esoterica>Really Esoteric Stuff</a>
<li><a href=special-method-names.html#furtherreading>Further Reading</a>
</ol>
<li id=where-to-go-from-here><a href=where-to-go-from-here.html>Where to Go From Here</a>
<ol>
<li><a href=where-to-go-from-here.html#things-to-read>Things to Read</a>
<li><a href=where-to-go-from-here.html#code>Where To Look For Python 3-Compatible Code</a>
</ol>
</ol>
<!-- /toc -->
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
<!DOCTYPE html>
<meta charset=utf-8>
<title>Table of contents - Dive Into Python 3</title>
<link rel=stylesheet href=dip3.css>
<style>
h1:before{content:''}
ol,ul{font-weight:bold}
li ol{font-weight:normal}
#porting-code-to-python-3-with-2to3,#special-method-names,#where-to-go-from-here{list-style:none;margin:0 0 0 -2em}
#porting-code-to-python-3-with-2to3 > ol,#special-method-names > ol,#where-to-go-from-here > ol{margin:0;padding:0 0 0 4.5em}
#porting-code-to-python-3-with-2to3:before{content:'A. \00a0 \00a0'}
#special-method-names:before{content:'B. \00a0 \00a0'}
#where-to-go-from-here:before{content:'C. \00a0 \00a0'}
</style>
<link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
<link rel=stylesheet media=print href=print.css>
<meta name=viewport content='initial-scale=1.0'>
<form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8><input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=sa value=Search></div></form>
<p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> Dive Into Python 3 <span class=u>&#8227;</span>
<h1>Table of Contents</h1>
<!-- toc -->
<ol start=-1>
<li id=whats-new><a href=whats-new.html>What&#8217;s New in &#8220;Dive Into Python 3&#8221;</a>
<ol>
<li><a href=whats-new.html#divingin><i>a.k.a.</i> &#8220;the minus level&#8221;</a>
</ol>
<li id=installing-python><a href=installing-python.html>Installing Python</a>
<ol>
<li><a href=installing-python.html#divingin>Diving In</a>
<li><a href=installing-python.html#which>Which Python Is Right For You?</a>
<li><a href=installing-python.html#windows>Installing on Microsoft Windows</a>
<li><a href=installing-python.html#macosx>Installing on Mac OS X</a>
<li><a href=installing-python.html#ubuntu>Installing on Ubuntu Linux</a>
<li><a href=installing-python.html#other>Installing on Other Platforms</a>
<li><a href=installing-python.html#idle>Using The Python Shell</a>
<li><a href=installing-python.html#editors>Python Editors and IDEs</a>
</ol>
<li id=your-first-python-program><a href=your-first-python-program.html>Your First Python Program</a>
<ol>
<li><a href=your-first-python-program.html#divingin>Diving In</a>
<li><a href=your-first-python-program.html#declaringfunctions>Declaring Functions</a>
<ol>
<li><a href=your-first-python-program.html#optional-arguments>Optional and Named Arguments</a>
</ol>
<li><a href=your-first-python-program.html#readability>Writing Readable Code</a>
<ol>
<li><a href=your-first-python-program.html#docstrings>Documentation Strings</a>
</ol>
<li><a href=your-first-python-program.html#importsearchpath>The <code>import</code> Search Path</a>
<li><a href=your-first-python-program.html#everythingisanobject>Everything Is An Object</a>
<ol>
<li><a href=your-first-python-program.html#whatsanobject>What&#8217;s An Object?</a>
</ol>
<li><a href=your-first-python-program.html#indentingcode>Indenting Code</a>
<li><a href=your-first-python-program.html#exceptions>Exceptions</a>
<ol>
<li><a href=your-first-python-program.html#importerror>Catching Import Errors</a>
</ol>
<li><a href=your-first-python-program.html#nameerror>Unbound Variables</a>
<li><a href=your-first-python-program.html#case>Everything is Case-Sensitive</a>
<li><a href=your-first-python-program.html#runningscripts>Running Scripts</a>
<li><a href=your-first-python-program.html#furtherreading>Further Reading</a>
</ol>
<li id=native-datatypes><a href=native-datatypes.html>Native Datatypes</a>
<ol>
<li><a href=native-datatypes.html#divingin>Diving In</a>
<li><a href=native-datatypes.html#booleans>Booleans</a>
<li><a href=native-datatypes.html#numbers>Numbers</a>
<ol>
<li><a href=native-datatypes.html#number-coercion>Coercing Integers To Floats And Vice-Versa</a>
<li><a href=native-datatypes.html#common-numerical-operations>Common Numerical Operations</a>
<li><a href=native-datatypes.html#fractions>Fractions</a>
<li><a href=native-datatypes.html#trig>Trigonometry</a>
<li><a href=native-datatypes.html#numbers-in-a-boolean-context>Numbers In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#lists>Lists</a>
<ol>
<li><a href=native-datatypes.html#creatinglists>Creating A List</a>
<li><a href=native-datatypes.html#slicinglists>Slicing A List</a>
<li><a href=native-datatypes.html#extendinglists>Adding Items To A List</a>
<li><a href=native-datatypes.html#searchinglists>Searching For Values In A List</a>
<li><a href=native-datatypes.html#removingfromlists>Removing Items From A List</a>
<li><a href=native-datatypes.html#popgoestheweasel>Removing Items From A List: Bonus Round</a>
<li><a href=native-datatypes.html#lists-in-a-boolean-context>Lists In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#tuples>Tuples</a>
<ol>
<li><a href=native-datatypes.html#tuples-in-a-boolean-context>Tuples In A Boolean Context</a>
<li><a href=native-datatypes.html#multivar>Assigning Multiple Values At Once</a>
</ol>
<li><a href=native-datatypes.html#sets>Sets</a>
<ol>
<li><a href=native-datatypes.html#creating-a-set>Creating A Set</a>
<li><a href=native-datatypes.html#modifying-sets>Modifying A Set</a>
<li><a href=native-datatypes.html#removing-from-sets>Removing Items From A Set</a>
<li><a href=native-datatypes.html#common-set-operations>Common Set Operations</a>
<li><a href=native-datatypes.html#sets-in-a-boolean-context>Sets In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#dictionaries>Dictionaries</a>
<ol>
<li><a href=native-datatypes.html#creating-dictionaries>Creating A Dictionary</a>
<li><a href=native-datatypes.html#modifying-dictionaries>Modifying A Dictionary</a>
<li><a href=native-datatypes.html#mixed-value-dictionaries>Mixed-Value Dictionaries</a>
<li><a href=native-datatypes.html#dictionaries-in-a-boolean-context>Dictionaries In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#none><code>None</code></a>
<ol>
<li><a href=native-datatypes.html#none-in-a-boolean-context><code>None</code> In A Boolean Context</a>
</ol>
<li><a href=native-datatypes.html#furtherreading>Further Reading</a>
</ol>
<li id=comprehensions><a href=comprehensions.html>Comprehensions</a>
<ol>
<li><a href=comprehensions.html#divingin>Diving In</a>
<li><a href=comprehensions.html#os>Working With Files And Directories</a>
<ol>
<li><a href=comprehensions.html#getcwd>The Current Working Directory</a>
<li><a href=comprehensions.html#ospath>Working With Filenames and Directory Names</a>
<li><a href=comprehensions.html#glob>Listing Directories</a>
<li><a href=comprehensions.html#osstat>Getting File Metadata</a>
<li><a href=comprehensions.html#abspath>Constructing Absolute Pathnames</a>
</ol>
<li><a href=comprehensions.html#listcomprehension>List Comprehensions</a>
<li><a href=comprehensions.html#dictionarycomprehension>Dictionary Comprehensions</a>
<ol>
<li><a href=comprehensions.html#stupiddicttricks>Other Fun Stuff To Do With Dictionary Comprehensions</a>
</ol>
<li><a href=comprehensions.html#setcomprehension>Set Comprehensions</a>
<li><a href=comprehensions.html#furtherreading>Further Reading</a>
</ol>
<li id=strings><a href=strings.html>Strings</a>
<ol>
<li><a href=strings.html#boring-stuff>Some Boring Stuff You Need To Understand Before You Can Dive In</a>
<li><a href=strings.html#one-ring-to-rule-them-all>Unicode</a>
<li><a href=strings.html#divingin>Diving In</a>
<li><a href=strings.html#formatting-strings>Formatting Strings</a>
<ol>
<li><a href=strings.html#compound-field-names>Compound Field Names</a>
<li><a href=strings.html#format-specifiers>Format Specifiers</a>
</ol>
<li><a href=strings.html#common-string-methods>Other Common String Methods</a>
<ol>
<li><a href=strings.html#slicingstrings>Slicing A String</a>
</ol>
<li><a href=strings.html#byte-arrays>Strings vs. Bytes</a>
<li><a href=strings.html#py-encoding>Postscript: Character Encoding Of Python Source Code</a>
<li><a href=strings.html#furtherreading>Further Reading</a>
</ol>
<li id=regular-expressions><a href=regular-expressions.html>Regular Expressions</a>
<ol>
<li><a href=regular-expressions.html#divingin>Diving In</a>
<li><a href=regular-expressions.html#streetaddresses>Case Study: Street Addresses</a>
<li><a href=regular-expressions.html#romannumerals>Case Study: Roman Numerals</a>
<ol>
<li><a href=regular-expressions.html#thousands>Checking For Thousands</a>
<li><a href=regular-expressions.html#hundreds>Checking For Hundreds</a>
</ol>
<li><a href=regular-expressions.html#nmsyntax>Using The <code>{n,m}</code> Syntax</a>
<ol>
<li><a href=regular-expressions.html#tensandones>Checking For Tens And Ones</a>
</ol>
<li><a href=regular-expressions.html#verbosere>Verbose Regular Expressions</a>
<li><a href=regular-expressions.html#phonenumbers>Case study: Parsing Phone Numbers</a>
<li><a href=regular-expressions.html#summary>Summary</a>
</ol>
<li id=generators><a href=generators.html>Closures <i class=baa>&amp;</i> Generators</a>
<ol>
<li><a href=generators.html#divingin>Diving In</a>
<li><a href=generators.html#i-know>I Know, Let&#8217;s Use Regular Expressions!</a>
<li><a href=generators.html#a-list-of-functions>A List Of Functions</a>
<li><a href=generators.html#a-list-of-patterns>A List Of Patterns</a>
<li><a href=generators.html#a-file-of-patterns>A File Of Patterns</a>
<li><a href=generators.html#generators>Generators</a>
<ol>
<li><a href=generators.html#a-fibonacci-generator>A Fibonacci Generator</a>
<li><a href=generators.html#a-plural-rule-generator>A Plural Rule Generator</a>
</ol>
<li><a href=generators.html#furtherreading>Further Reading</a>
</ol>
<li id=iterators><a href=iterators.html>Classes <i class=baa>&amp;</i> Iterators</a>
<ol>
<li><a href=iterators.html#divingin>Diving In</a>
<li><a href=iterators.html#defining-classes>Defining Classes</a>
<ol>
<li><a href=iterators.html#init-method>The <code>__init__()</code> Method</a>
</ol>
<li><a href=iterators.html#instantiating-classes>Instantiating Classes</a>
<li><a href=iterators.html#instance-variables>Instance Variables</a>
<li><a href=iterators.html#a-fibonacci-iterator>A Fibonacci Iterator</a>
<li><a href=iterators.html#a-plural-rule-iterator>A Plural Rule Iterator</a>
<li><a href=iterators.html#furtherreading>Further Reading</a>
</ol>
<li id=advanced-iterators><a href=advanced-iterators.html>Advanced Iterators</a>
<ol>
<li><a href=advanced-iterators.html#divingin>Diving In</a>
<li><a href=advanced-iterators.html#re-findall>Finding all occurrences of a pattern</a>
<li><a href=advanced-iterators.html#unique-items>Finding the unique items in a sequence</a>
<li><a href=advanced-iterators.html#assert>Making assertions</a>
<li><a href=advanced-iterators.html#generator-expressions>Generator expressions</a>
<li><a href=advanced-iterators.html#permutations>Calculating Permutations&hellip; The Lazy Way!</a>
<li><a href=advanced-iterators.html#more-itertools>Other Fun Stuff in the <code>itertools</code> Module</a>
<li><a href=advanced-iterators.html#string-translate>A New Kind Of String Manipulation</a>
<li><a href=advanced-iterators.html#eval>Evaluating Arbitrary Strings As Python Expressions</a>
<li><a href=advanced-iterators.html#alphametics-finale>Putting It All Together</a>
<li><a href=advanced-iterators.html#furtherreading>Further Reading</a>
</ol>
<li id=unit-testing><a href=unit-testing.html>Unit Testing</a>
<ol>
<li><a href=unit-testing.html#divingin>(Not) Diving In</a>
<li><a href=unit-testing.html#romantest1>A Single Question</a>
<li><a href=unit-testing.html#romantest2>&#8220;Halt And Catch Fire&#8221;</a>
<li><a href=unit-testing.html#romantest3>More Halting, More Fire</a>
<li><a href=unit-testing.html#romantest4>And One More Thing&hellip;</a>
<li><a href=unit-testing.html#romantest5>A Pleasing Symmetry</a>
<li><a href=unit-testing.html#romantest6>More Bad Input</a>
</ol>
<li id=refactoring><a href=refactoring.html>Refactoring</a>
<ol>
<li><a href=refactoring.html#divingin>Diving In</a>
<li><a href=refactoring.html#changing-requirements>Handling Changing Requirements</a>
<li><a href=refactoring.html#refactoring>Refactoring</a>
<li><a href=refactoring.html#summary>Summary</a>
</ol>
<li id=files><a href=files.html>Files</a>
<ol>
<li><a href=files.html#divingin>Diving In</a>
<li><a href=files.html#reading>Reading From Text Files</a>
<ol>
<li><a href=files.html#encoding>Character Encoding Rears Its Ugly Head</a>
<li><a href=files.html#file-objects>Stream Objects</a>
<li><a href=files.html#read>Reading Data From A Text File</a>
<li><a href=files.html#close>Closing Files</a>
<li><a href=files.html#with>Closing Files Automatically</a>
<li><a href=files.html#for>Reading Data One Line At A Time</a>
</ol>
<li><a href=files.html#writing>Writing to Text Files</a>
<ol>
<li><a href=files.html#encoding-again>Character Encoding Again</a>
</ol>
<li><a href=files.html#binary>Binary Files</a>
<li><a href=files.html#file-like-objects>Stream Objects From Non-File Sources</a>
<ol>
<li><a href=files.html#gzip>Handling Compressed Files</a>
</ol>
<li><a href=files.html#stdio>Standard Input, Output, and Error</a>
<ol>
<li><a href=files.html#redirect>Redirecting Standard Output</a>
</ol>
<li><a href=files.html#furtherreading>Further Reading</a>
</ol>
<li id=xml><a href=xml.html>XML</a>
<ol>
<li><a href=xml.html#divingin>Diving In</a>
<li><a href=xml.html#xml-intro>A 5-Minute Crash Course in XML</a>
<li><a href=xml.html#xml-structure>The Structure Of An Atom Feed</a>
<li><a href=xml.html#xml-parse>Parsing XML</a>
<ol>
<li><a href=xml.html#xml-elements>Elements Are Lists</a>
<li><a href=xml.html#xml-attributes>Attributes Are Dictonaries</a>
</ol>
<li><a href=xml.html#xml-find>Searching For Nodes Within An XML Document</a>
<li><a href=xml.html#xml-lxml>Going Further With lxml</a>
<li><a href=xml.html#xml-generate>Generating XML</a>
<li><a href=xml.html#xml-custom-parser>Parsing Broken XML</a>
<li><a href=xml.html#furtherreading>Further Reading</a>
</ol>
<li id=serializing><a href=serializing.html>Serializing Python Objects</a>
<ol>
<li><a href=serializing.html#divingin>Diving In</a>
<ol>
<li><a href=serializing.html#administrivia>A Quick Note About The Examples in This Chapter</a>
</ol>
<li><a href=serializing.html#dump>Saving Data to a Pickle File</a>
<li><a href=serializing.html#load>Loading Data from a Pickle File</a>
<li><a href=serializing.html#dumps>Pickling Without a File</a>
<li><a href=serializing.html#protocol-versions>Bytes and Strings Rear Their Ugly Heads Again</a>
<li><a href=serializing.html#debugging>Debugging Pickle Files</a>
<li><a href=serializing.html#json>Serializing Python Objects to be Read by Other Languages</a>
<li><a href=serializing.html#json-dump>Saving Data to a <abbr>JSON</abbr> File</a>
<li><a href=serializing.html#json-types>Mapping of Python Datatypes to <abbr>JSON</abbr></a>
<li><a href=serializing.html#json-unknown-types>Serializing Datatypes Unsupported by <abbr>JSON</abbr></a>
<li><a href=serializing.html#json-load>Loading Data from a <abbr>JSON</abbr> File</a>
<li><a href=serializing.html#furtherreading>Further Reading</a>
</ol>
<li id=http-web-services><a href=http-web-services.html>HTTP Web Services</a>
<ol>
<li><a href=http-web-services.html#divingin>Diving In</a>
<li><a href=http-web-services.html#http-features>Features of HTTP</a>
<ol>
<li><a href=http-web-services.html#caching>Caching</a>
<li><a href=http-web-services.html#last-modified>Last-Modified Checking</a>
<li><a href=http-web-services.html#etags>ETag Checking</a>
<li><a href=http-web-services.html#compression>Compression</a>
<li><a href=http-web-services.html#redirects>Redirects</a>
</ol>
<li><a href=http-web-services.html#dont-try-this-at-home>How Not To Fetch Data Over HTTP</a>
<li><a href=http-web-services.html#whats-on-the-wire>What&#8217;s On The Wire?</a>
<li><a href=http-web-services.html#introducing-httplib2>Introducing <code>httplib2</code></a>
<ol>
<li><a href=http-web-services.html#why-bytes>A Short Digression To Explain Why <code>httplib2</code> Returns Bytes Instead of Strings</a>
<li><a href=http-web-services.html#httplib2-caching>How <code>httplib2</code> Handles Caching</a>
<li><a href=http-web-services.html#httplib2-etags>How <code>httplib2</code> Handles <code>Last-Modified</code> and <code>ETag</code> Headers</a>
<li><a href=http-web-services.html#httplib2-compression>How <code>http2lib</code> Handles Compression</a>
<li><a href=http-web-services.html#httplib2-redirects>How <code>httplib2</code> Handles Redirects</a>
</ol>
<li><a href=http-web-services.html#beyond-get>Beyond HTTP GET</a>
<li><a href=http-web-services.html#beyond-post>Beyond HTTP POST</a>
<li><a href=http-web-services.html#furtherreading>Further Reading</a>
</ol>
<li id=case-study-porting-chardet-to-python-3><a href=case-study-porting-chardet-to-python-3.html>Case Study: Porting <code>chardet</code> to Python 3</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#divingin>Diving In</a>
<li><a href=case-study-porting-chardet-to-python-3.html#faq.what>What is Character Encoding Auto-Detection?</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#faq.impossible>Isn&#8217;t That Impossible?</a>
<li><a href=case-study-porting-chardet-to-python-3.html#faq.who>Does Such An Algorithm Exist?</a>
</ol>
<li><a href=case-study-porting-chardet-to-python-3.html#divingin2>Introducing The <code>chardet</code> Module</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#how.bom><abbr>UTF-n</abbr> With A <abbr>BOM</abbr></a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.esc>Escaped Encodings</a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.mb>Multi-Byte Encodings</a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.sb>Single-Byte Encodings</a>
<li><a href=case-study-porting-chardet-to-python-3.html#how.windows1252><code>windows-1252</code></a>
</ol>
<li><a href=case-study-porting-chardet-to-python-3.html#running2to3>Running <code>2to3</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#multifile-modules>A Short Digression Into Multi-File Modules</a>
<li><a href=case-study-porting-chardet-to-python-3.html#manual>Fixing What <code>2to3</code> Can&#8217;t</a>
<ol>
<li><a href=case-study-porting-chardet-to-python-3.html#falseisinvalidsyntax><code>False</code> is invalid syntax</a>
<li><a href=case-study-porting-chardet-to-python-3.html#nomodulenamedconstants>No module named <code>constants</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#namefileisnotdefined>Name <var>'file'</var> is not defined</a>
<li><a href=case-study-porting-chardet-to-python-3.html#cantuseastringpattern>Can&#8217;t use a string pattern on a bytes-like object</a>
<li><a href=case-study-porting-chardet-to-python-3.html#cantconvertbytesobject>Can't convert <code>'bytes'</code> object to <code>str</code> implicitly</a>
<li><a href=case-study-porting-chardet-to-python-3.html#unsupportedoperandtypeforplus>Unsupported operand type(s) for +: <code>'int'</code> and <code>'bytes'</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#ordexpectedstring><code>ord()</code> expected string of length 1, but <code>int</code> found</a>
<li><a href=case-study-porting-chardet-to-python-3.html#unorderabletypes>Unorderable types: <code>int()</code> >= <code>str()</code></a>
<li><a href=case-study-porting-chardet-to-python-3.html#reduceisnotdefined>Global name <code>'reduce'</code> is not defined</a>
</ol>
<li><a href=case-study-porting-chardet-to-python-3.html#summary>Summary</a>
</ol>
<li id=packaging><a href=packaging.html>Packaging Python Libraries</a>
<ol>
<li><a href=packaging.html#divingin>Diving In</a>
<li><a href=packaging.html#cantdo>Things Distutils Can&#8217;t Do For You</a>
<li><a href=packaging.html#structure>Directory Structure</a>
<li><a href=packaging.html#setuppy>Writing Your Setup Script</a>
<li><a href=packaging.html#trove>Classifying Your Package</a>
<ol>
<li><a href=packaging.html#trove-examples>Examples of Good Package Classifiers</a>
</ol>
<li><a href=packaging.html#manifest>Specifying Additional Files With A Manifest</a>
<li><a href=packaging.html#check>Checking Your Setup Script for Errors</a>
<li><a href=packaging.html#sdist>Creating a Source Distribution</a>
<li><a href=packaging.html#bdist>Creating a Graphical Installer</a>
<ol>
<li><a href=packaging.html#linux>Building Installable Packages for Other Operating Systems</a>
</ol>
<li><a href=packaging.html#pypi>Adding Your Software to The Python Package Index</a>
<li><a href=packaging.html#future>The Many Possible Futures of Python Packaging</a>
<li><a href=packaging.html#furtherreading>Further Reading</a>
</ol>
<li id=porting-code-to-python-3-with-2to3><a href=porting-code-to-python-3-with-2to3.html>Porting Code to Python 3 with <code>2to3</code></a>
<ol>
<li><a href=porting-code-to-python-3-with-2to3.html#divingin>Diving In</a>
<li><a href=porting-code-to-python-3-with-2to3.html#print><code>print</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#unicodeliteral>Unicode string literals</a>
<li><a href=porting-code-to-python-3-with-2to3.html#unicode><code>unicode()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#long><code>long</code> data type</a>
<li><a href=porting-code-to-python-3-with-2to3.html#ne>&lt;> comparison</a>
<li><a href=porting-code-to-python-3-with-2to3.html#has_key><code>has_key()</code> dictionary method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#dict>Dictionary methods that return lists</a>
<li><a href=porting-code-to-python-3-with-2to3.html#imports>Modules that have been renamed or reorganized</a>
<ol>
<li><a href=porting-code-to-python-3-with-2to3.html#http><code>http</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#urllib><code>urllib</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#dbm><code>dbm</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#xmlrpc><code>xmlrpc</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#othermodules>Other modules</a>
</ol>
<li><a href=porting-code-to-python-3-with-2to3.html#import>Relative imports within a package</a>
<li><a href=porting-code-to-python-3-with-2to3.html#next><code>next()</code> iterator method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#filter><code>filter()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#map><code>map()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#reduce><code>reduce()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#apply><code>apply()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#intern><code>intern()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#exec><code>exec</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#execfile><code>execfile</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#repr><code>repr</code> literals (backticks)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#except><code>try...except</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#raise><code>raise</code> statement</a>
<li><a href=porting-code-to-python-3-with-2to3.html#throw><code>throw</code> method on generators</a>
<li><a href=porting-code-to-python-3-with-2to3.html#xrange><code>xrange()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#raw_input><code>raw_input()</code> and <code>input()</code> global functions</a>
<li><a href=porting-code-to-python-3-with-2to3.html#funcattrs><code>func_*</code> function attributes</a>
<li><a href=porting-code-to-python-3-with-2to3.html#xreadlines><code>xreadlines()</code> I/O method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#tuple_params><code>lambda</code> functions that take a tuple instead of multiple parameters</a>
<li><a href=porting-code-to-python-3-with-2to3.html#methodattrs>Special method attributes</a>
<li><a href=porting-code-to-python-3-with-2to3.html#nonzero><code>__nonzero__</code> special method</a>
<li><a href=porting-code-to-python-3-with-2to3.html#numliterals>Octal literals</a>
<li><a href=porting-code-to-python-3-with-2to3.html#renames><code>sys.maxint</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#callable><code>callable()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#zip><code>zip()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#standarderror><code>StandardError</code> exception</a>
<li><a href=porting-code-to-python-3-with-2to3.html#types><code>types</code> module constants</a>
<li><a href=porting-code-to-python-3-with-2to3.html#isinstance><code>isinstance()</code> global function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#basestring><code>basestring</code> datatype</a>
<li><a href=porting-code-to-python-3-with-2to3.html#itertools><code>itertools</code> module</a>
<li><a href=porting-code-to-python-3-with-2to3.html#sys_exc><code>sys.exc_type</code>, <code>sys.exc_value</code>, <code>sys.exc_traceback</code></a>
<li><a href=porting-code-to-python-3-with-2to3.html#paren>List comprehensions over tuples</a>
<li><a href=porting-code-to-python-3-with-2to3.html#getcwdu><code>os.getcwdu()</code> function</a>
<li><a href=porting-code-to-python-3-with-2to3.html#metaclass>Metaclasses</a>
<li><a href=porting-code-to-python-3-with-2to3.html#nitpick>Matters of style</a>
<ol>
<li><a href=porting-code-to-python-3-with-2to3.html#set_literal><code>set()</code> literals (explicit)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#buffer><code>buffer()</code> global function (explicit)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#wscomma>Whitespace around commas (explicit)</a>
<li><a href=porting-code-to-python-3-with-2to3.html#idioms>Common idioms (explicit)</a>
</ol>
</ol>
<li id=special-method-names><a href=special-method-names.html>Special Method Names</a>
<ol>
<li><a href=special-method-names.html#divingin>Diving In</a>
<li><a href=special-method-names.html#basics>Basics</a>
<li><a href=special-method-names.html#acts-like-iterator>Classes That Act Like Iterators</a>
<li><a href=special-method-names.html#computed-attributes>Computed Attributes</a>
<li><a href=special-method-names.html#acts-like-function>Classes That Act Like Functions</a>
<li><a href=special-method-names.html#acts-like-list>Classes That Act Like Sequences</a>
<li><a href=special-method-names.html#acts-like-dict>Classes That Act Like Dictionaries</a>
<li><a href=special-method-names.html#acts-like-number>Classes That Act Like Numbers</a>
<li><a href=special-method-names.html#rich-comparisons>Classes That Can Be Compared</a>
<li><a href=special-method-names.html#pickle>Classes That Can Be Serialized</a>
<li><a href=special-method-names.html#context-managers>Classes That Can Be Used in a <code>with</code> Block</a>
<li><a href=special-method-names.html#esoterica>Really Esoteric Stuff</a>
<li><a href=special-method-names.html#furtherreading>Further Reading</a>
</ol>
<li id=where-to-go-from-here><a href=where-to-go-from-here.html>Where to Go From Here</a>
<ol>
<li><a href=where-to-go-from-here.html#things-to-read>Things to Read</a>
<li><a href=where-to-go-from-here.html#code>Where To Look For Python 3-Compatible Code</a>
</ol>
</ol>
<!-- /toc -->
<p class=c>&copy; 2001&ndash;10 <a href=about.html>Mark Pilgrim</a>
<!--[if IE]><script src=j/html5.js></script><![endif]-->
util/lesscss.py
+3 -8
View File
@@ -1,4 +1,4 @@
#!/usr/bin/python2.6
#!/usr/bin/python2.5
from pyquery import PyQuery as pq
import glob
@@ -12,10 +12,7 @@ SELECTOR_EXCEPTIONS = ('.w', '.b', '.str', '.kwd', '.com', '.typ', '.lit', '.pun
filename = sys.argv[1]
cssfilename = sys.argv[2]
pqd = pq(filename=filename)
with open(filename, 'rb') as fopen:
raw_data = fopen.read()
raw_data = open(filename, 'rb').read()
if raw_data.count('</a><script src=j/'): # HACK HACK HACK
def keep(s):
for selector in SELECTOR_EXCEPTIONS:
@@ -26,9 +23,7 @@ else:
def keep(s):
return False
with open(cssfilename, 'rb') as fopen:
original_css = fopen.read()
original_css = open(cssfilename, 'rb').read();
new_css = ''
for rule in original_css.split('}')[:-1]:
selectors, properties = rule.split('{', 1)