mirror of
https://github.com/kennethreitz/dive-into-python3.git
synced 2026-06-05 23:10:17 +00:00
iterators and generators chapter
--HG-- rename : humansize.py => examples/humansize.py rename : roman1.py => examples/roman1.py rename : roman2.py => examples/roman2.py rename : roman3.py => examples/roman3.py rename : roman4.py => examples/roman4.py rename : roman5.py => examples/roman5.py rename : roman6.py => examples/roman6.py rename : roman7.py => examples/roman7.py rename : roman8.py => examples/roman8.py rename : romantest1.py => examples/romantest1.py rename : romantest2.py => examples/romantest2.py rename : romantest3.py => examples/romantest3.py rename : romantest4.py => examples/romantest4.py rename : romantest5.py => examples/romantest5.py rename : romantest6.py => examples/romantest6.py rename : romantest7.py => examples/romantest7.py rename : romantest8.py => examples/romantest8.py
This commit is contained in:
Mark Pilgrim
@@ -7541,469 +7541,6 @@ if __name__ == "__main__":
|
||||
<div class=footnote>
|
||||
<p><sup>[<a name="ftn.d0e36079" href="#d0e36079">8</a>] </sup>Again, I should point out that <code>map</code> can take a list, a tuple, or any object that acts like a sequence. See previous footnote about <code>filter</code>.
|
||||
<div class=chapter>
|
||||
<h2 id="plural">Chapter 17. Dynamic functions</h2>
|
||||
<h2 id="plural.divein">17.1. Diving in</h2>
|
||||
<p>I want to talk about plural nouns. Also, functions that return other functions, advanced regular expressions, and generators.
|
||||
Generators are new in Python 2.3. But first, let's talk about how to make plural nouns.
|
||||
<p>If you haven't read <a href="#re" title="Chapter 7. Regular Expressions">Chapter 7, <i>Regular Expressions</i></a>, now would be a good time. This chapter assumes you understand the basics of regular expressions, and quickly descends into
|
||||
more advanced uses.
|
||||
<p>English is a schizophrenic language that borrows from a lot of other languages, and the rules for making singular nouns into
|
||||
plural nouns are varied and complex. There are rules, and then there are exceptions to those rules, and then there are exceptions
|
||||
to the exceptions.
|
||||
<p>If you grew up in an English-speaking country or learned English in a formal school setting, you're probably familiar with
|
||||
the basic rules:
|
||||
<div class=orderedlist>
|
||||
<ol>
|
||||
<li>If a word ends in S, X, or Z, add ES. “Bass” becomes “basses”, “fax” becomes “faxes”, and “waltz” becomes “waltzes”.
|
||||
|
||||
<li>If a word ends in a noisy H, add ES; if it ends in a silent H, just add S. What's a noisy H? One that gets combined with
|
||||
other letters to make a sound that you can hear. So “coach” becomes “coaches” and “rash” becomes “rashes”, because you can hear the CH and SH sounds when you say them. But “cheetah” becomes “cheetahs”, because the H is silent.
|
||||
|
||||
<li>If a word ends in Y that sounds like I, change the Y to IES; if the Y is combined with a vowel to sound like something else,
|
||||
just add S. So “vacancy” becomes “vacancies”, but “day” becomes “days”.
|
||||
|
||||
<li>If all else fails, just add S and hope for the best.
|
||||
</ol>
|
||||
<p>(I know, there are a lot of exceptions. “Man” becomes “men” and “woman” becomes “women”, but “human” becomes “humans”. “Mouse” becomes “mice” and “louse” becomes “lice”, but “house” becomes “houses”. “Knife” becomes “knives” and “wife” becomes “wives”, but “lowlife” becomes “lowlifes”. And don't even get me started on words that are their own plural, like “sheep”, “deer”, and “haiku”.)
|
||||
<p>Other languages are, of course, completely different.
|
||||
<p>Let's design a module that pluralizes nouns. Start with just English nouns, and just these four rules, but keep in mind that
|
||||
you'll inevitably need to add more rules, and you may eventually need to add more languages.
|
||||
<h2 id="plural.stage1">17.2. <code>plural.py</code>, stage 1</h2>
|
||||
<p>So you're looking at words, which at least in English are strings of characters. And you have rules that say you need to
|
||||
find different combinations of characters, and then do different things to them. This sounds like a job for regular expressions.
|
||||
<div class=example><h3>Example 17.1. <code>plural1.py</code></h3><pre><code>
|
||||
import re
|
||||
|
||||
def plural(noun):
|
||||
if re.search('[sxz]$', noun): <span>①</span>
|
||||
return re.sub('$', 'es', noun) <span>②</span>
|
||||
elif re.search('[^aeioudgkprt]h$', noun):
|
||||
return re.sub('$', 'es', noun)
|
||||
elif re.search('[^aeiou]y$', noun):
|
||||
return re.sub('y$', 'ies', noun)
|
||||
else:
|
||||
return noun + 's'
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>OK, this is a regular expression, but it uses a syntax you didn't see in <a href="#re" title="Chapter 7. Regular Expressions">Chapter 7, <i>Regular Expressions</i></a>. The square brackets mean “match exactly one of these characters”. So <code>[sxz]</code> means “<code>s</code>, or <code>x</code>, or <code>z</code>”, but only one of them. The <code>$</code> should be familiar; it matches the end of string. So you're checking to see if <var>noun</var> ends with <code>s</code>, <code>x</code>, or <code>z</code>.
|
||||
<li>This <code>re.sub</code> function performs regular expression-based string substitutions. Let's look at it in more detail.
|
||||
<div class=example><h3>Example 17.2. Introducing <code>re.sub</code></h3><pre class=screen>
|
||||
<samp class=p>>>> </samp><kbd>import re</kbd>
|
||||
<samp class=p>>>> </samp><kbd>re.search('[abc]', 'Mark')</kbd> <span>①</span>
|
||||
<_sre.SRE_Match object at 0x001C1FA8>
|
||||
<samp class=p>>>> </samp><kbd>re.sub('[abc]', 'o', 'Mark')</kbd> <span>②</span>
|
||||
'Mork'
|
||||
<samp class=p>>>> </samp><kbd>re.sub('[abc]', 'o', 'rock')</kbd> <span>③</span>
|
||||
'rook'
|
||||
<samp class=p>>>> </samp><kbd>re.sub('[abc]', 'o', 'caps')</kbd> <span>④</span>
|
||||
'oops'
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>Does the string <code>Mark</code> contain <code>a</code>, <code>b</code>, or <code>c</code>? Yes, it contains <code>a</code>.
|
||||
<li>OK, now find <code>a</code>, <code>b</code>, or <code>c</code>, and replace it with <code>o</code>. <code>Mark</code> becomes <code>Mork</code>.
|
||||
<li>The same function turns <code>rock</code> into <code>rook</code>.
|
||||
<li>You might think this would turn <code>caps</code> into <code>oaps</code>, but it doesn't. <code>re.sub</code> replaces <em>all</em> of the matches, not just the first one. So this regular expression turns <code>caps</code> into <code>oops</code>, because both the <code>c</code> and the <code>a</code> get turned into <code>o</code>.
|
||||
<div class=example><h3>Example 17.3. Back to <code>plural1.py</code></h3><pre><code>
|
||||
import re
|
||||
|
||||
def plural(noun):
|
||||
if re.search('[sxz]$', noun):
|
||||
return re.sub('$', 'es', noun) <span>①</span>
|
||||
elif re.search('[^aeioudgkprt]h$', noun): <span>②</span>
|
||||
return re.sub('$', 'es', noun) <span>③</span>
|
||||
elif re.search('[^aeiou]y$', noun):
|
||||
return re.sub('y$', 'ies', noun)
|
||||
else:
|
||||
return noun + 's'
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>Back to the <code>plural</code> function. What are you doing? You're replacing the end of string with <code>es</code>. In other words, adding <code>es</code> to the string. You could accomplish the same thing with string concatenation, for example <code>noun + 'es'</code>, but I'm using regular expressions for everything, for consistency, for reasons that will become clear later in the chapter.
|
||||
<li>Look closely, this is another new variation. The <code>^</code> as the first character inside the square brackets means something special: negation. <code>[^abc]</code> means “any single character <em>except</em> <code>a</code>, <code>b</code>, or <code>c</code>”. So <code>[^aeioudgkprt]</code> means any character except <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, <code>u</code>, <code>d</code>, <code>g</code>, <code>k</code>, <code>p</code>, <code>r</code>, or <code>t</code>. Then that character needs to be followed by <code>h</code>, followed by end of string. You're looking for words that end in H where the H can be heard.
|
||||
<li>Same pattern here: match words that end in Y, where the character before the Y is <em>not</em> <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, or <code>u</code>. You're looking for words that end in Y that sounds like I.
|
||||
<div class=example><h3>Example 17.4. More on negation regular expressions</h3><pre class=screen>
|
||||
<samp class=p>>>> </samp><kbd>import re</kbd>
|
||||
<samp class=p>>>> </samp><kbd>re.search('[^aeiou]y$', 'vacancy')</kbd> <span>①</span>
|
||||
<_sre.SRE_Match object at 0x001C1FA8>
|
||||
<samp class=p>>>> </samp><kbd>re.search('[^aeiou]y$', 'boy')</kbd> <span>②</span>
|
||||
<samp class=p>>>> </samp><kbd></kbd>
|
||||
<samp class=p>>>> </samp><kbd>re.search('[^aeiou]y$', 'day')</kbd>
|
||||
<samp class=p>>>> </samp><kbd></kbd>
|
||||
<samp class=p>>>> </samp><kbd>re.search('[^aeiou]y$', 'pita')</kbd> <span>③</span>
|
||||
<samp class=p>>>> </samp><kbd></kbd>
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li><code>vacancy</code> matches this regular expression, because it ends in <code>cy</code>, and <code>c</code> is not <code>a</code>, <code>e</code>, <code>i</code>, <code>o</code>, or <code>u</code>.
|
||||
<li><code>boy</code> does not match, because it ends in <code>oy</code>, and you specifically said that the character before the <code>y</code> could not be <code>o</code>. <code>day</code> does not match, because it ends in <code>ay</code>.
|
||||
<li><code>pita</code> does not match, because it does not end in <code>y</code>.
|
||||
<div class=example><h3>Example 17.5. More on <code>re.sub</code></h3><pre class=screen>
|
||||
<samp class=p>>>> </samp><kbd>re.sub('y$', 'ies', 'vacancy')</kbd> <span>①</span>
|
||||
'vacancies'
|
||||
<samp class=p>>>> </samp><kbd>re.sub('y$', 'ies', 'agency')</kbd>
|
||||
'agencies'
|
||||
<samp class=p>>>> </samp><kbd>re.sub('([^aeiou])y$', r'\1ies', 'vacancy')</kbd> <span>②</span>
|
||||
'vacancies'
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>This regular expression turns <code>vacancy</code> into <code>vacancies</code> and <code>agency</code> into <code>agencies</code>, which is what you wanted. Note that it would also turn <code>boy</code> into <code>boies</code>, but that will never happen in the function because you did that <code>re.search</code> first to find out whether you should do this <code>re.sub</code>.
|
||||
<li>Just in passing, I want to point out that it is possible to combine these two regular expressions (one to find out if the
|
||||
rule applies, and another to actually apply it) into a single regular expression. Here's what that would look like. Most
|
||||
of it should look familiar: you're using a remembered group, which you learned in <a href="#re.phone" title="7.6. Case study: Parsing Phone Numbers">Section 7.6, “Case study: Parsing Phone Numbers”</a>, to remember the character before the <code>y</code>. Then in the substitution string, you use a new syntax, <code>\1</code>, which means “hey, that first group you remembered? put it here”. In this case, you remember the <code>c</code> before the <code>y</code>, and then when you do the substitution, you substitute <code>c</code> in place of <code>c</code>, and <code>ies</code> in place of <code>y</code>. (If you have more than one remembered group, you can use <code>\2</code> and <code>\3</code> and so on.)
|
||||
<p>Regular expression substitutions are extremely powerful, and the <code>\1</code> syntax makes them even more powerful. But combining the entire operation into one regular expression is also much harder
|
||||
to read, and it doesn't directly map to the way you first described the pluralizing rules. You originally laid out rules
|
||||
like “if the word ends in S, X, or Z, then add ES”. And if you look at this function, you have two lines of code that say “if the word ends in S, X, or Z, then add ES”. It doesn't get much more direct than that.
|
||||
<h2 id="plural.stage2">17.3. <code>plural.py</code>, stage 2</h2>
|
||||
<p>Now you're going to add a level of abstraction. You started by defining a list of rules: if this, then do that, otherwise
|
||||
go to the next rule. Let's temporarily complicate part of the program so you can simplify another part.
|
||||
<div class=example><h3>Example 17.6. <code>plural2.py</code></h3><pre><code>
|
||||
import re
|
||||
|
||||
def match_sxz(noun):
|
||||
return re.search('[sxz]$', noun)
|
||||
|
||||
def apply_sxz(noun):
|
||||
return re.sub('$', 'es', noun)
|
||||
|
||||
def match_h(noun):
|
||||
return re.search('[^aeioudgkprt]h$', noun)
|
||||
|
||||
def apply_h(noun):
|
||||
return re.sub('$', 'es', noun)
|
||||
|
||||
def match_y(noun):
|
||||
return re.search('[^aeiou]y$', noun)
|
||||
|
||||
def apply_y(noun):
|
||||
return re.sub('y$', 'ies', noun)
|
||||
|
||||
def match_default(noun):
|
||||
return 1
|
||||
|
||||
def apply_default(noun):
|
||||
return noun + 's'
|
||||
|
||||
rules = ((match_sxz, apply_sxz),
|
||||
(match_h, apply_h),
|
||||
(match_y, apply_y),
|
||||
(match_default, apply_default)
|
||||
) <span>①</span>
|
||||
|
||||
def plural(noun):
|
||||
for matchesRule, applyRule in rules: <span>②</span>
|
||||
if matchesRule(noun):<span>③</span>
|
||||
return applyRule(noun) <span>④</span>
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>This version looks more complicated (it's certainly longer), but it does exactly the same thing: try to match four different
|
||||
rules, in order, and apply the appropriate regular expression when a match is found. The difference is that each individual
|
||||
match and apply rule is defined in its own function, and the functions are then listed in this <var>rules</var> variable, which is a tuple of tuples.
|
||||
<li>Using a <code>for</code> loop, you can pull out the match and apply rules two at a time (one match, one apply) from the <var>rules</var> tuple. On the first iteration of the <code>for</code> loop, <var>matchesRule</var> will get <code>match_sxz</code>, and <var>applyRule</var> will get <code>apply_sxz</code>. On the second iteration (assuming you get that far), <var>matchesRule</var> will be assigned <code>match_h</code>, and <var>applyRule</var> will be assigned <code>apply_h</code>.
|
||||
<li>Remember that <a href="#odbchelper.objects" title="2.4. Everything Is an Object">everything in Python is an object</a>, including functions. <var>rules</var> contains actual functions; not names of functions, but actual functions. When they get assigned in the <code>for</code> loop, then <var>matchesRule</var> and <var>applyRule</var> are actual functions that you can call. So on the first iteration of the <code>for</code> loop, this is equivalent to calling <code>matches_sxz(noun)</code>.
|
||||
<li>On the first iteration of the <code>for</code> loop, this is equivalent to calling <code>apply_sxz(noun)</code>, and so forth.
|
||||
<p>If this additional level of abstraction is confusing, try unrolling the function to see the equivalence. This <code>for</code> loop is equivalent to the following:
|
||||
<div class=example><h3>Example 17.7. Unrolling the <code>plural</code> function</h3><pre><code>
|
||||
def plural(noun):
|
||||
if match_sxz(noun):
|
||||
return apply_sxz(noun)
|
||||
if match_h(noun):
|
||||
return apply_h(noun)
|
||||
if match_y(noun):
|
||||
return apply_y(noun)
|
||||
if match_default(noun):
|
||||
return apply_default(noun)
|
||||
</pre><p>The benefit here is that that <code>plural</code> function is now simplified. It takes a list of rules, defined elsewhere, and iterates through them in a generic fashion.
|
||||
Get a match rule; does it match? Then call the apply rule. The rules could be defined anywhere, in any way. The <code>plural</code> function doesn't care.
|
||||
<p>Now, was adding this level of abstraction worth it? Well, not yet. Let's consider what it would take to add a new rule to
|
||||
the function. Well, in the previous example, it would require adding an <code>if</code> statement to the <code>plural</code> function. In this example, it would require adding two functions, <code>match_foo</code> and <code>apply_foo</code>, and then updating the <var>rules</var> list to specify where in the order the new match and apply functions should be called relative to the other rules.
|
||||
<p>This is really just a stepping stone to the next section. Let's move on.
|
||||
<h2 id="plural.stage3">17.4. <code>plural.py</code>, stage 3</h2>
|
||||
<p>Defining separate named functions for each match and apply rule isn't really necessary. You never call them directly; you
|
||||
define them in the <var>rules</var> list and call them through there. Let's streamline the rules definition by anonymizing those functions.
|
||||
<div class=example><h3>Example 17.8. <code>plural3.py</code></h3><pre><code>
|
||||
import re
|
||||
|
||||
rules = \
|
||||
(
|
||||
(
|
||||
lambda word: re.search('[sxz]$', word),
|
||||
lambda word: re.sub('$', 'es', word)
|
||||
),
|
||||
(
|
||||
lambda word: re.search('[^aeioudgkprt]h$', word),
|
||||
lambda word: re.sub('$', 'es', word)
|
||||
),
|
||||
(
|
||||
lambda word: re.search('[^aeiou]y$', word),
|
||||
lambda word: re.sub('y$', 'ies', word)
|
||||
),
|
||||
(
|
||||
lambda word: re.search('$', word),
|
||||
lambda word: re.sub('$', 's', word)
|
||||
)
|
||||
) <span>①</span>
|
||||
|
||||
def plural(noun):
|
||||
for matchesRule, applyRule in rules: <span>②</span>
|
||||
if matchesRule(noun):
|
||||
return applyRule(noun)
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>This is the same set of rules as you defined in stage 2. The only difference is that instead of defining named functions
|
||||
like <code>match_sxz</code> and <code>apply_sxz</code>, you have “inlined” those function definitions directly into the <var>rules</var> list itself, using <a href="#apihelper.lambda" title="4.7. Using lambda Functions">lambda functions</a>.
|
||||
<li>Note that the <code>plural</code> function hasn't changed at all. It iterates through a set of rule functions, checks the first rule, and if it returns a
|
||||
true value, calls the second rule and returns the value. Same as above, word for word. The only difference is that the rule
|
||||
functions were defined inline, anonymously, using lambda functions. But the <code>plural</code> function doesn't care how they were defined; it just gets a list of rules and blindly works through them.
|
||||
<p>Now to add a new rule, all you need to do is define the functions directly in the <var>rules</var> list itself: one match rule, and one apply rule. But defining the rule functions inline like this makes it very clear that
|
||||
you have some unnecessary duplication here. You have four pairs of functions, and they all follow the same pattern. The
|
||||
match function is a single call to <code>re.search</code>, and the apply function is a single call to <code>re.sub</code>. Let's factor out these similarities.
|
||||
<h2 id="plural.stage4">17.5. <code>plural.py</code>, stage 4</h2>
|
||||
<p>Let's factor out the duplication in the code so that defining new rules can be easier.
|
||||
<div class=example><h3 id="plural.stage4.example.1">Example 17.9. <code>plural4.py</code></h3><pre><code>
|
||||
import re
|
||||
|
||||
def buildMatchAndApplyFunctions((pattern, search, replace)):
|
||||
matchFunction = lambda word: re.search(pattern, word) <span>①</span>
|
||||
applyFunction = lambda word: re.sub(search, replace, word) <span>②</span>
|
||||
return (matchFunction, applyFunction) <span>③</span>
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li><code>buildMatchAndApplyFunctions</code> is a function that builds other functions dynamically. It takes <var>pattern</var>, <var>search</var> and <var>replace</var> (actually it takes a tuple, but more on that in a minute), and you can build the match function using the <code>lambda</code> syntax to be a function that takes one parameter (<var>word</var>) and calls <code>re.search</code> with the <var>pattern</var> that was passed to the <code>buildMatchAndApplyFunctions</code> function, and the <var>word</var> that was passed to the match function you're building. Whoa.
|
||||
<li>Building the apply function works the same way. The apply function is a function that takes one parameter, and calls <code>re.sub</code> with the <var>search</var> and <var>replace</var> parameters that were passed to the <code>buildMatchAndApplyFunctions</code> function, and the <var>word</var> that was passed to the apply function you're building. This technique of using the values of outside parameters within a
|
||||
dynamic function is called <em>closures</em>. You're essentially defining constants within the apply function you're building: it takes one parameter (<var>word</var>), but it then acts on that plus two other values (<var>search</var> and <var>replace</var>) which were set when you defined the apply function.
|
||||
<li>Finally, the <code>buildMatchAndApplyFunctions</code> function returns a tuple of two values: the two functions you just created. The constants you defined within those functions
|
||||
(<var>pattern</var> within <var>matchFunction</var>, and <var>search</var> and <var>replace</var> within <var>applyFunction</var>) stay with those functions, even after you return from <code>buildMatchAndApplyFunctions</code>. That's insanely cool.
|
||||
<p>If this is incredibly confusing (and it should be, this is weird stuff), it may become clearer when you see how to use it.
|
||||
<div class=example><h3>Example 17.10. <code>plural4.py</code> continued</h3><pre><code>
|
||||
patterns = \
|
||||
(
|
||||
('[sxz]$', '$', 'es'),
|
||||
('[^aeioudgkprt]h$', '$', 'es'),
|
||||
('(qu|[^aeiou])y$', 'y$', 'ies'),
|
||||
('$', '$', 's')
|
||||
) <span>①</span>
|
||||
rules = map(buildMatchAndApplyFunctions, patterns) <span>②</span>
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>Our pluralization rules are now defined as a series of strings (not functions). The first string is the regular expression
|
||||
that you would use in <code>re.search</code> to see if this rule matches; the second and third are the search and replace expressions you would use in <code>re.sub</code> to actually apply the rule to turn a noun into its plural.
|
||||
<li>This line is magic. It takes the list of strings in <var>patterns</var> and turns them into a list of functions. How? By mapping the strings to the <code>buildMatchAndApplyFunctions</code> function, which just happens to take three strings as parameters and return a tuple of two functions. This means that <var>rules</var> ends up being exactly the same as the previous example: a list of tuples, where each tuple is a pair of functions, where
|
||||
the first function is the match function that calls <code>re.search</code>, and the second function is the apply function that calls <code>re.sub</code>.
|
||||
<p>I swear I am not making this up: <var>rules</var> ends up with exactly the same list of functions as the previous example. Unroll the <var>rules</var> definition, and you'll get this:
|
||||
<div class=example><h3>Example 17.11. Unrolling the rules definition</h3><pre><code>
|
||||
rules = \
|
||||
(
|
||||
(
|
||||
lambda word: re.search('[sxz]$', word),
|
||||
lambda word: re.sub('$', 'es', word)
|
||||
),
|
||||
(
|
||||
lambda word: re.search('[^aeioudgkprt]h$', word),
|
||||
lambda word: re.sub('$', 'es', word)
|
||||
),
|
||||
(
|
||||
lambda word: re.search('[^aeiou]y$', word),
|
||||
lambda word: re.sub('y$', 'ies', word)
|
||||
),
|
||||
(
|
||||
lambda word: re.search('$', word),
|
||||
lambda word: re.sub('$', 's', word)
|
||||
)
|
||||
)
|
||||
</pre><div class=example><h3 id="plural.finishing.up">Example 17.12. <code>plural4.py</code>, finishing up</h3><pre><code>
|
||||
def plural(noun):
|
||||
for matchesRule, applyRule in rules: <span>①</span>
|
||||
if matchesRule(noun):
|
||||
return applyRule(noun)
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>Since the <var>rules</var> list is the same as the previous example, it should come as no surprise that the <code>plural</code> function hasn't changed. Remember, it's completely generic; it takes a list of rule functions and calls them in order.
|
||||
It doesn't care how the rules are defined. In <a href="#plural.stage2" title="17.3. plural.py, stage 2">stage 2</a>, they were defined as seperate named functions. In <a href="#plural.stage3" title="17.4. plural.py, stage 3">stage 3</a>, they were defined as anonymous <code>lambda</code> functions. Now in stage 4, they are built dynamically by mapping the <code>buildMatchAndApplyFunctions</code> function onto a list of raw strings. Doesn't matter; the <code>plural</code> function still works the same way.
|
||||
<p>Just in case that wasn't mind-blowing enough, I must confess that there was a subtlety in the definition of <code>buildMatchAndApplyFunctions</code> that I skipped over. Let's go back and take another look.
|
||||
<div class=example><h3>Example 17.13. Another look at <code>buildMatchAndApplyFunctions</code></h3><pre><code>
|
||||
def buildMatchAndApplyFunctions((pattern, search, replace)): <span>①</span>
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>Notice the double parentheses? This function doesn't actually take three parameters; it actually takes one parameter, a tuple
|
||||
of three elements. But the tuple is expanded when the function is called, and the three elements of the tuple are each assigned
|
||||
to different variables: <var>pattern</var>, <var>search</var>, and <var>replace</var>. Confused yet? Let's see it in action.
|
||||
<div class=example><h3>Example 17.14. Expanding tuples when calling functions</h3><pre class=screen>
|
||||
<samp class=p>>>> </samp><kbd>def foo((a, b, c)):</kbd>
|
||||
<samp class=p>... </samp>print c
|
||||
<samp class=p>... </samp>print b
|
||||
<samp class=p>... </samp>print a
|
||||
<samp class=p>>>> </samp><kbd>parameters = ('apple', 'bear', 'catnap')</kbd>
|
||||
<samp class=p>>>> </samp><kbd>foo(parameters)</kbd> <span>①</span>
|
||||
catnap
|
||||
bear
|
||||
apple
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>The proper way to call the function <code>foo</code> is with a tuple of three elements. When the function is called, the elements are assigned to different local variables within
|
||||
<code>foo</code>.
|
||||
<p>Now let's go back and see why this auto-tuple-expansion trick was necessary. <var>patterns</var> was a list of tuples, and each tuple had three elements. When you called <code>map(buildMatchAndApplyFunctions, patterns)</code>, that means that <code>buildMatchAndApplyFunctions</code> is <em>not</em> getting called with three parameters. Using <code>map</code> to map a single list onto a function always calls the function with a single parameter: each element of the list. In the
|
||||
case of <var>patterns</var>, each element of the list is a tuple, so <code>buildMatchAndApplyFunctions</code> always gets called with the tuple, and you use the auto-tuple-expansion trick in the definition of <code>buildMatchAndApplyFunctions</code> to assign the elements of that tuple to named variables that you can work with.
|
||||
<h2 id="plural.stage5">17.6. <code>plural.py</code>, stage 5</h2>
|
||||
<p>You've factored out all the duplicate code and added enough abstractions so that the pluralization rules are defined in a
|
||||
list of strings. The next logical step is to take these strings and put them in a separate file, where they can be maintained
|
||||
separately from the code that uses them.
|
||||
<p>First, let's create a text file that contains the rules you want. No fancy data structures, just space- (or tab-)delimited
|
||||
strings in three columns. You'll call it <code>rules.en</code>; “en” stands for English. These are the rules for pluralizing English nouns. You could add other rule files for other languages
|
||||
later.
|
||||
<div class=example><h3>Example 17.15. <code>rules.en</code></h3><pre><code>
|
||||
[sxz]$$ es
|
||||
[^aeioudgkprt]h$ $ es
|
||||
[^aeiou]y$ y$ ies
|
||||
$ $ s
|
||||
</pre><p>Now let's see how you can use this rules file.
|
||||
<div class=example><h3>Example 17.16. <code>plural5.py</code></h3><pre><code>
|
||||
import re
|
||||
import string
|
||||
|
||||
def buildRule((pattern, search, replace)):
|
||||
return lambda word: re.search(pattern, word) and re.sub(search, replace, word) <span>①</span>
|
||||
|
||||
def plural(noun, language='en'): <span>②</span>
|
||||
lines = file('rules.%s' % language).readlines() <span>③</span>
|
||||
patterns = map(string.split, lines) <span>④</span>
|
||||
rules = map(buildRule, patterns) <span>⑤</span>
|
||||
for rule in rules:
|
||||
result = rule(noun) <span>⑥</span>
|
||||
if result: return result
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>You're still using the closures technique here (building a function dynamically that uses variables defined outside the function),
|
||||
but now you've combined the separate match and apply functions into one. (The reason for this change will become clear in
|
||||
the next section.) This will let you accomplish the same thing as having two functions, but you'll need to call it differently,
|
||||
as you'll see in a minute.
|
||||
<li>Our <code>plural</code> function now takes an optional second parameter, <var>language</var>, which defaults to <code>en</code>.
|
||||
<li>You use the <var>language</var> parameter to construct a filename, then open the file and read the contents into a list. If <var>language</var> is <code>en</code>, then you'll open the <code>rules.en</code> file, read the entire thing, break it up by carriage returns, and return a list. Each line of the file will be one element
|
||||
in the list.
|
||||
<li>As you saw, each line in the file really has three values, but they're separated by whitespace (tabs or spaces, it makes no
|
||||
difference). Mapping the <code>string.split</code> function onto this list will create a new list where each element is a tuple of three strings. So a line like <code>[sxz]$ $ es</code> will be broken up into the tuple <code>('[sxz]$', '$', 'es')</code>. This means that <var>patterns</var> will end up as a list of tuples, just like you hard-coded it in <a href="#plural.stage4" title="17.5. plural.py, stage 4">stage 4</a>.
|
||||
<li>If <var>patterns</var> is a list of tuples, then <var>rules</var> will be a list of the functions created dynamically by each call to <code>buildRule</code>. Calling <code>buildRule(('[sxz]$', '$', 'es'))</code> returns a function that takes a single parameter, <var>word</var>. When this returned function is called, it will execute <code>re.search('[sxz]$', word) and re.sub('$', 'es', word)</code>.
|
||||
<li>Because you're now building a combined match-and-apply function, you need to call it differently. Just call the function,
|
||||
and if it returns something, then that's the plural; if it returns nothing (<code>None</code>), then the rule didn't match and you need to try another rule.
|
||||
<p>So the improvement here is that you've completely separated the pluralization rules into an external file. Not only can the
|
||||
file be maintained separately from the code, but you've set up a naming scheme where the same <code>plural</code> function can use different rule files, based on the <var>language</var> parameter.
|
||||
<p>The downside here is that you're reading that file every time you call the <code>plural</code> function. I thought I could get through this entire book without using the phrase “left as an exercise for the reader”, but here you go: building a caching mechanism for the language-specific rule files that auto-refreshes itself if the rule
|
||||
files change between calls <em>is left as an exercise for the reader</em>. Have fun.
|
||||
<h2 id="plural.stage6">17.7. <code>plural.py</code>, stage 6</h2>
|
||||
<p>Now you're ready to talk about generators.
|
||||
<div class=example><h3>Example 17.17. <code>plural6.py</code></h3><pre><code>
|
||||
import re
|
||||
|
||||
def rules(language):
|
||||
for line in file('rules.%s' % language):
|
||||
pattern, search, replace = line.split()
|
||||
yield lambda word: re.search(pattern, word) and re.sub(search, replace, word)
|
||||
|
||||
def plural(noun, language='en'):
|
||||
for applyRule in rules(language):
|
||||
result = applyRule(noun)
|
||||
if result: return result
|
||||
</pre><p>This uses a technique called generators, which I'm not even going to try to explain until you look at a simpler example first.
|
||||
<div class=example><h3 id="plural.introducing.generators">Example 17.18. Introducing generators</h3><pre class=screen>
|
||||
<samp class=p>>>> </samp><kbd>def make_counter(x):</kbd>
|
||||
<samp class=p>... </samp>print 'entering make_counter'
|
||||
<samp class=p>... </samp>while 1:
|
||||
<samp class=p>... </samp> yield x <span>①</span>
|
||||
<samp class=p>... </samp> print 'incrementing x'
|
||||
<samp class=p>... </samp> x = x + 1
|
||||
<samp class=p>... </samp>
|
||||
<samp class=p>>>> </samp><kbd>counter = make_counter(2)</kbd> <span>②</span>
|
||||
<samp class=p>>>> </samp><kbd>counter</kbd> <span>③</span>
|
||||
<generator object at 0x001C9C10>
|
||||
<samp class=p>>>> </samp><kbd>counter.next()</kbd> <span>④</span>
|
||||
<samp>entering make_counter
|
||||
2</samp>
|
||||
<samp class=p>>>> </samp><kbd>counter.next()</kbd> <span>⑤</span>
|
||||
<samp>incrementing x
|
||||
3</samp>
|
||||
<samp class=p>>>> </samp><kbd>counter.next()</kbd> <span>⑥</span>
|
||||
<samp>incrementing x
|
||||
4</span>
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>The presence of the <code>yield</code> keyword in <code>make_counter</code> means that this is not a normal function. It is a special kind of function which generates values one at a time. You can
|
||||
think of it as a resumable function. Calling it will return a generator that can be used to generate successive values of
|
||||
<var>x</var>.
|
||||
<li>To create an instance of the <code>make_counter</code> generator, just call it like any other function. Note that this does not actually execute the function code. You can tell
|
||||
this because the first line of <code>make_counter</code> is a <code>print</code> statement, but nothing has been printed yet.
|
||||
<li>The <code>make_counter</code> function returns a generator object.
|
||||
<li>The first time you call the <code>next()</code> method on the generator object, it executes the code in <code>make_counter</code> up to the first <code>yield</code> statement, and then returns the value that was yielded. In this case, that will be <code>2</code>, because you originally created the generator by calling <code>make_counter(2)</code>.
|
||||
<li>Repeatedly calling <code>next()</code> on the generator object <em>resumes where you left off</em> and continues until you hit the next <code>yield</code> statement. The next line of code waiting to be executed is the <code>print</code> statement that prints <code>incrementing x</code>, and then after that the <code>x = x + 1</code> statement that actually increments it. Then you loop through the <code>while</code> loop again, and the first thing you do is <code>yield x</code>, which returns the current value of <var>x</var> (now 3).
|
||||
<li>The second time you call <code>counter.next()</code>, you do all the same things again, but this time <var>x</var> is now <code>4</code>. And so forth. Since <code>make_counter</code> sets up an infinite loop, you could theoretically do this forever, and it would just keep incrementing <var>x</var> and spitting out values. But let's look at more productive uses of generators instead.
|
||||
<div class=example><h3 id="plural.fib.example">Example 17.19. Using generators instead of recursion</h3><pre><code>
|
||||
def fibonacci(max):
|
||||
a, b = 0, 1 <span>①</span>
|
||||
while a < max:
|
||||
yield a <span>②</span>
|
||||
a, b = b, a+b <span>③</span>
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>The Fibonacci sequence is a sequence of numbers where each number is the sum of the two numbers before it. It starts with
|
||||
<code>0</code> and <code>1</code>, goes up slowly at first, then more and more rapidly. To start the sequence, you need two variables: <var>a</var> starts at <code>0</code>, and <var>b</var> starts at <code>1</code>.
|
||||
<li><var>a</var> is the current number in the sequence, so yield it.
|
||||
<li><var>b</var> is the next number in the sequence, so assign that to <var>a</var>, but also calculate the next value (<code>a+b</code>) and assign that to <var>b</var> for later use. Note that this happens in parallel; if <var>a</var> is <code>3</code> and <var>b</var> is <code>5</code>, then <code>a, b = b, a+b</code> will set <var>a</var> to <code>5</code> (the previous value of <var>b</var>) and <var>b</var> to <code>8</code> (the sum of the previous values of <var>a</var> and <var>b</var>).
|
||||
<p>So you have a function that spits out successive Fibonacci numbers. Sure, you could do that with recursion, but this way
|
||||
is easier to read. Also, it works well with <code>for</code> loops.
|
||||
<div class=example><h3>Example 17.20. Generators in <code>for</code> loops</h3><pre class=screen>
|
||||
<samp class=p>>>> </samp><kbd>for n in fibonacci(1000):</kbd> <span>①</span>
|
||||
<samp class=p>... </samp>print n, <span>②</span>
|
||||
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li>You can use a generator like <code>fibonacci</code> in a <code>for</code> loop directly. The <code>for</code> loop will create the generator object and successively call the <code>next()</code> method to get values to assign to the <code>for</code> loop index variable (<var>n</var>).
|
||||
<li>Each time through the <code>for</code> loop, <var>n</var> gets a new value from the <code>yield</code> statement in <code>fibonacci</code>, and all you do is print it out. Once <code>fibonacci</code> runs out of numbers (<var>a</var> gets bigger than <var>max</var>, which in this case is <code>1000</code>), then the <code>for</code> loop exits gracefully.
|
||||
<p>OK, let's go back to the <code>plural</code> function and see how you're using this.
|
||||
<div class=example><h3>Example 17.21. Generators that generate dynamic functions</h3><pre><code>
|
||||
def rules(language):
|
||||
for line in file('rules.%s' % language): <span>①</span>
|
||||
pattern, search, replace = line.split() <span>②</span>
|
||||
yield lambda word: re.search(pattern, word) and re.sub(search, replace, word) <span>③</span>
|
||||
|
||||
def plural(noun, language='en'):
|
||||
for applyRule in rules(language): <span>④</span>
|
||||
result = applyRule(noun)
|
||||
if result: return result
|
||||
</pre><div class=calloutlist>
|
||||
<ol>
|
||||
<li><code>for line in file(...)</code> is a common idiom for reading lines from a file, one line at a time. It works because <em><code>file</code> actually returns a generator</em> whose <code>next()</code> method returns the next line of the file. That is so insanely cool, I wet myself just thinking about it.
|
||||
<li>No magic here. Remember that the lines of the rules file have three values separated by whitespace, so <code>line.split()</code> returns a tuple of 3 values, and you assign those values to 3 local variables.
|
||||
<li><em>And then you yield.</em> What do you yield? A function, built dynamically with <code>lambda</code>, that is actually a closure (it uses the local variables <var>pattern</var>, <var>search</var>, and <var>replace</var> as constants). In other words, <code>rules</code> is a generator that spits out rule functions.
|
||||
<li>Since <code>rules</code> is a generator, you can use it directly in a <code>for</code> loop. The first time through the <code>for</code> loop, you will call the <code>rules</code> function, which will open the rules file, read the first line out of it, dynamically build a function that matches and applies
|
||||
the first rule defined in the rules file, and yields the dynamically built function. The second time through the <code>for</code> loop, you will pick up where you left off in <code>rules</code> (which was in the middle of the <code>for line in file(...)</code> loop), read the second line of the rules file, dynamically build another function that matches and applies the second rule
|
||||
defined in the rules file, and yields it. And so forth.
|
||||
<p>What have you gained over <a href="#plural.stage5" title="17.6. plural.py, stage 5">stage 5</a>? In stage 5, you read the entire rules file and built a list of all the possible rules before you even tried the first one.
|
||||
Now with generators, you can do everything lazily: you open the first and read the first rule and create a function to try
|
||||
it, but if that works you don't ever read the rest of the file or create any other functions.
|
||||
<div class=itemizedlist>
|
||||
<h3>Further reading</h3>
|
||||
<ul>
|
||||
<li><a href="http://www.python.org/peps/pep-0255.html">PEP 255</a> defines generators.
|
||||
|
||||
<li><a href="http://www.activestate.com/ASPN/Python/Cookbook/" title="growing archive of annotated code samples">Python Cookbook</a> has <a href="http://www.google.com/search?q=generators+cookbook+site:aspn.activestate.com">many more examples of generators</a>.
|
||||
|
||||
</ul>
|
||||
<h2 id="plural.summary">17.8. Summary</h2>
|
||||
<p>You talked about several different advanced techniques in this chapter. Not all of them are appropriate for every situation.
|
||||
<p>You should now be comfortable with all of these techniques:
|
||||
<div class=itemizedlist>
|
||||
<ul>
|
||||
<li>Performing <a href="#plural.stage1" title="17.2. plural.py, stage 1">string substitution with regular expressions</a>.
|
||||
|
||||
<li>Treating <a href="#plural.stage2" title="17.3. plural.py, stage 2">functions as objects</a>, storing them in lists, assigning them to variables, and calling them through those variables.
|
||||
|
||||
<li>Building <a href="#plural.stage3" title="17.4. plural.py, stage 3">dynamic functions with <code>lambda</code></a>.
|
||||
|
||||
<li>Building <a href="#plural.stage4" title="17.5. plural.py, stage 4">closures</a>, dynamic functions that contain surrounding variables as constants.
|
||||
|
||||
<li>Building <a href="#plural.stage6" title="17.7. plural.py, stage 6">generators</a>, resumable functions that perform incremental logic and return different values each time you call them.
|
||||
|
||||
</ul>
|
||||
<p>Adding abstractions, building functions dynamically, building closures, and using generators can all make your code simpler,
|
||||
more readable, and more flexible. But they can also end up making it more difficult to debug later. It's up to you to find
|
||||
the right balance between simplicity and power.
|
||||
<div class=chapter>
|
||||
<h2 id="soundex">Chapter 18. Performance Tuning</h2>
|
||||
<p>Performance tuning is a many-splendored thing. Just because Python is an interpreted language doesn't mean you shouldn't worry about code optimization. But don't worry about it <em>too</em> much.
|
||||
<h2 id="soundex.divein">18.1. Diving in</h2>
|
||||
|
||||
Reference in New Issue
Block a user