You are here: Home ‣ Dive Into Python 3 ‣

Advanced Iterators

❝ Great fleas have little fleas upon their backs to bite ’em,
And little fleas have lesser fleas, and so ad infinitum. ❞
— Augustus De Morgan

Diving In

FIXME

original recipe by Raymond Hettinger, ported to Python 3 and used as the basis for this chapter with his permission.

[download alphametics.py]

import re
import itertools

def solve(puzzle):
    words = re.findall('[A-Z]+', puzzle.upper())
    unique_characters = {c for c in ''.join(words)}
    assert len(unique_characters) <= 10
    first_letters = {word[0] for word in words}
    n = len(first_letters)
    sorted_characters = ''.join(first_letters) + \
        ''.join(unique_characters - first_letters)
    characters = tuple(ord(c) for c in sorted_characters)
    digits = tuple(ord(c) for c in '0123456789')
    zero = digits[0]
    for guess in itertools.permutations(digits, len(characters)):
        if zero not in guess[:n]:
            equation = puzzle.translate(dict(zip(characters, guess)))
            if eval(equation):
                return equation

if __name__ == '__main__':
    import sys
    for puzzle in sys.argv[1:]:
        print(puzzle)
        solution = solve(puzzle)
        if solution:
            print(solution)

you@localhost:~$ python3 alphametics.py "SEND + MORE == MONEY"
SEND + MORE == MONEY
9567 + 1085 == 10652
you@localhost:~$ python3 alphametics.py "I + LOVE + YOU == DORA"
I + LOVE + YOU == DORA
1 + 2784 + 975 == 3760

Finding all occurrences of a pattern

FIXME

>>> import re
>>> re.findall('[A-Z]+', 'SEND + MORE == MONEY')
['SEND', 'MORE', 'MONEY']

FIXME

Finding the unique items in a sequence

>>> a_list = ['a', 'c', 'b', 'a', 'd', 'b']
>>> {c for c in a_list}
{'a', 'c', 'b', 'd'}
>>> a_string = 'EAST IS EAST'
>>> {c for c in a_string}
{'A', ' ', 'E', 'I', 'S', 'T'}
>>> words = ['SEND', 'MORE', 'MONEY']
>>> ''.join(words)
'SENDMOREMONEY'
>>> {c for c in ''.join(words)}
{'E', 'D', 'M', 'O', 'N', 'S', 'R', 'Y'}

FIXME

Making assertions

FIXME

>>> assert 1 + 1 = 2
>>> assert 1 + 1 = 3
Traceback (most recent call last):
  File "<stdin>", line 1, in 
AssertionError

FIXME

assert len(unique_characters) <= 10

…is equivalent to…

if len(unique_characters) > 10:
    raise AssertionError

Generator objects

FIXME

>>> unique_characters = {'E', 'D', 'M', 'O', 'N', 'S', 'R', 'Y'}
>>> gen = (ord(c) for c in unique_characters)
>>> gen
<generator object <genexpr> at 0x00BADC10>
>>> next(gen)
69
>>> next(gen)
68
>>> tuple(ord(c) for c in unique_characters)
(69, 68, 77, 79, 78, 83, 82, 89)

FIXME

Calculating Permutations… The Lazy Way!

First of all, what the heck are permutations? Permutations are a mathematical concept. (There are actually several definitions, depending on what kind of math you're doing. Here I'm talking about combinatorics, but if that doesn't mean anything to you, don't worry about it. As always, Wikipedia is your friend.)

The idea is that you take a list of things (could be numbers, could be letters, could be dancing bears) and find all the possible ways to split them up into smaller lists. All the smaller lists have the same size, which can be as small as 1 and as large as the total number of items. Oh, and nothing can be repeated. Mathematicians say things like "let's find the permutations of 3 different items taken 2 at a time," which means you have a sequence of 3 items and you want to find all the possible ordered pairs.

>>> import itertools                              ①
>>> perms = itertools.permutations([1, 2, 3], 2)  ②
>>> next(perms)                                   ③
(1, 2)
>>> next(perms)
(1, 3)
>>> next(perms)
(2, 1)                                            ④
>>> next(perms)
(2, 3)
>>> next(perms)
(3, 1)
>>> next(perms)
(3, 2)
>>> next(perms)                                   ⑤
Traceback (most recent call last):
  File "<stdin>", line 1, in 
StopIteration

The itertools module has all kinds of fun stuff in it, including a permutations() function that does all the hard work of finding permutations.
The permutations() function takes a sequence (here a list of three integers) and a number, which is the number of items you want in each smaller group. The function returns an iterator, which you can use in a foor loop or any old place that iterates. Here I'll step through the iterator manually to show all the values.
The first permutation of [1, 2, 3] taken 2 at a time is (1, 2).
Note that permutations are ordered: (2, 1) is different than (1, 2).
That's it! Those are all the permutations of [1, 2, 3] taken 2 at a time. Pairs like (1, 1) and (2, 2) never show up, because they contain repeats so they aren't valid permutations. When there are no more permutations, the iterator raises a StopIteration exception.

The permutations() function doesn't have to take a list. It can take any sequence — even a string.

>>> import itertools
>>> perms = itertools.permutations('ABC', 3)  ①
>>> next(perms)
('A', 'B', 'C')                               ②
>>> next(perms)
('A', 'C', 'B')
>>> next(perms)
('B', 'A', 'C')
>>> next(perms)
('B', 'C', 'A')
>>> next(perms)
('C', 'A', 'B')
>>> next(perms)
('C', 'B', 'A')
>>> next(perms)
Traceback (most recent call last):
  File "<stdin>", line 1, in 
StopIteration
>>> list(itertools.permutations('ABC', 3))    ③
[('A', 'B', 'C'), ('A', 'C', 'B'),
 ('B', 'A', 'C'), ('B', 'C', 'A'),
 ('C', 'A', 'B'), ('C', 'B', 'A')]

A string is just a sequence of characters. For the purposes of finding permutations, the string 'ABC' is equivalent to the list ['A', 'B', 'C'].
The first permutation of the 3 items ['A', 'B', 'C'], taken 3 at a time, is ('A', 'B', 'C'). There are five other permutations — the same three characters in every conceivable order.
Since the permutations() function always returns an iterator, an easy way to debug permutations is to pass that iterator to the built-in list() function to see all the permutations immediately.

Other Fun Stuff in the `itertools` Module

>>> import itertools
>>> list(itertools.product('ABC', '123'))
[('A', '1'), ('A', '2'), ('A', '3'), 
 ('B', '1'), ('B', '2'), ('B', '3'), 
 ('C', '1'), ('C', '2'), ('C', '3')]
>>> list(itertools.combinations('ABC', 2))
[('A', 'B'), ('A', 'C'), ('B', 'C')]

FIXME

>>> names = list(open('examples/favorite-people.txt'))
>>> names
['Dora\n', 'Ethan\n', 'Wesley\n', 'John\n', 'Anne\n',
'Mike\n', 'Chris\n', 'Sarah\n', 'Alex\n', 'Lizzie\n']
>>> names = [name[:-1] for name in names]
>>> names
['Dora', 'Ethan', 'Wesley', 'John', 'Anne',
'Mike', 'Chris', 'Sarah', 'Alex', 'Lizzie']
>>> names = sorted(names)
>>> names
['Alex', 'Anne', 'Chris', 'Dora', 'Ethan',
'John', 'Lizzie', 'Mike', 'Sarah', 'Wesley']
>>> names = sorted(names, key=len)
>>> names
['Alex', 'Anne', 'Dora', 'John', 'Mike',
'Chris', 'Ethan', 'Sarah', 'Lizzie', 'Wesley']
>>> import itertools
>>> groups = itertools.groupby(names, len)
>>> groups
<itertools.groupby object at 0x00BB20C0>
>>> list(groups)
[(4, <itertools._grouper object at 0x00BA8BF0>),
 (5, <itertools._grouper object at 0x00BB4050>),
 (6, <itertools._grouper object at 0x00BB4030>)]
>>> groups = itertools.groupby(names, len)
>>> for name_length, name_iter in groups:
...     print('Names with {0:d} letters:'.format(name_length))
...     for name in name_iter:
...         print(name)
... 
Names with 4 letters:
Alex
Anne
Dora
John
Mike
Names with 5 letters:
Chris
Ethan
Sarah
Names with 6 letters:
Lizzie
Wesley

FIXME

Combining Iterators

FIXME

>>> list(range(0, 3))
[0, 1, 2]
>>> list(range(10, 13))
[10, 11, 12]
>>> list(itertools.chain(range(0, 3), range(10, 13)))
[0, 1, 2, 10, 11, 12]
>>> list(zip(range(0, 3), range(10, 13)))
[(0, 10), (1, 11), (2, 12)]
>>> list(zip(range(0, 3), range(10, 14)))
[(0, 10), (1, 11), (2, 12)]
>>> list(itertools.zip_longest(range(0, 3), range(10, 14)))
[(0, 10), (1, 11), (2, 12), (None, 13)]

FIXME

>>> characters = ('S', 'M', 'E', 'D', 'O', 'N', 'R', 'Y')
>>> guess = ('1', '2', '0', '3', '4', '5', '6', '7')
>>> tuple(zip(characters, guess))
(('S', '1'), ('M', '2'), ('E', '0'), ('D', '3'),
 ('O', '4'), ('N', '5'), ('R', '6'), ('Y', '7'))
>>> dict(zip(characters, guess))
{'E': '0', 'D': '3', 'M': '2', 'O': '4',
 'N': '5', 'S': '1', 'R': '6', 'Y': '7'}

A New Kind Of String Manipulation

>>> characters = tuple(ord(c) for c in 'SMEDONRY')
>>> characters
(83, 77, 69, 68, 79, 78, 82, 89)
>>> digits = tuple(ord(c) for c in '0123456789')
>>> digits
(48, 49, 50, 51, 52, 53, 54, 55, 56, 57)
>>> guess = (49, 50, 48, 51, 52, 53, 54, 55)
>>> translation_table = dict(zip(characters, guess))
>>> translation_table
{68: 51, 69: 48, 77: 50, 78: 53, 79: 52, 82: 54, 83: 49, 89: 55}
>>> "SEND + MORE == MONEY".translate(translation_table)
'1053 + 2460 == 24507'

FIXME

>>> translation_table = {ord("A"): ord("O")}
>>> translation_table
{65: 79}
>>> 'MARK'.translate(translation_table)
'MORK'

FIXME

Evaluating Arbitrary Strings As Python Expressions

FIXME

Putting It All Together

FIXME

Advanced Iterators

Diving In

Finding all occurrences of a pattern

Finding the unique items in a sequence

Making assertions

Generator objects

Calculating Permutations… The Lazy Way!

Other Fun Stuff in the itertools Module

Combining Iterators

A New Kind Of String Manipulation

Evaluating Arbitrary Strings As Python Expressions

Putting It All Together

Further Reading

Other Fun Stuff in the `itertools` Module