Porting code to Python 3 with 2to3

❝ Life is pleasant. Death is peaceful. It's the transition that's troublesome. ❞
— Isaac Asimov (attributed)

1. Diving in
2. print statement
3. Unicode string literals
4. long data type
5. <> comparison
6. has_key() dictionary method
7. Dictionary methods that return lists
8. Modules that have been renamed or reorganized
   1. http package
   2. urllib package
   3. dbm package
   4. xmlrpc package
   5. Other modules
9. Relative imports within a package
10. filter() global function
11. map() global function
12. reduce() global function (3.1+)
13. apply() global function
14. intern() global function
15. exec statement
16. execfile statement (3.1+)
17. repr literals (backticks)
18. try...except statement
19. raise statement
20. throw statement
21. xrange() global function
22. raw_input() and input() global functions
23. func_* function attributes
24. xreadlines() I/O method
25. lambda functions with multiple parameters
26. Special method attributes
27. next() iterator method
28. __nonzero__ special class attribute
29. Number literals
30. sys.maxint
31. unicode() global function
32. callable() global function
33. zip() global function
34. StandardError() exception
35. types module constants
36. isinstance global function (3.1+)
37. basestring datatype
38. itertools module
39. sys.exc_type, sys.exc_value, sys.exc_traceback
40. List comprehensions over tuples
41. os.getcwdu() function
42. Metaclasses
43. set() literals
44. buffer() global function
45. Whitespace around commas
46. Common idioms

Diving in

Python 3 comes with a utility script called 2to3, which takes your actual Python 2 source code as input and auto-converts as much as it can to Python 3. Case study: porting chardet to Python 3 describes how to run the 2to3 script, then shows some things it can't fix automatically. This appendix documents what it can fix automatically.

print statement

In Python 2, print was a statement -- whatever you wanted to print simply followed the print keyword. In Python 3, print() is a function -- whatever you want to print is passed to print() like any other function.

skip over this table

1. To print a blank line, call print() without any arguments.
2. To print a single value, call print() with one argument
3. To print two values separated by a space, call print() with two arguments.
4. This one is a little tricky. In Python 2, if you ended a print statement with a comma, it would print the values separated by spaces, then print a trailing space, then stop without printing a carriage return. In Python 3, the way to do this is to pass end=' ' as a keyword argument to the print() function. The end argument defaults to '\n' (a carriage return), so overriding it will suppress the carriage return after printing the other arguments.
5. In Python 2, you could redirect the output to a pipe -- like sys.stderr -- by using the >>pipe_name syntax. In Python 3, the way to do this is to pass the pipe in the file keyword argument. The file argument defaults to sys.stdout (standard out), so overriding it will output to a different pipe instead.

Unicode string literals

Python 2 had two string types: Unicode strings and non-Unicode strings. Python 3 has one string type: Unicode strings.

skip over this table

1. Unicode string literals are simply converted into string literals, which, in Python 3, are always Unicode.
2. Unicode "raw" strings (in which Python does not auto-escape backslashes) are converted to raw strings. In Python 3, "raw" strings are also Unicode.

long data type

Python 2 had separate int and long types for non-floating-point numbers. An int could not be any larger than sys.maxint, which varied by platform. Longs were defined by appending an L to the end of the number, and they could be, well, longer than ints. In Python 3, there is only one integer type, called int, which mostly behaves like the long type in Python 2.

Since there are no longer two types, there is no need for special syntax to distinguish them.

Further reading: PEP 237: Unifying Long Integers and Integers.

skip over this table

1. Base 10 long integer literals become base 10 integer literals.
2. Base 16 long integer literals become base 16 integer literals.
3. In Python 3, the old long() function no longer exists, since longs don't exist. To coerce a variable to an integer, use the int() function.
4. To check whether a variable is an integer, get its type and compare it to int, not long.
5. You can also use the isinstance() function to check data types; again, use int, not long, to check for integers.

<> comparison

Python 2 supported <> as a synonym for !=, the not-equals comparison operator. Python 3 supports the != operator, but not <>.

skip over this table

1. A simple comparison.
2. A more complex comparison between three values.

has_key() dictionary method

In Python 2, dictionaries had a has_key() method to test whether the dictionary had a certain key. In Python 3, this method no longer exists. Instead, you need to use the in operator.

skip over this table

1. The simplest form.
2. The or operator takes precedence over the in operator, so there is no need for parentheses here.
3. On the other hand, you do need parentheses here, for the same reason -- or takes precedence over in.
4. The in operator takes precedence over the + operator, so this form needs parentheses too.
5. Again with the parentheses, for the same reason.

Dictionary methods that return lists

In Python 2, many dictionary methods returned lists. The most frequently used methods were keys(), items(), and values(). In Python 3, all of these methods return dynamic views. In some contexts, this is not a problem. If the method's return value is immediately passed to another function that iterates through the entire sequence, it makes no difference whether the actual type is a list or a view. In other contexts, it matters a great deal. If you were expecting a complete list with individually addressable elements, your code will choke, because views do not support indexing.

skip over this table

1. 2to3 errs on the side of safety, converting the return value from keys() to a static list with the list() function. This will always work, but it will be less efficient than using a view. You should examine the converted code to see if a list is absolutely necessary, or if a view would do.
2. Another view-to-list conversion, with the items() method. 2to3 will do the same thing with the values() method.
3. Python 3 does not support the iterkeys() method anymore. Use keys(), and if necessary, convert the view to an iterator with the iter() function.
4. 2to3 recognizes when the iterkeys() method is used inside a list comprehension, and converts it to the keys() method (without wrapping it in an extra call to iter()). This works because views are iterable.
5. 2to3 recognizes that the keys() method is immediately passed to a function which iterates through an entire sequence, so there is no need to convert the return value to a list first. The min() function will happily iterate through the view instead. This applies to min(), max(), sum(), list(), tuple(), set(), sorted(), any(), and all().

Modules that have been renamed or reorganized

Several modules in the Python Standard Library have been renamed. Several other modules which are related to each other have been combined or reorganized to make their association more logical.

http package

In Python 3, several related HTTP modules have been combined into a single package, http.

skip over this table

import BaseHTTPServer
import SimpleHTTPServer
import CGIHttpServer

1. The http.client module implements a low-level library that can request HTTP resources and interpret HTTP responses.
2. The http.cookies module provides a Pythonic interface to "cookies" that are sent in a Set-Cookie: HTTP header.
3. The http.cookiejar module manipulates the actual files on disk that popular web browsers use to store cookies.
4. The http.server module provides a basic HTTP server.

urllib package

Python 2 had a rat's nest of overlapping modules to parse, encode, and fetch URLs. In Python 3, these have all been refactored and combined in a single package, urllib.

skip over this table

from urllib import FancyURLopener
from urllib import urlencode

from urllib.request import FancyURLopener
from urllib.parse import urlencode

from urllib2 import Request
from urllib2 import HTTPError

from urllib.request import Request
from urllib.error import HTTPError

1. The old urllib module in Python 2 had a variety of functions, including urlopen() for fetching data and splittype(), splithost(), and splituser() for splitting a URL into its constituent parts. These functions have been reorganized more logically within the new urllib package. 2to3 will also change all calls to these functions so they use the new naming scheme.
2. The old urllib2 module in Python 2 has been folded into into the urllib package in Python 3. All your urllib2 favorites -- the build_opener() method, Request objects, and HTTPBasicAuthHandler and friends -- are still available.
3. The urllib.parse module in Python 3 contains all the parsing functions from the old urlparse module in Python 2.
4. The urllib.robotparser module parses robots.txt files.
5. The FancyURLopener class, which handles HTTP redirects and other status codes, is still available in the new urllib.request module. The urlencode function has moved to urllib.parse.
6. The Request object is still available in urllib.request, but constants like HTTPError have been moved to urllib.error.

dbm package

All the various DBM clones are now in a single package, dbm. If you need a specific variant like GNU DBM, you can import the appropriate module within the dbm package.

skip over this table

import anydbm
import whichdb

xmlrpc package

XML-RPC is a lightweight method of performing remote RPC calls over HTTP. The XML-RPC client library and several XML-RPC server implementations are now combined in a single package, xmlrpc.

skip over this table

import DocXMLRPCServer
import SimpleXMLRPCServer

Other modules

skip over this table

try:
    import cStringIO as StringIO
except ImportError:
    import StringIO

try:
    import cPickle as pickle
except ImportError:
    import pickle

1. A common idiom in Python 2 was to try to import cStringIO as StringIO, and if that failed, to import StringIO instead. Do not do this in Python 3; the io module does it for you. It will find the fastest implementation available and use it automatically.
2. A similar idiom was used to import the fastest pickle implementation. Do not do this in Python 3; the pickle module does it for you.
3. The builtins module contains the "global" functions, classes, and constants used throughout the Python language. Redefining a function in the builtins module will redefine the "global" function everywhere. That is exactly as powerful and scary as it sounds.
4. The copyreg module adds pickle support for custom types defined in C.
5. The queue module implements a multi-producer, multi-consumer queue.
6. The socketserver module provides generic base classes for implementing different kinds of socket servers.
7. The configparser module parses INI-style configuration files.
8. The reprlib module reimplements the built-in repr() function, but with limits on how many values are represented.
9. The subprocess module allows you to spawn processes, connect to their pipes, and obtain their return codes.

Relative imports within a package

A package is a group of related modules that function as a single entity. In Python 2, when modules within a package need to reference each other, you use import foo or from foo import Bar. The Python 2 interpreter first searches within the current package to find foo.py, and then moves on to the other directories in the Python search path (sys.path). Python 3 works a bit differently. Instead of searching the current package, it goes directly to the Python search path. If you want one module within a package to import another module in the same package, you need to explicitly provide the relative path between the two modules.

Suppose you had this package, with multiple files in the same directory:

skip over this ASCII art

chardet/
|
+--__init__.py
|
+--constants.py
|
+--mbcharsetprober.py
|
+--universaldetector.py

Now suppose that universaldetector.py needs to import the entire constants.py file and one class from mbcharsetprober.py. How do you do it?

skip over this table

1. When you need to import an entire module from elsewhere in your package, use the new from . import syntax. The period is actually a relative path from this file (universaldetector.py) to the file you want to import (constants.py). In this case, they are in the same directory, thus the single period. You can also import from the parent directory (from .. import anothermodule) or a subdirectory.
2. To import a specific class or function from another module directly into your module's namespace, prefix the target module with a relative path, minus the trailing slash. In this case, mbcharsetprober.py is in the same directory as universaldetector.py, so the path is a single period. You can also import form the parent directory (from ..anothermodule import AnotherClass) or a subdirectory.

filter() global function

In Python 2, the filter() function returned a list, the result of "filtering" a sequence through a function that returned True or False for each item in the sequence. In Python 3, the filter() function returns an interator, not a list.

skip over this table

1. In the most basic case, 2to3 will wrap a call to filter() with a call to list(), which simply iterates through its argument and returns a real list.
2. However, if the call to filter() is already wrapped in list(), 2to3 will do nothing, since the fact that filter() is returning an iterator is irrelevant.
3. For the special syntax of filter(None, ...), 2to3 will transform the call into a semantically equivalent list comprehension.
4. In contexts like for loops, which iterate through the entire sequence anyway, no changes are necessary.
5. Again, no changes are necessary, because the list comprehension will iterate through the entire sequence, and it can do that just as well if filter() returns an iterator as if it returns a list.

map() global function

In much the same way as filter(), the map() function now returns an iterator. (In Python 2, it returned a list.)

skip over this table

1. As with filter(), in the most basic case, 2to3 will wrap a call to map() with a call to list().
2. For the special syntax of map(None, ...), the identity function, 2to3 will convert it to an equivalent call to list().
3. If the first argument to map() is a lambda function, 2to3 will convert it to an equivalent list comprehension.
4. In contexts like for loops, which iterate through the entire sequence anyway, no changes are necessary.
5. Again, no changes are necessary, because the list comprehension will iterate through the entire sequence, and it can do that just as well if map() returns an iterator as if it returns a list.

reduce() global function (3.1+)

In Python 3, the reduce() function has been removed from the global namespace and placed in the functools module.

☞

The version of 2to3 that shipped with Python 3.0 would not fix the reduce() function automatically. The fix first appeared in the 2to3 script that shipped with Python 3.1.

skip over this table

from functtools import reduce
reduce(a, b, c)

apply() global function

Python 2 had a global function called apply(), which took a function f and a list [a, b, c] and returned f(a, b, c). In Python 3, the apply() function no longer exists. Instead, there is a new function calling syntax that allows you to pass a list and have Python apply the list as the function's arguments.

skip over this table

1. In the simplest form, you can call a function with a list of arguments (an actual list like [a, b, c]) by prepending the list with an asterisk (*). This is exactly equivalent to the old apply() function in Python 2.
2. In Python 2, the apply() function could actually take three parameters: a function, a list of arguments, and a dictionary of named arguments. In Python 3, you can accomplish the same thing by prepending the list of arguments with an asterisk (*) and the dictionary of named arguments with two asterisks (**).
3. The + operator, used here for list concatenation, takes precedence over the * operator, so there is no need for extra parentheses around a_list_of_args + z.
4. The 2to3 script is smart enough to convert complex apply() calls, including calling functions within imported modules.

intern() global function

In Python 2, you could call the intern() function on a string to intern it as a performance optimization. In Python 3, the intern() function has been moved to the sys module.

skip over this table

exec statement

Just as the print statement became a function in Python 3, so too has the exec statement. The exec() function takes a string which contains arbitrary Python code and executes it as if it were just another statement or expression.

skip over this table

1. In the simplest form, the 2to3 script simply encloses the code-as-a-string in parentheses, since exec() is now a function instead of a statement.
2. The old exec statement could take a namespace, a private environment of globals in which the code-as-a-string would be executed. Python 3 can also do this; just pass the namespace as the second argument to the exec() function.
3. Even fancier, the old exec statement could also take a local namespace (like the variables defined within a function). In Python 3, the exec() function can do that too.

execfile statement (3.1+)

Like the old exec statement, the old execfile statement will execute strings as if they were Python code. Where exec took a string, execfile took a filename. In Python 3, the execfile statement has been eliminated. If you really need to take a file of Python code and execute it (but you're not willing to simply import it), you can accomplish the same thing by opening the file, reading its contents, calling the global compile() function to force the Python interpreter to compile the code, and then call the new exec() function.

☞

The version of 2to3 that shipped with Python 3.0 would not fix the execfile statement automatically. The fix first appeared in the 2to3 script that shipped with Python 3.1.

skip over this table

repr literals (backticks)

In Python 2, there was a special syntax of wrapping any object in backticks (like `x`) to get a representation of the object. In Python 3, this capability still exists, but you can no longer use backticks to get it. Instead, use the global repr() function.

skip over this table

1. Remember, x can be anything -- a class, a function, a module, a primitive data type, etc. The repr() function works on everything.
2. In Python 2, backticks could be nested, leading to this sort of confusing (but valid) expression. The 2to3 tool is smart enough to convert this into nested calls to repr().

try...except statement

The syntax for catching exceptions has changed slightly between Python 2 and 3.

skip over this table

try:
    import mymodule
except ImportError, e
    pass

try:
    import mymodule
except ImportError as e:
    pass

try:
    import mymodule
except (RuntimeError, ImportError), e
    pass

try:
    import mymodule
except (RuntimeError, ImportError) as e:
    pass

try:
    import mymodule
except ImportError:
    pass

try:
    import mymodule
except:
    pass

1. Instead of a comma after the exception type, Python 3 uses a new keyword, as.
2. The as keyword also works for catching multiple types of exceptions at once.
3. If you catch an exception but don't actually care about accessing the exception object itself, the syntax is identical between Python 2 and 3.
4. Similarly, if you use a fallback to catch all exceptions, the syntax is identical.

☞

You should never use a fallback to catch all exceptions when importing modules (or most other times), because it will also catch things like KeyboardInterrupt (if the user pressed Ctrl-C to interrupt the program) and can make it more difficult to debug errors.

raise statement

FIXME intro

skip over this table

1. ...
2. ...
3. ...

throw statement

FIXME intro

skip over this table

1. ...
2. ...
3. ...

xrange() global function

In Python 2, there were two ways to get a range of numbers: range(), which returned a list, and xrange(), which returned an iterator. In Python 3, range() returns an iterator, and xrange() doesn't exist.

skip over this table

1. In the simplest case, the 2to3 script will simply convert xrange() to range().
2. If your Python 2 code used range(), the 2to3 script does not know whether you needed a list, or whether an iterator would do. It errs on the side of caution and coerces the return value into a list by calling the list() function.
3. If the xrange() function was inside a list comprehension, there is no need to coerce the result to a list, since the list comprehension will work just fine with an iterator.
4. Similarly, a for loop will work just fine with an iterator, so there is no need to change anything here.
5. The sum() function will also work with an iterator, so 2to3 makes no changes here either. Like dictionary methods that return views instead of lists, this applies to min(), max(), sum(), list(), tuple(), set(), sorted(), any(), and all().

raw_input() and input() global functions

FIXME intro

skip over this table

1. ...
2. ...
3. ...
4. ...

func_* function attributes

FIXME intro

skip over this table

1. ...
2. ...
3. ...
4. ...
5. ...
6. ...
7. ...

xreadlines() I/O method

FIXME intro

skip over this table

1. ...
2. ...

lambda functions with multiple parameters

FIXME intro

skip over this table

1. ...
2. ...
3. ...

Special method attributes

FIXME intro

skip over this table

1. ...
2. ...
3. ...

next() iterator method

FIXME intro

skip over this table

class A:
    def next(self):
        pass

class A:
    def __next__(self):
        pass

class A:
    def next(self, x, y):
        pass

next = 42
for an_iterator in a_sequence_of_iterators:
    an_iterator.next()

next = 42
for an_iterator in a_sequence_of_iterators:
    an_iterator.__next__()

1. ...
2. ...
3. ...
4. ...
5. ...

__nonzero__ special class attribute

FIXME intro

skip over this table

class A:
    def __nonzero__(self):
        pass

class A:
    def __bool__(self):
        pass

class A:
    def __nonzero__(self, x, y):
        pass

1. ...
2. ...

Number literals

FIXME intro

skip over this table

1. ...
2. ...

sys.maxint

FIXME intro

skip over this table

import sys
a_function(sys.maxint)

import sys
a_function(sys.maxsize)

1. ...
2. ...

unicode() global function

FIXME intro

skip over this table

1. ...

callable() global function

FIXME intro

skip over this table

1. ...

zip() global function

FIXME intro

skip over this table

1. ...
2. ...

StandardError() exception

FIXME intro

skip over this table

1. ...
2. ...

types module constants

FIXME intro

skip over this table

1. ...
2. ...
3. ...
4. ...
5. ...
6. ...

isinstance global function (3.1+)

FIXME intro

☞

The version of 2to3 that shipped with Python 3.0 would not fix these cases of isinstance() automatically. The fix first appeared in the 2to3 script that shipped with Python 3.1.

skip over this table

1. ...

basestring datatype

FIXME intro

skip over this table

1. ...

itertools module

FIXME intro

1. ...
2. ...
3. ...
4. ...
5. ...

sys.exc_type, sys.exc_value, sys.exc_traceback

FIXME intro

skip over this table

1. ...
2. ...
3. ...

List comprehensions over tuples

FIXME intro

skip over this table

1. ...

os.getcwdu() function

FIXME intro

skip over this table

1. ...

Metaclasses

FIXME intro

skip over this table

class Whip:
    __metaclass__ = PapayaMeta

class Whip(metaclass=PapayaMeta):
    pass

class Whip(Whipper):
    __metaclass__ = PapayaMeta

class Whip(Whipper, metaclass=PapayaMeta):
    pass

1. ...
2. ...

set() literals

FIXME intro

skip over this table

1. ...
2. ...
3. ...

buffer() global function

FIXME intro

skip over this table

1. ...

Whitespace around commas

FIXME intro

skip over this table

1. ...
2. ...

Common idioms

FIXME intro

skip over this table

while 1:
    do_stuff()

while True:
    do_stuff()

a_list = list(a_sequence)
a_list.sort()
do_stuff(a_list)

a_list = sorted(a_sequence)
do_stuff(a_list)

1. ...
2. ...
3. ...
4. ...

FIXME: once the rest of the book is written, this appendix should contain copious links back to any chapter or section that touches on these features.