Porting code to Python 3 with 2to3

  1. Diving in
  2. print statement
  3. <> comparison
  4. has_key() dictionary method
  5. Dictionary methods that return lists
  6. Modules that have been renamed or reorganized
    1. http package
    2. urllib package
    3. dbm package
    4. xmlrpc package
    5. Other modules
  7. Relative imports within a package
  8. filter() global function
  9. map() global function
  10. reduce() global function (3.1+)
  11. apply() global function
  12. intern() global function
  13. exec statement
  14. execfile statement (3.1+)
  15. repr literals (backticks)
  16. try...except statement
  17. raise statement
  18. throw statement
  19. long data type
  20. xrange() global function
  21. raw_input() and input() global functions
  22. func_* function attributes
  23. xreadlines() I/O method
  24. lambda functions with multiple parameters
  25. Special method attributes
  26. next() iterator method
  27. __nonzero__ special class attribute
  28. Number literals
  29. sys.maxint
  30. unicode() global function
  31. Unicode string literals
  32. callable() global function
  33. zip() global function
  34. StandardError() exception
  35. types module constants
  36. isinstance global function (3.1+)
  37. basestring datatype
  38. itertools module
  39. sys.exc_type, sys.exc_value, sys.exc_traceback
  40. List comprehensions over tuples
  41. os.getcwdu() function
  42. Metaclasses
  43. set() literals
  44. buffer() global function
  45. Whitespace around commas
  46. Common idioms

Diving in

FIXME intro

...

print statement

In Python 2, print was a statement -- whatever you wanted to print simply followed the print keyword. In Python 3, print() is a function -- whatever you want to print is passed to print() like any other function.

Notes Python 2 Python 3
① print print()
② print 1 print(1)
③ print 1, 2 print(1, 2)
④ print 1, 2, print(1, 2, end=' ')
⑤ print >>sys.stderr, 1, 2, 3 print(1, 2, 3, file=sys.stderr)
  1. To print a blank line, call print() without any arguments.
  2. To print a single value, call print() with one argument
  3. To print two values separated by a space, call print() with two arguments.
  4. This one is a little tricky. In Python 2, if you ended a print statement with a comma, it would print the values separated by spaces, then print a trailing space, then stop without printing a carriage return. In Python 3, the way to do this is to pass end=' ' as a keyword argument to the print() function. The end argument defaults to '\n' (a carriage return), so overriding it will suppress the carriage return after printing the other arguments.
  5. In Python 2, you could redirect the output to a pipe -- like sys.stderr -- by using the >>pipe_name syntax. In Python 3, the way to do this is to pass the pipe in the file keyword argument. The file argument defaults to sys.stdout (standard out), so overriding it will output to a different pipe instead.

<> comparison

Python 2 supported <> as a synonym for !=, the not-equals comparison operator. Python 3 supports the != operator, but not <>.

Notes Python 2 Python 3
① if x <> y: if x != y:
② if x <> y <> z: if x != y != z:
  1. A simple comparison.
  2. A more complex comparison between three values.

has_key() dictionary method

In Python 2, dictionaries had a has_key() method to test whether the dictionary had a certain key. In Python 3, this method no longer exists. Instead, you need to use the in operator.

Notes Python 2 Python 3
① a_dictionary.has_key("PapayaWhip") "PapayaWhip" in a_dictionary
② a_dictionary.has_key(x) or a_dictionary.has_key(y) x in a_dictionary or y in a_dictionary
③ a_dictionary.has_key(x or y) (x or y) in a_dictionary
④ a_dictionary.has_key(x + y) (x + y) in a_dictionary
⑤ x + a_dictionary.has_key(y) x + (y in a_dictionary)
  1. The simplest form.
  2. The or operator takes precedence over the in operator, so there is no need for parentheses here.
  3. On the other hand, you do need parentheses here, for the same reason -- or takes precedence over in.
  4. The in operator takes precedence over the + operator, so this form needs parentheses too.
  5. Again with the parentheses, for the same reason.

Dictionary methods that return lists

In Python 2, many dictionary methods returned lists. The most frequently used methods were keys(), items(), and values(). In Python 3, all of these methods return dynamic views. In some contexts, this is not a problem. If the method's return value is immediately passed to another function that iterates through the entire sequence, it makes no difference whether the actual type is a list or a view. In other contexts, it matters a great deal. If you were expecting a complete list with individually addressable elements, your code will choke, because views do not support indexing.

Notes Python 2 Python 3
① a_dictionary.keys() list(a_dictionary.keys())
② a_dictionary.items() list(a_dictionary.items())
③ a_dictionary.iterkeys() iter(a_dictionary.keys())
④ [i for i in a_dictionary.iterkeys()] [i for i in a_dictionary.keys()]
⑤ min(a_dictionary.keys()) no change
  1. 2to3 errs on the side of safety, converting the return value from keys() to a static list with the list() function. This will always work, but it will be less efficient than using a view. You should examine the converted code to see if a list is absolutely necessary, or if a view would do.
  2. Another view-to-list conversion, with the items() method. 2to3 will do the same thing with the values() method.
  3. Python 3 does not support the iterkeys() method anymore. Use keys(), and if necessary, convert the view to an iterator with the iter() function.
  4. 2to3 recognizes when the iterkeys() method is used inside a list comprehension, and converts it to the keys() method (without wrapping it in an extra call to iter()). This works because views are iterable.
  5. 2to3 recognizes that the keys() method is immediately passed to a function which iterates through an entire sequence, so there is no need to convert the return value to a list first. The min() function will happily iterate through the view instead. This applies to min(), max(), sum(), list(), tuple(), set(), sorted(), any(), and all().

Modules that have been renamed or reorganized

Several modules in the Python Standard Library have been renamed. Several other modules which are related to each other have been combined or reorganized to make their association more logical.

FIXME: once the rest of the book is written, these should link back to the chapters and sections that explain these modules.

http package

In Python 3, several related HTTP modules have been combined into a single package, http.

Notes Python 2 Python 3
① import httplib import http.client
② import Cookie import http.cookies
③ import cookielib import http.cookiejar
④ 
import BaseHTTPServer
import SimpleHTTPServer
import CGIHttpServer
 import http.server
  1. The http.client module implements a low-level library that can request HTTP resources and interpret HTTP responses.
  2. The http.cookies module provides a Pythonic interface to "cookies" that are sent in a Set-Cookie: HTTP header.
  3. The http.cookiejar module manipulates the actual files on disk that popular web browsers use to store cookies.
  4. The http.server module provides a basic HTTP server.

urllib package

Python 2 had a rat's nest of overlapping modules to parse, encode, and fetch URLs. In Python 3, these have all been refactored and combined in a single package, urllib.

Notes Python 2 Python 3
① import urllib import urllib.request, urllib.parse, urllib.error
② import urllib2 import urllib.request, urllib.error
③ import urlparse import urllib.parse
④ import robotparser import urllib.robotparser
⑤ 
from urllib import FancyURLopener
from urllib import urlencode
 
from urllib.request import FancyURLopener
from urllib.parse import urlencode
⑥ 
from urllib2 import Request
from urllib2 import HTTPError
 
from urllib.request import Request
from urllib.error import HTTPError
  1. The old urllib module in Python 2 had a variety of functions, including urlopen() for fetching data and splittype(), splithost(), and splituser() for splitting a URL into its constituent parts. These functions have been reorganized more logically within the new urllib package. 2to3 will also change all calls to these functions so they use the new naming scheme.
  2. The old urllib2 module in Python 2 has been folded into into the urllib package in Python 3. All your urllib2 favorites -- the build_opener() method, Request objects, and HTTPBasicAuthHandler and friends -- are still available.
  3. The urllib.parse module in Python 3 contains all the parsing functions from the old urlparse module in Python 2.
  4. The urllib.robotparser module parses robots.txt files.
  5. The FancyURLopener class, which handles HTTP redirects and other status codes, is still available in the new urllib.request module. The urlencode function has moved to urllib.parse.
  6. The Request object is still available in urllib.request, but constants like HTTPError have been moved to urllib.error.

dbm package

All the various DBM clones are now in a single package, dbm. If you need a specific variant like GNU DBM, you can import the appropriate module within the dbm package.

Notes Python 2 Python 3
import dbm import dbm.ndbm
import gdbm import dbm.gnu
import dbhash import dbm.bsd
import dumbdbm import dbm.dumb 
import anydbm
import whichdb
 import dbm

xmlrpc package

XML-RPC is a lightweight method of performing remote RPC calls over HTTP. The XML-RPC client library and several XML-RPC server implementations are now combined in a single package, xmlrpc.

Notes Python 2 Python 3
import xmlrpclib import xmlrpc.client 
import DocXMLRPCServer
import SimpleXMLRPCServer
 import xmlrpc.server

Other modules

Notes Python 2 Python 3
① 
try:
    import cStringIO as StringIO
except ImportError:
    import StringIO
 import io
② 
try:
    import cPickle as pickle
except ImportError:
    import pickle
 import pickle
③ import __builtin__ import builtins
④ import copy_reg import copyreg
⑤ import Queue import queue
⑥ import SocketServer import socketserver
⑦ import ConfigParser import configparser
⑧ import repr import reprlib
⑨ import commands import subprocess
  1. A common idiom in Python 2 was to try to import cStringIO as StringIO, and if that failed, to import StringIO instead. Do not do this in Python 3; the io module does it for you. It will find the fastest implementation available and use it automatically.
  2. A similar idiom was used to import the fastest pickle implementation. Do not do this in Python 3; the pickle module does it for you.
  3. The builtins module contains the "global" functions, classes, and constants used throughout the Python language. Redefining a function in the builtins module will redefine the "global" function everywhere. That is exactly as powerful and scary as it sounds.
  4. The copyreg module adds pickle support for custom types defined in C.
  5. The queue module implements a multi-producer, multi-consumer queue.
  6. The socketserver module provides generic base classes for implementing different kinds of socket servers.
  7. The configparser module parses INI-style configuration files.
  8. The reprlib module reimplements the built-in repr() function, but with limits on how many values are represented.
  9. The subprocess module allows you to spawn processes, connect to their pipes, and obtain their return codes.

Relative imports within a package

FIXME intro

Notes Python 2 Python 3
① FIXME FIXME
  1. ...

filter() global function

FIXME intro

Notes Python 2 Python 3
① filter(a_function, a_sequence) list(filter(a_function, a_sequence))
② list(filter(a_function, a_sequence)) no change
③ filter(None, a_sequence) [i for i in a_sequence if i]
④ for i in filter(None, a_sequence) no change
⑤ [i for i in filter(a_function, a_sequence)] no change
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...

map() global function

FIXME intro

Notes Python 2 Python 3
① map(a_function, 'PapayaWhip') list(map(a_function, 'PapayaWhip'))
② map(None, 'PapayaWhip') list('PapayaWhip')
③ map(lambda x: x+1, range(42)) [x+1 for x in range(42)]
④ for i in map(a_function, a_sequence): unchanged
⑤ [i for i in map(a_function, a_sequence)] unchanged
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...

reduce() global function (3.1+)

FIXME intro

Notes Python 2 Python 3
① reduce(a, b, c) 
from functtools import reduce
reduce(a, b, c)
  1. ...

apply() global function

FIXME intro

Notes Python 2 Python 3
① apply(a_function, args) a_function(*args)
② apply(a_function, args, kwds) a_function(*args, **kwds)
③ apply(a_function, args + z) a_function(*args + z)
④ apply(aModule.a_function, args) aModule.a_function(*args)
  1. ...
  2. ...
  3. ...
  4. ...

intern() global function

FIXME intro

Notes Python 2 Python 3
① intern(aString) sys.intern(aString)
  1. ...

exec statement

FIXME intro

Notes Python 2 Python 3
① exec codeString exec(codeString)
② exec codeString in a_global_namespace exec(codeString, a_global_namespace)
③ exec codeString in a_global_namespace, a_local_namespace exec(codeString, a_global_namespace, a_local_namespace)
  1. ...
  2. ...
  3. ...

execfile statement (3.1+)

FIXME intro

Notes Python 2 Python 3
① execfile("a_filename") execfile(compile(open("a_filename").read(), "a_filename", "exec"))
  1. ...

repr literals (backticks)

FIXME intro

Notes Python 2 Python 3
① `x` repr(x)
② `1 + 2` repr(1 + 2)
③ `"PapayaWhip" + `2`` repr("PapayaWhip" + repr(2))
  1. ...
  2. ...
  3. ...

try...except statement

FIXME intro

Notes Python 2 Python 3
① 
try:
    import mymodule
except ImportError, e
    pass
 
try:
    import mymodule
except ImportError as e:
    pass
② 
try:
    import mymodule
except (RuntimeError, ImportError), e
    pass
 
try:
    import mymodule
except (RuntimeError, ImportError) as e:
    pass
③ 
try:
    import mymodule
except ImportError:
    pass
 unchanged
④ 
try:
    import mymodule
except:
    pass
 unchanged
  1. ...
  2. ...
  3. ...
  4. ...

raise statement

FIXME intro

Notes Python 2 Python 3
① raise MyException, "error message" raise MyException("error message")
② raise MyException, "error message", a_traceback raise MyException("error message").with_traceback(a_traceback)
③ raise "error message" unsupported
  1. ...
  2. ...
  3. ...

throw statement

FIXME intro

Notes Python 2 Python 3
① aGenerator.throw(MyException) unchanged
② aGenerator.throw(MyException, "error message") aGenerator.throw(MyException("error message"))
③ aGenerator.throw("error message") unsupported
  1. ...
  2. ...
  3. ...

long data type

FIXME intro

Notes Python 2 Python 3
① x = 1000000000000L x = 1000000000000
② x = 0xFFFFFFFFFFFFL x = 0xFFFFFFFFFFFF
③ long(x) int(x)
④ type(x) is long type(x) is int
⑤ isinstance(x, long) isinstance(x, int)
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...

xrange() global function

FIXME intro

Notes Python 2 Python 3
① xrange(10) range(10)
② a_sequence = range(10) a_sequence = list(range(10))
③ [i for i in xrange(10)] [i for i in range(10)]
④ for i in range(10): unchanged
⑤ sum(range(10)) unchanged
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...

raw_input() and input() global functions

FIXME intro

Notes Python 2 Python 3
① raw_input() input()
② raw_input("prompt") input("prompt")
③ input() eval(input())
④ input("prompt") eval(input("prompt"))
  1. ...
  2. ...
  3. ...
  4. ...

func_* function attributes

FIXME intro

Notes Python 2 Python 3
① a_function.func_closure a_function.__closure__
② a_function.func_doc a_function.__doc__
③ a_function.func_name a_function.__name__
④ a_function.func_defaults a_function.__defaults__
⑤ a_function.func_code a_function.__code__
⑥ a_function.func_globals a_function.__globals__
⑦ a_function.func_dict a_function.__dict__
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...
  6. ...
  7. ...

xreadlines() I/O method

FIXME intro

Notes Python 2 Python 3
① for line in a_file.xreadlines(): for line in a_file:
② for line in a_file.xreadlines(5): unchanged
  1. ...
  2. ...

lambda functions with multiple parameters

FIXME intro

Notes Python 2 Python 3
① lambda (x,): x + f(x) lambda x1: x1[0] + f(x1[1])
② lambda (x, y): x + f(y) lambda x_y: x_y[0] + f(x_y[1])
③ lambda (x, (y, z)): x + y + z lambda x_y_z: x_y_z[0] + x_y_z[1][0] + x_y_z[1][1]
  1. ...
  2. ...
  3. ...

Special method attributes

FIXME intro

Notes Python 2 Python 3
① aClassInstance.aClassMethod.im_func aClassInstance.aClassMethod.__func__
② aClassInstance.aClassMethod.im_self aClassInstance.aClassMethod.__self__
③ aClassInstance.aClassMethod.im_class aClassInstance.aClassMethod.self.__class__
  1. ...
  2. ...
  3. ...

next() iterator method

FIXME intro

Notes Python 2 Python 3
① anIterator.next() next(anIterator)
② a_function_that_returns_an_iterator().next() next(a_function_that_returns_an_iterator())
③ 
class A:
    def next(self):
        pass
 
class A:
    def __next__(self):
        pass
④ 
class A:
    def next(self, x, y):
        pass
 unchanged
⑤ 
next = 42
for an_iterator in a_sequence_of_iterators:
    an_iterator.next()
 
next = 42
for an_iterator in a_sequence_of_iterators:
    an_iterator.__next__()
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...

__nonzero__ special class attribute

FIXME intro

Notes Python 2 Python 3
① 
class A:
    def __nonzero__(self):
        pass
 
class A:
    def __bool__(self):
        pass
② 
class A:
    def __nonzero__(self, x, y):
        pass
 unchanged
  1. ...
  2. ...

Number literals

FIXME intro

Notes Python 2 Python 3
① x = 12L x = 12
② x = 0755 x = 0o755
  1. ...
  2. ...

sys.maxint

FIXME intro

Notes Python 2 Python 3
① from sys import maxint from sys import maxsize
② 
import sys
a_function(sys.maxint)
 
import sys
a_function(sys.maxsize)
  1. ...
  2. ...

unicode() global function

FIXME intro

Notes Python 2 Python 3
① unicode(anything) str(anything)
  1. ...

Unicode string literals

FIXME intro

Notes Python 2 Python 3
① u"PapayaWhip" "PapayaWhip"
① ur"PapayaWhip\foo" r"PapayaWhip\foo"
  1. ...
  2. ...

callable() global function

FIXME intro

Notes Python 2 Python 3
① callable(anything) hasattr(anything, "__call__")
  1. ...

zip() global function

FIXME intro

Notes Python 2 Python 3
① zip(a, b, c) list(zip(a, b, c))
② d.join(zip(a, b, c)) unchanged
  1. ...
  2. ...

StandardError() exception

FIXME intro

Notes Python 2 Python 3
① x = StandardError() x = Exception()
② x = StandardError(a, b, c) x = Exception(a, b, c)
  1. ...
  2. ...

types module constants

FIXME intro

Notes Python 2 Python 3
① types.StringType bytes
② types.DictType dict
③ types.IntType int
④ types.LongType int
⑤ types.ListType list
⑥ types.NoneType type(None)
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...
  6. ...

isinstance global function (3.1+)

FIXME intro

Notes Python 2 Python 3
① isinstance(x, (int, float, int)) isinstance(x, (int, float))
  1. ...

basestring datatype

FIXME intro

Notes Python 2 Python 3
① isinstance(x, basestring) isinstance(x, str)
  1. ...

itertools module

FIXME intro

Notes Python 2 Python 3
① itertools.izip(a, b) zip(a, b)
② itertools.imap(a, b) map(a, b)
③ itertools.ifilter(a, b) filter(a, b)
④ itertools.ifilterfalse(a, b) filterfalse(a, b)
⑤ from itertools import imap, izip, foo from itertools import foo
  1. ...
  2. ...
  3. ...
  4. ...
  5. ...

sys.exc_type, sys.exc_value, sys.exc_traceback

FIXME intro

Notes Python 2 Python 3
① sys.exc_type sys.exc_info()[0]
② sys.exc_value sys.exc_info()[1]
③ sys.exc_traceback sys.exc_info()[2]
  1. ...
  2. ...
  3. ...

List comprehensions over tuples

FIXME intro

Notes Python 2 Python 3
① [i for i in 1, 2] [i for i in (1, 2)]
  1. ...

os.getcwdu() function

FIXME intro

Notes Python 2 Python 3
① os.getcwdu() os.getcwd()
  1. ...

Metaclasses

FIXME intro

Notes Python 2 Python 3
① 
class Whip:
    __metaclass__ = PapayaMeta
 
class Whip(metaclass=PapayaMeta):
    pass
② 
class Whip(Whipper):
    __metaclass__ = PapayaMeta
 
class Whip(Whipper, metaclass=PapayaMeta):
    pass
  1. ...
  2. ...

set() literals

FIXME intro

Notes Python 2 Python 3
① set([1, 2, 3]) {1, 2, 3}
② set((1, 2, 3)) {1, 2, 3}
③ set([i for i in a_sequence]) {i for i in a_sequence}
  1. ...
  2. ...
  3. ...

buffer() global function

FIXME intro

Notes Python 2 Python 3
① x = buffer(y) x = memoryview(y)
  1. ...

Whitespace around commas

FIXME intro

Notes Python 2 Python 3
① a ,b a, b
② {a :b} {a: b}
  1. ...
  2. ...

Common idioms

FIXME intro

Notes Python 2 Python 3
① 
while 1:
    do_stuff()
 
while True:
    do_stuff()
② type(x) == T isinstance(x, T)
③ type(x) is T isinstance(x, T)
④ 
a_list = list(a_sequence)
a_list.sort()
do_stuff(a_list)
 
a_list = sorted(a_sequence)
do_stuff(a_list)
  1. ...
  2. ...
  3. ...
  4. ...