You are here: Home ‣ Dive Into Python 3 ‣

Difficulty level: ♦♦♦♦♢

Packaging Python Libraries

❝ FIXME ❞
— FIXME

 

Diving In

So you want to release a Python script, library, framework, or application. Excellent. The world needs more Python code.

Python 3 comes with a packaging framework called Distutils. Distutils is many things: a build tool (for you), an installation tool (for your users), a package metadata format (for search engines), and more. It integrates with the Python Package Index (“PyPI”), a central repository for open source Python libraries.

All of these facets of Distutils center around the setup script, traditionally called setup.py. In fact, you’ve already seen a Distutils setup script: you used one to install httplib2 in the “HTTP Web Services” chapter.

In this chapter, you’ll learn how the setup script for httplib2 works and step through the process of releasing your own Python software. 

# httplib2's setup.py
from distutils.core import setup
VERSION = '0.5.0'
setup(name='httplib2',
      version=VERSION, 
      author='Joe Gregorio',
      author_email='joe@example.com',
      url='http://code.google.com/p/httplib2/',
      download_url='http://httplib2.googlecode.com/files/httplib2-python3-{}.tar.gz'.format(VERSION),
      description='A comprehensive HTTP client library.',
      license='MIT',
      packages=['httplib2'],
      classifiers=[
          'Development Status :: 4 - Beta',
          'Environment :: Web Environment',
          'Intended Audience :: Developers',
          'License :: OSI Approved :: MIT License',
          'Operating System :: OS Independent',
          'Programming Language :: Python',
          'Programming Language :: Python :: 3',
          'Topic :: Internet :: WWW/HTTP',
          'Topic :: Software Development :: Libraries',
      ],
        long_description="""

A comprehensive HTTP client library, ``httplib2`` supports many features left out of other HTTP libraries.

**HTTP and HTTPS**
  HTTPS support is only available if the socket module was compiled with SSL support. 
 

**Keep-Alive**
  Supports HTTP 1.1 Keep-Alive, keeping the socket open and performing multiple requests over the same connection if possible. 


**Authentication**
  The following three types of HTTP Authentication are supported. These can be used over both HTTP and HTTPS.

  * Digest
  * Basic
  * WSSE

**Caching**
  The module can optionally operate with a private cache that understands the Cache-Control: 
  header and uses both the ETag and Last-Modified cache validators. Both file system
  and memcached based caches are supported.


**All Methods**
  The module can handle any HTTP request method, not just GET and POST.


**Redirects**
  Automatically follows 3XX redirects on GETs.


**Compression**
  Handles both 'deflate' and 'gzip' types of compression.


**Lost update support**
  Automatically adds back ETags into PUT requests to resources we have already cached. This implements Section 3.2 of Detecting the Lost Update Problem Using Unreserved Checkout


**Unit Tested**
  A large and growing set of unit tests.
        """
)

☞httplib2 is open source, but there’s no requirement that you release your own Python libraries under any particular license. The process described in this chapter will work for any Python software, regardless of license.

⁂

Things Distutils Can’t Do For You

Releasing your first Python package is a daunting process. (Releasing your second one is easier.) Distutils tries to automate as much of it as possible, but there are some things you simply must do yourself.

Directory Structure

To start packaging your Python software, you need to get your files and directories in order. The httplib2 directory looks like this: 

httplib2/                 ①
|
+--README.txt             ②
|
+--setup.py               ③
|
+--httplib2/              ④
   |
   +--__init__.py
   |
   +--iri2uri.py
  1. Make a root directory to hold everything. Give it the same name as your Python module.
  2. To accomodate Windows users, your “read me” file should include a .txt extension, and it should use Windows-style carriage returns. Just because you use a fancy text editor that runs from the command line and includes its own macro language, that doesn’t mean you need to make life difficult for your users. (Your users use Notepad. Sad but true.) Even if you’re on Linux or Mac OS X, your fancy text editor undoubtedly has an option to save files with Windows-style carriage returns.
  3. Your Distutils setup script should be named setup.py unless you have a good reason not to. You do not have a good reason not to.
  4. If your Python software is a single .py file, you should put it in the root directory along with your “read me” file and your setup script. If it’s a multi-file module (i.e. a directory with a main __init__.py script), like httplib2, you should put the entire directory here. Yes, that means you’ll have an httplib2/ directory within an httplib2/ directory. Trust me, that’s not a problem. In fact, any other arrangement would be a problem.

Depending on the license you chose, you may include the license text within your .py files themselves, or you may have a separate file containing license text, or both. GPL-licensed programs generally include a file called COPYING that includes the entire text of the GPL. If you have a separate license file, it should go in the root directory along with your “read me” file and your setup script.

Writing Your Setup Script

The Distutils setup script is a Python script. In theory, it can do anything Python can do. In practice, it should do as little as possible, in as standard a way as possible. Setup scripts should be boring. The more exotic your installation process is, the more exotic your bug reports will be.

The first line of every Distutils setup script is always the same: 

from distutils.core import setup

This imports the setup() function, which is the main entry point into Distutils. 95% of all Distutils setup scripts consist of a single call to setup() and nothing else. (I totally just made up that statistic, but if your Distutils setup script is doing more than calling the Distutils setup() function, you should have a good reason.)

...FIXME...required setup() parameters, optional but recommended setup() parameters, always use named parameters, etc.

Classifying Your Package

The Python Package Index (“PyPI”) contains thousands of Python libraries. Proper classification metadata will allow people to find yours more easily.

The PyPI classification system is based on SourceForce’s software map. Classifiers are strings, but they are not freeform.

☞All of your classifiers should come from this master list on PyPI.

You should always include at least these four classifiers:

I strongly recommend that you also include the following classifications:

FIXME - look at several examples, including httplib2

Checking Your Setup Script for Errors

Writing Your Setup Configuration File

Creating Source Distributions

Creating Binary Distributions

Building A Windows Installer

Building a Linux RPM Package

Adding Your Package to The Python Package Index

Registering Yourself

Registering Your Package

Uploading New Versions

The Many Possible Futures of Python Packaging

Distutils is not the be-all and end-all of Python packaging, but as of this writing (August 2009), it’s the only packaging framework that works in Python 3. There are a number of other frameworks for Python 2; some focus on installation, others on testing and deployment. Some or all of these may end up being ported to Python 3 in the future.

These frameworks focus on installation:

These focus on testing and deployment:

Further Reading

On Distutils:

On other packaging frameworks:

© 2001–9 Mark Pilgrim