You are here: Home ‣ Dive Into Python 3 ‣

Difficulty level: ♦♦♦♦♢

Packaging Python Libraries

❝ FIXME ❞
— FIXME

Diving In

So you want to release a Python script, library, framework, or application. Excellent. The world needs more Python code.

Python 3 comes with a packaging framework called Distutils. Distutils is many things: a build tool (for you), an installation tool (for your users), a package metadata format (for search engines), and more. It integrates with the Python Package Index (“PyPI”), a central repository for open source Python libraries.

All of these facets of Distutils center around the setup script, traditionally called setup.py. In fact, you’ve already seen a Distutils setup script: you used one to install httplib2 in the “HTTP Web Services” chapter.

In this chapter, you’ll learn how the setup script for httplib2 works and step through the process of releasing your own Python software.

# httplib2's setup.py
from distutils.core import setup
VERSION = '0.5.0'
setup(name='httplib2',
      version=VERSION, 
      author='Joe Gregorio',
      author_email='joe@example.com',
      url='http://code.google.com/p/httplib2/',
      download_url='http://httplib2.googlecode.com/files/httplib2-python3-{}.tar.gz'.format(VERSION),
      description='A comprehensive HTTP client library.',
      license='MIT',
      packages=['httplib2'],
      classifiers=[
          'Development Status :: 4 - Beta',
          'Environment :: Web Environment',
          'Intended Audience :: Developers',
          'License :: OSI Approved :: MIT License',
          'Operating System :: OS Independent',
          'Programming Language :: Python',
          'Programming Language :: Python :: 3',
          'Topic :: Internet :: WWW/HTTP',
          'Topic :: Software Development :: Libraries :: Python Modules',
      ],
        long_description="""

A comprehensive HTTP client library, ``httplib2`` supports many features left out of other HTTP libraries.

**HTTP and HTTPS**
  HTTPS support is only available if the socket module was compiled with SSL support. 
 

**Keep-Alive**
  Supports HTTP 1.1 Keep-Alive, keeping the socket open and performing multiple requests over the same connection if possible. 


**Authentication**
  The following three types of HTTP Authentication are supported. These can be used over both HTTP and HTTPS.

  * Digest
  * Basic
  * WSSE

**Caching**
  The module can optionally operate with a private cache that understands the Cache-Control: 
  header and uses both the ETag and Last-Modified cache validators. Both file system
  and memcached based caches are supported.


**All Methods**
  The module can handle any HTTP request method, not just GET and POST.


**Redirects**
  Automatically follows 3XX redirects on GETs.


**Compression**
  Handles both 'deflate' and 'gzip' types of compression.


**Lost update support**
  Automatically adds back ETags into PUT requests to resources we have already cached. This implements Section 3.2 of Detecting the Lost Update Problem Using Unreserved Checkout


**Unit Tested**
  A large and growing set of unit tests.
        """
)

☞httplib2 is open source, but there’s no requirement that you release your own Python libraries under any particular license. The process described in this chapter will work for any Python software, regardless of license.

⁂

Things Distutils Can’t Do For You

Releasing your first Python package is a daunting process. (Releasing your second one is easier.) Distutils tries to automate as much of it as possible, but there are some things you simply must do yourself.

Choose a license. This is a complicated topic, fraught with politics and peril. If you wish to release your software as open source, I humbly offer five pieces of advice:
1. Don’t write your own license.
2. Don’t write your own license.
3. Don’t write your own license.
4. It doesn’t need to be GPL, but it needs to be GPL-compatible.
5. Don’t write your own license.
Classify your software using the PyPI classification system. I’ll explain what this means later in this chapter.
Write a “read me” file. Don’t skimp on this. At a minimum, it should give your users an overview of what your software does and how to install it.

Directory Structure

To start packaging your Python software, you need to get your files and directories in order. The httplib2 directory looks like this:

httplib2/                 ①
|
+--README.txt             ②
|
+--setup.py               ③
|
+--httplib2/              ④
   |
   +--__init__.py
   |
   +--iri2uri.py

Make a root directory to hold everything. Give it the same name as your Python module.
To accomodate Windows users, your “read me” file should include a .txt extension, and it should use Windows-style carriage returns. Just because you use a fancy text editor that runs from the command line and includes its own macro language, that doesn’t mean you need to make life difficult for your users. (Your users use Notepad. Sad but true.) Even if you’re on Linux or Mac OS X, your fancy text editor undoubtedly has an option to save files with Windows-style carriage returns.
Your Distutils setup script should be named setup.py unless you have a good reason not to. You do not have a good reason not to.
If your Python software is a single .py file, you should put it in the root directory along with your “read me” file and your setup script. If it’s a multi-file module (i.e. a directory with a main __init__.py script), like httplib2, you should put the entire directory here. Yes, that means you’ll have an httplib2/ directory within an httplib2/ directory. Trust me, that’s not a problem. In fact, any other arrangement would be a problem.

Depending on the license you chose, you may include the license text within your .py files themselves, or you may have a separate file containing license text, or both. GPL-licensed programs generally include a file called COPYING that includes the entire text of the GPL. If you have a separate license file, it should go in the root directory along with your “read me” file and your setup script.

Writing Your Setup Script

The Distutils setup script is a Python script. In theory, it can do anything Python can do. In practice, it should do as little as possible, in as standard a way as possible. Setup scripts should be boring. The more exotic your installation process is, the more exotic your bug reports will be.

The first line of every Distutils setup script is always the same:

from distutils.core import setup

This imports the setup() function, which is the main entry point into Distutils. 95% of all Distutils setup scripts consist of a single call to setup() and nothing else. (I totally just made up that statistic, but if your Distutils setup script is doing more than calling the Distutils setup() function, you should have a good reason.)

...FIXME...required setup() parameters, optional but recommended setup() parameters, always use named parameters, etc.

Classifying Your Package

The Python Package Index (“PyPI”) contains thousands of Python libraries. Proper classification metadata will allow people to find yours more easily.

The PyPI classification system is based on SourceForce’s software map. Classifiers are strings, but they are not freeform.

Development Status. Is your software beta quality? Alpha quality? Pre-alpha?
Environment. FIXME (optional, multiples allowed)
Framework. Does your software rely on a larger Python framework like Django or Zope? If not, you can omit this classifier.
Intended Audience. Who would download your software? End users looking for desktop software? System administrators? Scientists? Other Python developers?
License. FIXME (strongly recommended, probably just one unless your software is available under multiple licenses)
Natural Language. FIXME (optional, multiples allowed)
Operating System. FIXME (strongly recommended, multiples allowed)
Programming Language. FIXME (strongly recommended, multiples allowed and encouraged)
Topic. FIXME (recommended, multiples allowed and encouraged)

☞All of your classifiers should come from this master list on PyPI.

You should always include at least these four classifiers:

"Programming Language :: Python" and "Programming Language :: Python :: 3". If you do not include these, your package will not show up in this list of Python 3-compatible libraries, which linked from the sidebar of every single page of pypi.python.org.
"License :: whatever". This is the absolute first thing I look for when I’m evaluating third-party libraries. Don’t make me hunt for this vital information. Don’t include more than one license classifier unless your software is explicitly available under multiple licenses (and don’t release software under multiple licenses unless you’re forced to do so).
"Operating System :: whatever". If your software only runs on Windows (or Mac OS X, or Linux), I want to know sooner rather than later. If your software works anywhere without any platform-specific code, use the classifier "Operating System :: OS Independent". Multiple classifiers are allowed.

I strongly recommend that you also include the following classifications:

"Development Status :: whatever". Pick one. Be honest.
"Intended Audience :: whatever". Pick one. The most common choices are Developers, End Users/Desktop, Science/Research, and System Administrators.
"Topic :: whatever". There are a large number of topics available. Choose all that apply.

Examples of Good Package Classifiers

By way of example, here are the classifiers for Django, a production-ready, cross-platform, BSD-licensed content management system that runs on your web server. (Django is not yet compatible with Python 3, so the Programming Language :: Python :: Python 3 classifier is not listed.)

Programming Language :: Python
License :: OSI Approved :: BSD License
Operating System :: OS Independent
Development Status :: 5 - Production/Stable
Environment :: Web Environment
Framework :: Django
Intended Audience :: Developers
Topic :: Internet :: WWW/HTTP
Topic :: Internet :: WWW/HTTP :: Dynamic Content
Topic :: Internet :: WWW/HTTP :: WSGI
Topic :: Software Development :: Libraries :: Python Modules

Here are the classifiers for chardet, the character encoding detection library covered in Case Study: Porting chardet to Python 3. chardet is beta quality, cross-platform, Python 3-compatible, LGPL-licensed, and intended for developers to integrate into their own products.

Programming Language :: Python
Programming Language :: Python :: Python 3
License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
Operating System :: OS Independent
Development Status :: 4 - Beta
Environment :: Other Environment
Intended Audience :: Developers
Topic :: Text Processing :: Linguistic
Topic :: Software Development :: Libraries :: Python Modules

And here are the classifiers for httplib2, the HTTP module I mentioned at the beginning of this chapter. httplib2 is beta quality, cross-platform, MIT-licensed, and intended for Python developers.

Programming Language :: Python
Programming Language :: Python :: 3
License :: OSI Approved :: MIT License
Operating System :: OS Independent
Development Status :: 4 - Beta
Environment :: Web Environment
Intended Audience :: Developers
Topic :: Internet :: WWW/HTTP
Topic :: Software Development :: Libraries :: Python Modules

Checking Your Setup Script for Errors

Writing Your Setup Configuration File

Creating Source Distributions

Creating Binary Distributions

Building A Windows Installer

Building a Linux RPM Package

Adding Your Package to The Python Package Index

Registering Yourself

Registering Your Package

Uploading New Versions

The Many Possible Futures of Python Packaging

Distutils is not the be-all and end-all of Python packaging, but as of this writing (August 2009), it’s the only packaging framework that works in Python 3. There are a number of other frameworks for Python 2; some focus on installation, others on testing and deployment. Some or all of these may end up being ported to Python 3 in the future.

These frameworks focus on installation:

These focus on testing and deployment: