You are here: Home ‣ Dive Into Python 3 ‣

Strings

❝ I’m telling you this ’cause you’re one of my friends.
My alphabet starts where your alphabet ends! ❞
— Dr. Seuss, On Beyond Zebra!

Some Boring Stuff You Need To Understand Before You Can Dive In

Did you know that the people of Bougainville have the smallest alphabet in the world? Their Rotokas alphabet is composed of only 12 letters: A, E, G, I, K, O, P, R, S, T, U, and V. On the other end of the spectrum, languages like Chinese, Japanese, and Korean have thousands of characters. English, of course, has 26 letters — 52 if you count uppercase and lowercase separately — plus a handful of !@#$%& punctuation marks.

When people talk about “text,” they’re thinking of “characters and symbols on the computer screen.” But computers don’t deal in characters and symbols; they deal in bits and bytes. Every piece of text you’ve ever seen on a computer screen is actually stored in a particular character encoding. Very roughly speaking, the character encoding provides a mapping between the stuff you see on your screen and the stuff your computer actually stores in memory and on disk. There are many different character encodings, some optimized for particular languages like Russian or Chinese or English, and others that can be used for multiple languages.

In reality, it’s more complicated than that. Many characters are common to multiple encodings, but each encoding may use a different sequence of bytes to actually store those characters in memory or on disk. So you can think of the character encoding as a kind of decryption key. Whenever someone gives you a sequence of bytes — a file, a web page, whatever — and claims it’s “text,” you need to know what character encoding they used so you can decode the bytes into characters. If they give you the wrong key or no key at all, you’re left with the unenviable task of cracking the code yourself. Chances are you’ll get it wrong, and the result will be gibberish.

Surely you’ve seen web pages like this, with strange question-mark-like characters where apostrophes should be. That usually means the page author didn’t declare their character encoding correctly, your browser was left guessing, and the result was a mix of expected and unexpected characters. In English it’s merely annoying; in other languages, the result can be completely unreadable.

There are character encodings for each major language in the world. Since each language is different, and memory and disk space have historically been expensive, each character encoding is optimized for a particular language. By that, I mean each encoding using the same numbers (0–255) to represent that language’s characters. For instance, you’re probably familiar with the ASCII encoding, which stores English characters as numbers ranging from 0 to 127. (65 is capital “A”, 97 is lowercase “a”, &c.) English has a very simple alphabet, so it can be completely expressed in less than 128 numbers. For those of you who can count in base 2, that’s 7 out of the 8 bits in a byte.

Western European languages like French, Spanish, and German have more letters than English. Or, more precisely, they have letters combined with various diacritical marks, like the ñ character in Spanish. The most common encoding for these languages is CP-1252, also called “windows-1252” because it is widely used on Microsoft Windows. The CP-1252 encoding shares characters with ASCII in the 0–127 range, but then extends into the 128–255 range for characters like n-with-a-tilde-over-it (241), u-with-two-dots-over-it (252), &c. It’s still a single-byte encoding, though; the highest possible number, 255, still fits in one byte.

Then there are languages like Chinese, Japanese, and Korean, which have so many characters that they require multiple-byte character sets. That is, each “character” is represented by a two-byte number from 0–65535. But different multi-byte encodings still share the same problem as different single-byte encodings, namely that they each use the same numbers to mean different things. It’s just that the range of numbers is broader, because there are many more characters to represent.

That was mostly OK in a non-networked world, where “text” was something you typed yourself and occasionally printed. There wasn’t much “plain text”. Source code was ASCII, and everyone else used word processors, which defined their own (non-text) formats that tracked character encoding information along with rich styling, &c. People read these documents with the same word processing program as the original author, so everything worked, more or less.

Now think about the rise of global networks like email and the web. Lots of “plain text” flying around the globe, being authored on one computer, transmitted through a second computer, and received and displayed by a third computer. Computers can only see numbers, but the numbers could mean different things. Oh no! What to do? Well, systems had to be designed to carry encoding information along with every piece of “plain text.” Remember, it’s the decryption key that maps computer-readable numbers to human-readable characters. A missing decryption key means garbled text, gibberish, or worse.

Now think about trying to store multiple pieces of text in the same place, like in the same database table that holds all the email you’ve ever received. You still need to store the character encoding alongside each piece of text so you can display it properly. Think that’s hard? Try searching your email database, which means converting between multiple encodings on the fly. Doesn’t that sound fun?

Now think about the possibility of multilingual documents, where characters from several languages are next to each other in the same document. (Hint: programs that tried to do this typically used escape codes to switch “modes.” Poof, you’re in Russian koi8-r mode, so 241 means this character; poof, now you’re in Mac Greek mode, so 241 means some other character.) And of course you’ll want to search those documents, too.

Now cry a lot, because everything you thought you knew about strings is wrong, and there ain’t no such thing as “plain text.”

Unicode

Enter Unicode.

Unicode is a system designed to represent every character from every language. Unicode represents each letter, character, or ideograph as a 4-byte number, from 0–4294967295. (That's 2³²−1.) Each 4-byte number represents a unique character used in at least one of the world's languages. Not all the numbers are used, but more than 65535 of them are, so 2 bytes wouldn't be sufficient. Characters that are used in multiple languages generally have the same number, unless there is a good etymological reason not to. Regardless, there is exactly 1 number per character, and exactly 1 character per number. Every number always means just one thing; there are no “modes” to keep track of. U+0041 is always 'A', even if your language doesn't have an 'A' in it.

Right away, the obvious question should leap out at you. Four bytes? For every single character‽ That seems awfully wasteful, especially for languages like English and Spanish, which need less than 256 numbers to express every possible character. [FIXME incomplete paragraph]

Of course, there is still the matter of all those legacy encoding systems. [FIXME incomplete paragraph]

[FIXME stuff about UTF-32, UTF-16, and finally UTF-8]

[FIXME FIXME FIXME, damn it!]

>>> s = u'Dive in'            ①
>>> s
u'Dive in'
>>> print s ②
Dive in

>>> s = u'La Pe\xf1a'         ①
>>> print s ②
Traceback (innermost last):
  File "<interactive input>", line 1, in ?
UnicodeError: ASCII encoding error: ordinal not in range(128)
>>> print s.encode('latin-1') ③
La Peña

Diving In

Let's take another look at humansize.py:

[download humansize.py]

SUFFIXES = {1000: ['KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'],         ①
            1024: ['KiB', 'MiB', 'GiB', 'TiB', 'PiB', 'EiB', 'ZiB', 'YiB']}

def approximate_size(size, a_kilobyte_is_1024_bytes=True):
    """Convert a file size to human-readable form.                          ②

    Keyword arguments:
    size -- file size in bytes
    a_kilobyte_is_1024_bytes -- if True (default), use multiples of 1024
                                if False, use multiples of 1000

    Returns: string

    """                                                                     ③
    if size < 0:
        raise ValueError('number must be non-negative')                     ④

    multiple = 1024 if a_kilobyte_is_1024_bytes else 1000
    for suffix in SUFFIXES[multiple]:
        size /= multiple
        if size < multiple:
            return "{0:.1f} {1}".format(size, suffix)                       ⑤

    raise ValueError('number too large')

1. 'KB', 'MB', 'GB'… those are each strings. Python strings can be defined with either single quotes (') or double quotes (").
2. Function docstrings are strings. This docstring spans multiple lines, so it uses three-in-a-row quotes to start and end the string.
3. These three-in-a-row quotes end the docstring.
4. There's another string, being passed to the exception as a human-readable error message.
5. There's a… whoa, what the heck is that?

Formatting Strings

Python 3 supports formatting values into strings. Although this can include very complicated expressions, the most basic usage is to insert a value into a string with single placeholder.

>>> username = "mark"
>>> password = "PapayaWhip"                             ①
>>> "{0}'s password is {1}".format(username, password)  ②
"mark's password is PapayaWhip"

1. No, my password is not really PapayaWhip.
2. There's a lot going on here. First, that's a method call on a string literal. Strings are objects, and objects have methods. Second, the whole expression evaluates to a string. Third, {0} and {1} are replacement fields, which are replaced by the arguments passed to the format() method.

Compound Field Names

The previous example shows the simplest case, where the replacement fields are simply integers. Integer replacement fields are treated as positional indices into the argument list of the format() method. That means that {0} is replaced by the first argument (username in this case), {1} is replaced by the second argument (password), &c. You can have as many positional indices as you have arguments, and you can have as many arguments as you want. But replacement fields are much more powerful than that.

>>> import humansize
>>> si_suffixes = humansize.SUFFIXES[1000]      ①
>>> si_suffixes
['KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB']
>>> "1000{0[0]} = 1{0[1]}".format(si_suffixes)  ②
'1000KB = 1MB'

1. Rather than calling any function in the humansize module, you're just grabbing one of the data structures it defines: the list of "SI" (powers-of-1000) suffixes.
2. This looks complicated, but it's not. {0} would refer to the first argument passed to the format() method, si_suffixes. But si_suffixes is a list. So {0[0]} refers to the first item of the list which is the first argument passed to the format() method: 'KB'. Meanwhile, {0[1]} refers to the second item of the same list: 'MB'. Everything outside the curly braces — including 1000, the equals sign, and the spaces — is untouched. The final result is the string '1000KB = 1MB'.

What this example shows is that format specifers can access items and properties of data structures using (almost) Python syntax. This is called compound field names. The following compound field names "just work":

• Passing a list, and accessing an item of the list by index (as in the previous example)
• Passing a dictionary, and accessing a value of the dictionary by key
• Passing a module, and accessing its variables and functions by name
• Passing a class instance, and accessing its properties and methods by name
• Any combination of the above

Just to blow your mind, here's an example that combines all of the above:

>>> import humansize
>>> import sys
>>> "1MB = 1000{0.modules[humansize].SUFFIXES[1000][0]}".format(sys)
'1MB = 1000KB'

Here's how it works:

• The sys module holds information about the currently running Python instance. Since you just imported it, you can pass the sys module itself as an argument to the format() method. So the replacement field {0} refers to the sys module.
• sys.modules is a dictionary of all the modules that have been imported in this Python instance. The keys are the module names as strings; the values are the module objects themselves. So the replacement field {0.modules} refers to the dictionary of imported modules.
• sys.modules["humansize"] is the humansize module which you just imported. The replacement field {0.modules[humansize]} refers to the humansize module. Note the slight difference in syntax here. In real Python code, the keys of the sys.modules dictionary are strings; to refer to them, you need to put quotes around the module name (e.g. "humansize"). But within a replacement field, you skip the quotes around the dictionary key name (e.g. humansize).
• sys.modules["humansize"].SUFFIXES is the dictionary defined at the top of the humansize module. The replacement field {0.modules[humansize].SUFFIXES} refers to that dictionary.
• sys.modules["humansize"].SUFFIXES[1000] is a list of SI suffixes: ['KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB']. So the replacement field {0.modules[humansize].SUFFIXES[1000]} refers to that list.
• sys.modules["humansize"].SUFFIXES[1000][0] is the first item of the list of SI suffixes: 'KB'. Therefore, the complete replacement field {0.modules[humansize].SUFFIXES[1000][0]} is replaced by the two-character string KB.

Format Specifiers

But wait! There's more! Let's take another look at that strange line of code from humansize.py:

if size < multiple:
    return "{0:.1f} {1}".format(size, suffix)

{1} is replaced with the second argument passed to the format() method, which is suffix. But what is {0:.1f}? It's two things: {0}, which you recognize, and :.1f, which you don't. The second half (including and after the colon) defines the format specifier, which further refines how the replaced variable should be formatted.

☞Format specifiers allow you to munge the replacement text in a variety of useful ways, like the printf() function in C. You can add zero- or space-padding, align strings, control decimal precision, and even convert numbers to hexadecimal.

Within a replacement field, a colon (:) marks the start of the format specifier. The format specifier “.1” means “round to the nearest tenth” (i.e. display only one digit after the decimal point). The format specifier “f” means “fixed-point number” (as opposed to exponential notation or some other decimal representation). Thus, given a size of 698.25 and suffix of 'GB', the formatted string would be '698.3 GB', because 698.25 gets rounded to one decimal place, then the suffix is appended after the number.

>>> "{0:.1f} {1}".format(698.25, 'GB')
'698.3 GB'

For all the gory details on format specifiers, consult the Format Specification Mini-Language in the official Python documentation.

Other Common String Methods

Besides formatting, strings can do a number of other useful tricks.

>>> s = """Finished files are the re-  ①
... sult of years of scientif-
... ic study combined with the
... experience of years."""
>>> s.splitlines()                     ②
['Finished files are the re-',
 'sult of years of scientif-',
 'ic study combined with the',
 'experience of years.']
>>> print(s.lower())                   ③
finished files are the re-
sult of years of scientif-
ic study combined with the
experience of years.
>>> s.lower().count("f")               ④
6

1. You can input multi-line strings in the Python interactive shell. Once you start a multi-line string with triple quotation marks, just hit ENTER and the interactive shell will prompt you to continue the string. Typing the closing triple quotation marks ends the string, and the next ENTER will execute the command (in this case, assigning the string to s).
2. The splitlines() method takes one multi-line string and returns a list of strings, one for each line of the original. Note that the carriage returns at the end of each line are not included.
3. The lower() method converts the entire string to lowercase. (Similarly, the upper() method converts a string to uppercase.)
4. The count() method counts the number of occurrences of a substring. Yes, there really are six “f”s in that sentence!

The string Module

[FIXME is this worth keeping? The module still exists in 3.0; check if it's going away in 3.1 or something.]

Strings vs. Bytes

FIXME

Character Encoding Of Python Source Code

Python 3 assumes that your source code — i.e. each .py file — is encoded in UTF-8.

☞In Python 2, the default encoding for .py files was ASCII. In Python 3, the default encoding is UTF-8.

If you would like to use a different encoding within your Python code, you can put an encoding declaration on the first line of each file. This declaration defines a .py file to be windows-1252:

# -*- coding: windows-1252 -*-

Technically, the character encoding override can also be on the second line, if the first line is a UNIX-like hash-bang command.

#!/usr/bin/python3
# -*- coding: windows-1252 -*-

For more information, consult PEP 263: Defining Python Source Code Encodings.

Further Reading

On Unicode in Python:

• Python Unicode HOWTO
• What's New In Python 3: Text vs. Data Instead Of Unicode vs. 8-bit

On Unicode in general:

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
• On the Goodness of Unicode
• On Character Strings
• Characters vs. Bytes

On character encoding in other formats:

• Character encoding in XML
• Character encoding in HTML

On strings and string formatting:

• string — Common string operations
• Format String Syntax
• Format Specification Mini-Language
• PEP 3101: Advanced String Formatting

© 2001–9 Mark Pilgrim