# Serializing Python Objects - Dive into Python 3

You are here: Home ‣ Dive Into Python 3 ‣

Difficulty level: ♦♦♦♦♢

# Serializing Python Objects

> ❝ FIXME ❞
> — FIXME

## Diving In

FIXME

    >>> shell = 1

FIXME

    >>> shell = 2

⁂

## Serializing Simple Python Objects

FIXME - introduction to pickle module, concepts, what datatypes can be pickled w/o additional work

### Saving to (and Loading from) a File

The `pickle` module works with data structures. Let’s build one.

    >>> shell
    1
    >>> entry = {}
    >>> entry['title'] = 'Dive into history, 2009 edition'
    >>> entry['article_link'] = 'http://diveintomark.org/archives/2009/03/27/dive-into-history-2009-edition'
    >>> entry['comments_link'] = None
    >>> entry['internal_id'] = b'\xde\xd5\xb4\xf8'
    >>> entry['tags'] = ('diveintopython', 'docbook', 'html')
    >>> entry['published'] = True
    >>> import time
    >>> entry['published_date'] = time.strptime('Fri Mar 27 22:20:42 2009')
    >>> entry['published_date']
    time.struct_time(tm_year=2009, tm_mon=3, tm_mday=27, tm_hour=22, tm_min=20, tm_sec=42, tm_wday=4, tm_yday=86, tm_isdst=-1)

1. FIXME

    >>> shell
    1
    >>> with open('entry.pickle', 'wb') as f:
    ...     pickle.dump(entry, f)
    ...

1. FIXME

    you@localhost:~/diveintopython3/examples$ ls -l entry.pickle
    -rw-r--r-- 1 you  you  324 Aug  3 13:34 entry.pickle
    you@localhost:~/diveintopython3/examples$ cat entry.pickle
    comments_linkqNXtagsqXdiveintopythonqXdocbookqXhtmlq?qX publishedq?
    XlinkXJhttp://diveintomark.org/archives/2009/03/27/dive-into-history-2009-edition
    q   Xpublished_dateq
    ctime
    struct_time
    ?qRqXtitleqXDive into history, 2009 editionqu.

1. FIXME

FIXME now switch to your second Python Shell

    >>> shell
    2
    >>> entry
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'entry' is not defined
    >>> import pickle
    >>> with open('entry.pickle', 'rb') as f:
    ...     entry = pickle.load(f)
    ...
    >>> entry
    FIXME

1. FIXME

FIXME

    >>> shell
    1
    >>> with open('entry.pickle', 'rb') as f:
    ...     entry2 = pickle.load(f)
    ...
    >>> entry2 == entry
    True
    >>> entry2['tags']
    ('diveintopython', 'docbook', 'html')
    >>> entry2['internal_id']
    b'\xde\xd5\xb4\xf8'

1. FIXME

### Saving to (and Loading from) an Object in Memory

FIXME

### Bytes and Strings Rear Their Ugly Heads (Again!)

FIXME - discussion of pickle protocol versions, backward incompatibility of protocol version 3 due to bytes/strings separation in Python 3, link to http://docs.python.org/3.1/library/pickle.html#data-stream-format

⁂

## Serializing Complex Python Objects

FIXME - discussion of pickling class instances, stateful objects, __getstate__ and __setstate__, links to http://docs.python.org/3.1/library/pickle.html#pickle-inst and http://docs.python.org/3.1/library/pickle.html#pickle-state

## Security Concerns with Pickled Objects

FIXME - pickled objects can be modified in memory, in transit, or on disk; no checksums; no built-in guarantee that the pickle you're loading is the pickle you dumped; never unpickle untrusted input; xref to "eval() is evil" discussion in advanced-iterators chapter

## Serializing Python Objects to be Read by Other Languages

The data format used by the `pickle` module is Python-specific. It makes no attempt to be compatible with other programming languages. If cross-language compatibility is one of your requirements, you need to look at other serialization formats.

One format that is designed to be used by multiple programming languages is JSON.

FIXME - pickle format is python-specific; JSON format is designed to be cross-language (in fact, it was originally designed for JavaScript, hence the name); differences with pickle format (table or list); json module implements dumping and loading JSON-formatted data structures; JSON format is string-based (and always encoded as UTF-8 where bytes are required); compact vs. pretty-printing; JSONEncoder; JSONDecoder; iterencode

### Mapping of Python Datatypes to JSON

    [source: help(json)]

    +---------------+-------------------+
    | JSON          | Python            |
    +===============+===================+
    | object        | dict              |
    +---------------+-------------------+
    | array         | list              |
    +---------------+-------------------+
    | string        | unicode           |
    +---------------+-------------------+
    | number (int)  | int, long         |
    +---------------+-------------------+
    | number (real) | float             |
    +---------------+-------------------+
    | true          | True              |
    +---------------+-------------------+
    | false         | False             |
    +---------------+-------------------+
    | null          | None              |
    +---------------+-------------------+

### Serializing Datatypes Unsupported by JSON

    >>> shell
    1
    >>> entry
    FIXME
    >>> import json
    >>> with open('entry.json', 'w', encoding='utf-8') as f:
    ...     json.dump(entry)
    ...
    FIXME

1. FIXME

FIXME

    # customserializer.py
    def to_json(python_object):
        if isinstance(python_object, bytes):
            return {'__class__': 'bytes',
                    '__value__': list(python_object)}
        raise TypeError(repr(python_object) + ' is not JSON serializable')

1. FIXME

FIXME

    >>> shell
    1
    >>> import customserializer
    >>> with open('entry.json', 'w', encoding='utf-8') as f:
    ...     json.dump(entry, default = customserializer.to_json)
    ...
    FIXME

1. FIXME

    # customserializer.py
    def to_json(python_object):
        if isinstance(python_object, time.struct_time):
            return {'__class__': 'time.asctime',
                    '__value__': time.asctime(python_object)}
        if isinstance(python_object, bytes):
            return {'__class__': 'bytes',
                    '__value__': list(python_object)}
        raise TypeError(repr(python_object) + ' is not JSON serializable')

1. FIXME

FIXME

    >>> shell
    1
    >>> with open('entry.json', 'w', encoding='utf-8') as f:
    ...     json.dump(entry, default = customserializer.to_json)
    ...

1. FIXME

FIXME

    you@localhost:~/diveintopython3/examples$ ls -l example.json
    FIXME
    you@localhost:~/diveintopython3/examples$ cat example.json
    FIXME

1. FIXME

FIXME

    >>> shell
    2
    >>> del entry
    >>> entry
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'entry' is not defined
    >>> import json
    >>> with open('entry.json', 'r', encoding='utf-8') as f:
    ...     entry = json.load(f)
    ...
    FIXME

1. FIXME

FIXME

    # customserializer.py
    def from_json(json_object):
        if '__class__' in json_object:
            if json_object['__class__'] == 'time.asctime':
                return time.strptime(json_object['__value__'])
            if json_object['__class__'] == 'bytes':
                return bytes(json_object['__value__'])
        return json_object

    >>> shell
    2
    >>> import customserializer
    >>> with open('entry.json', 'r', encoding='utf-8') as f:
    ...     entry = json.load(f, object_hook = customserializer.from_json)
    ...
    >>> entry
    FIXME

1. FIXME

FIXME

    >>> shell
    1
    >>> import customserializer
    >>> with open('entry.json', 'r', encoding='utf-8') as f:
    ...     entry2 = json.load(f, object_hook = customserializer.from_json)
    ...
    >>> entry2 == entry
    False
    >>> entry['tags']
    ('diveintopython', 'docbook', 'html')
    >>> entry2['tags']
    ['diveintopython', 'docbook', 'html']

1. FIXME

FIXME

## Further Reading

> ☞Many articles about the `pickle` module make references to `cPickle`. In Python 2, there were two implementations of the `pickle` module, one written in pure Python and another written in C (but still callable from Python). In Python 3, these two modules have been consolidated, so you should always just `import pickle`. You may find these articles useful, but you should ignore the now-obsolete information about `cPickle`.

- `pickle` module
- `pickle` and `cPickle` — Python object serialization
- Using `pickle`
- Python persistence management
- `json` — JavaScript Object Notation Serializer
- JSON encoding and ecoding with custom objects in Python

☜ ☞

© 2001–9 Mark Pilgrim