From 879310bebb955590d10ae59d0b82f0a5fdb1efc0 Mon Sep 17 00:00:00 2001
From: Mark Pilgrim <mark@diveintomark.org>
Date: Tue, 18 Aug 2009 15:00:22 -0400
Subject: [PATCH] finished #debugging section

---
 serializing.html | 37 +++++++++++++++++++++++++++++++------
 1 file changed, 31 insertions(+), 6 deletions(-)

diff --git a/serializing.html b/serializing.html
index 602a0cc..7d48901 100644
--- a/serializing.html
+++ b/serializing.html
@@ -6,6 +6,7 @@
 <link rel=stylesheet href=dip3.css>
 <style>
 body{counter-reset:h1 13}
+mark{display:inline}
 </style>
 <link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
 <link rel=stylesheet media=print href=print.css>
@@ -45,7 +46,7 @@ body{counter-reset:h1 13}
 <p>What can the <code>pickle</code> module store?
 
 <ul>
-<li>All the <a href=native-datatypes.html>native datatypes</a> that Python supports: booleans, integers, floating point numbers, complex numbers, strings, <code>byte</code> objects, byte arrays, and <code>None</code>.
+<li>All the <a href=native-datatypes.html>native datatypes</a> that Python supports: booleans, integers, floating point numbers, complex numbers, strings, <code>bytes</code> objects, byte arrays, and <code>None</code>.
 <li>Lists, tuples, dictionaries, and sets containing any combination of native datatypes.
 <li>Lists, tuples, dictionaries, and sets containing any combination of lists, tuples, dictionaries, and sets containing any combination of native datatypes (and so on, to <a title='sys.getrecursionlimit()' href=http://docs.python.org/3.1/library/sys.html#sys.getrecursionlimit>the maximum nesting level that Python supports</a>).
 <li>Functions, classes, and instances of classes (with caveats that I&#8217;ll explain shortly).
@@ -178,10 +179,17 @@ NameError: name 'entry' is not defined</samp>
 <li>The end result is the same: a perfect replica of the original dictionary.
 </ol>
 
-<h3 id=protocol-versions>Bytes and Strings Rear Their Ugly Heads (Again!)</h3>
+<h3 id=protocol-versions>Bytes and Strings Rear Their Ugly Heads Again</h3>
 
-<p>The pickle protocol has been around for many years, and it has matured as Python itself has matured. 
-<p>FIXME - discussion of pickle protocol versions, backward incompatibility of protocol version 3 due to bytes/strings separation in Python 3, link to http://docs.python.org/3.1/library/pickle.html#data-stream-format
+<p>The pickle protocol has been around for many years, and it has matured as Python itself has matured. There are now <a href=http://docs.python.org/3.1/library/pickle.html#data-stream-format>four different versions</a> of the pickle protocol.
+
+<ul>
+<li>Python 1.x had two pickle protocols, a text-based format (&#8220;version 0&#8221;) and a binary format (&#8220;version 1&#8221;).
+<li>Python 2.3 introduced a new pickle protocol (&#8220;version 2&#8221;) to handle new functionality in Python class objects. It is a binary format.
+<li>Python 3.0 introduced another pickle protocol (&#8220;version 3&#8221;) with explicit support for <code>bytes</code> objects and byte arrays. It is a binary format.
+</ul>
+
+<p>Oh look, <a href=strings.html#byte-arrays>the difference between bytes and strings</a> rears its ugly head again. (If you&#8217;re surprised, you haven&#8217;t been paying attention.) What this means in practice is that, while Python 3 can read data pickled with protocol version 2, Python 2 can not read data pickled with protocol version 3.
 
 <h3 id=debugging>Debugging Pickle Files</h3>
 
@@ -262,9 +270,26 @@ struct_time
   355: \x88     NEWTRUE
   356: u        SETITEMS   (MARK at 5)
   357: .    STOP
-highest protocol among opcodes = 3</samp></pre>
+<mark>highest protocol among opcodes = 3</mark></samp></pre>
 
-<p>FIXME more here about <i>fix_imports</i> and such?
+<p>The most interesting piece of information in that disassembly is on the last line, because it includes the version of the pickle protocol with which this file was saved. There is no explicit version marker in the pickle protocol. To determine which protocol version was used to store a pickle file, you need to look at the markers (&#8220;opcodes&#8221;) within the pickled data and use hard-coded knowledge of which opcodes were introduced with each version of the pickle protocol. The <code>pickle.dis()</code> function does exactly that, and it prints the result in the last line of the disassembly output. Here is a function that returns just the version number, without printing anything:
+
+<p class=d>[<a href=examples/pickleversion.py>download <code>pickleversion.py</code></a>]
+<pre class=pp><code>import pickletools
+
+def protocol_version(file_object):
+    maxproto = -1
+    for opcode, arg, pos in pickletools.genops(file_object):
+        maxproto = max(maxproto, opcode.proto)
+    return maxproto</code></pre>
+
+<p>And here it is in action:</p>
+
+<pre class=screen>
+<samp class=p>>>> </samp><kbd class=pp>import pickleversion</kbd>
+<samp class=p>>>> </samp><kbd class=pp>with open('entry.pickle', 'rb') as f:</kbd>
+<samp class=p>... </samp><kbd class=pp>    pickleversion.protocol_version(f)</kbd>
+<samp class=pp>3</samp></pre>
 
 <p class=a>&#x2042;