mirror of
https://github.com/kennethreitz/dive-into-python3.git
synced 2026-06-05 15:00:18 +00:00
added a few asides
This commit is contained in:
Mark Pilgrim
+17
-1
@@ -52,6 +52,8 @@ mark{display:inline}
|
||||
|
||||
<p>The most important thing to understand about any type of web service is that network access is incredibly expensive. I don’t mean “dollars and cents” expensive (although bandwidth ain’t free). I mean that it takes an extraordinary long time to open a connection, send a request, and retrieve a response from a remote server. Even on the fastest broadband connection, <i>latency</i> (the time it takes to send a request and start retrieving data in a response) can still be higher than you anticipated. A router misbehaves, a packet is dropped, an intermediate proxy is under attack — there’s <a href=http://isc.sans.org/>never a dull moment</a> on the public internet, and there may be nothing you can do about it.
|
||||
|
||||
<aside><code>Cache-Control</code> means “don't bug me until next week.”</aside>
|
||||
|
||||
<p><abbr>HTTP</abbr> is designed with caching in mind. There is an entire class of devices (called “caching proxies”) whose only job is to sit between you and the rest of the world and minimize network access. Your company or <abbr>ISP</abbr> almost certainly maintains caching proxies, even if you’re unaware of them. They work because caching built into the <abbr>HTTP</abbr> protocol.
|
||||
|
||||
<p>Here’s a concrete example of how caching works. You visit <a href=http://diveintomark.org/><code>diveintomark.org</code></a> in your browser. That page includes a background image, <a href=http://wearehugh.com/m.jpg><code>wearehugh.com/m.jpg</code></a>. When your browser downloads that image, the server includes the following <abbr>HTTP</abbr> headers:
|
||||
@@ -82,6 +84,8 @@ Content-Type: image/jpeg</code></pre>
|
||||
|
||||
<p>Some data never changes, while other data changes all the time. In between, there is a vast field of data that <em>might</em> have changed, but hasn’t. CNN.com’s feed is updated every few minutes, but my weblog’s feed may not change for days or weeks at a time. In the latter case, I don’t want to tell clients to cache my feed for weeks at a time, because then when I do actually post something, people may not read it for weeks (because they’re respecting my cache headers which said “don’t bother checking this feed for weeks”). On the other hand, I don’t want clients downloading my entire feed once an hour if it hasn’t changed!
|
||||
|
||||
<aside><code>Last-Modified</code> means “same shit, different day.”</aside>
|
||||
|
||||
<p><abbr>HTTP</abbr> has a solution to this, too. When you request data for the first time, the server can send back a <code>Last-Modified</code> header. This is exactly what it sounds like: the date that the data was changed. That background image referenced from <code>diveintomark.org</code> included a <code>Last-Modified</code> header.
|
||||
|
||||
<pre class=nd><code>HTTP/1.1 200 OK
|
||||
@@ -130,7 +134,9 @@ Connection: close
|
||||
Content-Type: image/jpeg
|
||||
</code></pre>
|
||||
|
||||
The second time you request the same data, you include the ETag hash in an <code>If-None-Match</code> header of your request. If the data hasn’t changed, the server will send you back a <code>304</code> status code. As with the last-modified date checking, the server sends back <em>only</em> the <code>304</code> status code; it doesn’t send you the same data a second time. By including the ETag hash in your second request, you’re telling the server that there’s no need to re-send the same data if it still matches this hash, since <a href=#caching>you still have the data from the last time</a>.
|
||||
<aside><code>ETag</code> means “there’s nothing new under the sun.”</aside>
|
||||
|
||||
<p>The second time you request the same data, you include the ETag hash in an <code>If-None-Match</code> header of your request. If the data hasn’t changed, the server will send you back a <code>304</code> status code. As with the last-modified date checking, the server sends back <em>only</em> the <code>304</code> status code; it doesn’t send you the same data a second time. By including the ETag hash in your second request, you’re telling the server that there’s no need to re-send the same data if it still matches this hash, since <a href=#caching>you still have the data from the last time</a>.
|
||||
|
||||
<p>Again with the <kbd>curl</kbd>:
|
||||
|
||||
@@ -161,6 +167,8 @@ Cache-Control: max-age=31536000, public</samp></pre>
|
||||
|
||||
<p><a href=http://www.w3.org/Provider/Style/URI>Cool <abbr>URI</abbr>s don’t change</a>, but many <abbr>URI</abbr>s are seriously uncool. Web sites get reorganized, pages move to new addresses. Even web services can reorganize. A syndicated feed at <code>http://example.com/index.xml</code> might be moved to <code>http://example.com/xml/atom.xml</code>. Or an entire domain might move, as an organization expands and reorganizes; <code>http://www.example.com/index.xml</code> becomes <code>http://server-farm-1.example.com/index.xml</code>.
|
||||
|
||||
<aside><code>Location</code> means “look over there!”</aside>
|
||||
|
||||
<p>Every time you request any kind of resource from an <abbr>HTTP</abbr> server, the server includes a status code in its response. Status code <code>200</code> means “everything’s normal, here’s the page you asked for”. Status code <code>404</code> means “page not found”. (You’ve probably seen 404 errors while browsing the web.) Status codes in the 300’s indicate some form of redirection.
|
||||
|
||||
<p><abbr>HTTP</abbr> has several different ways of signifying that a resource has moved. The two most common techiques are status codes <code>302</code> and <code>301</code>. Status code <code>302</code> is a <i>temporary redirect</i>; it means “oops, that got moved over here temporarily” (and then gives the temporary address in a <code>Location</code> header). Status code <code>301</code> is a <i>permanent redirect</i>; it means “oops, that got moved permanently” (and then gives the new address in a <code>Location</code> header). If you get a <code>302</code> status code and a new address, the <abbr>HTTP</abbr> specification says you should use the new address to get what you asked for, but the next time you want to access the same resource, you should retry the old address. But if you get a <code>301</code> status code and a new address, you’re supposed to use the new address from then on.
|
||||
@@ -224,6 +232,8 @@ reply: 'HTTP/1.1 200 OK'
|
||||
<li>The fourth line specifies the name of the library that is making the request. By default, this is <code>Python-urllib</code> plus a version number. Both <code>urllib.request</code> and <code>httplib2</code> support changing the user agent, simply by adding a <code>User-Agent</code> header to the request (which will override the default value).
|
||||
</ol>
|
||||
|
||||
<aside>We’re downloading 3070 bytes when we could have just downloaded 941.</aside>
|
||||
|
||||
<p>Now let’s look at what the server sent back in its response.
|
||||
|
||||
<pre class=screen>
|
||||
@@ -393,6 +403,8 @@ Writing /usr/local/lib/python3.1/dist-packages/httplib2-python3_0.5.0.egg-info</
|
||||
|
||||
<p>If you know what sort of resource you’re expecting (an <abbr>XML</abbr> document in this case), perhaps you could “just” pass the returned <code>bytes</code> object to the <a href=xml.html#xml-parse><code>xml.etree.ElementTree.parse()</code> function</a>. That’ll work as long as the <abbr>XML</abbr> document includes information on its own character encoding (as this one does), but that’s an optional feature and not all <abbr>XML</abbr> documents do that. If an <abbr>XML</abbr> document doesn’t include encoding information, the client is supposed to look at the enclosing transport — <i>i.e.</i> the <code>Content-Type</code> <abbr>HTTP</abbr> header, which can include a <code>charset</code> parameter.
|
||||
|
||||
<p class=ss><a style=border:0 href=http://www.cafepress.com/feedparser><img src=http://feedparser.org/img/feedparser.jpg alt="[I support RFC 3023 t-shirt]" width=150 height=150></a>
|
||||
|
||||
<p>But it’s worse than that. Now character encoding information can be in two places: within the <abbr>XML</abbr> document itself, and within the <code>Content-Type</code> <abbr>HTTP</abbr> header. If the information is in <em>both</em> places, which one wins? According to <a href=http://www.ietf.org/rfc/rfc3023.txt>RFC 3023</a> (I swear I am not making this up), if the media type given in the <code>Content-Type</code> <abbr>HTTP</abbr> header is <code>application/xml</code>, <code>application/xml-dtd</code>, <code>application/xml-external-parsed-entity</code>, or any one of the subtypes of <code>application/xml</code> such as <code>application/atom+xml</code> or <code>application/rss+xml</code> or even <code>application/rdf+xml</code>, then the encoding is
|
||||
|
||||
<ol>
|
||||
@@ -456,6 +468,8 @@ Writing /usr/local/lib/python3.1/dist-packages/httplib2-python3_0.5.0.egg-info</
|
||||
<li>Here’s the rub: this “response” was generated from <code>httplib2</code>’s local cache. That directory name you passed in when you created the <code>httplib2.Http</code> object — that directory holds <code>httplib2</code>’s cache of all the operations it’s ever performed.
|
||||
</ol>
|
||||
|
||||
<aside>What’s on the wire? Absolutely nothing.</aside>
|
||||
|
||||
<blockquote class=note>
|
||||
<p><span class=u>☞</span>If you want to turn on <code>httplib2</code> debugging, you need to set a module-level constant (<code>httplib2.debuglevel</code>), then create a new <code>httplib2.Http</code> object. If you want to turn off debugging, you need to change the same module-level constant, then create a new <code>httplib2.Http</code> object.
|
||||
</blockquote>
|
||||
@@ -576,6 +590,8 @@ user-agent: Python-httplib2/$Rev: 259 $'
|
||||
|
||||
<h3 id=httplib2-compression>How <code>http2lib</code> Handles Compression</h3>
|
||||
|
||||
<aside>“We have both kinds of music, country AND western.”</aside>
|
||||
|
||||
<p><abbr>HTTP</abbr> supports <a href=#compression>two types of compression</a>. <code>httplib2</code> supports both of them.
|
||||
|
||||
<pre class=screen>
|
||||
|
||||
Reference in New Issue
Block a user