kennethreitz/requests3
1
0
Fork 0
mirror of https://github.com/kennethreitz/requests3.git synced 2026-06-05 23:10:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/develop' into develop

Browse Source
This commit is contained in:
Mike Waldner
2011-08-23 22:22:51 -04:00
parent c56acf0e32 75027f9557
commit 68a2954c67
25 changed files with 1077 additions and 325 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AUTHORS
+10 -1
View File
@@ -31,4 +31,13 @@ Patches and Suggestions
- Tamás Gulácsi
- Rubén Abad
- Peter Manser
- Jeremy Selie
- Jeremy Selier
- Jens Diemer
- Alex <@alopatin>
- Tom Hogans <tomhsx@gmail.com>
- Armin Ronacher
- Shrikant Sharat Kandula
- Mikko Ohtamaa
- Den Shabalin
- Alejandro Giacometti
- Rick Mak
HISTORY.rst
+23 -3
View File
@@ -1,13 +1,33 @@
History
-------
0.5.2 (2011-09-??)
* Automatic decoding of unicode, when available.
* New ``decode_unicode`` setting
* Removal of ``r.read/close`` methods
* New ``r.fo`` interface for advanced response usage.
0.6.1 (2011-08-20)
++++++++++++++++++
* status code reference dict?
* Enhanced status codes experience ``\o/``
* Set a maximum number of redirects (``settings.max_redirects``)
* Full Unicode URL support
* Support for protocol-less redirects.
* Allow for arbitrary request types.
* Bugfixes
0.6.0 (2011-08-17)
++++++++++++++++++
* New callback hook system
* New persistient sessions object and context manager
* Transparent Dict-cookie handling
* Status code reference object
* Removed Response.cached
* Added Response.request
* all args are kwargs
* All args are kwargs
* Relative redirect support
* HTTPError handling improvements
* Improved https testing
MANIFEST.in
+1 -1
View File
@@ -1 +1 @@
include README.rst LICENSE HISTORY.rst
include README.rst LICENSE HISTORY.rst test_requests.py
README.rst
+27 -83
View File
@@ -1,11 +1,35 @@
Requests: HTTP for Humans
=========================
Most existing Python modules for dealing HTTP requests are insane. I have to look up *everything* that I want to do. Most of my worst Python experiences are a result of the various built-in HTTP libraries (yes, even worse than Logging).
Requests is an ISC Licensed HTTP library, written in Python, for human
beings.
But this one's different. This one's going to be awesome. And simple.
Most existing Python modules for sending HTTP requests are extremely
verbose and cumbersome. Python's builtin urllib2 module provides most of
the HTTP capabilities you should need, but the api is thoroughly broken.
It requires an enormous amount of work (even method overrides) to
perform the simplest of tasks.
Things shouldn't be this way. Not in Python.
::
>>> r = requests.get('https://api.github.com', auth=('user', 'pass'))
>>> r.status_code
204
>>> r.headers['content-type']
'application/json'
>>> r.content
...
See `the same code, without Requests <https://gist.github.com/973705>`_.
Requests allow you to send **HEAD**, **GET**, **POST**, **PUT**,
**PATCH**, and **DELETE** HTTP requests. You can add headers, form data,
multipart files, and parameters with simple Python dictionaries, and access the
response data in the same way. It's powered by urllib2, but it does
all the hard work and crazy hacks for you.
Really simple.
Features
--------
@@ -55,80 +79,6 @@ Uh oh, we're not authorized! Let's add authentication. ::
'{"authenticated": true, "user": "user"}'
API
---
**Requests:**
All request functions return a Response object (see below).
If a {filename: fileobject} dictionary is passed in (files=...), a multipart_encode upload will be performed.
If CookieJar object is is passed in (cookies=...), the cookies will be sent with the request.
HEAD Requests
>>> requests.head(url, params={}, headers={}, cookies=None, auth=None, timeout=None, proxies={})
<Response [200]>
GET Requests
>>> requests.get(url, params={}, headers={}, cookies=None, auth=None, timeout=None, proxies={})
<Response [200]>
POST Requests
>>> requests.post(url, data={}, headers={}, files={}, cookies=None, auth=None, timeout=None, allow_redirects=False, params{}, proxies={})
<Response [200]>
PUT Requests
>>> requests.put(url, data={}, headers={}, files={}, cookies=None, auth=None, timeout=None, allow_redirects=False, params{}, proxies={})
<Response [200]>
PATCH Requests
>>> requests.patch(url, data={}, headers={}, files={}, cookies=None, auth=None, timeout=None, allow_redirects=False, params{}, proxies={})
<Response [200]>
DELETE Requests
>>> requests.delete(url, params={}, headers={}, cookies=None, auth=None, timeout=None, allow_redirects=False, params{}, proxies={})
<Response [200]>
**Responses:**
Response.status_code
(Integer) Received HTTP Status Code Response
Response.headers
((CaseInsensitive) Dictionary) Received HTTP Response Headers.
Response.content
(Bytes) Received Content.
Response.history
(List of Responses) Redirection History.
Response.url
(String) URL of response. Useful for detecting redirects.
Response.ok
(Bool) True if no errors occurred during the request, and the status_code is kosher.
Response.cached
(Bool) True if Response.content is stored within the object.
Response.error
(HTTPError) If an HTTPError occurred (e.g. status of 404), Otherwise this is None.
Response.raise_for_status()
Raises HTTPError if a request is not kosher.
**HTTP Authentication Registry:**
You can register AuthObjects to automatically enable HTTP Authentication on requests that contain a registered base URL string.
>>> requests.auth_manager.add_auth(url, authobject)
Installation
------------
@@ -150,11 +100,5 @@ Contribute
If you'd like to contribute, simply fork `the repository`_, commit your changes to the **develop** branch (or branch off of it), and send a pull request. Make sure you add yourself to AUTHORS_.
Roadmap
-------
- Sphinx Documentation
.. _`the repository`: http://github.com/kennethreitz/requests
.. _AUTHORS: http://github.com/kennethreitz/requests/blob/master/AUTHORS
docs/api.rst
+39 -18
View File
@@ -13,9 +13,10 @@ important right here and provide links to the canonical documentation.
Main Interface
--------------
All of Request's functionality can be accessed by these 6 methods. They
all return a :class:`Response <models.Response>` object.
All of Request's functionality can be accessed by these 7 methods.
They all return an instance of the :class:`Response <Response>` object.
.. autofunction:: request
.. autofunction:: head
.. autofunction:: get
.. autofunction:: post
@@ -27,20 +28,33 @@ all return a :class:`Response <models.Response>` object.
-----------
.. autoclass:: requests.models.Response
.. autoclass:: Response
:inherited-members:
Exceptions
----------
.. autoexception:: HTTPError
Utilities
---------
.. autoexception:: RequestException
These functions are used internally, but may be useful outside of
Requests.
.. autoexception:: requests.models.AuthenticationError
.. autoexception:: requests.models.URLRequired
.. autoexception:: requests.models.InvalidMethod
.. module:: requests.utils
Cookies
~~~~~~~
.. autofunction:: dict_from_cookiejar
.. autofunction:: cookiejar_from_dict
.. autofunction:: add_dict_to_cookiejar
Encodings
~~~~~~~~~
.. autofunction:: get_encodings_from_content
.. autofunction:: get_encoding_from_headers
.. autofunction:: get_unicode_from_response
.. autofunction:: decode_gzip
Internals
@@ -50,19 +64,26 @@ These items are an internal component to Requests, and should never be
seen by the end user (developer). This part of the API documentation
exists for those who are extending the functionality of Requests.
Functions
~~~~~~~~~
Exceptions
~~~~~~~~~~
.. module:: requests
.. autoexception:: HTTPError
.. autoexception:: RequestException
.. autoexception:: AuthenticationError
.. autoexception:: URLRequired
.. autoexception:: InvalidMethod
.. autoexception:: TooManyRedirects
.. autofunction:: request
Classes
~~~~~~~
.. autoclass:: requests.models.Request
.. autoclass:: requests.Request
:inherited-members:
Structures
~~~~~~~~~~
.. autoclass:: requests.structures.CaseInsensitiveDict
:inherited-members:
docs/community/faq.rst
+67
View File
@@ -0,0 +1,67 @@
.. _faq:
Frequently Asked Questions
==========================
This part of the documentation answers common questions about Requests.
Encoded Data?
-------------
Requests automatically decompresses gzip-encoded responses, and does
it's best to decodes response content to unicode when possible.
You can get direct access to the raw response (and even the socket),
if needed as well.
Custom User-Agents?
-------------------
Requests allows you to easily override User-Agent strings, along with
any other HTTP Header.
Why not Httplib2?
-----------------
Chris Adams gave an excellent summary on
`Hacker News <http://news.ycombinator.com/item?id=2884406>`_:
httplib2 is part of why you should use requests: it's far more respectable
as a client but not as well documented and it still takes way too much code
for basic operations. I appreciate what httplib2 is trying to do, that
there's a ton of hard low-level annoyances in building a modern HTTP
client, but really, just use requests instead. Kenneth Reitz is very
motivated and he gets the degree to which simple things should be simple
whereas httplib2 feels more like an academic exercise than something
people should use to build production systems[1].
Disclosure: I'm listed in the requests AUTHORS file but can claim credit
for, oh, about 0.0001% of the awesomeness.
1. http://code.google.com/p/httplib2/issues/detail?id=96 is a good example:
an annoying bug which affect many people, there was a fix available for
months, which worked great when I applied it in a fork and pounded a couple
TB of data through it, but it took over a year to make it into trunk and
even longer to make it onto PyPI where any other project which required "
httplib2" would get the working version.
Python 3 Support?
-----------------
It's on the way. Here's a list of `supported interpreters <interpreters>`_.
Keep-alive Support?
-------------------
It's on the way.
Proxy Support?
--------------
You bet!
docs/community/support.rst
+37
View File
@@ -0,0 +1,37 @@
.. _support:
Support
=======
If you have a questions or issues about Requests, there are serveral options:
Send a Tweet
------------
If your question is less than 140 characters, feel free to send a tweet to
`@kennethreitz <http://twitter.com/kennethreitz>`_.
File an Issue
-------------
If you notice some unexpected behavior in Requests, or want to see support
for a new feature,
`file an issue on GitHub <https://github.com/kennethreitz/requests/issues>`_.
E-mail
------
I'm more than happy to answer any personal or in-depth questions about
Requests. Feel free to email
`requests@kennethreitz.com <mailto:requests@kennethreitz.com>`_.
IRC
---
The official Freenode channel for Requests is
`#python-requests <irc://irc.freenode.net/python-requests>`_
I'm also available as **kennethreitz** on Freenode.
docs/community/updates.rst
+30
View File
@@ -0,0 +1,30 @@
.. _updates:
Updates
=======
If you'd like to stay up to date on the community and development of Requests,
there are serveral options:
GitHub
------
The best way to track the development of Requests is through
`the GitHub repo <https://github.com/kennethreitz/requests>`_.
Twitter
-------
I often tweet about new features and releases of Requests.
Follow `@kennethreitz <https://twitter.com/kennethreitz>`_ for updates.
Mailing List
------------
There's a low-volume mailing list for Requests. To subscribe to the
mailing list, send an email to
`requests@librelist.org <mailto:requests@librelist.org>`_.
docs/conf.py
+1 -1
View File
@@ -155,7 +155,7 @@ html_sidebars = {
#html_split_index = False
# If true, links to the reST sources are added to the pages.
html_show_sourcelink = True
html_show_sourcelink = False
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
html_show_sphinx = False
docs/dev/authors.rst
+5
View File
@@ -0,0 +1,5 @@
Authors
=======
.. include:: ../../AUTHORS
docs/index.rst
+26 -6
View File
@@ -13,7 +13,8 @@ Requests is an :ref:`ISC Licensed <isc>` HTTP library, written in Python, for hu
Most existing Python modules for sending HTTP requests are extremely verbose
and cumbersome. Python's builtin **urllib2** module provides most of
the HTTP capabilities you should need, but the api is thoroughly **broken**.
It requires an *enormous* amount of work (even method overrides) to perform the simplest of tasks.
It requires an *enormous* amount of work (even method overrides) to perform
the simplest of tasks.
Things shouldn’t be this way. Not in Python.
@@ -41,32 +42,50 @@ Testimonals
`Twitter, Inc <http://twitter.com>`_ uses Requests internally.
**Daniel Greenfeld**
Nuked a 1200 LOC spaghetti code library with 10 lines of code thanks to @kennethreitz's request library. Today has been AWESOME.
Nuked a 1200 LOC spaghetti code library with 10 lines of code thanks to
@kennethreitz's request library. Today has been AWESOME.
**Kenny Meyers**
Python HTTP: When in doubt, or when not in doubt, use Requests. Beautiful, simple, Pythonic.
Python HTTP: When in doubt, or when not in doubt, use Requests. Beautiful,
simple, Pythonic.
**Rich Leland**
Requests is awesome. That is all.
**Steve Pike**
I can never remember how to do it the regular way. ``import requests; requests.get()`` is just so easy!
I can never remember how to do it the regular way.
``import requests; requests.get()`` is just so easy!
User Guide
----------
This part of the documentation, which is mostly prose, begins with some background information about Requests, then focuses on step-by-step instructions for getting the most out of Requests.
This part of the documentation, which is mostly prose, begins with some
background information about Requests, then focuses on step-by-step
instructions for getting the most out of Requests.
.. toctree::
:maxdepth: 2
user/intro
user/install
.. user/quickstart
user/quickstart
user/advanced
Community Guide
-----------------
This part of the documentation, which is mostly prose, details the
Requests ecosystem and community.
.. toctree::
:maxdepth: 2
community/faq
community/support
community/updates
API Documentation
-----------------
@@ -90,3 +109,4 @@ you.
dev/internals
dev/todo
dev/authors
docs/user/advanced.rst
+137
View File
@@ -0,0 +1,137 @@
.. _advanced:
Advanced Usage
==============
This document covers some of Requests more advanced features.
Session Objects
---------------
The Session object allows you to persist certain parameters across
requests. It also establishes a CookieJar and passes it along
to any requests made from the Session instance.
A session object has all the methods of the main Requests API.
Let's persist some cookies across requests::
with requests.session() as s:
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")
print r.content
# '{"cookies": {"sessioncookie": "123456789"}}'
Sessions can also be used to provide default data to the request methods::
headers = {'x-test': 'true'}
auth = ('user', 'pass')
with requests.session(auth=auth, headers=headers) as c:
# both 'x-test' and 'x-test2' are sent
c.get('http://httpbin.org/headers', headers={'x-test2': 'true'})
.. admonition:: Global Settings
Certain parameters are best set at the ``request.config`` level
(e.g.. a global proxy, user agent header).
Event Hooks
-----------
Requests has a hook system that you can use to manipulate portions of
the request process, or signal event handling.
Available hooks:
``args``:
A dictionary of the arguments being sent to Request().
``pre_request``:
The Request object, directly before being sent.
``post_request``:
The Request object, directly after being sent.
``response``:
The response generated from a Request.
You can assign a hook function on a per-request basis by passing a
``{hook_name: callback_function}`` dictionary to the ``hooks`` request
parameter::
hooks=dict(args=print_url)
That ``callback_function`` will receive a chunk of data as its first
argument.
::
def print_url(args):
print args['url']
If an error occurs while executing your callback, a warning is given.
If the callback function returns a value, it is assumed that it is to
replace the data that was passed in. If the function doesn't return
anything, nothing else is effected.
Let's print some request method arguments at runtime::
>>> requests.get('http://httpbin', hooks=dict(args=print_url))
http://httpbin
<Response [200]>
Let's hijack some arguments this time with a new callback::
def hack_headers(args):
if not args[headers]:
args['headers'] = dict()
args['headers'].update({'X-Testing': 'True'})
return args
hooks = dict(args=hack_headers)
headers = dict(yo=dawg)
And give it a try::
>>> requests.get('http://httpbin/headers', hooks=hooks, headers=headers)
{
"headers": {
"Content-Length": "",
"Accept-Encoding": "gzip",
"Yo": "dawg",
"X-Forwarded-For": "::ffff:24.127.96.129",
"Connection": "close",
"User-Agent": "python-requests.org",
"Host": "httpbin.org",
"X-Testing": "True",
"X-Forwarded-Protocol": "",
"Content-Type": ""
}
}
Verbose Logging
---------------
If you want to get a good look at what HTTP requests are being sent
by your application, you can turn on verbose logging.
To do so, just configure Requests with a stream to write to::
>>> requests.settings.verbose = sys.stderr
>>> requests.get('http://httpbin.org/headers')
2011-08-17T03:04:23.380175 GET http://httpbin.org/headers
<Response [200]>
docs/user/install.rst
+6 -3
View File
@@ -3,7 +3,8 @@
Installation
============
This part of the documentation covers the installation of Requests. The first step to using any software package is getting it properly installed.
This part of the documentation covers the installation of Requests.
The first step to using any software package is getting it properly installed.
Distribute & Pip
@@ -24,7 +25,8 @@ But, you really `shouldn't do that <http://www.pip-installer.org/en/latest/index
Cheeseshop Mirror
-----------------
If the Cheeseshop is down, you can also install Requests from Kenneth Reitz's personal `Cheeseshop mirror <http://pip.kreitz.co/>`_::
If the Cheeseshop is down, you can also install Requests from Kenneth Reitz's
personal `Cheeseshop mirror <http://pip.kreitz.co/>`_::
$ pip install -i http://pip.kreitz.co/simple requests
@@ -48,6 +50,7 @@ Or, download the `zipball <https://github.com/kennethreitz/requests/zipball/mast
$ curl -O https://github.com/kennethreitz/requests/zipball/master
Once you have a copy of the source, you can embed it in your Python package, or install it into your site-packages easily::
Once you have a copy of the source, you can embed it in your Python package,
or install it into your site-packages easily::
$ python setup.py install
docs/user/intro.rst
+18 -3
View File
@@ -1,3 +1,5 @@
.. _introduction:
Introduction
============
@@ -19,10 +21,16 @@ All contributions to Requests should keep these important rules in mind.
ISC License
-----------
A large number of open source projects you find today are `GPL Licensed`_.
While the GPL has its time and place, it should most certainly not be your
go-to license for your next open source project.
A large number of open source projects you find today are `GPL Licensed`_. While the GPL has its time and place, it should most certainly not be your go-to license for your next open source project.
A project that is released as GPL cannot be used in any commercial product
without the product itself also being offered as open source.
A project that is released as GPL cannot be used in any commercial product without the product itself also being offered as open source. The MIT, BSD, ISC, and Apache2 licenses are great alternatives to the GPL that allow your open-source software to be used freely in proprietary, closed-source software.
The MIT, BSD, ISC, and Apache2 licenses are great alternatives to the GPL
that allow your open-source software to be used freely in proprietary,
closed-source software.
Requests is released under terms of `The ISC License`_.
@@ -40,16 +48,23 @@ Requests License
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
.. _interpreters:
Python Interpreters
-------------------
At this time, the following Python platforms are officially supported:
* cPython 2.5
* cPython 2.5.5
* cPython 2.5.6
* cPython 2.6
* cPython 2.6.6
* cPython 2.6.7
* cPython 2.7
* cPython 2.7.1
* cPython 2.7.2
* PyPy-c 1.4
* PyPy-c 1.5
Support for Python 3.x is planned.
Support for Python 3.x is planned.
docs/user/quickstart.rst
+113 -59
View File
@@ -1,84 +1,138 @@
Feature Overview
================
Requests is designed to solve a 90% use case — making simple requests. While most
HTTP libraries are extremely extensible, they often attempt to support the entire HTTP Spec.
This often leads to extremely messy and cumbersome APIs, as is the case with urllib2. Requests abandons support for edge-cases, and focuses on the essentials.
.. _features:
Requests Can:
-------------
- Make **GET**, **POST**, **PUT**, **DELETE**, and **HEAD** requests.
- Handle HTTP and HTTPS Requests
- Add Request headers (with a simple dictionary)
- URLEncode your Form Data (with a simple dictionary)
- Add Multi-part File Uploads (with a simple dictionary)
- Handle CookieJars (with a single parameter)
- Add HTTP Authentication (with a single parameter)
- Handle redirects (with history)
- Automatically decompress GZip'd responses
- Support Unicode URLs
- Gracefully timeout
- Interface with Eventlet & Gevent
Requests Can't:
---------------
- Handle Caching
- Handle Keep-Alives
.. _quickstart:
Quickstart
==========
.. module:: requests.models
GET Request
-----------
Eager to get started? This page gives a good introduction in how to get started
with Requests. This assumes you already have Requests installed. If you do not,
head over to the :ref:`Installation <install>` section.
First, make sure that:
* Requests is :ref:`installed <install>`
* Requests is :ref:`up-to-date <updates>`
Adding Parameters
-----------------
Lets gets started with some simple use cases and examples.
Adding Headers
--------------
HTTP Basic Auth
---------------
Tracking Redirects
Make a GET Request
------------------
Making a standard request with Requests is very simple.
Let's get GitHub's public timeline ::
r = requests.get('https://github.com/timeline.json')
Now, we have a :class:`Response` object called ``r``. We can get all the
information we need from this.
Response Content
----------------
HTTP POST (Form Data)
We can read the content of the server's response::
>>> r.content
'[{"repository":{"open_issues":0,"url":"https://github.com/...
Response Status Codes
---------------------
We can check the response status code::
HTTP POST (Binary Data)
-----------------------
>>> r.status_code
200
Requests also comes with a built-in status code lookup object for easy
reference::
>>> r.status_code == requests.codes.ok
True
If we made a bad request, we can raise it with
:class:`Response.raise_for_status()`::
>>> _r = requests.get('http://httpbin.org/status/404')
>>> _r.status_code
404
>>> _r.raise_for_status()
Traceback (most recent call last):
File "requests/models.py", line 394, in raise_for_status
raise self.error
urllib2.HTTPError: HTTP Error 404: NOT FOUND
But, since our ``status_code`` was ``200``, when we call it::
>>> r.raise_for_status()
None
All is well.
HTTP POST (Multipart Files)
---------------------------
Response Headers
----------------
We can view the server's response headers with a simple Python dictionary
interface::
>>> r.headers
{
'status': '200 OK',
'content-encoding': 'gzip',
'transfer-encoding': 'chunked',
'connection': 'close',
'server': 'nginx/1.0.4',
'x-runtime': '148ms',
'etag': '"e1ca502697e5c9317743dc078f67693f"',
'content-type': 'application/json; charset=utf-8'
}
The dictionary is special, though: it's made just for HTTP headers. According to
`RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>`_, HTTP
Headers are case-insensitive.
So, we can access the headers using any capitalization we want::
>>> r.headers['Content-Type']
'application/json; charset=utf-8'
>>> r.headers.get('content-type')
'application/json; charset=utf-8'
If a header doesn't exist in the Response, its value defaults to ``None``::
>>> r.headers['X-Random']
None
HTTP PUT
--------
Cookies
-------
If a response contains some Cookies, you can get quick access to them::
HTTP DELETE
-----------
>>> url = 'http://httpbin.org/cookies/set/requests-is/awesome'
>>> r = requests.get(url)
>>> print r.cookies
{'requests-is': 'awesome'}
HTTP HEAD
---------
The underlying CookieJar is also available for more advanced handling::
>>> r.request.cookiejar
<cookielib.CookieJar>
To send your own cookies to the server, you can use the ``cookies``
parameter::
>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')
>>> r = requests.get(url, cookies=cookies)
>>> r.content
'{"cookies": {"cookies_are": "working"}}'
requests/api.py
+45 -59
View File
@@ -12,24 +12,28 @@ This module impliments the Requests API.
"""
import config
from .models import Request, Response, AuthManager, AuthObject, auth_manager
from .models import Request, Response, AuthObject
from .status_codes import codes
from .hooks import dispatch_hook
from .utils import cookiejar_from_dict
from urlparse import urlparse
__all__ = ('request', 'get', 'head', 'post', 'patch', 'put', 'delete')
def request(method, url,
params=None, data=None, headers=None, cookies=None, files=None, auth=None,
timeout=None, allow_redirects=False, proxies=None):
timeout=None, allow_redirects=False, proxies=None, hooks=None):
"""Constructs and sends a :class:`Request <models.Request>`. Returns :class:`Response <models.Response>` object.
"""Constructs and sends a :class:`Request <Request>`.
Returns :class:`Response <Response>` object.
:param method: method for the new :class:`Request` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
:param data: (optional) Dictionary or bytes to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
:param cookies: (optional) CookieJar object to send with the :class:`Request`.
:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
:param files: (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.
:param auth: (optional) AuthObject to enable Basic HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
@@ -37,7 +41,14 @@ def request(method, url,
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
"""
r = Request(
method = str(method).upper()
if cookies is None:
cookies = {}
cookies = cookiejar_from_dict(cookies)
args = dict(
method = method,
url = url,
data = data,
@@ -45,14 +56,29 @@ def request(method, url,
headers = headers,
cookiejar = cookies,
files = files,
auth = auth or auth_manager.get_auth(url),
auth = auth,
timeout = timeout or config.settings.timeout,
allow_redirects = allow_redirects,
proxies = proxies or config.settings.proxies
proxies = proxies or config.settings.proxies,
)
# Arguments manipulation hook.
args = dispatch_hook('args', hooks, args)
r = Request(**args)
# Pre-request hook.
r = dispatch_hook('pre_request', hooks, r)
# Send the HTTP Request.
r.send()
# Post-request hook.
r = dispatch_hook('post_request', hooks, r)
# Response manipulation hook.
r.response = dispatch_hook('response', hooks, r.response)
return r.response
@@ -61,50 +87,31 @@ def get(url, **kwargs):
"""Sends a GET request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary of parameters, or bytes, to be sent in the query string for the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
:param cookies: (optional) CookieJar object to send with the :class:`Request`.
:param auth: (optional) AuthObject to enable Basic HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param **kwargs: Optional arguments that ``request`` takes.
"""
return request('GET', url, **kwargs)
return request('get', url, **kwargs)
def head(url, **kwargs):
"""Sends a HEAD request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary of parameters, or bytes, to be sent in the query string for the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to sent with the :class:`Request`.
:param cookies: (optional) CookieJar object to send with the :class:`Request`.
:param auth: (optional) AuthObject to enable Basic HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param **kwargs: Optional arguments that ``request`` takes.
"""
return request('HEAD', url, **kwargs)
return request('head', url, **kwargs)
def post(url, data='', **kwargs):
"""Sends a POST request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary or bytes to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to sent with the :class:`Request`.
:param files: (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.
:param cookies: (optional) CookieJar object to send with the :class:`Request`.
:param auth: (optional) AuthObject to enable Basic HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
:param allow_redirects: (optional) Boolean. Set to True if redirect following is allowed.
:param params: (optional) Dictionary of parameters, or bytes, to be sent in the query string for the :class:`Request`.
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param **kwargs: Optional arguments that ``request`` takes.
"""
return request('POST', url, data=data, **kwargs)
return request('post', url, data=data, **kwargs)
def put(url, data='', **kwargs):
@@ -112,17 +119,10 @@ def put(url, data='', **kwargs):
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary or bytes to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to sent with the :class:`Request`.
:param files: (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.
:param cookies: (optional) CookieJar object to send with the :class:`Request`.
:param auth: (optional) AuthObject to enable Basic HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
:param allow_redirects: (optional) Boolean. Set to True if redirect following is allowed.
:param params: (optional) Dictionary of parameters, or bytes, to be sent in the query string for the :class:`Request`.
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param **kwargs: Optional arguments that ``request`` takes.
"""
return request('PUT', url, data=data, **kwargs)
return request('put', url, data=data, **kwargs)
def patch(url, data='', **kwargs):
@@ -130,31 +130,17 @@ def patch(url, data='', **kwargs):
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary or bytes to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to sent with the :class:`Request`.
:param files: (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.
:param cookies: (optional) CookieJar object to send with the :class:`Request`.
:param auth: (optional) AuthObject to enable Basic HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
:param allow_redirects: (optional) Boolean. Set to True if redirect following is allowed.
:param params: (optional) Dictionary of parameters, or bytes, to be sent in the query string for the :class:`Request`.
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param **kwargs: Optional arguments that ``request`` takes.
"""
return request('PATCH', url, **kwargs)
return request('patch', url, **kwargs)
def delete(url, **kwargs):
"""Sends a DELETE request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary of parameters, or bytes, to be sent in the query string for the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to sent with the :class:`Request`.
:param cookies: (optional) CookieJar object to send with the :class:`Request`.
:param auth: (optional) AuthObject to enable Basic HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
:param allow_redirects: (optional) Boolean. Set to True if redirect following is allowed.
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param **kwargs: Optional arguments that ``request`` takes.
"""
return request('DELETE', url, **kwargs)
return request('delete', url, **kwargs)
requests/config.py
+10 -1
View File
@@ -12,7 +12,7 @@ class Settings(object):
_singleton = {}
# attributes with defaults
__attrs__ = ('timeout', 'verbose')
__attrs__ = []
def __init__(self, **kwargs):
super(Settings, self).__init__()
@@ -53,7 +53,16 @@ class Settings(object):
return None
return object.__getattribute__(self, key)
settings = Settings()
settings.base_headers = {'User-Agent': 'python-requests.org'}
settings.accept_gzip = True
settings.proxies = None
settings.verbose = None
settings.timeout = None
settings.max_redirects = 30
settings.decode_unicode = True
#: Use socket.setdefaulttimeout() as fallback?
settings.timeout_fallback = True
requests/core.py
+7 -4
View File
@@ -12,15 +12,18 @@ This module implements the main Requests system.
"""
__title__ = 'requests'
__version__ = '0.5.1'
__build__ = 0x000501
__version__ = '0.6.1'
__build__ = 0x000601
__author__ = 'Kenneth Reitz'
__license__ = 'ISC'
__copyright__ = 'Copyright 2011 Kenneth Reitz'
from models import HTTPError, auth_manager
from models import HTTPError, Request, Response
from api import *
from exceptions import *
from sessions import session
from status_codes import codes
from config import settings
from config import settings
import utils
requests/exceptions.py
+3
View File
@@ -21,3 +21,6 @@ class URLRequired(RequestException):
class InvalidMethod(RequestException):
"""An inappropriate method was attempted."""
class TooManyRedirects(RequestException):
"""Too many redirects."""
requests/hooks.py
+40
View File
@@ -0,0 +1,40 @@
# -*- coding: utf-8 -*-
"""
requests.hooks
~~~~~~~~~~~~~~
This module provides the capabilities for the Requests hooks system.
Available hooks:
``args``:
A dictionary of the arguments being sent to Request().
``pre_request``:
The Request object, directly before being sent.
``post_request``:
The Request object, directly after being sent.
``response``:
The response generated from a Request.
"""
import warnings
def dispatch_hook(key, hooks, hook_data):
"""Dipatches a hook dictionary on a given peice of data."""
hooks = hooks or dict()
if key in hooks:
try:
return hooks.get(key).__call__(hook_data) or hook_data
except Exception, why:
warnings.warn(str(why))
return hook_data
requests/models.py
+143 -59
View File
@@ -11,6 +11,7 @@ import urllib2
import socket
import zlib
from urllib2 import HTTPError
from urlparse import urlparse, urlunparse, urljoin
from datetime import datetime
@@ -20,52 +21,63 @@ from .monkeys import Request as _Request, HTTPBasicAuthHandler, HTTPForcedBasicA
from .structures import CaseInsensitiveDict
from .packages.poster.encode import multipart_encode
from .packages.poster.streaminghttp import register_openers, get_handlers
from .exceptions import RequestException, AuthenticationError, Timeout, URLRequired, InvalidMethod
from .utils import dict_from_cookiejar, get_unicode_from_response, decode_gzip
from .status_codes import codes
from .exceptions import RequestException, AuthenticationError, Timeout, URLRequired, InvalidMethod, TooManyRedirects
REDIRECT_STATI = (301, 302, 303, 307)
REDIRECT_STATI = (codes.moved, codes.found, codes.other, codes.temporary_moved)
class Request(object):
"""The :class:`Request <models.Request>` object. It carries out all functionality of
"""The :class:`Request <Request>` object. It carries out all functionality of
Requests. Recommended interface is with the Requests functions.
"""
_METHODS = ('GET', 'HEAD', 'PUT', 'POST', 'DELETE', 'PATCH')
def __init__(self,
url=None, headers=dict(), files=None, method=None, data=dict(),
params=dict(), auth=None, cookiejar=None, timeout=None, redirect=False,
allow_redirects=False, proxies=None):
socket.setdefaulttimeout(timeout)
#: Float describ the timeout of the request.
# (Use socket.setdefaulttimeout() as fallback)
self.timeout = timeout
#: Request URL.
self.url = url
#: Dictonary of HTTP Headers to attach to the :class:`Request <models.Request>`.
#: Dictonary of HTTP Headers to attach to the :class:`Request <Request>`.
self.headers = headers
#: Dictionary of files to multipart upload (``{filename: content}``).
self.files = files
#: HTTP Method to use. Available: GET, HEAD, PUT, POST, DELETE.
#: HTTP Method to use.
self.method = method
#: Dictionary or byte of request body data to attach to the
#: :class:`Request <models.Request>`.
#: :class:`Request <Request>`.
self.data = None
#: Dictionary or byte of querystring data to attach to the
#: :class:`Request <models.Request>`.
#: :class:`Request <Request>`.
self.params = None
#: True if :class:`Request <models.Request>` is part of a redirect chain (disables history
#: True if :class:`Request <Request>` is part of a redirect chain (disables history
#: and HTTPError storage).
self.redirect = redirect
#: Set to True if full redirects are allowed (e.g. re-POST-ing of data at new ``Location``)
self.allow_redirects = allow_redirects
# Dictionary mapping protocol to the URL of the proxy (e.g. {'http': 'foo.bar:3128'})
self.proxies = proxies
self.data, self._enc_data = self._encode_params(data)
self.params, self._enc_params = self._encode_params(params)
#: :class:`Response <models.Response>` instance, containing
#: :class:`Response <Response>` instance, containing
#: content and metadata of HTTP Response, once :attr:`sent <send>`.
self.response = Response()
@@ -73,10 +85,13 @@ class Request(object):
auth = AuthObject(*auth)
if not auth:
auth = auth_manager.get_auth(self.url)
#: :class:`AuthObject` to attach to :class:`Request <models.Request>`.
#: :class:`AuthObject` to attach to :class:`Request <Request>`.
self.auth = auth
#: CookieJar to attach to :class:`Request <models.Request>`.
#: CookieJar to attach to :class:`Request <Request>`.
self.cookiejar = cookiejar
#: True if Request has been sent.
self.sent = False
@@ -102,14 +117,6 @@ class Request(object):
return '<Request [%s]>' % (self.method)
def __setattr__(self, name, value):
if (name == 'method') and (value):
if not value in self._METHODS:
raise InvalidMethod()
object.__setattr__(self, name, value)
def _checks(self):
"""Deterministic checks for consistency."""
@@ -126,8 +133,16 @@ class Request(object):
_handlers.append(urllib2.HTTPCookieProcessor(self.cookiejar))
if self.auth:
if not isinstance(self.auth.handler, (urllib2.AbstractBasicAuthHandler, urllib2.AbstractDigestAuthHandler)):
auth_manager.add_password(self.auth.realm, self.url, self.auth.username, self.auth.password)
if not isinstance(self.auth.handler,
(urllib2.AbstractBasicAuthHandler,
urllib2.AbstractDigestAuthHandler)):
# TODO: REMOVE THIS COMPLETELY
auth_manager.add_password(
self.auth.realm, self.url,
self.auth.username,
self.auth.password)
self.auth.handler = self.auth.handler(auth_manager)
auth_manager.add_auth(self.url, self.auth)
@@ -159,7 +174,10 @@ class Request(object):
def _build_response(self, resp, is_error=False):
"""Build internal :class:`Response <models.Response>` object from given response."""
"""Build internal :class:`Response <Response>` object
from given response.
"""
def build(resp):
@@ -168,8 +186,13 @@ class Request(object):
try:
response.headers = CaseInsensitiveDict(getattr(resp.info(), 'dict', None))
response.read = resp.read
response.close = resp.close
response.fo = resp
if self.cookiejar:
response.cookies = dict_from_cookiejar(self.cookiejar)
except AttributeError:
pass
@@ -190,20 +213,31 @@ class Request(object):
while (
('location' in r.headers) and
((self.method in ('GET', 'HEAD')) or
(r.status_code is 303) or
(r.status_code is codes.see_other) or
(self.allow_redirects))
):
r.fo.close()
if not len(history) < settings.max_redirects:
raise TooManyRedirects()
history.append(r)
url = r.headers['location']
# Handle redirection without scheme (see: RFC 1808 Section 4)
if url.startswith('//'):
parsed_rurl = urlparse(r.url)
url = '%s:%s' % (parsed_rurl.scheme, url)
# Facilitate non-RFC2616-compliant 'location' headers
# (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
url = urljoin(r.url, url)
if not urlparse(url).netloc:
url = urljoin(r.url, urllib.quote(urllib.unquote(url)))
# http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4
if r.status_code is 303:
if r.status_code is codes.see_other:
method = 'GET'
else:
method = self.method
@@ -232,8 +266,8 @@ class Request(object):
Otherwise, assumes the data is already encoded appropriately, and
returns it twice.
"""
if hasattr(data, 'items'):
result = []
for k, vs in data.items():
@@ -246,12 +280,15 @@ class Request(object):
def _build_url(self):
"""Build the actual URL to use"""
"""Build the actual URL to use."""
# Support for unicode domain names.
parsed_url = list(urlparse(self.url))
parsed_url[1] = parsed_url[1].encode('idna')
self.url = urlunparse(parsed_url)
# Support for unicode domain names and paths.
scheme, netloc, path, params, query, fragment = urlparse(self.url)
netloc = netloc.encode('idna')
if isinstance(path, unicode):
path = path.encode('utf-8')
path = urllib.quote(urllib.unquote(path))
self.url = str(urlunparse([ scheme, netloc, path, params, query, fragment ]))
if self._enc_params:
if urlparse(self.url).query:
@@ -272,6 +309,7 @@ class Request(object):
:param anyway: If True, request will be sent, even if it has
already been sent.
"""
self._checks()
success = False
@@ -300,13 +338,32 @@ class Request(object):
req = _Request(url, data=self._enc_data, method=self.method)
if self.headers:
req.headers.update(self.headers)
for k,v in self.headers.iteritems():
req.add_header(k, v)
if not self.sent or anyway:
try:
opener = self._get_opener()
resp = opener(req)
try:
resp = opener(req, timeout=self.timeout)
except TypeError, err:
# timeout argument is new since Python v2.6
if not 'timeout' in str(err):
raise
if settings.timeout_fallback:
# fall-back and use global socket timeout (This is not thread-safe!)
old_timeout = socket.getdefaulttimeout()
socket.setdefaulttimeout(self.timeout)
resp = opener(req)
if settings.timeout_fallback:
# restore gobal timeout
socket.setdefaulttimeout(old_timeout)
if self.cookiejar is not None:
self.cookiejar.extract_cookies(resp, req)
@@ -318,7 +375,6 @@ class Request(object):
self._build_response(why, is_error=True)
else:
self._build_response(resp)
self.response.ok = True
@@ -365,38 +421,48 @@ class Request(object):
#: Params handled in _build_url
return curl_cmd + method_opt + data + header + '"' + self._build_url() + '"'
class Response(object):
"""The core :class:`Response <models.Response>` object. All
:class:`Request <models.Request>` objects contain a
:class:`response <models.Response>` attribute, which is an instance
"""The core :class:`Response <Response>` object. All
:class:`Request <Request>` objects contain a
:class:`response <Response>` attribute, which is an instance
of this class.
"""
def __init__(self):
#: Raw content of the response, in bytes.
#: If ``content-encoding`` of response was set to ``gzip``, the
#: response data will be automatically deflated.
self._content = None
#: Integer Code of responded HTTP Status.
self.status_code = None
#: Case-insensitive Dictionary of Response Headers.
#: For example, ``headers['content-encoding']`` will return the
#: value of a ``'Content-Encoding'`` response header.
self.headers = CaseInsensitiveDict()
#: File-like object representation of response (for advanced usage).
self.fo = None
#: Final URL location of Response.
self.url = None
#: True if no :attr:`error` occured.
self.ok = False
#: Resulting :class:`HTTPError` of request, if one occured.
self.error = None
#: A list of :class:`Response <models.Response>` objects from
#: A list of :class:`Response <Response>` objects from
#: the history of the Request. Any redirect responses will end
#: up here.
self.history = []
#: The Request that created the Response.
#: The :class:`Request <Request>` that created the Response.
self.request = None
#: A dictionary of Cookies the server sent back.
self.cookies = None
def __repr__(self):
return '<Response [%s]>' % (self.status_code)
@@ -404,22 +470,35 @@ class Response(object):
def __nonzero__(self):
"""Returns true if :attr:`status_code` is 'OK'."""
return not self.error
def __getattr__(self, name):
"""Read and returns the full stream when accessing to :attr: `content`"""
if name == 'content':
if self._content is not None:
return self._content
self._content = self.read()
if self.headers.get('content-encoding', '') == 'gzip':
try:
self._content = zlib.decompress(self._content, 16+zlib.MAX_WBITS)
except zlib.error:
pass
@property
def content(self):
"""Content of the response, in bytes or unicode
(if available).
"""
if self._content is not None:
return self._content
# Read the contents.
self._content = self.fo.read()
# Decode GZip'd content.
if 'gzip' in self.headers.get('content-encoding', ''):
try:
self._content = decode_gzip(self._content)
except zlib.error:
pass
# Decode unicode content.
if settings.decode_unicode:
self._content = get_unicode_from_response(self)
return self._content
def raise_for_status(self):
"""Raises stored :class:`HTTPError` or :class:`URLError`, if one occured."""
@@ -427,6 +506,7 @@ class Response(object):
raise self.error
class AuthManager(object):
"""Requests Authentication Manager."""
@@ -500,8 +580,10 @@ class AuthManager(object):
def reduce_uri(self, uri, default_port=True):
"""Accept authority or URI and extract only the authority and path."""
# note HTTP URLs do not have a userinfo component
parts = urllib2.urlparse.urlsplit(uri)
if parts[1]:
# URI
scheme = parts[0]
@@ -512,7 +594,9 @@ class AuthManager(object):
scheme = None
authority = uri
path = '/'
host, port = urllib2.splitport(authority)
if default_port and port is None and scheme is not None:
dport = {"http": 80,
"https": 443,
requests/sessions.py
+84
View File
@@ -0,0 +1,84 @@
# -*- coding: utf-8 -*-
"""
requests.session
~~~~~~~~~~~~~~~
This module provides a Session object to manage and persist settings across
requests (cookies, auth, proxies).
"""
import cookielib
from . import api
from .utils import add_dict_to_cookiejar
class Session(object):
"""A Requests session."""
__attrs__ = ['headers', 'cookies', 'auth', 'timeout', 'proxies', 'hooks']
def __init__(self, **kwargs):
# Set up a CookieJar to be used by default
self.cookies = cookielib.FileCookieJar()
# Map args from kwargs to instance-local variables
map(lambda k, v: (k in self.__attrs__) and setattr(self, k, v),
kwargs.iterkeys(), kwargs.itervalues())
# Map and wrap requests.api methods
self._map_api_methods()
def __repr__(self):
return '<requests-client at 0x%x>' % (id(self))
def __enter__(self):
return self
def __exit__(self, *args):
# print args
pass
def _map_api_methods(self):
"""Reads each available method from requests.api and decorates
them with a wrapper, which inserts any instance-local attributes
(from __attrs__) that have been set, combining them with **kwargs.
"""
def pass_args(func):
def wrapper_func(*args, **kwargs):
inst_attrs = dict((k, v) for k, v in self.__dict__.iteritems()
if k in self.__attrs__)
# Combine instance-local values with kwargs values, with
# priority to values in kwargs
kwargs = dict(inst_attrs.items() + kwargs.items())
# If a session request has a cookie_dict, inject the
# values into the existing CookieJar instead.
if isinstance(kwargs.get('cookies', None), dict):
kwargs['cookies'] = add_dict_to_cookiejar(
inst_attrs['cookies'], kwargs['cookies']
)
if kwargs.get('headers', None) and inst_attrs.get('headers', None):
kwargs['headers'].update(inst_attrs['headers'])
return func(*args, **kwargs)
return wrapper_func
# Map and decorate each function available in requests.api
map(lambda fn: setattr(self, fn, pass_args(getattr(api, fn))),
api.__all__)
def session(**kwargs):
"""Returns a :class:`Session` for context-managment."""
return Session(**kwargs)
requests/status_codes.py
+4 -4
View File
@@ -22,9 +22,9 @@ _codes = {
# Redirection.
300: ('multiple_choices',),
301: ('moved_pemanently', 'moved'),
301: ('moved_permanently', 'moved', '\\o-'),
302: ('found',),
302: ('see_other', 'other'),
303: ('see_other', 'other'),
304: ('not_modified',),
305: ('use_proxy',),
306: ('switch_proxy',),
@@ -36,7 +36,7 @@ _codes = {
401: ('unauthorized',),
402: ('payment_required', 'payment'),
403: ('forbidden',),
404: ('not_found',),
404: ('not_found', '-o-'),
405: ('method_not_allowed', 'not_allowed'),
406: ('not_acceptable',),
407: ('proxy_authentication_required', 'proxy_auth', 'proxy_authentication'),
@@ -62,7 +62,7 @@ _codes = {
499: ('client_closed_request',),
# Server Error.
500: ('iternal_server_error', 'server_error'),
500: ('internal_server_error', 'server_error', '/o\\'),
501: ('not_implemented',),
502: ('bad_gateway',),
503: ('service_unavailable', 'unavailable'),
requests/utils.py
+165
View File
@@ -0,0 +1,165 @@
# -*- coding: utf-8 -*-
"""
requests.utils
~~~~~~~~~~~~~~
This module provides utlity functions that are used within Requests
that are also useful for external consumption.
"""
import cgi
import cookielib
import re
import zlib
def dict_from_cookiejar(cj):
"""Returns a key/value dictionary from a CookieJar.
:param cj: CookieJar object to extract cookies from.
"""
cookie_dict = {}
for _, cookies in cj._cookies.items():
for _, cookies in cookies.items():
for cookie in cookies.values():
# print cookie
cookie_dict[cookie.name] = cookie.value
return cookie_dict
def cookiejar_from_dict(cookie_dict):
"""Returns a CookieJar from a key/value dictionary.
:param cookie_dict: Dict of key/values to insert into CookieJar.
"""
# return cookiejar if one was passed in
if isinstance(cookie_dict, cookielib.CookieJar):
return cookie_dict
# create cookiejar
cj = cookielib.CookieJar()
cj = add_dict_to_cookiejar(cj, cookie_dict)
return cj
def add_dict_to_cookiejar(cj, cookie_dict):
"""Returns a CookieJar from a key/value dictionary.
:param cj: CookieJar to insert cookies into.
:param cookie_dict: Dict of key/values to insert into CookieJar.
"""
for k, v in cookie_dict.items():
cookie = cookielib.Cookie(
version=0,
name=k,
value=v,
port=None,
port_specified=False,
domain='',
domain_specified=False,
domain_initial_dot=False,
path='/',
path_specified=True,
secure=False,
expires=None,
discard=True,
comment=None,
comment_url=None,
rest={'HttpOnly': None},
rfc2109=False
)
# add cookie to cookiejar
cj.set_cookie(cookie)
return cj
def get_encodings_from_content(content):
"""Returns encodings from given content string.
:param content: bytestring to extract encodings from.
"""
charset_re = re.compile(r'<meta.*?charset=["\']*(.+?)["\'>]', flags=re.I)
return charset_re.findall(content)
def get_encoding_from_headers(headers):
"""Returns encodings from given HTTP Header Dict.
:param headers: dictionary to extract encoding from.
"""
content_type = headers.get('content-type')
content_type, params = cgi.parse_header(content_type)
if 'charset' in params:
return params['charset'].strip("'\"")
def get_unicode_from_response(r):
"""Returns the requested content back in unicode.
:param r: Reponse object to get unicode content from.
Tried:
1. charset from content-type
2. every encodings from ``<meta ... charset=XXX>``
3. fall back and replace all unicode characters
"""
tried_encodings = []
# Try charset from content-type
encoding = get_encoding_from_headers(r.headers)
if encoding:
try:
print '!'
return unicode(r.content, encoding)
except UnicodeError:
tried_encodings.append(encoding)
# Try every encodings from <meta ... charset=XXX>
encodings = get_encodings_from_content(r.content)
for encoding in encodings:
if encoding in tried_encodings:
continue
try:
return unicode(r.content, encoding)
except (UnicodeError, TypeError):
tried_encodings.append(encoding)
# Fall back:
try:
return unicode(r.content, encoding, errors='replace')
except TypeError:
return r.content
def decode_gzip(content):
"""Return gzip-decoded string.
:param content: bytestring to gzip-decode.
"""
return zlib.decompress(content, 16+zlib.MAX_WBITS)
test_requests.py
+36 -20
View File
@@ -13,6 +13,7 @@ except ImportError:
import requests
from requests.sessions import Session
HTTPBIN_URL = 'http://httpbin.org/'
@@ -131,22 +132,21 @@ class RequestsTestSuite(unittest.TestCase):
self.assertEqual(r.status_code, 200)
def test_AUTH_HTTPS_200_OK_GET(self):
def test_AUTH_HTTP_200_OK_GET(self):
for service in SERVICES:
auth = ('user', 'pass')
url = service('basic-auth', 'user', 'pass')
r = requests.get(url, auth=auth)
r = requests.get(url, auth=auth)
# print r.__dict__
self.assertEqual(r.status_code, 200)
r = requests.get(url)
self.assertEqual(r.status_code, 200)
# reset auto authentication
requests.auth_manager.empty()
def test_POSTBIN_GET_POST_FILES(self):
@@ -244,21 +244,6 @@ class RequestsTestSuite(unittest.TestCase):
r.content.decode('ascii')
def test_autoauth(self):
http_auth = ('user', 'pass')
requests.auth_manager.add_auth('httpbin.org', http_auth)
r = requests.get(httpbin('basic-auth', 'user', 'pass'))
self.assertEquals(r.status_code, 200)
requests.auth_manager.add_auth('httpbin.ep.io', http_auth)
r = requests.get(httpsbin('basic-auth', 'user', 'pass'))
self.assertEquals(r.status_code, 200)
def test_unicode_get(self):
for service in SERVICES:
@@ -455,5 +440,36 @@ class RequestsTestSuite(unittest.TestCase):
self.assertEquals(len(r.history), 3)
def test_session_HTTP_200_OK_GET(self):
s = Session()
r = s.get(httpbin('/'))
self.assertEqual(r.status_code, 200)
def test_session_HTTPS_200_OK_GET(self):
s = Session()
r = s.get(httpsbin('/'))
self.assertEqual(r.status_code, 200)
def test_session_persistent_headers(self):
heads = {'User-agent': 'Mozilla/5.0'}
s = Session()
s.headers = heads
# Make 2 requests from Session object, should send header both times
r1 = s.get(httpbin('user-agent'))
assert heads['User-agent'] in r1.content
r2 = s.get(httpbin('user-agent'))
assert heads['User-agent'] in r2.content
self.assertEqual(r2.status_code, 200)
if __name__ == '__main__':
unittest.main()