kennethreitz/requests
1
0
Fork 0
mirror of https://github.com/kennethreitz/requests.git synced 2026-06-05 22:50:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into fix-1859

Browse Source
This commit is contained in:
Cory Benfield
2015-12-18 09:14:13 +00:00
parent 87abd9c609 b32c3bc10f
commit 96a068b58e
83 changed files with 7953 additions and 3013 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -21,3 +21,4 @@ env/
t.py
t2.py
dist
AUTHORS.rst
+51 -35
View File
@@ -1,10 +1,13 @@
Requests is written and maintained by Kenneth Reitz and
various contributors:
Development Lead
````````````````
Keepers of the Three Crystals
`````````````````````````````
- Kenneth Reitz <me@kennethreitz.org> `@kennethreitz <https://github.com/kennethreitz>`_, Keeper of the Master Crystal.
- Cory Benfield <cory@lukasa.co.uk> `@lukasa <https://github.com/lukasa>`_
- Ian Cordasco <graffatcolmingov@gmail.com> `@sigmavirus24 <https://github.com/sigmavirus24>`_
- Kenneth Reitz <me@kennethreitz.com>
Urllib3
@@ -39,7 +42,7 @@ Patches and Suggestions
- Peter Manser
- Jeremy Selier
- Jens Diemer
- Alex <@alopatin>
- Alex (`@alopatin <https://github.com/alopatin>`_)
- Tom Hogans <tomhsx@gmail.com>
- Armin Ronacher
- Shrikant Sharat Kandula
@@ -76,11 +79,10 @@ Patches and Suggestions
- Chase Sterling
- Marty Alchin
- takluyver
- Ben Toews (mastahyeti)
- Ben Toews (`@mastahyeti <https://github.com/mastahyeti>`_)
- David Kemp
- Brendon Crawford
- Denis (Telofy)
- Cory Benfield (Lukasa)
- Denis (`@Telofy <https://github.com/Telofy>`_)
- Matt Giuca
- Adam Tauber
- Honza Javorek
@@ -97,53 +99,67 @@ Patches and Suggestions
- Jonty Wareing <jonty@jonty.co.uk>
- Shivaram Lingamneni
- Miguel Turner
- Rohan Jain (crodjer)
- Rohan Jain (`@crodjer <https://github.com/crodjer>`_)
- Justin Barber <barber.justin@gmail.com>
- Roman Haritonov <@reclosedev>
- Roman Haritonov (`@reclosedev <https://github.com/reclosedev>`_)
- Josh Imhoff <joshimhoff13@gmail.com>
- Arup Malakar <amalakar@gmail.com>
- Danilo Bargen (dbrgn)
- Danilo Bargen (`@dbrgn <https://github.com/dbrgn>`_)
- Torsten Landschoff
- Michael Holler (apotheos)
- Michael Holler (`@apotheos <https://github.com/apotheos>`_)
- Timnit Gebru
- Sarah Gonzalez
- Victoria Mo
- Leila Muhtasib
- Matthias Rahlf <matthias@webding.de>
- Jakub Roztocil <jakub@roztocil.name>
- Ian Cordasco <graffatcolmingov@gmail.com> @sigmavirus24
- Rhys Elsmore
- André Graf (dergraf)
- Stephen Zhuang (everbird)
- André Graf (`@dergraf <https://github.com/dergraf>`_)
- Stephen Zhuang (`@everbird <https://github.com/everbird>`_)
- Martijn Pieters
- Jonatan Heyman
- David Bonner <dbonner@gmail.com> @rascalking
- David Bonner <dbonner@gmail.com> (`@rascalking <https://github.com/rascalking>`_)
- Vinod Chandru
- Johnny Goodnow <j.goodnow29@gmail.com>
- Denis Ryzhkov <denisr@denisr.com>
- Wilfred Hughes <me@wilfred.me.uk> @dontYetKnow
- Wilfred Hughes <me@wilfred.me.uk>
- Dmitry Medvinsky <me@dmedvinsky.name>
- Bryce Boe <bbzbryce@gmail.com> @bboe
- Colin Dunklau <colin.dunklau@gmail.com> @cdunklau
- Bob Carroll <bob.carroll@alum.rit.edu> @rcarz
- Hugo Osvaldo Barrera <hugo@osvaldobarrera.com.ar> @hobarrera
- Łukasz Langa <lukasz@langa.pl> @llanga
- Bryce Boe <bbzbryce@gmail.com> (`@bboe <https://github.com/bboe>`_)
- Colin Dunklau <colin.dunklau@gmail.com> (`@cdunklau <https://github.com/cdunklau>`_)
- Bob Carroll <bob.carroll@alum.rit.edu> (`@rcarz <https://github.com/rcarz>`_)
- Hugo Osvaldo Barrera <hugo@osvaldobarrera.com.ar> (`@hobarrera <https://github.com/hobarrera>`_)
- Łukasz Langa <lukasz@langa.pl>
- Dave Shawley <daveshawley@gmail.com>
- James Clarke (jam)
- James Clarke (`@jam <https://github.com/jam>`_)
- Kevin Burke <kev@inburke.com>
- Flavio Curella
- David Pursehouse <david.pursehouse@gmail.com> @dpursehouse
- Jon Parise
- Alexander Karpinsky @homm86
- Marc Schlaich @schlamar
- Park Ilsu <daftonshady@gmail.com> @daftshady
- Matt Spitz @mattspitz
- Vikram Oberoi @voberoi
- Can Ibanoglu <can.ibanoglu@gmail.com> @canibanoglu
- Thomas Weißschuh <thomas@t-8ch.de> @t-8ch
- Jayson Vantuyl <jayson@aggressive.ly> @kagato
- David Pursehouse <david.pursehouse@gmail.com> (`@dpursehouse <https://github.com/dpursehouse>`_)
- Jon Parise (`@jparise <https://github.com/jparise>`_)
- Alexander Karpinsky (`@homm86 <https://twitter.com/homm86>`_)
- Marc Schlaich (`@schlamar <https://github.com/schlamar>`_)
- Park Ilsu <daftonshady@gmail.com> (`@daftshady <https://github.com/daftshady>`_)
- Matt Spitz (`@mattspitz <https://github.com/mattspitz>`_)
- Vikram Oberoi (`@voberoi <https://github.com/voberoi>`_)
- Can Ibanoglu <can.ibanoglu@gmail.com> (`@canibanoglu <https://github.com/canibanoglu>`_)
- Thomas Weißschuh <thomas@t-8ch.de> (`@t-8ch <https://github.com/t-8ch>`_)
- Jayson Vantuyl <jayson@aggressive.ly>
- Pengfei.X <pengphy@gmail.com>
- Kamil Madac <kamil.madac@gmail.com>
- Michael Becker <mike@beckerfuffle.com> @beckerfuffle
- Erik Wickstrom <erik@erikwickstrom.com> @erikwickstrom
- Константин Подшумок @podshumok
- Michael Becker <mike@beckerfuffle.com> (`@beckerfuffle <https://twitter.com/beckerfuffle>`_)
- Erik Wickstrom <erik@erikwickstrom.com> (`@erikwickstrom <https://github.com/erikwickstrom>`_)
- Константин Подшумок (`@podshumok <https://github.com/podshumok>`_)
- Ben Bass (`@codedstructure <https://github.com/codedstructure>`_)
- Jonathan Wong <evolutionace@gmail.com> (`@ContinuousFunction <https://github.com/ContinuousFunction>`_)
- Martin Jul (`@mjul <https://github.com/mjul>`_)
- Joe Alcorn (`@buttscicles <https://github.com/buttscicles>`_)
- Syed Suhail Ahmed <ssuhail.ahmed93@gmail.com> (`@syedsuhail <https://github.com/syedsuhail>`_)
- Scott Sadler (`@ssadler <https://github.com/ssadler>`_)
- Arthur Darcet (`@arthurdarcet <https://github.com/arthurdarcet>`_)
- Ulrich Petri (`@ulope <https://github.com/ulope>`_)
- Muhammad Yasoob Ullah Khalid <yasoob.khld@gmail.com> (`@yasoob <https://github.com/yasoob>`_)
- Paul van der Linden (`@pvanderlinden <https://github.com/pvanderlinden>`_)
- Colin Dickson (`@colindickson <https://github.com/colindickson>`_)
- Smiley Barry (`@smiley <https://github.com/smiley>`_)
- Shagun Sodhani (`@shagunsodhani <https://github.com/shagunsodhani>`_)
- Robin Linderborg (`@vienno <https://github.com/vienno>`_)
- Brian Samek(`@bsamek <https://github.com/bsamek`_)
CONTRIBUTING.md
+57
View File
@@ -0,0 +1,57 @@
# Contribution Guidelines
Before opening any issues or proposing any pull requests, please do the
following:
1. Read our [Contributor's Guide](http://docs.python-requests.org/en/latest/dev/contributing/).
2. Understand our [development philosophy](http://docs.python-requests.org/en/latest/dev/philosophy/).
To get the greatest chance of helpful responses, please also observe the
following additional notes.
## Questions
The GitHub issue tracker is for *bug reports* and *feature requests*. Please do
not use it to ask questions about how to use Requests. These questions should
instead be directed to [Stack Overflow](https://stackoverflow.com/). Make sure
that your question is tagged with the `python-requests` tag when asking it on
Stack Overflow, to ensure that it is answered promptly and accurately.
## Good Bug Reports
Please be aware of the following things when filing bug reports:
1. Avoid raising duplicate issues. *Please* use the GitHub issue search feature
to check whether your bug report or feature request has been mentioned in
the past. Duplicate bug reports and feature requests are a huge maintenance
burden on the limited resources of the project. If it is clear from your
report that you would have struggled to find the original, that's ok, but
if searching for a selection of words in your issue title would have found
the duplicate then the issue will likely be closed extremely abruptly.
2. When filing bug reports about exceptions or tracebacks, please include the
*complete* traceback. Partial tracebacks, or just the exception text, are
not helpful. Issues that do not contain complete tracebacks may be closed
without warning.
3. Make sure you provide a suitable amount of information to work with. This
means you should provide:
- Guidance on **how to reproduce the issue**. Ideally, this should be a
*small* code sample that can be run immediately by the maintainers.
Failing that, let us know what you're doing, how often it happens, what
environment you're using, etc. Be thorough: it prevents us needing to ask
further questions.
- Tell us **what you expected to happen**. When we run your example code,
what are we expecting to happen? What does "success" look like for your
code?
- Tell us **what actually happens**. It's not helpful for you to say "it
doesn't work" or "it fails". Tell us *how* it fails: do you get an
exception? A hang? A non-200 status code? How was the actual result
different from your expected result?
- Tell us **what version of Requests you're using**, and
**how you installed it**. Different versions of Requests behave
differently and have different bugs, and some distributors of Requests
ship patches on top of the code we supply.
If you do not provide all of these things, it will take us much longer to
fix your problem. If we ask you to clarify these and you never respond, we
will close your issue without fixing it.
HISTORY.rst
+361 -9
View File
@@ -3,6 +3,356 @@
Release History
---------------
2.9.0 (2015-12-15)
++++++++++++++++++
**Minor Improvements** (Backwards compatible)
- The ``verify`` keyword argument now supports being passed a path to a
directory of CA certificates, not just a single-file bundle.
- Warnings are now emitted when sending files opened in text mode.
- Added the 511 Network Authentication Required status code to the status code
registry.
**Bugfixes**
- For file-like objects that are not seeked to the very beginning, we now
send the content length for the number of bytes we will actually read, rather
than the total size of the file, allowing partial file uploads.
- When uploading file-like objects, if they are empty or have no obvious
content length we set ``Transfer-Encoding: chunked`` rather than
``Content-Length: 0``.
- We correctly receive the response in buffered mode when uploading chunked
bodies.
- We now handle being passed a query string as a bytestring on Python 3, by
decoding it as UTF-8.
- Sessions are now closed in all cases (exceptional and not) when using the
functional API rather than leaking and waiting for the garbage collector to
clean them up.
- Correctly handle digest auth headers with a malformed ``qop`` directive that
contains no token, by treating it the same as if no ``qop`` directive was
provided at all.
- Minor performance improvements when removing specific cookies by name.
2.8.1 (2015-10-13)
++++++++++++++++++
**Bugfixes**
- Update certificate bundle to match ``certifi`` 2015.9.6.2's weak certificate
bundle.
- Fix a bug in 2.8.0 where requests would raise ``ConnectTimeout`` instead of
``ConnectionError``
- When using the PreparedRequest flow, requests will now correctly respect the
``json`` parameter. Broken in 2.8.0.
- When using the PreparedRequest flow, requests will now correctly handle a
Unicode-string method name on Python 2. Broken in 2.8.0.
2.8.0 (2015-10-05)
++++++++++++++++++
**Minor Improvements** (Backwards Compatible)
- Requests now supports per-host proxies. This allows the ``proxies``
dictionary to have entries of the form
``{'<scheme>://<hostname>': '<proxy>'}``. Host-specific proxies will be used
in preference to the previously-supported scheme-specific ones, but the
previous syntax will continue to work.
- ``Response.raise_for_status`` now prints the URL that failed as part of the
exception message.
- ``requests.utils.get_netrc_auth`` now takes an ``raise_errors`` kwarg,
defaulting to ``False``. When ``True``, errors parsing ``.netrc`` files cause
exceptions to be thrown.
- Change to bundled projects import logic to make it easier to unbundle
requests downstream.
- Changed the default User-Agent string to avoid leaking data on Linux: now
contains only the requests version.
**Bugfixes**
- The ``json`` parameter to ``post()`` and friends will now only be used if
neither ``data`` nor ``files`` are present, consistent with the
documentation.
- We now ignore empty fields in the ``NO_PROXY`` environment variable.
- Fixed problem where ``httplib.BadStatusLine`` would get raised if combining
``stream=True`` with ``contextlib.closing``.
- Prevented bugs where we would attempt to return the same connection back to
the connection pool twice when sending a Chunked body.
- Miscellaneous minor internal changes.
- Digest Auth support is now thread safe.
**Updates**
- Updated urllib3 to 1.12.
2.7.0 (2015-05-03)
++++++++++++++++++
This is the first release that follows our new release process. For more, see
`our documentation
<http://docs.python-requests.org/en/latest/community/release-process/>`_.
**Bugfixes**
- Updated urllib3 to 1.10.4, resolving several bugs involving chunked transfer
encoding and response framing.
2.6.2 (2015-04-23)
++++++++++++++++++
**Bugfixes**
- Fix regression where compressed data that was sent as chunked data was not
properly decompressed. (#2561)
2.6.1 (2015-04-22)
++++++++++++++++++
**Bugfixes**
- Remove VendorAlias import machinery introduced in v2.5.2.
- Simplify the PreparedRequest.prepare API: We no longer require the user to
pass an empty list to the hooks keyword argument. (c.f. #2552)
- Resolve redirects now receives and forwards all of the original arguments to
the adapter. (#2503)
- Handle UnicodeDecodeErrors when trying to deal with a unicode URL that
cannot be encoded in ASCII. (#2540)
- Populate the parsed path of the URI field when performing Digest
Authentication. (#2426)
- Copy a PreparedRequest's CookieJar more reliably when it is not an instance
of RequestsCookieJar. (#2527)
2.6.0 (2015-03-14)
++++++++++++++++++
**Bugfixes**
- CVE-2015-2296: Fix handling of cookies on redirect. Previously a cookie
without a host value set would use the hostname for the redirected URL
exposing requests users to session fixation attacks and potentially cookie
stealing. This was disclosed privately by Matthew Daley of
`BugFuzz <https://bugfuzz.com>`_. This affects all versions of requests from
v2.1.0 to v2.5.3 (inclusive on both ends).
- Fix error when requests is an ``install_requires`` dependency and ``python
setup.py test`` is run. (#2462)
- Fix error when urllib3 is unbundled and requests continues to use the
vendored import location.
- Include fixes to ``urllib3``'s header handling.
- Requests' handling of unvendored dependencies is now more restrictive.
**Features and Improvements**
- Support bytearrays when passed as parameters in the ``files`` argument.
(#2468)
- Avoid data duplication when creating a request with ``str``, ``bytes``, or
``bytearray`` input to the ``files`` argument.
2.5.3 (2015-02-24)
++++++++++++++++++
**Bugfixes**
- Revert changes to our vendored certificate bundle. For more context see
(#2455, #2456, and http://bugs.python.org/issue23476)
2.5.2 (2015-02-23)
++++++++++++++++++
**Features and Improvements**
- Add sha256 fingerprint support. (`shazow/urllib3#540`_)
- Improve the performance of headers. (`shazow/urllib3#544`_)
**Bugfixes**
- Copy pip's import machinery. When downstream redistributors remove
requests.packages.urllib3 the import machinery will continue to let those
same symbols work. Example usage in requests' documentation and 3rd-party
libraries relying on the vendored copies of urllib3 will work without having
to fallback to the system urllib3.
- Attempt to quote parts of the URL on redirect if unquoting and then quoting
fails. (#2356)
- Fix filename type check for multipart form-data uploads. (#2411)
- Properly handle the case where a server issuing digest authentication
challenges provides both auth and auth-int qop-values. (#2408)
- Fix a socket leak. (`shazow/urllib3#549`_)
- Fix multiple ``Set-Cookie`` headers properly. (`shazow/urllib3#534`_)
- Disable the built-in hostname verification. (`shazow/urllib3#526`_)
- Fix the behaviour of decoding an exhausted stream. (`shazow/urllib3#535`_)
**Security**
- Pulled in an updated ``cacert.pem``.
- Drop RC4 from the default cipher list. (`shazow/urllib3#551`_)
.. _shazow/urllib3#551: https://github.com/shazow/urllib3/pull/551
.. _shazow/urllib3#549: https://github.com/shazow/urllib3/pull/549
.. _shazow/urllib3#544: https://github.com/shazow/urllib3/pull/544
.. _shazow/urllib3#540: https://github.com/shazow/urllib3/pull/540
.. _shazow/urllib3#535: https://github.com/shazow/urllib3/pull/535
.. _shazow/urllib3#534: https://github.com/shazow/urllib3/pull/534
.. _shazow/urllib3#526: https://github.com/shazow/urllib3/pull/526
2.5.1 (2014-12-23)
++++++++++++++++++
**Behavioural Changes**
- Only catch HTTPErrors in raise_for_status (#2382)
**Bugfixes**
- Handle LocationParseError from urllib3 (#2344)
- Handle file-like object filenames that are not strings (#2379)
- Unbreak HTTPDigestAuth handler. Allow new nonces to be negotiated (#2389)
2.5.0 (2014-12-01)
++++++++++++++++++
**Improvements**
- Allow usage of urllib3's Retry object with HTTPAdapters (#2216)
- The ``iter_lines`` method on a response now accepts a delimiter with which
to split the content (#2295)
**Behavioural Changes**
- Add deprecation warnings to functions in requests.utils that will be removed
in 3.0 (#2309)
- Sessions used by the functional API are always closed (#2326)
- Restrict requests to HTTP/1.1 and HTTP/1.0 (stop accepting HTTP/0.9) (#2323)
**Bugfixes**
- Only parse the URL once (#2353)
- Allow Content-Length header to always be overridden (#2332)
- Properly handle files in HTTPDigestAuth (#2333)
- Cap redirect_cache size to prevent memory abuse (#2299)
- Fix HTTPDigestAuth handling of redirects after authenticating successfully
(#2253)
- Fix crash with custom method parameter to Session.request (#2317)
- Fix how Link headers are parsed using the regular expression library (#2271)
**Documentation**
- Add more references for interlinking (#2348)
- Update CSS for theme (#2290)
- Update width of buttons and sidebar (#2289)
- Replace references of Gittip with Gratipay (#2282)
- Add link to changelog in sidebar (#2273)
2.4.3 (2014-10-06)
++++++++++++++++++
**Bugfixes**
- Unicode URL improvements for Python 2.
- Re-order JSON param for backwards compat.
- Automatically defrag authentication schemes from host/pass URIs. (`#2249 <https://github.com/kennethreitz/requests/issues/2249>`_)
2.4.2 (2014-10-05)
++++++++++++++++++
**Improvements**
- FINALLY! Add json parameter for uploads! (`#2258 <https://github.com/kennethreitz/requests/pull/2258>`_)
- Support for bytestring URLs on Python 3.x (`#2238 <https://github.com/kennethreitz/requests/pull/2238>`_)
**Bugfixes**
- Avoid getting stuck in a loop (`#2244 <https://github.com/kennethreitz/requests/pull/2244>`_)
- Multiple calls to iter* fail with unhelpful error. (`#2240 <https://github.com/kennethreitz/requests/issues/2240>`_, `#2241 <https://github.com/kennethreitz/requests/issues/2241>`_)
**Documentation**
- Correct redirection introduction (`#2245 <https://github.com/kennethreitz/requests/pull/2245/>`_)
- Added example of how to send multiple files in one request. (`#2227 <https://github.com/kennethreitz/requests/pull/2227/>`_)
- Clarify how to pass a custom set of CAs (`#2248 <https://github.com/kennethreitz/requests/pull/2248/>`_)
2.4.1 (2014-09-09)
++++++++++++++++++
- Now has a "security" package extras set, ``$ pip install requests[security]``
- Requests will now use Certifi if it is available.
- Capture and re-raise urllib3 ProtocolError
- Bugfix for responses that attempt to redirect to themselves forever (wtf?).
2.4.0 (2014-08-29)
++++++++++++++++++
**Behavioral Changes**
- ``Connection: keep-alive`` header is now sent automatically.
**Improvements**
- Support for connect timeouts! Timeout now accepts a tuple (connect, read) which is used to set individual connect and read timeouts.
- Allow copying of PreparedRequests without headers/cookies.
- Updated bundled urllib3 version.
- Refactored settings loading from environment -- new `Session.merge_environment_settings`.
- Handle socket errors in iter_content.
2.3.0 (2014-05-16)
++++++++++++++++++
**API Changes**
- New ``Response`` property ``is_redirect``, which is true when the
library could have processed this response as a redirection (whether
or not it actually did).
- The ``timeout`` parameter now affects requests with both ``stream=True`` and
``stream=False`` equally.
- The change in v2.0.0 to mandate explicit proxy schemes has been reverted.
Proxy schemes now default to ``http://``.
- The ``CaseInsensitiveDict`` used for HTTP headers now behaves like a normal
dictionary when references as string or viewed in the interpreter.
**Bugfixes**
- No longer expose Authorization or Proxy-Authorization headers on redirect.
Fix CVE-2014-1829 and CVE-2014-1830 respectively.
- Authorization is re-evaluated each redirect.
- On redirect, pass url as native strings.
- Fall-back to autodetected encoding for JSON when Unicode detection fails.
- Headers set to ``None`` on the ``Session`` are now correctly not sent.
- Correctly honor ``decode_unicode`` even if it wasn't used earlier in the same
response.
- Stop advertising ``compress`` as a supported Content-Encoding.
- The ``Response.history`` parameter is now always a list.
- Many, many ``urllib3`` bugfixes.
2.2.1 (2014-01-23)
++++++++++++++++++
**Bugfixes**
- Fixes incorrect parsing of proxy credentials that contain a literal or encoded '#' character.
- Assorted urllib3 fixes.
2.2.0 (2014-01-09)
++++++++++++++++++
@@ -14,7 +364,7 @@ Release History
**Bugfixes**
- Avoid many many exceptions from the buggy implementation of ``proxy_bypass`` on OS X in Python 2.6.
- Avoid crashing when attempting to get authentication credentions from ~/.netrc when running as a user without a home directory.
- Avoid crashing when attempting to get authentication credentials from ~/.netrc when running as a user without a home directory.
- Use the correct pool size for pools of connections to proxies.
- Fix iteration of ``CookieJar`` objects.
- Ensure that cookies are persisted over redirect.
@@ -120,6 +470,8 @@ Release History
1.2.1 (2013-05-20)
++++++++++++++++++
- 301 and 302 redirects now change the verb to GET for all verbs, not just
POST, improving browser compatibility.
- Python 3.3.2 compatibility
- Always percent-encode location headers
- Fix connection adapter matching to be most-specific first
@@ -204,7 +556,7 @@ This is not a backwards compatible change.
- Improved mime-compatible JSON handling
- Proxy fixes
- Path hack fixes
- Case-Insensistive Content-Encoding headers
- Case-Insensitive Content-Encoding headers
- Support for CJK parameters in form posts
@@ -240,7 +592,7 @@ This is not a backwards compatible change.
- Digest Authentication improvements.
- Ensure proxy exclusions work properly.
- Clearer UnicodeError exceptions.
- Automatic casting of URLs to tsrings (fURL and such)
- Automatic casting of URLs to strings (fURL and such)
- Bugfixes.
0.13.6 (2012-08-06)
@@ -291,8 +643,8 @@ This is not a backwards compatible change.
+++++++++++++++++++
- Removal of Requests.async in favor of `grequests <https://github.com/kennethreitz/grequests>`_
- Allow disabling of cookie persistiance.
- New implimentation of safe_mode
- Allow disabling of cookie persistence.
- New implementation of safe_mode
- cookies.get now supports default argument
- Session cookies not saved when Session.request is called with return_response=False
- Env: no_proxy support.
@@ -409,7 +761,7 @@ This is not a backwards compatible change.
* ``Response.content`` is now bytes-only. (*Backwards Incompatible*)
* New ``Response.text`` is unicode-only.
* If no ``Response.encoding`` is specified and ``chardet`` is available, ``Respoonse.text`` will guess an encoding.
* If no ``Response.encoding`` is specified and ``chardet`` is available, ``Response.text`` will guess an encoding.
* Default to ISO-8859-1 (Western) encoding for "text" subtypes.
* Removal of `decode_unicode`. (*Backwards Incompatible*)
* New multiple-hooks system.
@@ -529,7 +881,7 @@ This is not a backwards compatible change.
0.7.5 (2011-11-04)
++++++++++++++++++
* Response.content = None if there was an invalid repsonse.
* Response.content = None if there was an invalid response.
* Redirection auth handling.
0.7.4 (2011-10-26)
@@ -616,7 +968,7 @@ This is not a backwards compatible change.
++++++++++++++++++
* New callback hook system
* New persistient sessions object and context manager
* New persistent sessions object and context manager
* Transparent Dict-cookie handling
* Status code reference object
* Removed Response.cached
@@ -650,7 +1002,7 @@ This is not a backwards compatible change.
* Redirect Fixes
* settings.verbose stream writing
* Querystrings for all methods
* URLErrors (Connection Refused, Timeout, Invalid URLs) are treated as explicity raised
* URLErrors (Connection Refused, Timeout, Invalid URLs) are treated as explicitly raised
``r.requests.get('hwe://blah'); r.raise_for_status()``
LICENSE
+1 -1
View File
@@ -1,4 +1,4 @@
Copyright 2014 Kenneth Reitz
Copyright 2015 Kenneth Reitz
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Makefile
+5 -9
View File
@@ -4,7 +4,9 @@ init:
pip install -r requirements.txt
test:
py.test
# This runs all of the tests. To run an individual test, run py.test with
# the -k flag, like "py.test -k test_path_is_not_double_encoded"
py.test test_requests.py
coverage:
py.test --verbose --cov-report term --cov=requests test_requests.py
@@ -18,16 +20,10 @@ certs:
deps: urllib3 chardet
urllib3:
rm -fr requests/packages/urllib3
git clone https://github.com/shazow/urllib3.git
mv urllib3/urllib3 requests/packages/
rm -fr urllib3
git clone https://github.com/shazow/urllib3.git && rm -fr requests/packages/urllib3 && mv urllib3/urllib3 requests/packages/ && rm -fr urllib3
chardet:
rm -fr requests/packages/chardet
git clone https://github.com/chardet/chardet.git
mv chardet/chardet requests/packages/
rm -fr chardet
git clone https://github.com/chardet/chardet.git && rm -fr requests/packages/chardet && mv chardet/chardet requests/packages/ && rm -fr chardet
publish:
python setup.py register
NOTICE
+23 -64
View File
@@ -8,24 +8,27 @@ This is the MIT license: http://www.opensource.org/licenses/mit-license.php
Copyright 2008-2011 Andrey Petrov and contributors (see CONTRIBUTORS.txt),
Modifications copyright 2012 Kenneth Reitz.
Permission is hereby granted, free of charge, to any person obtaining a copy of this
software and associated documentation files (the "Software"), to deal in the Software
without restriction, including without limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
to whom the Software is furnished to do so, subject to the following conditions:
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Charade License
================
Chardet License
===============
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
@@ -43,53 +46,9 @@ Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
02110-1301 USA
CA Bundle License
=================
Bundle of CA Root Certificates
==============================
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
02110-1301
OAuthlib
========
Copyright (c) 2011 Idan Gazit and contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the name of this project nor the names of its contributors may
be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
README.rst
+7 -14
View File
@@ -1,11 +1,13 @@
Requests: HTTP for Humans
=========================
.. image:: https://badge.fury.io/py/requests.png
:target: http://badge.fury.io/py/requests
.. image:: https://img.shields.io/pypi/v/requests.svg
:target: https://pypi.python.org/pypi/requests
.. image:: https://img.shields.io/pypi/dm/requests.svg
:target: https://pypi.python.org/pypi/requests
.. image:: https://pypip.in/d/requests/badge.png
:target: https://crate.io/packages/requests/
Requests is an Apache2 Licensed HTTP library, written in Python, for human
@@ -19,7 +21,7 @@ perform the simplest of tasks.
Things shouldn't be this way. Not in Python.
.. code-block:: pycon
.. code-block:: python
>>> r = requests.get('https://api.github.com', auth=('user', 'pass'))
>>> r.status_code
@@ -64,14 +66,6 @@ To install Requests, simply:
$ pip install requests
Or, if you absolutely must:
.. code-block:: bash
$ easy_install requests
But, you really shouldn't do that.
Documentation
-------------
@@ -83,7 +77,6 @@ Contribute
----------
#. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a `Contributor Friendly`_ tag for issues that should be ideal for people who are not very familiar with the codebase yet.
#. If you feel uncomfortable or uncertain about an issue or your changes, feel free to email @sigmavirus24 and he will happily help you via email, Skype, remote pairing or whatever you are comfortable with.
#. Fork `the repository`_ on GitHub to start making your changes to the **master** branch (or branch off of it).
#. Write a test which shows that the bug was fixed or that the feature works as expected.
#. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS_.
docs/_templates/sidebarintro.html
Vendored
+12 -13
View File
@@ -15,23 +15,17 @@
</p>
<p>
<a href="https://gumroad.com/l/RRZc" class="gumroad-button">Buy Requests Pro</a>
</p>
<h3>Get Updates</h3>
<p>Receive updates on new releases and upcoming projects.</p>
<p><a href="http://tinyletter.com/kennethreitz">Subscribe to Newsletter</a></p>
<h3>Get Support</h3>
<p>
If your organization uses Requests, consider financial support:
</p>
<p>
<a href="http://gum.co/RRZc" class="gumroad-button">Requests Pro</a><script type="text/javascript" src="https://gumroad.com/js/gumroad-button.js"></script>
</p>
<h3>Translations</h3>
<ul>
<li><a href="http://docs.python-requests.org/">English</a></li>
@@ -40,13 +34,18 @@
<li><a href="http://jp.python-requests.org/">Japanese</a></li>
<li><a href="http://cn.python-requests.org/">Chinese</a></li>
<li><a href="http://pt.python-requests.org/">Portuguese</a></li>
<li><a href="http://it.python-requests.org/">Italian</a></li>
<li><a href="http://es.python-requests.org/">Spanish</a></li>
</ul>
<h3>Useful Links</h3>
<ul>
<li><a href="http://docs.python-requests.org/en/latest/community/recommended/">Recommended Packages and Extensions</a></li>
<p></p>
<li><a href="http://github.com/kennethreitz/requests">Requests @ GitHub</a></li>
<li><a href="http://pypi.python.org/pypi/requests">Requests @ PyPI</a></li>
<li><a href="http://github.com/kennethreitz/requests/issues">Issue Tracker</a></li>
<li><a href="http://docs.python-requests.org/en/latest/community/updates/#software-updates">Release History</a></li>
</ul>
docs/_templates/sidebarlogo.html
Vendored
+4 -8
View File
@@ -14,17 +14,13 @@
development release.
</p>
<p>
<a href="https://gumroad.com/l/RRZc" class="gumroad-button">Buy Requests Pro</a>
</p>
<h3>Get Updates</h3>
<p>Receive updates on new releases and upcoming projects.</p>
<p><a href="http://tinyletter.com/kennethreitz">Subscribe to Newsletter</a></p>
<h3>Get Support</h3>
<p>
If your organization uses Requests, consider financial support:
</p>
<p>
<a href="http://gum.co/RRZc" class="gumroad-button">Requests Pro</a><script type="text/javascript" src="https://gumroad.com/js/gumroad-button.js"></script>
</p>
docs/_themes/kr/layout.html
Vendored
+13
View File
@@ -15,6 +15,9 @@
<a href="https://github.com/kennethreitz/requests" class="github">
<img style="position: absolute; top: 0; right: 0; border: 0;" src="http://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" class="github"/>
</a>
<script type="text/javascript" src="https://gumroad.com/js/gumroad.js"></script>
<script type="text/javascript">
/* <![CDATA[ */
(function() {
@@ -68,5 +71,15 @@
})();
</script>
<script type="text/javascript">
(function() {
window._pa = window._pa || {};
_pa.productId = "requests-docs";
var pa = document.createElement('script'); pa.type = 'text/javascript'; pa.async = true;
pa.src = ('https:' == document.location.protocol ? 'https:' : 'http:') + "//tag.perfectaudience.com/serve/5226171f87bc6890da0000a0.js";
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(pa, s);
})();
</script>
{%- endblock %}
docs/api.rst
+21 -19
View File
@@ -5,7 +5,7 @@ Developer Interface
.. module:: requests
This part of the documentation covers all the interfaces of Requests. For
This part of the documentation covers all the interfaces of Requests. For
parts where Requests depends on external libraries, we document the most
important right here and provide links to the canonical documentation.
@@ -44,6 +44,13 @@ Request Sessions
.. autoclass:: requests.adapters.HTTPAdapter
:inherited-members:
Authentication
--------------
.. autoclass:: requests.auth.AuthBase
.. autoclass:: requests.auth.HTTPBasicAuth
.. autoclass:: requests.auth.HTTPProxyAuth
.. autoclass:: requests.auth.HTTPDigestAuth
Exceptions
~~~~~~~~~~
@@ -53,6 +60,9 @@ Exceptions
.. autoexception:: requests.exceptions.HTTPError
.. autoexception:: requests.exceptions.URLRequired
.. autoexception:: requests.exceptions.TooManyRedirects
.. autoexception:: requests.exceptions.ConnectTimeout
.. autoexception:: requests.exceptions.ReadTimeout
.. autoexception:: requests.exceptions.Timeout
Status Code Lookup
@@ -71,6 +81,8 @@ Status Code Lookup
>>> requests.codes['\o/']
200
.. _api-cookies:
Cookies
~~~~~~~
@@ -78,6 +90,12 @@ Cookies
.. autofunction:: requests.utils.cookiejar_from_dict
.. autofunction:: requests.utils.add_dict_to_cookiejar
.. autoclass:: requests.cookies.RequestsCookieJar
:inherited-members:
.. autoclass:: requests.cookies.CookieConflictError
:inherited-members:
Encodings
~~~~~~~~~
@@ -239,27 +257,11 @@ API Changes
requests.get("http://example.org", proxies=proxies)
Behavioral Changes
~~~~~~~~~~~~~~~~~~
Behavioural Changes
~~~~~~~~~~~~~~~~~~~~~~~
* Keys in the ``headers`` dictionary are now native strings on all Python
versions, i.e. bytestrings on Python 2 and unicode on Python 3. If the
keys are not native strings (unicode on Python2 or bytestrings on Python 3)
they will be converted to the native string type assuming UTF-8 encoding.
* Timeouts behave slightly differently. On streaming requests, the timeout
only applies to the connection attempt. On regular requests, the timeout
is applied to the connection process and downloading the full body.
::
tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'
# One second timeout for the connection attempt
# Unlimited time to download the tarball
r = requests.get(tarball_url, stream=True, timeout=1)
# One second timeout for the connection attempt
# Another full second timeout to download the tarball
r = requests.get(tarball_url, timeout=1)
docs/community/faq.rst
+26
View File
@@ -59,4 +59,30 @@ supported:
* Python 3.1
* Python 3.2
* Python 3.3
* Python 3.4
* PyPy 1.9
* PyPy 2.2
What are "hostname doesn't match" errors?
-----------------------------------------
These errors occur when :ref:`SSL certificate verification <verification>`
fails to match the certificate the server responds with to the hostname
Requests thinks it's contacting. If you're certain the server's SSL setup is
correct (for example, because you can visit the site with your browser) and
you're using Python 2.6 or 2.7, a possible explanation is that you need
Server-Name-Indication.
`Server-Name-Indication`_, or SNI, is an official extension to SSL where the
client tells the server what hostname it is contacting. This is important
when servers are using `Virtual Hosting`_. When such servers are hosting
more than one SSL site they need to be able to return the appropriate
certificate based on the hostname the client is connecting to.
Python3 and Python 2.7.9+ include native support for SNI in their SSL modules.
For information on using SNI with Requests on Python < 2.7.9 refer to this
`Stack Overflow answer`_.
.. _`Server-Name-Indication`: https://en.wikipedia.org/wiki/Server_Name_Indication
.. _`virtual hosting`: https://en.wikipedia.org/wiki/Virtual_hosting
.. _`Stack Overflow answer`: https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484
docs/community/out-there.rst
+2 -2
View File
@@ -25,9 +25,9 @@ To give it a try, simply::
Articles & Talks
================
- `Python for the Web <http://gun.io/blog/python-for-the-web/>`_ teaches how to use Python to interact with the web, using Requests.
- `Daniel Greenfield's Review of Requests <http://pydanny.blogspot.com/2011/05/python-http-requests-for-humans.html>`_
- `Daniel Greenfeld's Review of Requests <http://pydanny.blogspot.com/2011/05/python-http-requests-for-humans.html>`_
- `My 'Python for Humans' talk <http://python-for-humans.heroku.com>`_ ( `audio <http://codeconf.s3.amazonaws.com/2011/pycodeconf/talks/PyCodeConf2011%20-%20Kenneth%20Reitz.m4a>`_ )
- `Issac Kelly's 'Consuming Web APIs' talk <http://issackelly.github.com/Consuming-Web-APIs-with-Python-Talk/slides/slides.html>`_
- `Blog post about Requests via Yum <http://arunsag.wordpress.com/2011/08/17/new-package-python-requests-http-for-humans/>`_
- `Russian blog post introducing Requests <http://habrahabr.ru/blogs/python/126262/>`_
- `French blog post introducing Requests <http://www.nicosphere.net/requests-urllib2-de-python-simplifie-2432/>`_
- `Sending JSON in Requests <http://www.coglib.com/~icordasc/blog/2014/11/sending-json-in-requests.html>`_
docs/community/recommended.rst
+57
View File
@@ -0,0 +1,57 @@
.. _recommended:
Recommended Packages and Extensions
===================================
Requests has a great variety of powerful and useful third-party extensions.
This page provides an overview of some of the best of them.
Certifi CA Bundle
-----------------
`Certifi`_ is a carefully curated collection of Root Certificates for
validating the trustworthiness of SSL certificates while verifying the
identity of TLS hosts. It has been extracted from the Requests project.
.. _Certifi: http://certifi.io/en/latest/
CacheControl
------------
`CacheControl`_ is an extension that adds a full HTTP cache to Requests. This
makes your web requests substantially more efficient, and should be used
whenever you're making a lot of web requests.
.. _CacheControl: https://cachecontrol.readthedocs.org/en/latest/
Requests-Toolbelt
-----------------
`Requests-Toolbelt`_ is a collection of utilities that some users of Requests may desire,
but do not belong in Requests proper. This library is actively maintained
by members of the Requests core team, and reflects the functionality most
requested by users within the community.
.. _Requests-Toolbelt: http://toolbelt.readthedocs.org/en/latest/index.html
Requests-OAuthlib
-----------------
`requests-oauthlib`_ makes it possible to do the OAuth dance from Requests
automatically. This is useful for the large number of websites that use OAuth
to provide authentication. It also provides a lot of tweaks that handle ways
that specific OAuth providers differ from the standard specifications.
.. _requests-oauthlib: https://requests-oauthlib.readthedocs.org/en/latest/
Betamax
-------
`Betamax`_ records your HTTP interactions so the NSA does not have to.
A VCR imitation designed only for Python-Requests.
.. _betamax: https://github.com/sigmavirus24/betamax
docs/community/release-process.rst
+54
View File
@@ -0,0 +1,54 @@
Release Process and Rules
=========================
.. versionadded:: v2.6.2
Starting with the version to be released after ``v2.6.2``, the following rules
will govern and describe how the Requests core team produces a new release.
Major Releases
--------------
A major release will include breaking changes. When it is versioned, it will
be versioned as ``vX.0.0``. For example, if the previous release was
``v10.2.7`` the next version will be ``v11.0.0``.
Breaking changes are changes that break backwards compatibility with prior
versions. If the project were to change the ``text`` attribute on a
``Response`` object to a method, that would only happen in a Major release.
Major releases may also include miscellaneous bug fixes and upgrades to
vendored packages. The core developers of Requests are committed to providing
a good user experience. This means we're also committed to preserving
backwards compatibility as much as possible. Major releases will be infrequent
and will need strong justifications before they are considered.
Minor Releases
--------------
A minor release will not include breaking changes but may include
miscellaneous bug fixes and upgrades to vendored packages. If the previous
version of Requests released was ``v10.2.7`` a minor release would be
versioned as ``v10.3.0``.
Minor releases will be backwards compatible with releases that have the same
major version number. In other words, all versions that would start with
``v10.`` should be compatible with each other.
Hotfix Releases
---------------
A hotfix release will only include bug fixes that were missed when the project
released the previous version. If the previous version of Requests released
``v10.2.7`` the hotfix release would be versioned as ``v10.2.8``.
Hotfixes will **not** include upgrades to vendored dependences after
``v2.6.2``
Reasoning
---------
In the 2.5 and 2.6 release series, the Requests core team upgraded vendored
dependencies and caused a great deal of headaches for both users and the core
team. To reduce this pain, we're forming a concrete set of procedures so
expectations will be properly set.
docs/community/support.rst
+18 -4
View File
@@ -5,17 +5,26 @@ Support
If you have questions or issues about Requests, there are several options:
StackOverflow
-------------
If your question does not contain sensitive (possibly proprietary)
information or can be properly anonymized, please ask a question on
`StackOverflow <https://stackoverflow.com/questions/tagged/python-requests>`_
and use the tag ``python-requests``.
Send a Tweet
------------
If your question is less than 140 characters, feel free to send a tweet to
`@kennethreitz <http://twitter.com/kennethreitz>`_.
`@kennethreitz <https://twitter.com/kennethreitz>`_,
`@sigmavirus24 <https://twitter.com/sigmavirus24>`_, or
`@lukasaoz <https://twitter.com/lukasaoz>`_.
File an Issue
-------------
If you notice some unexpected behavior in Requests, or want to see support
If you notice some unexpected behaviour in Requests, or want to see support
for a new feature,
`file an issue on GitHub <https://github.com/kennethreitz/requests/issues>`_.
@@ -34,4 +43,9 @@ IRC
The official Freenode channel for Requests is
`#python-requests <irc://irc.freenode.net/python-requests>`_
I'm also available as **kennethreitz** on Freenode.
The core developers of requests are on IRC throughout the day.
You can find them in ``#python-requests`` as:
- kennethreitz
- lukasa
- sigmavirus24
docs/community/updates.rst
+3 -14
View File
@@ -1,8 +1,6 @@
.. _updates:
Community Updates
=================
@@ -19,23 +17,14 @@ The best way to track the development of Requests is through
Twitter
-------
I often tweet about new features and releases of Requests.
The author, Kenneth Reitz, often tweets about new features and releases of Requests.
Follow `@kennethreitz <https://twitter.com/kennethreitz>`_ for updates.
Mailing List
------------
There's a low-volume mailing list for Requests. To subscribe to the
mailing list, send an email to
`requests@librelist.org <mailto:requests@librelist.org>`_.
Software Updates
================
Release and Version History
===========================
.. include:: ../../HISTORY.rst
docs/community/vulnerabilities.rst
+105
View File
@@ -0,0 +1,105 @@
Vulnerability Disclosure
========================
If you think you have found a potential security vulnerability in requests,
please email `sigmavirus24 <mailto:graffatcolmingov@gmail.com>`_ and
`Lukasa <mailto:cory@lukasa.co.uk>`_ directly. **Do not file a public issue.**
Our PGP Key fingerprints are:
- 0161 BB7E B208 B5E0 4FDC 9F81 D9DA 0A04 9113 F853 (@sigmavirus24)
- 90DC AE40 FEA7 4B14 9B70 662D F25F 2144 EEC1 373D (@lukasa)
If English is not your first language, please try to describe the problem and
its impact to the best of your ability. For greater detail, please use your
native language and we will try our best to translate it using online services.
Please also include the code you used to find the problem and the shortest
amount of code necessary to reproduce it.
Please do not disclose this to anyone else. We will retrieve a CVE identifier
if necessary and give you full credit under whatever name or alias you provide.
We will only request an identifier when we have a fix and can publish it in a
release.
We will respect your privacy and will only publicize your involvement if you
grant us permission.
Process
-------
This following information discusses the process the requests project follows
in response to vulnerability disclosures. If you are disclosing a
vulnerability, this section of the documentation lets you know how we will
respond to your disclosure.
Timeline
~~~~~~~~
When you report an issue, one of the project members will respond to you within
two days *at the outside*. In most cases responses will be faster, usually
within 12 hours. This initial response will at the very least confirm receipt
of the report.
If we were able to rapidly reproduce the issue, the initial response will also
contain confirmation of the issue. If we are not, we will often ask for more
information about the reproduction scenario.
Our goal is to have a fix for any vulnerability released within two weeks of
the initial disclosure. This may potentially involve shipping an interim
release that simply disables function while a more mature fix can be prepared,
but will in the vast majority of cases mean shipping a complete release as soon
as possible.
Throughout the fix process we will keep you up to speed with how the fix is
progressing. Once the fix is prepared, we will notify you that we believe we
have a fix. Often we will ask you to confirm the fix resolves the problem in
your environment, especially if we are not confident of our reproduction
scenario.
At this point, we will prepare for the release. We will obtain a CVE number
if one is required, providing you with full credit for the discovery. We will
also decide on a planned release date, and let you know when it is. This
release date will *always* be on a weekday.
At this point we will reach out to our major downstream packagers to notify
them of an impending security-related patch so they can make arrangements. In
addition, these packagers will be provided with the intended patch ahead of
time, to ensure that they are able to promptly release their downstream
packages. Currently the list of people we actively contact *ahead of a public
release* is:
- Ralph Bean, Red Hat (@ralphbean)
- Daniele Tricoli, Debian (@eriol)
We will notify these individuals at least a week ahead of our planned release
date to ensure that they have sufficient time to prepare. If you believe you
should be on this list, please let one of the maintainers know at one of the
email addresses at the top of this article.
On release day, we will push the patch to our public repository, along with an
updated changelog that describes the issue and credits you. We will then issue
a PyPI release containing the patch.
At this point, we will publicise the release. This will involve mails to
mailing lists, Tweets, and all other communication mechanisms available to the
core team.
We will also explicitly mention which commits contain the fix to make it easier
for other distributors and users to easily patch their own versions of requests
if upgrading is not an option.
Previous CVEs
-------------
- Fixed in 2.6.0
- `CVE 2015-2296 <http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2015-2296>`_,
reported by Matthew Daley of `BugFuzz <https://bugfuzz.com/>`_.
- Fixed in 2.3.0
- `CVE 2014-1829 <http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2014-1829>`_
- `CVE 2014-1830 <http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2014-1830>`_
docs/conf.py
+1 -1
View File
@@ -46,7 +46,7 @@ master_doc = 'index'
# General information about the project.
project = u'Requests'
copyright = u'2014. A <a href="http://kennethreitz.com/pages/open-projects.html">Kenneth Reitz</a> Project'
copyright = u'2015. A <a href="http://kennethreitz.com/pages/open-projects.html">Kenneth Reitz</a> Project'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
docs/dev/contributing.rst
+145
View File
@@ -0,0 +1,145 @@
.. _contributing:
Contributor's Guide
===================
If you're reading this you're probably interested in contributing to
Requests. First, We'd like to say: thank you! Open source projects
live-and-die based on the support they receive from others, and the fact that
you're even considering supporting Requests is very generous of
you.
This document lays out guidelines and advice for contributing to Requests.
If you're thinking of contributing, start by reading this thoroughly and
getting a feel for how contributing to the project works. If you have any
questions, feel free to reach out to either `Ian Cordasco`_ or `Cory Benfield`_,
the primary maintainers.
The guide is split into sections based on the type of contribution you're
thinking of making, with a section that covers general guidelines for all
contributors.
.. _Ian Cordasco: http://www.coglib.com/~icordasc/
.. _Cory Benfield: https://lukasa.co.uk/about
All Contributions
-----------------
Be Cordial
~~~~~~~~~~
**Be cordial or be on your way.**
Requests has one very important rule governing all forms of contribution,
including reporting bugs or requesting features. This golden rule is
`be cordial or be on your way`_. **All contributions are welcome**, as long as
everyone involved is treated with respect.
.. _be cordial or be on your way: http://kennethreitz.org/be-cordial-or-be-on-your-way/
.. _early-feedback:
Get Early Feedback
~~~~~~~~~~~~~~~~~~
If you are contributing, do not feel the need to sit on your contribution until
it is perfectly polished and complete. It helps everyone involved for you to
seek feedback as early as you possibly can. Submitting an early, unfinished
version of your contribution for feedback in no way prejudices your chances of
getting that contribution accepted, and can save you from putting a lot of work
into a contribution that is not suitable for the project.
Contribution Suitability
~~~~~~~~~~~~~~~~~~~~~~~~
The project maintainer has the last word on whether or not a contribution is
suitable for Requests. All contributions will be considered, but from time
to time contributions will be rejected because they do not suit the project.
If your contribution is rejected, don't despair! So long as you followed these
guidelines, you'll have a much better chance of getting your next contribution
accepted.
Code Contributions
------------------
Steps
~~~~~
When contributing code, you'll want to follow this checklist:
1. Fork the repository on GitHub.
2. Run the tests to confirm they all pass on your system. If they don't, you'll
need to investigate why they fail. If you're unable to diagnose this
yourself, raise it as a bug report by following the guidelines in this
document: :ref:`bug-reports`.
3. Write tests that demonstrate your bug or feature. Ensure that they fail.
4. Make your change.
5. Run the entire test suite again, confirming that all tests pass *including
the ones you just added*.
6. Send a GitHub Pull Request to the main repository's ``master`` branch.
GitHub Pull Requests are the expected method of code collaboration on this
project.
The following sub-sections go into more detail on some of the points above.
Code Review
~~~~~~~~~~~
Contributions will not be merged until they've been code reviewed. You should
implement any code review feedback unless you strongly object to it. In the
event that you object to the code review feedback, you should make your case
clearly and calmly. If, after doing so, the feedback is judged to still apply,
you must either apply the feedback or withdraw your contribution.
New Contributors
~~~~~~~~~~~~~~~~
If you are new or relatively new to Open Source, welcome! Requests aims to
be a gentle introduction to the world of Open Source. If you're concerned about
how best to contribute, please consider mailing a maintainer (listed above) and
asking for help.
Please also check the :ref:`early-feedback` section.
Documentation Contributions
---------------------------
Documentation improvements are always welcome! The documentation files live in
the ``docs/`` directory of the codebase. They're written in
`reStructuredText`_, and use `Sphinx`_ to generate the full suite of
documentation.
When contributing documentation, please attempt to follow the style of the
documentation files. This means a soft-limit of 79 characters wide in your text
files and a semi-formal prose style.
.. _reStructuredText: http://docutils.sourceforge.net/rst.html
.. _Sphinx: http://sphinx-doc.org/index.html
.. _bug-reports:
Bug Reports
-----------
Bug reports are hugely important! Before you raise one, though, please check
through the `GitHub issues`_, **both open and closed**, to confirm that the bug
hasn't been reported before. Duplicate bug reports are a huge drain on the time
of other contributors, and should be avoided as much as possible.
.. _GitHub issues: https://github.com/kennethreitz/requests/issues
Feature Requests
----------------
Requests is in a perpetual feature freeze. The maintainers believe that
requests contains every major feature currently required by the vast majority
of users.
If you believe there is a feature missing, feel free to raise a feature
request, but please do be aware that the overwhelming likelihood is that your
feature request will not be accepted.
docs/dev/internals.rst
View File
docs/dev/philosophy.rst
+5 -3
View File
@@ -4,10 +4,12 @@ Development Philosophy
Requests is an open but opinionated library, created by an open but opinionated developer.
Benevolent Dictator
~~~~~~~~~~~~~~~~~~~
Management Style
~~~~~~~~~~~~~~~~
`Kenneth Reitz <http://kennethreitz.org>`_ is the BDFL. He has final say in any decision related to Requests.
`Kenneth Reitz <http://kennethreitz.org>`_ is the BDFL. He has final say in any decision related to the Requests project. Kenneth is responsible for the direction and form of the library. In addition to making decisions based on technical merit, he is responsible for making decisions based on the development philosophy of Requests. Only Kenneth may merge code into Requests.
`Ian Cordasco <http://www.coglib.com/~icordasc/>`_ and `Cory Benfield <https://lukasa.co.uk/about/>`_ are the core contributors. They are responsible for triaging bug reports, reviewing pull requests and ensuring that Kenneth is kept up to speed with developments around the library. The day-to-day managing of the project is done by the core contributors. They are responsible for making judgements about whether or not a feature request is likely to be accepted by Kenneth. They do not have the authority to change code or merge code changes, though they may change documentation. Their word is not final.
Values
~~~~~~
docs/dev/todo.rst
+1 -2
View File
@@ -24,8 +24,7 @@ Development Dependencies
You'll need to install py.test in order to run the Requests' test suite::
$ pip install -r requirements.txt
$ invoke test
py.test
$ py.test
platform darwin -- Python 2.7.3 -- pytest-2.3.4
collected 25 items
docs/index.rst
+20 -6
View File
@@ -8,11 +8,14 @@ Requests: HTTP for Humans
Release v\ |version|. (:ref:`Installation <install>`)
Requests is an :ref:`Apache2 Licensed <apache2>` HTTP library, written in Python, for human beings.
Requests is an :ref:`Apache2 Licensed <apache2>` HTTP library, written in
Python, for human beings.
Python's standard **urllib2** module provides most of
the HTTP capabilities you need, but the API is thoroughly **broken**.
It was built for a different time — and a different web. It requires an *enormous* amount of work (even method overrides) to perform the simplest of tasks.
It was built for a different time — and a different web. It requires an
*enormous* amount of work (even method overrides) to perform the simplest of
tasks.
Things shouldn’t be this way. Not in Python.
@@ -32,13 +35,20 @@ Things shouldn’t be this way. Not in Python.
See `similar code, without Requests <https://gist.github.com/973705>`_.
Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There's no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by `urllib3 <https://github.com/shazow/urllib3>`_, which is embedded within Requests.
Requests takes all of the work out of Python HTTP/1.1 — making your integration
with web services seamless. There's no need to manually add query strings to
your URLs, or to form-encode your POST data. Keep-alive and HTTP connection
pooling are 100% automatic, powered by `urllib3 <https://github.com/shazow/urllib3>`_,
which is embedded within Requests.
Testimonials
------------
Her Majesty's Government, Amazon, Google, Twilio, Runscope, Mozilla, Heroku, PayPal, NPR, Obama for America, Transifex, Native Instruments, The Washington Post, Twitter, SoundCloud, Kippt, Readability, and Federal US Institutions use Requests internally. It has been downloaded over 8,000,000 times from PyPI.
Her Majesty's Government, Amazon, Google, Twilio, Runscope, Mozilla, Heroku,
PayPal, NPR, Obama for America, Transifex, Native Instruments, The Washington
Post, Twitter, SoundCloud, Kippt, Readability, Sony, and Federal US Institutions that prefer to be unnamed
use Requests internally. It has been downloaded over 60,000,000 times from PyPI.
**Armin Ronacher**
Requests is the perfect example how beautiful an API can be with the
@@ -73,7 +83,7 @@ Requests is ready for today's web.
- Multipart File Uploads
- Connection Timeouts
- ``.netrc`` support
- Python 2.6—3.3
- Python 2.6—3.4
- Thread-safe.
@@ -104,9 +114,12 @@ Requests ecosystem and community.
:maxdepth: 1
community/faq
community/out-there.rst
community/recommended
community/out-there
community/support
community/vulnerabilities
community/updates
community/release-process
API Documentation
-----------------
@@ -129,6 +142,7 @@ you.
.. toctree::
:maxdepth: 1
dev/contributing
dev/philosophy
dev/todo
dev/authors
docs/user/advanced.rst
+267 -38
View File
@@ -5,13 +5,17 @@ Advanced Usage
This document covers some of Requests more advanced features.
.. _session-objects:
Session Objects
---------------
The Session object allows you to persist certain parameters across
requests. It also persists cookies across all requests made from the
Session instance.
Session instance, and will use ``urllib3``'s `connection pooling`_. So if
you're making several requests to the same host, the underlying TCP
connection will be reused, which can result in a significant performance
increase (see `HTTP persistent connection`_).
A Session object has all the methods of the main Requests API.
@@ -37,13 +41,47 @@ is done by providing data to the properties on a Session object::
s.get('http://httpbin.org/headers', headers={'x-test2': 'true'})
Any dictionaries that you pass to a request method will be merged with the session-level values that are set. The method-level parameters override session parameters.
Any dictionaries that you pass to a request method will be merged with the
session-level values that are set. The method-level parameters override session
parameters.
Note, however, that method-level parameters will *not* be persisted across
requests, even if using a session. This example will only send the cookies
with the first request, but not the second::
s = requests.Session()
r = s.get('http://httpbin.org/cookies', cookies={'from-my': 'browser'})
print(r.text)
# '{"cookies": {"from-my": "browser"}}'
r = s.get('http://httpbin.org/cookies')
print(r.text)
# '{"cookies": {}}'
If you want to manually add cookies to your session, use the
:ref:`Cookie utility functions <api-cookies>` to manipulate
:attr:`Session.cookies <requests.Session.cookies>`.
Sessions can also be used as context managers::
with requests.Session() as s:
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
This will make sure the session is closed as soon as the ``with`` block is
exited, even if unhandled exceptions occurred.
.. admonition:: Remove a Value From a Dict Parameter
Sometimes you'll want to omit session-level keys from a dict parameter. To do this, you simply set that key's value to ``None`` in the method-level parameter. It will automatically be omitted.
Sometimes you'll want to omit session-level keys from a dict parameter. To
do this, you simply set that key's value to ``None`` in the method-level
parameter. It will automatically be omitted.
All values that are contained within a session are directly available to you. See the :ref:`Session API Docs <sessionapi>` to learn more.
All values that are contained within a session are directly available to you.
See the :ref:`Session API Docs <sessionapi>` to learn more.
.. _request-and-response-objects:
Request and Response Objects
----------------------------
@@ -77,10 +115,12 @@ request, and then the request's headers::
{'Accept-Encoding': 'identity, deflate, compress, gzip',
'Accept': '*/*', 'User-Agent': 'python-requests/1.2.0'}
.. _prepared-requests:
Prepared Requests
-----------------
Whenever you receive a :class:`Response <requests.models.Response>` object
Whenever you receive a :class:`Response <requests.Response>` object
from an API call or a Session call, the ``request`` attribute is actually the
``PreparedRequest`` that was used. In some cases you may wish to do some extra
work to the body or headers (or anything else really) before sending a
@@ -111,13 +151,13 @@ request. The simple recipe for this is the following::
Since you are not doing anything special with the ``Request`` object, you
prepare it immediately and modify the ``PreparedRequest`` object. You then
send that with the other parameters you would have sent to ``requests.*`` or
``Sesssion.*``.
``Session.*``.
However, the above code will lose some of the advantages of having a Requests
:class:`Session <requests.Session>` object. In particular,
:class:`Session <requests.Session>`-level state such as cookies will
not get applied to your request. To get a
:class:`PreparedRequest <requests.models.PreparedRequest>` with that state
:class:`PreparedRequest <requests.PreparedRequest>` with that state
applied, replace the call to :meth:`Request.prepare()
<requests.Request.prepare>` with a call to
:meth:`Session.prepare_request() <requests.Session.prepare_request>`, like this::
@@ -145,10 +185,13 @@ applied, replace the call to :meth:`Request.prepare()
print(resp.status_code)
.. _verification:
SSL Cert Verification
---------------------
Requests can verify SSL certificates for HTTPS requests, just like a web browser. To check a host's SSL certificate, you can use the ``verify`` argument::
Requests can verify SSL certificates for HTTPS requests, just like a web browser.
To check a host's SSL certificate, you can use the ``verify`` argument::
>>> requests.get('https://kennethreitz.com', verify=True)
requests.exceptions.SSLError: hostname 'kennethreitz.com' doesn't match either of '*.herokuapp.com', 'herokuapp.com'
@@ -158,7 +201,11 @@ I don't have SSL setup on this domain, so it fails. Excellent. GitHub does thoug
>>> requests.get('https://github.com', verify=True)
<Response [200]>
You can also pass ``verify`` the path to a CA_BUNDLE file for private certs. You can also set the ``REQUESTS_CA_BUNDLE`` environment variable.
You can pass ``verify`` the path to a CA_BUNDLE file or directory with certificates of trusted CAs::
>>> requests.get('https://github.com', verify='/path/to/certfile')
This list of trusted CAs can also be specified through the ``REQUESTS_CA_BUNDLE`` environment variable.
Requests can also ignore verifying the SSL certificate if you set ``verify`` to False.
@@ -169,7 +216,9 @@ Requests can also ignore verifying the SSL certificate if you set ``verify`` to
By default, ``verify`` is set to True. Option ``verify`` only applies to host certs.
You can also specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both file's path::
You can also specify a local cert to use as client side certificate, as a single
file (containing the private key and the certificate) or as a tuple of both
file's path::
>>> requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key'))
<Response [200]>
@@ -179,49 +228,110 @@ If you specify a wrong path or an invalid cert::
>>> requests.get('https://kennethreitz.com', cert='/wrong_path/server.pem')
SSLError: [Errno 336265225] _ssl.c:347: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
.. _ca-certificates:
CA Certificates
---------------
By default Requests bundles a set of root CAs that it trusts, sourced from the
`Mozilla trust store`_. However, these are only updated once for each Requests
version. This means that if you pin a Requests version your certificates can
become extremely out of date.
From Requests version 2.4.0 onwards, Requests will attempt to use certificates
from `certifi`_ if it is present on the system. This allows for users to update
their trusted certificates without having to change the code that runs on their
system.
For the sake of security we recommend upgrading certifi frequently!
.. _HTTP persistent connection: https://en.wikipedia.org/wiki/HTTP_persistent_connection
.. _connection pooling: https://urllib3.readthedocs.org/en/latest/pools.html
.. _certifi: http://certifi.io/
.. _Mozilla trust store: https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt
.. _body-content-workflow:
Body Content Workflow
---------------------
By default, when you make a request, the body of the response is downloaded
immediately. You can override this behavior and defer downloading the response
immediately. You can override this behaviour and defer downloading the response
body until you access the :class:`Response.content <requests.Response.content>`
attribute with the ``stream`` parameter::
tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'
r = requests.get(tarball_url, stream=True)
At this point only the response headers have been downloaded and the connection remains open, hence allowing us to make content retrieval conditional::
At this point only the response headers have been downloaded and the connection
remains open, hence allowing us to make content retrieval conditional::
if int(r.headers['content-length']) < TOO_LONG:
content = r.content
...
You can further control the workflow by use of the :class:`Response.iter_content <requests.Response.iter_content>` and :class:`Response.iter_lines <requests.Response.iter_lines>` methods. Alternatively, you can read the undecoded body from the underlying urllib3 :class:`urllib3.HTTPResponse <urllib3.response.HTTPResponse>` at :class:`Response.raw <requests.Response.raw>`.
You can further control the workflow by use of the :class:`Response.iter_content <requests.Response.iter_content>`
and :class:`Response.iter_lines <requests.Response.iter_lines>` methods.
Alternatively, you can read the undecoded body from the underlying
urllib3 :class:`urllib3.HTTPResponse <urllib3.response.HTTPResponse>` at
:class:`Response.raw <requests.Response.raw>`.
If you set ``stream`` to ``True`` when making a request, Requests cannot
release the connection back to the pool unless you consume all the data or call
:class:`Response.close <requests.Response.close>`. This can lead to
inefficiency with connections. If you find yourself partially reading request
bodies (or not reading them at all) while using ``stream=True``, you should
consider using ``contextlib.closing`` (`documented here`_), like this::
from contextlib import closing
with closing(requests.get('http://httpbin.org/get', stream=True)) as r:
# Do things with the response here.
.. _`documented here`: http://docs.python.org/2/library/contextlib.html#contextlib.closing
.. _keep-alive:
Keep-Alive
----------
Excellent news — thanks to urllib3, keep-alive is 100% automatic within a session! Any requests that you make within a session will automatically reuse the appropriate connection!
Excellent news — thanks to urllib3, keep-alive is 100% automatic within a session!
Any requests that you make within a session will automatically reuse the appropriate
connection!
Note that connections are only released back to the pool for reuse once all body data has been read; be sure to either set ``stream`` to ``False`` or read the ``content`` property of the ``Response`` object.
Note that connections are only released back to the pool for reuse once all body
data has been read; be sure to either set ``stream`` to ``False`` or read the
``content`` property of the ``Response`` object.
.. _streaming-uploads:
Streaming Uploads
-----------------
Requests supports streaming uploads, which allow you to send large streams or files without reading them into memory. To stream and upload, simply provide a file-like object for your body::
Requests supports streaming uploads, which allow you to send large streams or
files without reading them into memory. To stream and upload, simply provide a
file-like object for your body::
with open('massive-body') as f:
with open('massive-body', 'rb') as f:
requests.post('http://some.url/streamed', data=f)
.. warning:: It is strongly recommended that you open files in `binary mode`_.
This is because Requests may attempt to provide the
``Content-Length`` header for you, and if it does this value will
be set to the number of *bytes* in the file. Errors may occur if
you open the file in *text mode*.
.. _binary mode: https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files
.. _chunk-encoding:
Chunk-Encoded Requests
----------------------
Requests also supports Chunked transfer encoding for outgoing and incoming requests. To send a chunk-encoded request, simply provide a generator (or any iterator without a length) for your body::
Requests also supports Chunked transfer encoding for outgoing and incoming requests.
To send a chunk-encoded request, simply provide a generator (or any iterator without
a length) for your body::
def gen():
yield 'hi'
@@ -229,6 +339,48 @@ Requests also supports Chunked transfer encoding for outgoing and incoming reque
requests.post('http://some.url/chunked', data=gen())
For chunked encoded responses, it's best to iterate over the data using
:meth:`Response.iter_content() <requests.models.Response.iter_content>`. In
an ideal situation you'll have set ``stream=True`` on the request, in which
case you can iterate chunk-by-chunk by calling ``iter_content`` with a chunk
size parameter of ``None``. If you want to set a maximum size of the chunk,
you can set a chunk size parameter to any integer.
.. _multipart:
POST Multiple Multipart-Encoded Files
-------------------------------------
You can send multiple files in one request. For example, suppose you want to
upload image files to an HTML form with a multiple file field 'images':
<input type="file" name="images" multiple="true" required="true"/>
To do that, just set files to a list of tuples of (form_field_name, file_info):
>>> url = 'http://httpbin.org/post'
>>> multiple_files = [('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),
('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))]
>>> r = requests.post(url, files=multiple_files)
>>> r.text
{
...
'files': {'images': 'data:image/png;base64,iVBORw ....'}
'Content-Type': 'multipart/form-data; boundary=3131623adb2043caaeb5538cc7aa0b3a',
...
}
.. warning:: It is strongly recommended that you open files in `binary mode`_.
This is because Requests may attempt to provide the
``Content-Length`` header for you, and if it does this value will
be set to the number of *bytes* in the file. Errors may occur if
you open the file in *text mode*.
.. _binary mode: https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files
.. _event-hooks:
Event Hooks
-----------
@@ -268,6 +420,7 @@ Let's print some request method arguments at runtime::
http://httpbin.org
<Response [200]>
.. _custom-auth:
Custom Authentication
---------------------
@@ -311,7 +464,7 @@ Streaming Requests
With :class:`requests.Response.iter_lines()` you can easily
iterate over streaming APIs such as the `Twitter Streaming
API <https://dev.twitter.com/docs/streaming-api>`_. Simply
API <https://dev.twitter.com/streaming/overview>`_. Simply
set ``stream`` to ``True`` and iterate over the response with
:class:`~requests.Response.iter_lines()`::
@@ -324,8 +477,22 @@ set ``stream`` to ``True`` and iterate over the response with
# filter out keep-alive new lines
if line:
print json.loads(line)
print(json.loads(line))
.. warning::
:class:`~requests.Response.iter_lines()` is not reentrant safe.
Calling this method multiple times causes some of the received data
being lost. In case you need to call it from multiple places, use
the resulting iterator object instead::
lines = r.iter_lines()
# Save the first line for later or just skip it
first_line = next(lines)
for line in lines:
print(line)
.. _proxies:
Proxies
-------
@@ -359,8 +526,20 @@ To use HTTP Basic Auth with your proxy, use the `http://user:password@host/` syn
"http": "http://user:pass@10.10.1.10:3128/",
}
To give a proxy for a specific scheme and host, use the
`scheme://hostname` form for the key. This will match for
any request to the given scheme and exact hostname.
::
proxies = {
"http://10.20.1.128": "http://10.10.1.10:5323",
}
Note that proxy URLs must include the scheme.
.. _compliance:
Compliance
----------
@@ -387,6 +566,8 @@ specification in this case. If you require a different encoding, you can
manually set the :attr:`Response.encoding <requests.Response.encoding>`
property, or use the raw :attr:`Response.content <requests.Response.content>`.
.. _http-verbs:
HTTP Verbs
----------
@@ -407,8 +588,8 @@ like so::
We should confirm that GitHub responded correctly. If it has, we want to work
out what type of content it is. Do this like so::
>>> if (r.status_code == requests.codes.ok):
... print r.headers['content-type']
>>> if r.status_code == requests.codes.ok:
... print(r.headers['content-type'])
...
application/json; charset=utf-8
@@ -418,11 +599,11 @@ So, GitHub returns JSON. That's great, we can use the :meth:`r.json
::
>>> commit_data = r.json()
>>> print commit_data.keys()
>>> print(commit_data.keys())
[u'committer', u'author', u'url', u'tree', u'sha', u'parents', u'message']
>>> print commit_data[u'committer']
>>> print(commit_data[u'committer'])
{u'date': u'2012-05-10T11:10:50-07:00', u'email': u'me@kennethreitz.com', u'name': u'Kenneth Reitz'}
>>> print commit_data[u'message']
>>> print(commit_data[u'message'])
makin' history
So far, so simple. Well, let's investigate the GitHub API a little bit. Now,
@@ -445,7 +626,7 @@ headers, e.g.
::
>>> verbs = requests.options('http://a-good-website.com/api/cats')
>>> print verbs.headers['allow']
>>> print(verbs.headers['allow'])
GET,HEAD,POST,OPTIONS
Turning to the documentation, we see that the only other method allowed for
@@ -462,9 +643,9 @@ already exists, we will use it as an example. Let's start by getting it.
>>> r.status_code
200
>>> issue = json.loads(r.text)
>>> print issue[u'title']
>>> print(issue[u'title'])
Feature any http verb in docs
>>> print issue[u'comments']
>>> print(issue[u'comments'])
3
Cool, we have three comments. Let's take a look at the last of them.
@@ -475,9 +656,9 @@ Cool, we have three comments. Let's take a look at the last of them.
>>> r.status_code
200
>>> comments = r.json()
>>> print comments[0].keys()
>>> print(comments[0].keys())
[u'body', u'url', u'created_at', u'updated_at', u'user', u'id']
>>> print comments[2][u'body']
>>> print(comments[2][u'body'])
Probably in the "advanced" section
Well, that seems like a silly place. Let's post a comment telling the poster
@@ -485,7 +666,7 @@ that he's silly. Who is the poster, anyway?
::
>>> print comments[2][u'user'][u'login']
>>> print(comments[2][u'user'][u'login'])
kennethreitz
OK, so let's tell this Kenneth guy that we think this example should go in the
@@ -512,7 +693,7 @@ the very common Basic Auth.
>>> r.status_code
201
>>> content = r.json()
>>> print content[u'body']
>>> print(content[u'body'])
Sounds great! I'll get right on it.
Brilliant. Oh, wait, no! I meant to add that it would take me a while, because
@@ -522,7 +703,7 @@ that.
::
>>> print content[u"id"]
>>> print(content[u"id"])
5804413
>>> body = json.dumps({u"body": u"Sounds great! I'll get right on it once I feed my cat."})
>>> url = u"https://api.github.com/repos/kennethreitz/requests/issues/comments/5804413"
@@ -551,7 +732,7 @@ headers.
::
>>> r = requests.head(url=url, auth=auth)
>>> print r.headers
>>> print(r.headers)
...
'x-ratelimit-remaining': '4995'
'x-ratelimit-limit': '5000'
@@ -560,12 +741,16 @@ headers.
Excellent. Time to write a Python program that abuses the GitHub API in all
kinds of exciting ways, 4995 more times.
.. _link-headers:
Link Headers
------------
Many HTTP APIs feature Link headers. They make APIs more self describing and discoverable.
Many HTTP APIs feature Link headers. They make APIs more self describing and
discoverable.
GitHub uses these for `pagination <http://developer.github.com/v3/#pagination>`_ in their API, for example::
GitHub uses these for `pagination <http://developer.github.com/v3/#pagination>`_
in their API, for example::
>>> url = 'https://api.github.com/users/kennethreitz/repos?page=1&per_page=10'
>>> r = requests.head(url=url)
@@ -580,6 +765,8 @@ Requests will automatically parse these link headers and make them easily consum
>>> r.links["last"]
{'url': 'https://api.github.com/users/kennethreitz/repos?page=7&per_page=10', 'rel': 'last'}
.. _transport-adapters:
Transport Adapters
------------------
@@ -645,9 +832,11 @@ SSLv3:
block=block,
ssl_version=ssl.PROTOCOL_SSLv3)
.. _`described here`: http://kennethreitz.org/exposures/the-future-of-python-http
.. _`described here`: http://www.kennethreitz.org/essays/the-future-of-python-http
.. _`urllib3`: https://github.com/shazow/urllib3
.. _blocking-or-nonblocking:
Blocking Or Non-Blocking?
-------------------------
@@ -664,3 +853,43 @@ Two excellent examples are `grequests`_ and `requests-futures`_.
.. _`grequests`: https://github.com/kennethreitz/grequests
.. _`requests-futures`: https://github.com/ross/requests-futures
.. _timeouts:
Timeouts
--------
Most requests to external servers should have a timeout attached, in case the
server is not responding in a timely manner. Without a timeout, your code may
hang for minutes or more.
The **connect** timeout is the number of seconds Requests will wait for your
client to establish a connection to a remote machine (corresponding to the
`connect()`_) call on the socket. It's a good practice to set connect timeouts
to slightly larger than a multiple of 3, which is the default `TCP packet
retransmission window <http://www.hjp.at/doc/rfc/rfc2988.txt>`_.
Once your client has connected to the server and sent the HTTP request, the
**read** timeout is the number of seconds the client will wait for the server
to send a response. (Specifically, it's the number of seconds that the client
will wait *between* bytes sent from the server. In 99.9% of cases, this is the
time before the server sends the first byte).
If you specify a single value for the timeout, like this::
r = requests.get('https://github.com', timeout=5)
The timeout value will be applied to both the ``connect`` and the ``read``
timeouts. Specify a tuple if you would like to set the values separately::
r = requests.get('https://github.com', timeout=(3.05, 27))
If the remote server is very slow, you can tell Requests to wait forever for
a response, by passing None as a timeout value and then retrieving a cup of
coffee.
.. code-block:: python
r = requests.get('https://github.com', timeout=None)
.. _`connect()`: http://linux.die.net/man/2/connect
docs/user/authentication.rst
+1 -1
View File
@@ -99,7 +99,7 @@ If you can't find a good implementation of the form of authentication you
want, you can implement it yourself. Requests makes it easy to add your own
forms of authentication.
To do so, subclass :class:`requests.auth.AuthBase` and implement the
To do so, subclass :class:`AuthBase <requests.auth.AuthBase>` and implement the
``__call__()`` method::
>>> import requests
docs/user/install.rst
+4 -13
View File
@@ -10,7 +10,8 @@ The first step to using any software package is getting it properly installed.
Distribute & Pip
----------------
Installing Requests is simple with `pip <http://www.pip-installer.org/>`_::
Installing Requests is simple with `pip <https://pip.pypa.io>`_, just run
this in your terminal::
$ pip install requests
@@ -18,17 +19,7 @@ or, with `easy_install <http://pypi.python.org/pypi/setuptools>`_::
$ easy_install requests
But, you really `shouldn't do that <http://www.pip-installer.org/en/latest/other-tools.html#pip-compared-to-easy-install>`_.
Cheeseshop (PyPI) Mirror
------------------------
If the Cheeseshop (a.k.a. PyPI) is down, you can also install Requests from one
of the mirrors. `Crate.io <http://crate.io>`_ is one of them::
$ pip install -i http://simple.crate.io/ requests
But, you really `shouldn't do that <https://stackoverflow.com/questions/3220404/why-use-pip-over-easy-install>`_.
Get the Code
@@ -39,7 +30,7 @@ Requests is actively developed on GitHub, where the code is
You can either clone the public repository::
git clone git://github.com/kennethreitz/requests.git
$ git clone git://github.com/kennethreitz/requests.git
Download the `tarball <https://github.com/kennethreitz/requests/tarball/master>`_::
docs/user/quickstart.rst
+91 -33
View File
@@ -6,8 +6,7 @@ Quickstart
.. module:: requests.models
Eager to get started? This page gives a good introduction in how to get started
with Requests. This assumes you already have Requests installed. If you do not,
head over to the :ref:`Installation <install>` section.
with Requests.
First, make sure that:
@@ -30,20 +29,20 @@ Begin by importing the Requests module::
Now, let's try to get a webpage. For this example, let's get GitHub's public
timeline ::
>>> r = requests.get('https://github.com/timeline.json')
>>> r = requests.get('https://api.github.com/events')
Now, we have a :class:`Response` object called ``r``. We can get all the
information we need from this object.
Now, we have a :class:`Response <requests.Response>` object called ``r``. We can
get all the information we need from this object.
Requests' simple API means that all forms of HTTP request are as obvious. For
example, this is how you make an HTTP POST request::
>>> r = requests.post("http://httpbin.org/post")
>>> r = requests.post("http://httpbin.org/post", data = {"key":"value"})
Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD and
OPTIONS? These are all just as simple::
>>> r = requests.put("http://httpbin.org/put")
>>> r = requests.put("http://httpbin.org/put", data = {"key":"value"})
>>> r = requests.delete("http://httpbin.org/delete")
>>> r = requests.head("http://httpbin.org/get")
>>> r = requests.options("http://httpbin.org/get")
@@ -74,6 +73,12 @@ You can see that the URL has been correctly encoded by printing the URL::
Note that any dictionary key whose value is ``None`` will not be added to the
URL's query string.
You can also pass a list of items as a value::
>>> payload = {'key1': 'value1', 'key2': ['value2', 'value3']}
>>> r = requests.get("http://httpbin.org/get", params=payload)
>>> print(r.url)
http://httpbin.org/get?key1=value1&key2=value2&key2=value3
Response Content
----------------
@@ -82,7 +87,7 @@ We can read the content of the server's response. Consider the GitHub timeline
again::
>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r = requests.get('https://api.github.com/events')
>>> r.text
u'[{"repository":{"open_issues":0,"url":"https://github.com/...
@@ -135,13 +140,19 @@ JSON Response Content
There's also a builtin JSON decoder, in case you're dealing with JSON data::
>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r = requests.get('https://api.github.com/events')
>>> r.json()
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...
In case the JSON decoding fails, ``r.json`` raises an exception. For example, if
the response gets a 401 (Unauthorized), attempting ``r.json`` raises ``ValueError:
No JSON object could be decoded``
the response gets a 204 (No Content), or if the response contains invalid JSON,
attempting ``r.json`` raises ``ValueError: No JSON object could be decoded``.
It should be noted that the success of the call to ``r.json`` does **not**
indicate the success of the response. Some servers may return a JSON object in a
failed response (e.g. error details with HTTP 500). Such JSON will be decoded
and returned. To check that a request is successful, use
``r.raise_for_status()`` or check ``r.status_code`` is what you expect.
Raw Response Content
@@ -151,7 +162,7 @@ In the rare case that you'd like to get the raw socket response from the
server, you can access ``r.raw``. If you want to do this, make sure you set
``stream=True`` in your initial request. Once you do, you can do this::
>>> r = requests.get('https://github.com/timeline.json', stream=True)
>>> r = requests.get('https://api.github.com/events', stream=True)
>>> r.raw
<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>
>>> r.raw.read(10)
@@ -176,14 +187,22 @@ Custom Headers
If you'd like to add HTTP headers to a request, simply pass in a ``dict`` to the
``headers`` parameter.
For example, we didn't specify our content-type in the previous example::
For example, we didn't specify our user-agent in the previous example::
>>> import json
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> headers = {'content-type': 'application/json'}
>>> headers = {'user-agent': 'my-app/0.0.1'}
>>> r = requests.post(url, data=json.dumps(payload), headers=headers)
>>> r = requests.get(url, headers=headers)
Note: Custom headers are given less precedence than more specific sources of information. For instance:
* Authorization headers will be overridden if credentials are passed via the ``auth`` parameter or are specified in a ``.netrc`` accessible in the environment.
* Authorization headers will be removed if you get redirected off-host.
* Proxy-Authorization headers will be overridden by proxy credentials provided in the URL.
* Content-Length headers will be overridden when we can determine the length of the content.
Furthermore, Requests does not change its behavior at all based on which custom headers are specified. The headers are simply passed on into the final request.
More complicated POST requests
@@ -195,7 +214,7 @@ dictionary of data will automatically be form-encoded when the request is made::
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> print r.text
>>> print(r.text)
{
...
"form": {
@@ -205,7 +224,8 @@ dictionary of data will automatically be form-encoded when the request is made::
...
}
There are many times that you want to send data that is not form-encoded. If you pass in a ``string`` instead of a ``dict``, that data will be posted directly.
There are many times that you want to send data that is not form-encoded. If
you pass in a ``string`` instead of a ``dict``, that data will be posted directly.
For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data::
@@ -215,6 +235,14 @@ For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data::
>>> r = requests.post(url, data=json.dumps(payload))
Instead of encoding the ``dict`` yourself, you can also pass it directly using
the ``json`` parameter (added in version 2.4.2) and it will be encoded automatically::
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> r = requests.post(url, json=payload)
POST a Multipart-Encoded File
-----------------------------
@@ -234,10 +262,10 @@ Requests makes it simple to upload Multipart-encoded files::
...
}
You can set the filename explicitly::
You can set the filename, content_type and headers explicitly::
>>> url = 'http://httpbin.org/post'
>>> files = {'file': ('report.xls', open('report.xls', 'rb'))}
>>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}
>>> r = requests.post(url, files=files)
>>> r.text
@@ -264,6 +292,23 @@ If you want, you can send strings to be received as files::
...
}
In the event you are posting a very large file as a ``multipart/form-data``
request, you may want to stream the request. By default, ``requests`` does not
support this, but there is a separate package which does -
``requests-toolbelt``. You should read `the toolbelt's documentation
<https://toolbelt.readthedocs.org>`_ for more details about how to use it.
For sending multiple files in one request refer to the :ref:`advanced <advanced>`
section.
.. warning:: It is strongly recommended that you open files in `binary mode`_.
This is because Requests may attempt to provide the
``Content-Length`` header for you, and if it does this value will
be set to the number of *bytes* in the file. Errors may occur if
you open the file in *text mode*.
.. _binary mode: https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files
Response Status Codes
---------------------
@@ -280,8 +325,9 @@ reference::
>>> r.status_code == requests.codes.ok
True
If we made a bad request (a 4XX client error or 5XX server error response), we can raise it with
:class:`Response.raise_for_status()`::
If we made a bad request (a 4XX client error or 5XX server error response), we
can raise it with
:meth:`Response.raise_for_status() <requests.Response.raise_for_status>`::
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
@@ -319,8 +365,8 @@ We can view the server's response headers using a Python dictionary::
}
The dictionary is special, though: it's made just for HTTP headers. According to
`RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>`_, HTTP
Headers are case-insensitive.
`RFC 7230 <http://tools.ietf.org/html/rfc7230#section-3.2>`_, HTTP Header names
are case-insensitive.
So, we can access the headers using any capitalization we want::
@@ -330,11 +376,20 @@ So, we can access the headers using any capitalization we want::
>>> r.headers.get('content-type')
'application/json'
It is also special in that the server could have sent the same header multiple
times with different values, but requests combines them so they can be
represented in the dictionary within a single mapping, as per
`RFC 7230 <http://tools.ietf.org/html/rfc7230#section-3.2>`_:
A recipient MAY combine multiple header fields with the same field name
into one "field-name: field-value" pair, without changing the semantics
of the message, by appending each subsequent field value to the combined
field value in order, separated by a comma.
Cookies
-------
If a response contains some Cookies, you can get quick access to them::
If a response contains some Cookies, you can quickly access them::
>>> url = 'http://example.com/some/cookie/setting/url'
>>> r = requests.get(url)
@@ -356,11 +411,17 @@ parameter::
Redirection and History
-----------------------
Requests will automatically perform location redirection for all verbs except
By default Requests will perform location redirection for all verbs except
HEAD.
GitHub redirects all HTTP requests to HTTPS. We can use the ``history`` method
of the Response object to track redirection. Let's see what GitHub does::
We can use the ``history`` property of the Response object to track redirection.
The :meth:`Response.history <requests.Response.history>` list contains the
:class:`Response <requests.Response>` objects that were created in order to
complete the request. The list is sorted from the oldest to the most recent
response.
For example, GitHub redirects all HTTP requests to HTTPS::
>>> r = requests.get('http://github.com')
>>> r.url
@@ -370,9 +431,6 @@ of the Response object to track redirection. Let's see what GitHub does::
>>> r.history
[<Response [301]>]
The :class:`Response.history` list contains the :class:`Request` objects that
were created in order to complete the request. The list is sorted from the
oldest to the most recent request.
If you're using GET, OPTIONS, POST, PUT, PATCH or DELETE, you can disable
redirection handling with the ``allow_redirects`` parameter::
@@ -385,7 +443,7 @@ redirection handling with the ``allow_redirects`` parameter::
If you're using HEAD, you can enable redirection as well::
>>> r = requests.post('http://github.com', allow_redirects=True)
>>> r = requests.head('http://github.com', allow_redirects=True)
>>> r.url
'https://github.com/'
>>> r.history
@@ -418,7 +476,7 @@ Errors and Exceptions
In the event of a network problem (e.g. DNS failure, refused connection, etc),
Requests will raise a :class:`~requests.exceptions.ConnectionError` exception.
In the event of the rare invalid HTTP response, Requests will raise an
In the rare event of an invalid HTTP response, Requests will raise an
:class:`~requests.exceptions.HTTPError` exception.
If a request times out, a :class:`~requests.exceptions.Timeout` exception is
ext/requests-logo.svg
+487
View File
@@ -0,0 +1,487 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 15.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" [
<!ENTITY ns_extend "http://ns.adobe.com/Extensibility/1.0/">
<!ENTITY ns_ai "http://ns.adobe.com/AdobeIllustrator/10.0/">
<!ENTITY ns_graphs "http://ns.adobe.com/Graphs/1.0/">
<!ENTITY ns_vars "http://ns.adobe.com/Variables/1.0/">
<!ENTITY ns_imrep "http://ns.adobe.com/ImageReplacement/1.0/">
<!ENTITY ns_sfw "http://ns.adobe.com/SaveForWeb/1.0/">
<!ENTITY ns_custom "http://ns.adobe.com/GenericCustomNamespace/1.0/">
<!ENTITY ns_adobe_xpath "http://ns.adobe.com/XPath/1.0/">
]>
<svg version="1.1" id="Layer_1" xmlns:x="&ns_extend;" xmlns:i="&ns_ai;" xmlns:graph="&ns_graphs;"
xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="300px" height="300px"
viewBox="0 0 300 300" enable-background="new 0 0 300 300" xml:space="preserve">
<metadata>
<sfw xmlns="&ns_sfw;">
<slices></slices>
<sliceSourceBounds height="226.344" width="297.902" x="2.418" y="-269.612" bottomLeftOrigin="true"></sliceSourceBounds>
</sfw>
</metadata>
<g>
<g>
<path fill-rule="evenodd" clip-rule="evenodd" d="M221.263,69.622c-2.289-1.402,1.148-3.191,0.734-5.145
c-1.289,2.596-3.376,3.761-5.145,1.47c9.042-5.844,25.363-2.561,33.81,2.205c-4.146,0.797-8.379-0.11-11.024-2.94
c-2.93,0.5-5.001,1.859-8.82,1.47c-1.225-1.46,0.066-0.935,0-2.94C228.052,66.122,225.694,68.909,221.263,69.622z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M213.912,64.477c0.98,0,1.96,0,2.94,0c-1.225,1.46,0.066,0.935,0,2.94
c-2.126,0.125-3.351,2.999-3.675-0.735c0.595,0.14,0.732,0.737,1.47,0.735C214.076,66.763,213.857,65.757,213.912,64.477z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M235.228,74.767c-1.663-1.522-3.552-2.818-2.94-6.615
c3.021-1.182,9.177-1.451,9.556,2.205c-1.4-0.315-1.847-1.583-3.675-1.47C241.371,73.46,233.52,70.113,235.228,74.767z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M244.782,68.887c2.452-0.002,2.51,2.391,5.146,2.205
c-1.225,1.461,0.066,0.935,0,2.94c-3.429-2.418-2.329-0.397-5.88-2.205C243.736,70.291,244.551,69.881,244.782,68.887z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M229.348,76.237c1.552,0.977,2.172-0.362,4.41,0.735
c0.996,4.658-3.604,1.532-4.41,2.94C227.627,79.724,228.924,76.683,229.348,76.237z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M219.058,78.442c0.73,2.699,1.646,5.213,2.939,7.35
c-4.096-1.83-1.685,0.925-2.939,2.94C217.267,85.036,218.043,81.979,219.058,78.442z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M181.572,85.057c-0.734,0-1.47,0-2.205,0c0.98-1.225-0.268-4.678,2.205-4.41
C181.572,82.117,181.572,83.587,181.572,85.057z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M238.168,88.732c1.606,0.108,1.578,1.852,1.47,3.675
c-1.374,0.586-2.692,1.228-4.41,1.47C234.684,90.639,237.606,90.866,238.168,88.732z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M181.572,89.467c-1.631,7.112-2.808,5.626-8.085,8.82
c1.475-2.201,2.611-4.739,5.88-5.145c-1.267-2.578-3.878-0.97-5.145,1.47C172.591,90.111,177.957,90.473,181.572,89.467z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M228.612,90.937c1.154,0.316,0.628,2.312,2.94,1.47
c-1.774,1.408,1.756,4.525-2.205,4.41C228.957,94.224,227.191,93.53,228.612,90.937z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M183.777,98.287c-1.3-0.661-2.146-1.774-3.675-2.205
C181.095,92.907,183.977,95.932,183.777,98.287z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M179.367,97.552c3.462,1.61,3.844,4.566,2.205,8.085
c-2.554,0.306-2.552-5.721-4.41-3.675C174.883,100.339,178.996,98.954,179.367,97.552z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M169.077,107.842c0.259-2.436,2.202-3.188,3.676-4.41
c-0.021-0.715-0.9-0.57-1.471-0.735c1.246-1.694,1.85-4.03,5.146-3.675C172.769,100.753,174.71,108.085,169.077,107.842z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M175.692,107.842c-0.607,2.078,1.036,1.904,0.735,3.675
c-3.048,0.199-2.864-0.965-5.146,0C171.74,109.28,172.915,107.759,175.692,107.842z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M168.343,109.312c2.133,3.143-2.463,5.845-4.41,7.35
C163.998,112.808,166.995,111.885,168.343,109.312z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M160.992,118.132c-1.385-2.153,1.098-6.717,4.41-6.615
C164.316,114.106,162.506,115.97,160.992,118.132z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M169.813,112.987c4.291,1.292,3.826,3.497,2.205,6.615
c-3.127-0.437-1.422-2.911-5.146-1.47C166.329,114.894,169.252,115.121,169.813,112.987z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M208.032,113.722c1.878,0.082,2.573,1.347,3.675,2.205
c-0.097,2.353-1.666,3.234-4.409,2.94C207.335,115.668,206.194,115.487,208.032,113.722z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M159.522,124.747c4.284,1.078,1.874,8.733-0.735,9.555
C156.646,131.634,159.036,127.229,159.522,124.747z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M91.902,126.217c0.998,0.228,0.61,1.84,0.735,2.94
c-1.318-0.094-1.55,0.899-2.94,0.735C89.35,127.584,91.676,127.95,91.902,126.217z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M89.697,130.627c0.735,0,1.47,0,2.205,0c0.312,1.536-0.503,1.946-0.735,2.94
C89.875,133.39,89.492,132.302,89.697,130.627z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M213.178,137.242c2.258,0.135,1.542,3.958-0.735,3.675
C212.208,139.212,212.561,138.095,213.178,137.242z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M210.237,140.182c-0.26,0.965-1.294,1.156-1.47,2.205
C205.697,141.895,208.426,137.824,210.237,140.182z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M211.707,143.857c1.188-0.207,1.266,0.695,2.205,0.735
c-0.431,1.529-1.544,2.375-2.205,3.675C208.994,147.677,211.098,145.241,211.707,143.857z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M194.067,154.882c-1.406,3.493-4.694,5.106-7.35,7.351
c-0.56-1.156-2.332-1.099-2.205-2.94C186.69,158.008,189.762,155.912,194.067,154.882z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M105.132,129.892c-0.259,0.965-1.294,1.156-1.47,2.205
c3.349-2.139,0.824-3.979,2.94-7.35c0.595,0.14,0.732,0.737,1.47,0.735c-0.205-1.51-1.963-1.467-2.94-2.205
c-2.181,0.513-1.164,3.346-0.735,3.675c-0.572,0.996-3.322-2.019-5.145-2.205c-1.422,1.273-2.098,3.292-3.675,4.41
c-2.716-2.528,2.432-3.406,2.205-5.88c-1.313-0.207-1.508,0.695-0.735,0.735c-1.332,0.627-2.027,1.893-3.675,2.205
c-0.207-1.187,0.694-1.265,0.735-2.205c-3.127,0.672-5.928-1.597-7.35,2.205c-1.828-0.11-0.215-2.489,0-2.94
c-2.025,1.16-1.901,4.468-5.145,4.41c-0.246-2.449-2.129-3.26-1.47-6.615c-1.355,1.585-2.261,3.62-3.675,5.145
c-0.679-1.536-0.178-3.51-1.47-5.88c-0.885,0.83-1.332,2.098-3.675,1.47c-0.311,1.536,0.504,1.946,0.735,2.94
c0.861-0.119,0.62-1.34,0.735-2.205c1.937-0.222,2.412,1.018,2.205,2.94c-1.536-0.311-1.946,0.504-2.94,0.735
c0.311,1.537-0.504,1.946-0.735,2.94c1.293-0.177,1.676-1.265,1.47-2.94c0.855,1.35,4.796-0.386,3.675,2.94
c1.646-2.38,1.878-2.348,2.205-5.145c2.976,0.792-0.225,3.5,0.735,6.615c-4.914-0.43-6.491,0.67-8.82,3.675
c-1.534-2.14-4.863-2.487-6.615-4.41c-0.405,1.629,1.403,1.046,1.47,2.205c0.271,1.25-0.573,1.387-1.47,1.47
c1.102,2.429,1.797-1.527,3.675-0.735c0.83,0.885,2.098,1.332,1.47,3.675c2.081-0.859,3.392-2.488,4.41-4.41
c3.192,2.351,6.123-1.243,11.76-1.47c-1.422,2.636,0.283,2.693,0.735,5.145c-0.629,1.034-4.261,4.201-3.675,0
c-1.059,0.167-0.256,2.194-1.47,2.205c1.967,2.986,0.771,5.375,0,8.82c1.187,0.207,1.266-0.694,2.205-0.735
c-1.272-4.998-1.169-4.912,2.94-8.085c-0.27,1.25,0.573,1.387,1.47,1.47c-0.75,0.966-0.721,2.709-1.47,3.675
c0.595-0.14,0.732-0.737,1.47-0.735c1.991-0.836-0.667-5.659,2.94-6.615c-0.152,2.112,0.186,3.734,1.47,4.41
c1.543-0.205-1.065-1.963,0.735-2.94c-1.671,2.174-2.07-3.811,0.735-3.675c-0.35,3.254,1.011,3.655,0,5.88
c0.965-0.259,1.156-1.294,2.205-1.47c-0.315-1.4-1.583-1.847-1.47-3.675c1.048,2.901,2.51-0.337,0.735-1.47
c-0.882,1.03-0.286,0.063,0-0.735c1.889,0.071,1.051,2.869,2.94,2.94c0.861-0.119,0.62-1.34,0.735-2.205
c-0.89-2.473,1.475,4.867,2.94,0.735c-4.998,6.275-17.181,11.727-25.725,16.905c-0.686-0.785-2.711-0.229-2.205-2.205
c0.654,0.571,1.66,0.79,2.94,0.735c1.607-0.108,1.578-1.852,1.47-3.675c-1.499,0.461-0.207,3.713-2.94,2.94
c0.393-2.843-1.312-3.588-0.735-6.615c0.853-0.617,1.97-0.969,3.675-0.735c0,1.225,0,2.45,0,3.675c2.399-0.76,1.558-1.861,3.675,0
c-1.601-3.604-4.412-3.501-4.41-7.35c-1.078,1.194-3.132,5.628-4.41,2.205c-2.347,1.503,5.125,12.203-5.145,10.29
c0.83-0.885,2.098-1.332,1.47-3.675c-0.994,0.722-1.469,1.961-2.205,2.94c-1.939-1.001-5.012-0.869-7.35-1.47
c-4.346-6.795-2.056-19.485,5.88-20.58c2.167-4.889,7.351-5.015,11.025-8.085c4.302-3.595,6.498-10.399,12.495-13.23
c17.674-27.161,41.896-47.773,74.97-59.535c1.187-0.207,1.266,0.694,2.205,0.735c8.635-8.454,21.879,0.769,30.135,0.735
c2.308,1.858,3.361,4.969,2.94,9.555c2.304-0.508,0.958-1.96,0.734-3.675c6.348,2.802,4.76,15.554,1.471,16.17
c-2.136-3.209-5.993-2.727-5.146,0.735c-0.14,0.834-0.737,0.408-0.735,0c-0.625-0.135-0.693,0.287-0.734,0.735
c-1.435-0.313,1.202-2.1-2.205-1.47c-1.091,0.321,0.29,0.94,0,2.205c-0.654-0.452-1.314-0.888-1.47,0
c-2.832-0.827,2.832-2.849,0-3.675c0.478-1.248,1.689-0.459,1.47,0.735c2.779,0.629,2.601-3.291,1.47-4.41
c-2.063,0.009,1.75,2.005-1.47,1.47c0.021-3.163,3.064-3.306,3.675-5.88c1.293,0.177,1.676,1.265,1.47,2.94
c-2.87-3.025-2.131,1.497-2.205,2.94c3.162-2,4.191-4.506,1.471-7.35c-0.687,0.539-1.114,1.336-1.471,2.205
c-1.046-0.898-1.67-2.805-0.734-3.675c-0.592,0.389-1.193,0.767-2.205,0.735c1.973,2.973-1.039,1.283-2.205,2.205
c4.338,2.115-0.647,1.666,1.47,5.88c-1.889-0.35-2.072,4.546-3.675,6.615c1.061,1.145,2.96,1.45,2.205,4.41
c-1.043-0.063-2.081-0.122-2.205,0.735c-1.309-0.539-0.106-1.336,0-2.205c-1.952,0.253-2.48,1.929-2.94,3.675
c2.514-3.24,0.045,1.961,1.471,4.41c2.166-0.773,2.555-3.325,3.675-5.145c1.678,0.576-2.923,5.305,0.735,5.88
c-1.456,0.995-2.859,2.042-2.205,5.145c0.854,0.12,1.308-0.162,1.47-0.735c1.53,1.72-3.937,3.109-4.41,0.735
c-1.748-1.476,0.817,0.325,1.47,0.735c1.279-3.729-3.189-1.71-2.939-4.41c1.303,0.167,3.681-0.741,2.939,1.47
c2.614-2.189-1.638-3.703-4.409-2.94c-2.379-0.846,1.974-0.832,2.939-0.735c0.507-1.976-1.52-1.42-2.205-2.205
c-0.15,1.46-1.334,2.01-1.47-1.47c0.735,0,1.47,0,2.205,0c0-0.98,0-1.96,0-2.94c-2.796-1.081-0.145,3.285-2.94,2.205
c1.225-1.46-0.066-0.935,0-2.94c3.064,0.369,3.242-2.148,6.615-1.47c-4.934-5.145-12.385,5.58-8.82,12.495
c-0.586,1.913-1.228-2.097-1.47-2.94c-0.596,0.14-0.732,0.738-1.47,0.735c-0.908,2.378,2.378,0.562,1.47,2.94
c-2.304-0.344-2.072,1.848-2.94,2.94c-0.485-4.088,2.15-2.542,0-7.35c4.121-0.981,1.96-5.823,3.676-4.41
c0.535-2.005-1.461-1.479-1.471-2.94c-0.164,1.064-2.053,2.823-2.939,5.145c-2.035,1.313,0.66-2.51,0.734-3.675
c-1.935,0.453-0.96,1.458-2.939,2.94c-0.987-0.185,0.263-3.894-2.205-4.41c-3.403,2.081,0.535,3.419-2.94,5.88
c2.597-0.085,1.178,1.798,2.205,1.47c0.748,1.517-3.451,4.364-5.145,5.88c-1.891,0.395,4.393-2.66,3.675-5.88
c-1.359,0.111-0.451,2.489-1.47,2.94c-0.799-0.182-0.499-1.461-1.47-1.47c-1.661,0.411,0.891,1.146-0.735,2.205
c-1.61,1.757,0.645-1.78-0.735-2.205c-1.054,0.906-1.393,2.527-1.47,4.41c-1.476-0.25-2.3,0.15-2.205,1.47
c-1.771,0.301-1.598-1.342-3.675-0.735c-0.136-0.625,0.286-0.693,0.734-0.735c0-0.49,0-0.98,0-1.47
c0.715,0.02,0.571,0.899,0.735,1.47c1.87-0.089,2.851-1.069,2.94-2.94c-2.971,3.455-4.581-0.72-4.41-3.675
c8.53-7.15,17.914-13.445,25.725-21.315c2.509,0.826-0.231,2.849,1.471,3.675c-0.299,1.416-2.85,0.581-2.205,2.94
c1.593-1.589,1.712-1.057,4.409-2.205c-2.733-3.893-3.063-8.246,3.676-10.29c-3.24,2.513,1.96,0.044,4.409,1.47
c0.631-2.345-1.156-2.273-1.47-3.675c-2.246-0.531-2.679,0.751-4.41,0.735c2.628-7.415-2.069,2.665-4.409,4.41
c-2.036,1.518-5.892,2.234-5.881,4.41c-1.059-0.166-0.256-2.193-1.47-2.205c-0.064-2.514,4.475-0.425,4.41-2.94
c-0.202-2.579-2.738,1.844-2.94-0.735c-1.049,5.431-4.216-5.936,0.735-2.94c0.711-4.173-2.864-0.627-3.675-2.205
c0.546,3.658-0.06,7.504-1.47,11.76c0.166,0.936,2.193,0.963,2.204,0c0.917,0.014-2.027,4.567-2.939,2.94
c-0.119,0.854,0.162,1.308,0.735,1.47c-2.996,1.442-3.296,0.729-4.41,3.675c-3.43-1.062-0.599,0.121-2.94,1.47
c-0.861-0.119-0.62-1.34-0.734-2.205c-1.857,1.39,0.678,2.577-0.735,4.41c-0.843-0.882-1.863-0.287-0.735,0
c-0.081,1.799-2.43,0.156-2.205-0.735c-2.006,0.664,2.236,1.644,0,2.94c-3.391-2.171-1.608-0.146-3.675,1.47
c-0.442-2.944-3.201-7.413-5.88-10.29c-1.689,3.547,6.799,5.9,4.41,10.29c-0.799-0.182-0.499-1.461-1.47-1.47
c-0.134,1.604,0.602,2.338,2.204,2.205c-0.979,4.41-7.415,3.365-7.35,8.82c-5.05-1.6-1.421-3.929-2.205-8.085
c-1.059,0.412-1.793,1.146-2.205,2.205c2.701,3.553-0.494,8.366-2.94,11.025c-0.904-0.321-1.509-0.94-1.47-2.205
c0.049,6.452-14.079,6.938-17.64,14.7c-4.031-1.282-3.27,3.562-5.145,0.735c0.364,1.05-2.512,3.711-4.41,2.205
c0,1.715,0,3.43,0,5.145c4.073-5.424,15.954-11.495,21.315-15.435c0.616,0.365,2.889-2.625,4.41-0.735
c0.839-2.63,5.369-4.042,8.085-8.085c-1.408,0.708,0.074,3.432-1.47,7.35c0.119,0.862,1.34,0.62,2.205,0.735
c-1.146-3.052,0.62-3.602-0.735-7.35c2.9-1.754,4.255-5.056,8.085-5.88c0.322,2.527-0.29,4.12-0.734,5.88
c0.912,3.335,2.137-1.27,3.675,0.735c-2.104,0.17-6.468,8.646-10.29,11.76c-2.046-0.404,1.032-5.932-2.94-4.41
c0.326,0.654,0.646,1.314,1.47,1.47c-0.205,1.737-1.963-0.843-2.94-0.735c0.009,1.46,2.005,0.935,1.47,2.94
c-2.859,1.306-4.711,3.619-8.085,4.41c-1.377-1.399-0.08-1.847,0-3.675c-2.508,0.432-1.813,4.066-3.675,5.145
c2.263,0.547,1.413-2.018,3.675-1.47c-1.159,1.781-2.629,3.25-4.41,4.41c0.441,1.328,3.77-0.57,3.675-2.205
c1.091,0.321-0.29,0.94,0,2.205c3.011-2.271,3.416-2.657,4.41-4.41c2.603,0.082-1.562,2.43,0.735,2.205
c-0.639,1.109-1.101-0.202-2.94,0.735c0.942,1.906,3.499,0.33,3.675,1.47c1.508,1.702-3.734-1.223-5.145,0
c1.96,1.39,5.419,2.093,6.615,2.205c-0.026-0.089-7.406,4.91-8.085,6.615c0.12-0.854-0.163-1.308-0.735-1.47
c0.216-1.532,1.989,1.09,2.205-1.47c-0.394-1.515-3.477-0.175-1.47-2.205c-2.228,1.376-2.064,0.579-4.41,1.47
c0.55,1.388,2.194,0.51,0.735,2.205c1.25,0.27,1.387-0.573,1.47-1.47c0.998,0.228,0.61,1.84,0.735,2.94
c-1.059-0.822-1.793-0.792-2.205,0.735c-0.414-1.596-0.542-1.116-4.41-1.47c0.228,0.998,1.84,0.61,2.94,0.735
c-1.983,2.152-2.853,0.632-5.145,0c0.478,0.748,1.69,0.76,1.47,2.205c-2.622-1.308-3.733,0.313-6.615-0.735
c-5.281,2.179-0.556,5.58-5.145,2.94c0.052,2.062-1.489,0.169-3.675,2.205c-1.996-0.669,0.749-2.576,0.735-3.675
c-2.034,0.092-2.523,1.403-1.47-0.735c-0.735,0-1.47,0-2.205,0c1.05,3.5,1.045,3.487,2.205,6.615c0.98,0,1.96,0,2.94,0
c2.247,1.136-3.188,0.647-4.41,0.735c0.389-0.591,0.767-1.193,0.735-2.205c-2.604,2.202-3.459,3.854-6.615,5.145
C101.729,132.744,103.628,129.698,105.132,129.892z M196.272,68.152c0.639-1.076,1.101-2.33,2.94-2.205
c-0.125,1.84,1.129,2.301,2.205,2.94c-0.536-2.005,1.46-1.479,1.47-2.94c-2.413,1.24-3.241-0.39-5.146-1.47
C197.78,65.825,193.976,66.966,196.272,68.152z M180.838,68.887c-0.301,2.15-2.516,2.385-3.676,3.675
c0.372,1.833,1.391,3.02,2.205,4.41c-0.956-2.902,2.697-1.779,0.735-3.675c2.033-0.091,2.523-1.403,1.47,0.735
c1.84,0.125,2.301-1.129,2.94-2.205C179.934,71.64,182.191,71.764,180.838,68.887z M174.957,72.562
c0.971,1.839-2.905,2.301-0.734,2.94c-0.707-0.002,0.481,2.963,3.675-0.735C176.398,74.551,176.456,72.778,174.957,72.562z
M184.513,71.092c0.436-2.345-1.38-2.273,0-3.675C181.553,66.699,181.553,71.81,184.513,71.092z M150.703,83.587
c3.646,0.031,1.324-2.756,0-3.675C148.651,76.459,152.112,83.266,150.703,83.587z M141.147,90.202
c-0.234,0.9,0.118,4.54,0.735,2.205c-0.402-1.068,0.859-3.799-0.735-3.675c-0.176,0.798-1.21,1.765-1.47,0.735
c0.408,0.002,0.834-0.595,0-0.735C138.316,90.013,139.867,91.563,141.147,90.202z M149.967,96.082
c0.604,2.797-2.454,2.765-0.735,5.145c0.216-1.499,1.989-1.441,2.205-2.94c-0.57-0.165-1.45-0.021-1.47-0.735
C150.709,97.543,150.892,96.264,149.967,96.082z M131.592,115.192c-1.375,0.203,1.547-5.177-0.735-5.88
c-0.155,0.888-0.815,0.452-1.47,0c-0.663,1.715-5.102,1.631-7.35,3.675c0.125,1.1-0.263,2.713,0.735,2.94c0-0.735,0-1.47,0-2.205
c1.88-1.423,4.617,0.982,2.94-1.47c1.012,0.032,1.614-0.346,2.205-0.735c-0.509,1.296-0.812,2.276,0,2.94
c0.356-0.869,0.784-1.667,1.47-2.205c-0.002,0.737,0.595,0.875,0.735,1.47c-1.715,0.41-1.715,2.211,0,0.735
c-1.156,1.538-1.748,2.762-0.735,3.675C129.604,116.633,131.376,116.691,131.592,115.192z M119.833,115.927
c-1.107,1.099-0.22,4.19-2.94,3.675c1.514-2.97-7.9,3.836-8.085-1.47c-0.458,1.993-3.071,1.83-2.94,4.41
c1.483-1.572,6.2-1.275,8.085,0.735c-1.516-0.646-2.784,2.423-1.47,2.205c0.913-1.538,2.138-2.762,3.675-3.675
c-4.265,4.662,5.502-0.267,1.47-1.47c1.611-0.661,4.083-4.677,5.145-1.47c-0.825,0.075-1.144-1.069-1.47,0
c0.924,0.301,1.502,0.948,1.47,2.205c1.878-0.082,2.573-1.347,3.675-2.205C123.449,117.17,123.415,117.041,119.833,115.927z
M178.633,48.307c1.1-0.125,2.712,0.262,2.939-0.735c-1.019-0.291-2.082-4.172-2.939-2.205
C180.527,45.544,177.623,46.631,178.633,48.307z M172.753,54.187c-1.59-0.616-2.508-1.903-5.146-1.47
C167.963,55.016,171.16,56.697,172.753,54.187z M163.933,57.127c-0.57,0.165-1.45,0.02-1.47,0.735
c2.007,2.019,2.93,1.296,4.409,2.94c0.082-1.388,2.431-0.51,2.205-2.205c-2.074,0.793-3.622-3.844-5.88-2.205
C163.646,56.434,164.067,56.502,163.933,57.127z M138.942,69.622c0.87-0.935,2.776-0.312,3.675,0.735
c-1.068,0.402-3.799-0.859-3.675,0.735c2.005-0.535,1.479,1.461,2.94,1.47c1.168-1.501,2.261-5.995,0-4.41
C143.881,69.625,136.646,67.545,138.942,69.622z M139.678,71.827c-0.431,1.529-1.545,2.375-2.205,3.675
c2.125,1.667,4.05-1.55,1.47,0C138.786,73.352,143.09,73.117,139.678,71.827z M146.292,76.237c2.824,1.12,1.403-8.217,0.735-5.88
C147.349,72.884,146.737,74.477,146.292,76.237z M120.567,83.587c-1.049-0.286-1.24-0.882-2.205,0
c-0.347,1.433,1.979,3.759,2.205,2.205c-1.213,0.475,2.511-5.403-0.735-2.94C120.281,82.894,120.703,82.962,120.567,83.587z
M131.592,82.117c-0.688,0.621-1.57,4.685,0,5.145c-0.392-1.059,2.864-1.794,0.735-2.205
C130.582,87.59,133.903,82.247,131.592,82.117z M127.183,96.082c-3.341-0.443-0.694-7.005-2.205-5.88
C123.392,92.253,125.284,97.864,127.183,96.082z M102.192,112.987c-1.499-0.216-1.441-1.989-2.94-2.205
c-0.646,2.361,2.423,1.007,2.205,2.94c-2.754,1.301-1.651,6.64-5.145,8.82c4.625-0.031,2.284-7.026,7.35-6.615
c-0.967-2.181-0.65-3.625,1.47-4.41c0.009,0.971,1.289,0.672,1.47,1.47c-1.516-0.646-2.784,2.423-1.47,2.205
c0.517-2.467,4.225-1.217,4.41-2.205c-0.856-0.124-0.798-1.162-0.735-2.205c-1.309-0.439-3.053,2.154-3.675,0
c0.57-0.165,1.45-0.021,1.47-0.735C104.117,110.011,103.414,111.758,102.192,112.987z M94.842,115.927
c0.042-0.448,0.11-0.87,0.735-0.735c0.165,0.57,0.02,1.45,0.735,1.47c-0.649-2.422,3.599-3.673,1.47-5.145
C98.161,113.905,93.23,114.739,94.842,115.927z M80.877,119.602c3.09-0.095,3.954-2.417,6.615-2.94
C84.711,115.294,83.058,118.278,80.877,119.602z M86.757,121.807c2.345,0.436,2.273-1.379,3.675,0c0.367-2.313-1.603-2.874,0-4.41
C87.913,117.572,88.968,121.323,86.757,121.807z M66.913,142.387c-0.449,0.042-0.87,0.11-0.735,0.735
c1.718-0.227,3.037-1.439,4.41,0c-1.831-1.845-1.914-5.436-3.675-7.35C67.708,138.377,67.816,141.08,66.913,142.387z
M63.237,146.062c1.901-2.899,0.251-5.01,1.47-8.82C63.007,138.465,62.123,142.961,63.237,146.062z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M170.548,84.322c2.603,0.246,1.807,2.022,4.409,0.735
c-1.325,5.079-1.02,2.865-4.409,5.88c0-1.225,0-2.45,0-3.675c0.715,0.02,0.57,0.899,0.734,1.47
c0.596-0.14,0.733-0.737,1.471-0.735c0.093-1.318-0.9-1.55-0.735-2.94c-2.268,1.652-4.007,3.833-6.615,5.145
C162.736,94.102,169.12,84.865,170.548,84.322z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M172.753,90.202c1.125,0.015-0.368,3.467-0.735,4.41
c-0.986-0.483-1.369-1.571-1.47-2.94C171.805,91.704,172.451,91.126,172.753,90.202z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M210.237,68.887c-0.539-0.687-1.336-1.114-2.205-1.47
c-1.469,4.521-1.293-1.97-2.939,1.47c-0.998-0.228-0.61-1.84-0.735-2.94c2.126,2.123,3.351-2.483,3.675,0.735
c0.596-0.14,0.732-0.737,1.471-0.735c-1.067-2.118-4.146-2.225-5.146-4.41c2.46,0.022,4.254,3.056,9.555,2.205
c-1.172,0.542,0.168,3.598-2.205,2.94c-0.207-1.187,0.695-1.266,0.735-2.205c-2.218-0.208-2.085,5.689-1.47,5.88
c-0.088,1.198-3.135-0.701-2.94-2.205C208.624,68.541,209.226,68.919,210.237,68.887z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M230.082,66.682c0.458,1.758,0.228,3.461,0.735,6.615
c-2.193,1.978-4.702-0.186-6.615-0.735C224.827,69.267,226.787,67.307,230.082,66.682z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M235.963,78.442c1.295,0.665,1.214,2.706,2.939,2.94
c1.318,0.094,1.551-0.899,2.94-0.735c-0.623-2.154-2.366,0.439-3.675,0c0.093-1.318-0.9-1.55-0.735-2.94
c4.72,0.203,9.964,2.318,6.615,8.085c2.408-0.286,2.282-3.107,3.675-4.41c-0.613,1.982-0.522,6.032-2.205,7.35
c0.497-0.389-3.614-3.934-1.47,0c-2.24-0.861-5.022,0.625-3.676-2.94c-1.078,2.326-2.949,1.144-2.939-1.47
c-1.423,1.027-4.009,0.891-3.675,3.675c-2.792,1.129-3.468-4.995-3.676,0c-1.075-0.639-2.329-1.101-2.204-2.94
c2.42-0.474,6.061-5.208,7.35-2.205C237.045,81.821,236.524,80.32,235.963,78.442z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M212.442,76.237c0.747,0.478,0.76,1.69,2.205,1.47
c-1.888,2.663-0.785,5.386,0,8.82c-0.687-0.416-1.113-2.043-1.47-0.735C211.199,82.283,214.195,79.971,212.442,76.237z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M256.543,79.912c1.641,3.357,2.67,10.924,0.734,20.58
c-1.971-0.969-1.221-4.659-2.205-6.615c1.861,2.816,2.767-7.364,0-9.555C254.721,82.01,256.119,81.449,256.543,79.912z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M201.418,85.792c1.976-0.203,2.094,5.177,0.734,5.88
c0.539,0.687,1.337,1.114,2.205,1.47c1.912-0.891,1.347-3.425,1.47-3.675c0.01-2.062,2.006,1.75,1.471-1.47
c3.115,1.971-0.72,8.058-0.735,6.615c-0.861,0.119-0.62,1.34-0.735,2.205c1.435-0.229,4.399-4.063,2.94-0.735
c3.793-2.429-0.295-5.48,2.205-9.555c0.798,0.182,0.498,1.461,1.47,1.47c-2.21,5.055-0.619,3.366,0.735,8.82
c-1.223-2.658-2.667-0.107-2.94,2.205c0.605,0.308,3.026,0.941,1.47,2.94c-2.074,0.989-1.842-3.895-2.939-1.47
c-1.309-0.539-0.106-1.337,0-2.205c-0.14-0.595-0.737-0.733-0.735-1.47c-2.062,0.01,1.75,2.005-1.47,1.47
c0.571,0.654,0.79,1.66,0.735,2.94c0.854,0.12,1.308-0.162,1.47-0.735c1.736,0.205-0.843,1.963-0.735,2.94
c-2.296-3.259-2.716-4.097-2.939-8.82c-3.109,1.554,0.032,6.509,1.47,9.555c2.006,0.066,1.479-1.224,2.94,0
c-1.799-2.022,1.098-1.55,0.734,0.735c2.765,0.615,2.602-3.29,4.41-1.47c-0.612-2.328-2.144-3.736-3.675-5.145
c2.071-2.26,3.477,2.361,4.41,3.675c1.46-1.224,0.935,0.066,2.939,0c-1.332-2.616,1.515,1.026,3.675-1.47
c-0.963,3.167,3.5,0.91,2.94,3.675c0.687-0.539,1.113-1.336,1.47-2.205c4.768,1.51,5.927,4.398,11.025,4.41
c0.687-0.539,1.113-1.336,1.47-2.205c-1.25-0.27-1.387,0.573-1.47,1.47c-2.604-0.082,1.562-2.43-0.735-2.205
c-0.641-1.543,3.74,1.065,1.471-0.735c1.627-1.607,0.648,1.393,1.47,2.205c1.537-0.913,2.763-2.137,3.675-3.675
c-0.65-2.014-3.261,1.435-1.47-1.47c-2.217,0.968-6.297,0.074-8.085,1.47c-2.091-0.98,0.566-3.269,0.734-4.41
c3.769-0.073,7.66,3.583,9.556,2.94c1.544,3.973-5.429,4.615-5.88,7.35c-4.734,0.324-7.626-1.194-9.556-3.675
c-2.109,3.28,0.611,11.391-1.47,14.7c-3.381-0.928-1.588-4.182-1.47-7.35c-1.044-0.063-2.082-0.122-2.205,0.735
c1.636-0.632,2.483,3.563-1.471,2.94c1.803-0.979-1.324-2.218,0.735-2.94c-0.182-0.798-1.461-0.499-1.47-1.47
c-0.546,0.071-0.546,2.869,0,2.94c-0.722,1.188-1.961,1.692-2.94,0.735c2.205-0.957,1.339-3.145,0-4.41
c-1.492-0.225,0.15,2.124,0.735,2.205c-2.854,2.326-2.703-4.619-5.88-4.41c2.38,3.048,1.534,3.766,3.675,5.88
c-0.217,1.532-1.989-1.09-2.205,1.47c0.853-0.587-0.962-5.455-2.205-5.145c-1.068-0.326,0.075-0.646,0-1.47
c-2.144-1.081-2.144,3.286,0,2.205c-1.832,1.354-5.572,0.797-6.615,2.94c3.111,0.081,0.013,1.377-0.734,0
c0.32-0.904,0.94-1.509,2.205-1.47c-0.669-2.011-2.576,1.304-3.676-0.735c-0.063,1.043-0.121,2.081,0.735,2.205
c0.225,2.297-2.123-1.868-2.205,0.735c-0.383-0.195,1.037-4.801,1.47-6.615c-3.607-0.668-2.799,3.081-5.88,2.94
c2.114-2.009,5.844-5.637,8.085-5.145C202.495,96.475,201.962,93.26,201.418,85.792z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M227.143,118.867c0.181,5.081-2.342,7.458-4.41,10.29
c-1.114,0.542,3.288-5.311,3.675-8.085c0.125-1.839-1.129-2.301-2.205-2.94C224.924,115.878,226.163,118.977,227.143,118.867z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M251.397,98.287c1.53-1.818-5.091-1.737-7.35-2.205
c3.146-1.916,6.013-4.558,3.675-8.085c2.915,1.98,2.307,4.2,6.615,3.675c0.413,2.863-2.324,2.576-2.94,4.41
c0.664,1.296,1.645,2.276,2.94,2.94c-1.007,1.16-3.481-0.601-5.881,1.47c-2.561-0.216,0.063-1.989-1.47-2.205
c-1.992,0.665,0.619,2.706,1.47,2.94c-1.638,1.827-4.189-1.648-4.409-3.675C246.281,98.013,250.379,96.611,251.397,98.287z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M220.527,95.347c0.925,0.182,0.741,1.461,0,1.47
c1.374,1.076,3.492,1.408,2.94,4.41C222.186,99.674,218.793,97.618,220.527,95.347z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M217.588,93.142c-0.822,1.059-0.793,1.793,0.734,2.205
c-2.169-0.701-1.332,3.871-2.939,0.735c-0.998,0.228-0.61,1.84-0.735,2.94c6.66-0.831-3.216,3.206-0.735-5.145
c1.024-2.366,0.32,3.561,2.205,0.735c0.207-1.187-0.694-1.265-0.734-2.205C216.167,89.905,215.611,94.452,217.588,93.142z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M192.598,91.672c1.71,1.103,1.121,1.797,0.734,3.675
c-1.721-2.255-0.713-0.571-2.204,0.735C190.006,98.108,191.473,92.079,192.598,91.672z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M167.607,101.227c1.772-1.412,2.918-3.452,4.41-5.145
c1.542,2.311-1.77,6.046-4.41,6.615c-0.66-2.151-0.66-2.258,0-4.41c-0.625-0.135-0.693,0.287-0.735,0.735
c-1.99-1.516,1.681-0.643,1.471-1.47C170.178,98.06,166.979,99.513,167.607,101.227z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M163.933,106.372c1.091,0.321-0.29,0.94,0,2.205
c2.444-1.23,1.972-5.378,5.88-5.145c-0.187,3.243-2.987,3.873-3.675,6.615c-1.059-1.626-1.794,0.925-2.205-0.735
c-1.362,1.333-1.827,3.563-3.675,4.41C159.598,111.107,162.779,108.16,163.933,106.372z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M177.162,102.697c-0.049,1.222,1.658,1.974,2.205,4.41
c-1.796,1.808-3.187-0.543-5.88,0C173.194,104.119,176.615,104.845,177.162,102.697z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M200.683,104.167c-1.456-0.933-2.859,0.44-2.205-3.675
c0.861,0.119,0.62,1.34,0.735,2.205c0.861-0.119,0.619-1.34,0.734-2.205c3.201,3.407-0.348,6.558-2.205,7.35
C196.005,109.11,199.516,104.705,200.683,104.167z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M152.173,104.167c1.606,0.108,1.578,1.852,1.47,3.675
c-1.666,0.723-2.671-0.252-2.94,2.94c-1.35-1.1-1.833-3.068-3.675-3.675c0.417-0.686,2.043-1.113,0.735-1.47
c-0.363-0.964,3.647-0.323,2.94,1.47C152.062,106.997,151.153,104.618,152.173,104.167z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M186.718,105.637c1.341,0.571-0.44,0.79-1.471,0.735
c0.021,0.715,0.9,0.57,1.471,0.735c-1.615,3.041-3.447,5.863-5.88,8.085c1.829,1.342,1.746-0.518,2.205,1.47
c-0.178,1.895-1.265-1.009-2.94,0c0.756,2.841,1.937,2.899,0.735,5.145c2.016-0.388,3.072-2.837,3.675,0
c2.201-0.484-0.703-1.571,0-2.94c0.654,0.571,1.659,0.79,2.939,0.735c-0.173-2.031-4.146-0.263-3.675-2.94
c1.84,1.064,2.887,1.78,5.146,0.735c-2.236-1.281,2.006-2.831,0-1.47c0.182,0.798,1.461,0.499,1.47,1.47
c2.299-3.745,5.449-0.806,4.41-7.35c-1.043-0.063-2.082-0.121-2.205,0.735c-0.408-3.128,3.24-0.956,5.88,0
c0.191,2.151-1.662,2.258-1.47,4.41c-4.872-0.175-1.821,2.456-2.205,4.41c-0.625,0.135-0.693-0.287-0.735-0.735
c-0.794,10.327-26.139,2.21-27.93,13.965c-5.521,0.604-6.028,6.222-8.82,9.555c-0.753-3.056,0.68-0.647-0.735-5.88
c-0.625-0.135-0.693,0.287-0.734,0.735c-2.417-1.09,2.181-1.144,2.205-2.205c2.141,1.031-1.304,2.532-0.735,4.41
c1.813-0.434,4.178-4.095,2.94-5.145c0.851-0.322,2.929-0.702,2.939-3.675c3.299,0.604,3.282-2.108,5.146-2.94
c-0.021-0.715-0.9-0.57-1.471-0.735c0.835-2.402,2.459,0.182,2.94-2.94c0.654,0.326,1.314,0.645,1.47,1.47
c1.607-0.108,1.578-1.852,1.471-3.675c0.653,0.326,1.314,0.646,1.47,1.47c3.038-3.467,0.999-3.431,5.88-3.675
c-0.722-2.06-1.961,1.067-2.94-0.735c0.417-0.687,2.044-1.114,0.735-1.47c1.51,0.194,0.623-1.673,2.94-2.205
c0.207-1.187-0.695-1.265-0.735-2.205c0.703-0.936,1.501,1.805,2.205-1.47c-0.411-1.661-1.146,0.891-2.205-0.735
C184.604,110.043,183.415,108.328,186.718,105.637z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M158.053,115.927c1.875-1.998,1.856,4.014-0.735,3.675
c0.776,1.183,2.913-1.197,1.47,2.205c2.148-0.547,1.423-3.968,4.41-3.675c0.852,1.088,0.852,1.853,0,2.94
c1.461-0.009,0.935-2.005,2.94-1.47c0.325,0.654,0.646,1.314,1.47,1.47c1.19,5.533-7.574,5.169-5.145,0.735
c-4.624,2.237-5.797,7.923-7.351,13.23c-3.857-1.697,1.651-8.464-2.939-8.82c0.011-2.684,2.089-3.301,2.939-5.145
c2.254,0.722-0.845,1.961-0.734,2.94C157.269,124.97,157.297,117.272,158.053,115.927z M158.053,122.542
c-0.945,0.829-3.58,3.692-2.205,4.41C155.947,125.279,160.271,124.104,158.053,122.542z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M146.292,118.867c-1.627,2.401,1.289-0.182,1.47,2.94
c1.091-0.321-0.29-0.94,0-2.205c1.293,0.177,1.675,1.265,1.47,2.94c-1.87,0.8-2.85-3.442-2.94,0c-0.654-0.452-1.314-0.888-1.47,0
c-0.834-0.14-0.408-0.737,0-0.735c0.094-1.318-0.899-1.55-0.735-2.94c-0.099,2.661-2.252,2.474-0.735-0.735
C144.006,118.703,145.012,118.922,146.292,118.867z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M147.027,126.217c-2.746,0.786-2.095-1.825-4.41-1.47
c2.307-1.646,2.204,1.484,5.88,0c0.135-0.625-0.287-0.693-0.735-0.735c0.119-0.861,1.34-0.62,2.205-0.735
C149.303,124.573,148.323,125.553,147.027,126.217z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M150.703,124.012c1.047,1.875-0.541,5.018,2.205,5.145
c0.203,3.927-3.142-0.253-2.94,3.675c0.639,2.171,1.1-1.706,2.94-0.735c-1.577,2.097,0.49,7.84-3.675,7.35c0-1.715,0-3.43,0-5.145
c-3.898,2.016,0.009,9.898-6.615,11.76c0.085,2.09,5.114-0.567,2.94-0.735c0.14-0.834,0.737-0.408,0.735,0
c0.625,0.135,0.693-0.287,0.735-0.735c-0.328,3.397-6.316,2.948-7.35,8.085c-1.152-0.26,0.643-1.294-0.735-1.47
c3.993-2.054,2.354-5.377,4.41-8.82c-2.3,2.979-7.474,7.924-5.145,11.025c-0.665,1.295-2.706,1.214-2.94,2.939
c-1.986-1.1,1.273-3.067,2.205-3.675c-1.903-0.945-1.414-2.416-0.735-4.41c-2.21,1.22-1.396,5.463-4.41,5.88
c-0.113,2.697,3.764,2.816,0.735,4.409c-1.543-0.205,1.065-1.963-0.735-2.939c-2.354,1.954-2.1,3.67-3.675,5.145
c0.14,0.835,0.737,0.408,0.735,0c1.309,0.539,0.106,1.337,0,2.205c-0.49,0-0.98,0-1.47,0c2.45-12.494,11.829-18.062,18.375-26.46
c-0.993,0.722-1.469,1.961-2.205,2.94c-0.644-3.111,5.124-6.133,5.88-10.29c-1.686-1.899-3.332,1.904-5.88-0.735
c-1.359,0.111-0.451,2.489-1.47,2.94c-3.668-1-3.68,2.417-10.29,0.735c3.521-2.188,0.104-1.269-0.735-3.675
c-0.692-1.587,2.19,1.703,3.675-0.735c0.093,1.318-0.9,1.55-0.735,2.94c2.874-3.932,10.509-1.78,16.17-3.675
C149.037,125.82,149.037,125.144,150.703,124.012z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M88.228,125.482c4.012,0.636-3.248,6.923,0.735,7.35
c-0.242,1.718-0.883,3.037-1.47,4.41C84.94,132.513,87.434,130.002,88.228,125.482z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M116.893,128.422c0.508,0.983,1.96-0.363,3.675,0.735c0,1.225,0,2.45,0,3.675
c-0.715-0.02-0.57-0.9-0.735-1.47c-0.965,0.882-1.156,0.287-2.205,0c0.26-0.965,1.294-1.156,1.47-2.205
c-1.829-0.114-2.275,1.154-3.675,1.47c-1.769-0.442,2.188-1.137,0.735-3.675C117,127.211,118.021,128.246,116.893,128.422z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M124.978,127.687c2.152,0.661,2.258,0.661,4.41,0
c-1.096-0.157-0.959,4.072,1.47,2.94c1.091,0.321-0.29,0.94,0,2.205c-2.347-2.365-4.171,0.786-7.35-2.205
C123.517,129.167,125.513,129.692,124.978,127.687z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M202.152,136.507c0.275,2.907,3.908-1.138,3.675,2.94
c1.695,0.225,0.816-2.123,2.205-2.205c1.166-1.502-1.186,4.812-4.41,3.675c0.507-1.977-1.519-1.42-2.204-2.205
c1.105-3.549,2.412-6.897,2.939-11.025c2.25,1.826-0.88,4.461,0,7.35c2.643-1.767,3.063-5.757,6.615-6.615
c-0.83,0.885-2.098,1.332-1.47,3.675c-1.499-1.477-1.441,0.325-2.94,0.735c1.749,0.501,2.271,4.078-0.735,3.675
c0.571-0.654,0.791-1.66,0.735-2.94C205.083,136.163,204.159,134.745,202.152,136.507z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M219.058,136.507c1.376-0.704-2.154,5.418-2.94,7.35
C213.548,140.76,217.194,137.612,219.058,136.507z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M197.742,137.242c1.502,2.578-5.185,1.526-2.205,4.41
c1.848,1.377-1.222,0.081-2.205,0c1.78-1.466,0.739-1.16,1.471-3.675C196.643,138.069,197.104,139.38,197.742,137.242z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M154.378,137.977c1.123,1.651-3.711,3.611-4.411,5.88
C144.878,143.354,153.333,142.019,154.378,137.977z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M203.622,146.062c0.235,1.705-0.118,2.822-0.734,3.675
c0.625,0.135,0.693-0.287,0.734-0.735c-0.768,4.558-3.626-5.897-5.145-3.675c-1.759-1.102,1.164-3.678,2.94-3.675
c-0.141-0.595-0.738-0.732-0.735-1.47c-0.752,0.718-1.487,1.453-2.205,2.205c-0.625,0.135-0.693-0.287-0.735-0.735
c0.136-0.625-0.286-0.693-0.734-0.735c0.461-1.499,3.713-0.207,2.939-2.94C203.798,139.573,199.335,145.898,203.622,146.062z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M205.827,149.737c0.515,2.262,0.515,1.413,0,3.675
c1.67-0.536,2.198-2.212,2.94-3.675c0.639,3.084-5.657,9.218-7.35,8.085c0.382,1.087,1.087,1.853,1.47,2.939
c0.965-0.26,1.156-1.294,2.205-1.47c1.034-1.25-2.403,4.648-5.146,4.41c1.156-0.56,1.099-2.332,2.94-2.205
c-0.835-0.705-2.458-4.309-2.94-2.94c0.008-2.459,3.756-7.502,5.88-10.29c-1.441-0.07-1.323-2.201-2.939-5.145
c0.654-0.326,1.314-0.646,1.47-1.47c0.308,0.906,0.941,2.034,2.94,2.94c0.021-1.988,1.634-0.024,2.939-2.94
c0.739,0.315,0.03,5.372-2.939,4.41C208.061,146.757,207.969,149.849,205.827,149.737z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M191.128,144.592c2.271,1.88,3.943-1.985,5.145-1.47
c1.982,2.27-2.469,4.359-3.675,5.88c-2.304-0.508-0.958-1.96-0.735-3.675c-1.694-0.225-0.816,2.123-2.205,2.205
C188.102,149.324,190.522,145.226,191.128,144.592z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M189.657,150.472c-0.362-0.964,3.647-0.323,2.94,1.47
c1.389-0.551,0.51-2.194,2.205-0.735c0.374-0.934,1.395-4.076,4.41-4.41c0.822,4.695-2.131,7.315-4.41,9.555
c-0.017-1.731,1.266-2.164,0.734-4.409c-0.997,0.227-0.609,1.84-0.734,2.939c-1.527-0.411-1.558-1.146-0.735-2.205
c-2.361-0.452-1.007,2.646-2.939,0.735c-2.671,2.188-3.962,4.627-7.351,4.41c0.69-2.981,3.169-5.582,5.88-4.41
C190.85,153.938,190.85,149.946,189.657,150.472z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M197.742,161.497c-1.842,0.127-1.784-1.646-2.939-2.205
c-1.337,0.851-0.664,3.716-3.675,3.675c1.133,1.79,5.077,0.236,5.88-0.734c-0.354,3.983-11.766,2.225-12.495,2.939
c0.118,0.861,1.34,0.62,2.205,0.735c2.378,0.846-1.974,0.832-2.94,0.734c-0.967-3.192-0.65-2.217,1.47-2.939
c0.218-1.933-2.852-0.579-2.204-2.94c1.707,0.498,1.967,2.442,3.675,2.94c5.322-4.528,11.612-5.481,13.965-13.965
c0.625-0.135,0.693,0.287,0.735,0.735c-0.136,0.625,0.286,0.693,0.734,0.735C200.91,153.552,197.779,156.027,197.742,161.497z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M213.912,147.532c3.728-0.668-7.716,10.658-8.085,10.29
c-0.692-0.693,18.229-18.229,2.94-2.94c1.044-1.044,1.567-2.663,2.939-2.939C211.719,149.749,213.618,149.443,213.912,147.532z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M255.072,80.647c-2.489,0.451-5.676,0.204-5.145,3.675
c-1.826-2.835,0.356-7.951,1.47-10.29C254.542,74.317,254.898,77.391,255.072,80.647z"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M224.938,77.707c0.69-3.513,2.378,0.297,2.94,2.94
c-0.478,0.748-1.69,0.76-1.471,2.205c-1.319-1.996-9.06-7.74-2.205-8.82C223.266,75.969,224.971,75.586,224.938,77.707z"/>
</g>
</g>
<g>
<path d="M18.655,159.962v39.241c0,1.453,0.049,2.668,0.15,3.645c0.1,0.979,0.275,1.668,0.526,2.068
c0.25,0.351,0.613,0.613,1.09,0.789c0.476,0.176,1.165,0.289,2.067,0.338l3.608,0.15v2.256c-1.955-0.051-3.822-0.102-5.601-0.15
c-1.779-0.051-3.546-0.076-5.3-0.076c-0.501,0-1.015,0-1.541,0s-1.065,0-1.616,0c-1.304,0.05-2.606,0.088-3.909,0.113
c-1.304,0.024-2.657,0.063-4.06,0.113v-2.256l3.608-0.226c0.902,0,1.603-0.087,2.105-0.263c0.5-0.176,0.852-0.438,1.052-0.789
c0.25-0.4,0.425-1.09,0.526-2.068c0.1-0.977,0.15-2.166,0.15-3.57v-33.828c0-1.402-0.05-2.58-0.15-3.533
c-0.101-0.951-0.276-1.627-0.526-2.029c-0.201-0.4-0.552-0.676-1.052-0.826c-0.502-0.15-1.203-0.25-2.105-0.301l-3.082-0.301
v-2.256c0.65,0.051,1.327,0.088,2.029,0.113c0.702,0.025,1.403,0.037,2.105,0.037c1.353,0.051,2.769,0.088,4.248,0.113
c1.478,0.025,2.943,0.037,4.397,0.037c1.503,0,2.981-0.012,4.435-0.037c1.453-0.025,2.906-0.063,4.36-0.113
c0.701,0,1.428-0.012,2.18-0.037c0.752-0.025,1.478-0.037,2.18-0.037c0.1,0,0.212,0,0.338,0c0.125,0,0.237,0,0.338,0
c5.212,0,8.845,1.014,10.9,3.044c2.054,2.03,3.232,4.272,3.533,6.728c0,0.252,0.012,0.502,0.038,0.752
c0.024,0.252,0.037,0.502,0.037,0.752c0,4.059-1.503,7.305-4.51,9.734c-3.007,2.432-6.314,3.998-9.923,4.699
c-0.05,0.05-0.088,0.074-0.113,0.074c-0.025,0-0.063,0-0.112,0l17.064,22.176c0.551,0.803,1.227,1.316,2.029,1.541
c0.801,0.227,1.653,0.39,2.556,0.489v2.181c-0.301,0-0.614-0.014-0.939-0.038c-0.326-0.026-0.639-0.038-0.939-0.038
c-0.602-0.051-1.191-0.088-1.767-0.112c-0.577-0.026-1.165-0.038-1.767-0.038c-0.451,0-0.891,0.012-1.315,0.038
c-0.427,0.024-0.84,0.038-1.24,0.038c-0.15,0-0.289,0.012-0.414,0.037c-0.125,0.024-0.263,0.037-0.413,0.037
c-0.301,0-0.59,0.012-0.865,0.038c-0.276,0.024-0.563,0.038-0.864,0.038c-0.251-0.301-0.501-0.627-0.752-0.979
c-0.251-0.35-0.501-0.676-0.751-0.977c-0.101-0.15-0.201-0.275-0.301-0.376c-0.101-0.1-0.176-0.2-0.226-0.301
c-0.201-0.25-0.389-0.488-0.564-0.714c-0.176-0.226-0.338-0.438-0.488-0.639l-13.832-18.643c-0.351-0.451-0.727-0.902-1.127-1.354
c-0.402-0.451-0.802-0.902-1.203-1.354c-0.251-0.35-0.514-0.676-0.789-0.977c-0.276-0.301-0.539-0.602-0.79-0.902l0.451-0.978
c0.35,0.101,0.701,0.15,1.053,0.15c0.35,0,0.701,0,1.052,0c0.5,0,1.002-0.012,1.503-0.038c0.5-0.024,1.002-0.062,1.503-0.112
c0.25-0.05,0.5-0.087,0.752-0.112c0.25-0.025,0.5-0.063,0.751-0.113c2.656-0.451,4.949-1.628,6.878-3.533
c1.929-1.904,2.894-4.234,2.894-6.99c0-3.959-1.191-6.665-3.571-8.119c-2.381-1.453-4.999-2.205-7.855-2.256
c-0.101,0-0.188,0-0.263,0s-0.139,0-0.188,0c-1.704,0-3.146,0.088-4.322,0.264c-1.178,0.176-2.293,0.389-3.345,0.639V159.962z"/>
<path d="M72.028,172.967c4.108,0.051,7.518,1.305,10.224,3.759c2.706,2.456,4.06,5.814,4.06,10.073c0,0.502,0,1.015,0,1.541
s-0.051,1.065-0.15,1.616c-1.104,0.2-2.23,0.364-3.383,0.489c-1.153,0.125-2.281,0.238-3.383,0.338
c-1.153,0.051-2.293,0.102-3.421,0.15c-1.127,0.051-2.268,0.075-3.42,0.075H61.579c0,4.438,1.115,8.015,3.346,10.733
c2.229,2.718,5.649,4.075,10.261,4.075c1.754,0,3.346-0.35,4.773-1.052c1.429-0.701,2.819-1.453,4.172-2.256
c0.101-0.049,0.176-0.087,0.226-0.112c0.05-0.024,0.101-0.063,0.15-0.112l0.978,0.676l-1.504,2.557
c-0.101,0.25-0.45,0.551-1.052,0.901s-1.329,0.752-2.181,1.202c-0.853,0.352-1.792,0.689-2.818,1.016
c-1.028,0.325-2.067,0.588-3.12,0.789c-0.501,0.1-0.99,0.176-1.466,0.226c-0.477,0.05-0.939,0.075-1.391,0.075
c-5.664,0-9.974-1.616-12.93-4.849c-2.958-3.232-4.436-7.379-4.436-12.44c0-3.658,0.477-6.627,1.429-8.908
c0.951-2.28,2.28-4.098,3.984-5.451c0.701-0.551,1.453-1.102,2.255-1.653c0.801-0.551,1.604-1.002,2.405-1.353
c0.701-0.351,1.391-0.65,2.067-0.902c0.677-0.25,1.341-0.476,1.992-0.677c0.551-0.15,1.09-0.263,1.616-0.339
c0.526-0.074,1.09-0.112,1.691-0.112V172.967z M71.652,175.447c-2.757,0.051-5.111,1.053-7.066,3.008
c-1.954,1.954-2.906,4.836-2.856,8.645c0,0.15,0,0.301,0,0.451s0,0.301,0,0.451c1.654,0.201,3.308,0.326,4.962,0.375
c1.653,0.051,3.257,0.076,4.811,0.076c1.453,0,2.856-0.025,4.21-0.076c1.354-0.049,2.655-0.1,3.909-0.15
c0-3.507-0.652-6.502-1.955-8.982c-1.304-2.48-3.308-3.721-6.014-3.721V175.447z"/>
<path d="M118.183,203.262l-7.818,4.961c-0.401,0.301-0.89,0.514-1.466,0.64c-0.576,0.124-1.141,0.212-1.691,0.263
c-0.101,0-0.2,0-0.301,0s-0.2,0-0.3,0c-0.051,0-0.113,0-0.188,0s-0.163,0-0.263,0c-5.213-0.302-8.921-1.908-11.126-4.821
c-2.206-2.913-3.434-6.103-3.684-9.567c-0.051-0.251-0.075-0.514-0.075-0.791c0-0.276,0-0.565,0-0.866
c0-3.817,0.701-6.905,2.105-9.267c1.402-2.36,3.131-4.193,5.187-5.499l6.314-4.144c0.551-0.25,1.303-0.5,2.255-0.752
c0.952-0.25,1.929-0.4,2.932-0.451c0.101,0,0.188,0,0.264,0s0.138,0,0.188,0c1.854,0,3.671,0.214,5.45,0.639
c1.778,0.427,3.496,1.041,5.149,1.842l3.684-2.405l1.429,0.827c-0.402,0.809-0.715,1.703-0.94,2.688s-0.389,1.981-0.488,2.992
c-0.101,0.857-0.176,1.703-0.226,2.535c-0.051,0.834-0.075,1.604-0.075,2.31v37.06c0,1.002,0.024,1.88,0.075,2.631
c0.05,0.752,0.15,1.328,0.301,1.729c0.15,0.351,0.425,0.602,0.827,0.752c0.4,0.15,0.927,0.25,1.578,0.301l3.458,0.226v2.255
c-0.451,0-0.915-0.013-1.391-0.038c-0.477-0.025-0.966-0.037-1.466-0.037c-0.952-0.051-1.968-0.088-3.045-0.113
c-1.078-0.025-2.243-0.037-3.495-0.037c-0.952,0-1.879,0.012-2.781,0.037s-1.805,0.038-2.706,0.038
c-0.201,0-0.414,0.012-0.64,0.038c-0.226,0.024-0.464,0.037-0.714,0.037c-0.552,0-1.116,0.012-1.691,0.037
c-0.576,0.025-1.165,0.038-1.767,0.038v-2.255l4.36-0.226c0.65-0.051,1.177-0.15,1.578-0.301c0.4-0.15,0.677-0.401,0.827-0.752
c0.15-0.401,0.25-0.978,0.301-1.729c0.05-0.751,0.075-1.629,0.075-2.631V203.262z M118.183,199.188v-19.672
c-1.003-0.904-2.131-1.582-3.383-2.035c-1.253-0.451-2.507-0.729-3.759-0.83c-0.401-0.049-0.802-0.087-1.202-0.113
c-0.402-0.023-0.803-0.037-1.203-0.037c-0.401,0-0.815,0.025-1.24,0.075c-0.427,0.051-0.84,0.126-1.24,0.228
c-2.105,0.502-4.022,1.721-5.751,3.654c-1.729,1.936-2.594,4.861-2.594,8.781c0,0.252,0,0.516,0,0.792s0.024,0.539,0.075,0.79
c0.15,3.518,1.09,6.747,2.819,9.686c1.729,2.94,4.647,4.41,8.758,4.41c0.25,0,0.525-0.012,0.826-0.038
c0.301-0.024,0.576-0.088,0.827-0.188c1.053-0.251,2.079-0.666,3.082-1.243c1.002-0.578,1.829-1.193,2.48-1.847
c0.451-0.401,0.814-0.816,1.091-1.244c0.274-0.428,0.413-0.791,0.413-1.094V199.188z"/>
<path d="M168.549,200.545c0,1.055,0.025,1.959,0.074,2.713c0.051,0.754,0.15,1.331,0.301,1.732c0.15,0.352,0.426,0.602,0.828,0.752
c0.4,0.15,0.926,0.252,1.578,0.301l3.082,0.15v2.256c-0.301,0-0.613-0.014-0.939-0.038c-0.326-0.026-0.664-0.038-1.016-0.038
c-0.5-0.051-0.99-0.088-1.465-0.112c-0.477-0.026-0.965-0.038-1.467-0.038c-0.201,0-0.412,0-0.639,0c-0.225,0-0.439,0-0.639,0
c-0.703,0-1.416,0.012-2.143,0.038c-0.727,0.024-1.441,0.062-2.143,0.112c-0.352,0-0.689,0.012-1.014,0.038
c-0.328,0.024-0.666,0.038-1.016,0.038c0-0.102,0-0.188,0-0.264s0.025-0.162,0.074-0.264c0.051-0.951,0.102-1.928,0.15-2.932
c0.051-1.002,0.076-2.129,0.076-3.383l-5.037,4.586c-1.053,1.004-2.268,1.742-3.645,2.218c-1.381,0.476-2.97,0.714-4.775,0.714
c-4.811,0-7.769-1.152-8.87-3.457c-1.103-2.305-1.68-4.861-1.729-7.668c0-0.25,0-0.514,0-0.789s0-0.564,0-0.865v-12.704
c0-1.152-0.038-2.13-0.113-2.932s-0.214-1.353-0.413-1.653c-0.201-0.301-0.515-0.514-0.939-0.64
c-0.427-0.124-1.041-0.188-1.842-0.188h-2.781v-2.254c0.1,0.05,0.199,0.063,0.3,0.037s0.176-0.037,0.226-0.037
c1.754-0.2,3.445-0.514,5.074-0.94c1.628-0.425,3.094-0.89,4.398-1.391c0.3-0.1,0.601-0.212,0.901-0.339
c0.301-0.124,0.576-0.236,0.827-0.338l0.677,0.226v22.229c0,3.014,0.5,5.299,1.503,6.857c1.002,1.557,3.232,2.336,6.691,2.336
c1.904,0,3.684-0.639,5.338-1.918c1.652-1.277,2.855-2.668,3.607-4.172c0.199-0.451,0.35-0.889,0.451-1.314
c0.1-0.426,0.15-0.84,0.15-1.241v-12.328c0-1.152-0.037-2.13-0.113-2.932c-0.074-0.802-0.213-1.353-0.412-1.653
c-0.201-0.301-0.516-0.514-0.941-0.64c-0.426-0.124-1.039-0.188-1.842-0.188h-2.781v-2.254c0.1,0.05,0.201,0.063,0.301,0.037
s0.176-0.037,0.227-0.037c1.754-0.2,3.445-0.514,5.074-0.94c1.627-0.425,3.094-0.89,4.396-1.391c0.301-0.1,0.602-0.212,0.902-0.339
c0.301-0.124,0.576-0.236,0.828-0.338l0.676,0.226V200.545z"/>
<path d="M195.461,172.967c4.107,0.051,7.518,1.305,10.223,3.759c2.707,2.456,4.061,5.814,4.061,10.073c0,0.502,0,1.015,0,1.541
s-0.051,1.065-0.15,1.616c-1.104,0.2-2.23,0.364-3.383,0.489c-1.154,0.125-2.281,0.238-3.383,0.338
c-1.154,0.051-2.293,0.102-3.422,0.15c-1.127,0.051-2.268,0.075-3.42,0.075h-10.975c0,4.438,1.115,8.015,3.346,10.733
c2.229,2.718,5.648,4.075,10.26,4.075c1.754,0,3.346-0.35,4.773-1.052c1.43-0.701,2.82-1.453,4.172-2.256
c0.102-0.049,0.176-0.087,0.227-0.112c0.049-0.024,0.1-0.063,0.15-0.112l0.977,0.676l-1.504,2.557
c-0.1,0.25-0.449,0.551-1.051,0.901s-1.33,0.752-2.182,1.202c-0.852,0.352-1.791,0.689-2.818,1.016
c-1.027,0.325-2.066,0.588-3.119,0.789c-0.502,0.1-0.99,0.176-1.467,0.226s-0.939,0.075-1.391,0.075
c-5.664,0-9.973-1.616-12.93-4.849s-4.436-7.379-4.436-12.44c0-3.658,0.477-6.627,1.43-8.908c0.951-2.28,2.279-4.098,3.984-5.451
c0.701-0.551,1.453-1.102,2.254-1.653c0.801-0.551,1.604-1.002,2.406-1.353c0.701-0.351,1.391-0.65,2.066-0.902
c0.678-0.25,1.342-0.476,1.992-0.677c0.551-0.15,1.09-0.263,1.617-0.339c0.525-0.074,1.09-0.112,1.691-0.112V172.967z
M195.084,175.447c-2.756,0.051-5.111,1.053-7.066,3.008c-1.953,1.954-2.906,4.836-2.855,8.645c0,0.15,0,0.301,0,0.451
s0,0.301,0,0.451c1.654,0.201,3.307,0.326,4.961,0.375c1.654,0.051,3.258,0.076,4.811,0.076c1.453,0,2.857-0.025,4.211-0.076
c1.354-0.049,2.654-0.1,3.908-0.15c0-3.507-0.652-6.502-1.955-8.982s-3.307-3.721-6.014-3.721V175.447z"/>
<path d="M215.154,197.474h2.256v3.082c0,2.005,0.789,3.509,2.367,4.511c1.58,1.002,3.32,1.604,5.225,1.804
c0.35,0.05,0.701,0.088,1.053,0.112c0.35,0.026,0.701,0.038,1.053,0.038c2.355,0,4.41-0.64,6.164-1.921
c1.754-1.279,2.631-2.926,2.631-4.936c0-1.205-0.313-2.258-0.939-3.162c-0.627-0.905-1.516-1.508-2.668-1.81
c-1.004-0.251-2.08-0.515-3.232-0.79c-1.154-0.276-2.307-0.565-3.459-0.867c-0.752-0.201-1.492-0.414-2.217-0.641
c-0.727-0.225-1.441-0.465-2.143-0.717c-1.805-0.701-3.32-1.668-4.549-2.899s-1.842-3.001-1.842-5.312
c0-4.068,1.479-6.918,4.436-8.551s6.189-2.449,9.697-2.449c1.754,0,3.445,0.176,5.074,0.526s3.17,0.802,4.623,1.353
c-0.102,1.354-0.176,2.732-0.225,4.135c-0.051,1.404-0.102,2.857-0.15,4.36l-2.105-0.075v-1.729c0-1.854-0.865-3.309-2.594-4.36
s-3.621-1.628-5.676-1.729c-0.1,0-0.188,0-0.262,0c-0.076,0-0.139,0-0.189,0c-2.506,0-4.271,0.453-5.299,1.357
s-1.666,1.908-1.916,3.013c-0.051,0.302-0.09,0.591-0.113,0.866c-0.025,0.277-0.037,0.565-0.037,0.866
c0,1.608,0.65,2.876,1.953,3.806c1.303,0.929,3.383,1.645,6.24,2.146c4.459,0.754,7.768,1.81,9.922,3.164
c2.154,1.355,3.232,3.516,3.232,6.479c0,4.118-1.729,7.245-5.186,9.38c-3.459,2.133-7.268,3.201-11.428,3.201
c-0.801,0-1.604-0.025-2.404-0.075c-0.803-0.05-1.58-0.15-2.33-0.301c-0.953-0.101-1.906-0.276-2.857-0.526
c-0.953-0.251-1.93-0.525-2.932-0.826c0.301-1.303,0.5-2.656,0.602-4.06c0.1-1.403,0.174-2.831,0.225-4.285
c0-0.351,0-0.701,0-1.052s0-0.701,0-1.053V197.474z"/>
<path d="M245.6,179.808v-1.579l5.863-2.631v-5.262c0-0.951-0.025-1.917-0.074-2.895c-0.051-0.977-0.127-1.966-0.227-2.969
c0.701-0.199,1.365-0.426,1.992-0.677c0.627-0.25,1.215-0.501,1.768-0.751c0.551-0.251,1.051-0.501,1.502-0.752
c0.451-0.25,0.828-0.477,1.129-0.678l0.902,0.752c-0.102,0.803-0.201,1.629-0.301,2.48c-0.102,0.854-0.178,1.705-0.227,2.557
c-0.102,1.003-0.176,1.992-0.225,2.969c-0.051,0.978-0.076,1.893-0.076,2.744v2.781l10.299-0.451l-0.676,4.061h-9.623v19.695
c0,1.754,0.301,3.07,0.902,3.945c0.602,0.878,1.678,1.316,3.232,1.316c0.752,0,1.439-0.1,2.066-0.301
c0.627-0.199,1.166-0.451,1.617-0.752c0.15-0.049,0.287-0.113,0.414-0.188c0.123-0.076,0.236-0.138,0.338-0.188l0.977,1.202
l-3.607,3.984c-0.201,0.25-0.59,0.438-1.166,0.564c-0.576,0.125-1.191,0.213-1.842,0.263c-0.301,0.05-0.613,0.075-0.939,0.075
s-0.639,0-0.939,0c-2.006,0-3.734-0.552-5.188-1.654c-1.453-1.102-2.18-3.182-2.18-6.238v-21.725L245.6,179.808z"/>
<path d="M271.533,197.474h2.256v3.082c0,2.005,0.789,3.509,2.367,4.511c1.58,1.002,3.32,1.604,5.225,1.804
c0.35,0.05,0.701,0.088,1.053,0.112c0.35,0.026,0.701,0.038,1.053,0.038c2.355,0,4.41-0.64,6.164-1.921
c1.754-1.279,2.631-2.926,2.631-4.936c0-1.205-0.313-2.258-0.939-3.162c-0.627-0.905-1.516-1.508-2.668-1.81
c-1.004-0.251-2.08-0.515-3.232-0.79c-1.154-0.276-2.307-0.565-3.459-0.867c-0.752-0.201-1.492-0.414-2.217-0.641
c-0.727-0.225-1.441-0.465-2.143-0.717c-1.805-0.701-3.32-1.668-4.549-2.899s-1.842-3.001-1.842-5.312
c0-4.068,1.479-6.918,4.436-8.551s6.189-2.449,9.697-2.449c1.754,0,3.445,0.176,5.074,0.526s3.17,0.802,4.623,1.353
c-0.102,1.354-0.176,2.732-0.225,4.135c-0.051,1.404-0.102,2.857-0.15,4.36l-2.105-0.075v-1.729c0-1.854-0.865-3.309-2.594-4.36
s-3.621-1.628-5.676-1.729c-0.1,0-0.188,0-0.262,0c-0.076,0-0.139,0-0.189,0c-2.506,0-4.271,0.453-5.299,1.357
s-1.666,1.908-1.916,3.013c-0.051,0.302-0.09,0.591-0.113,0.866c-0.025,0.277-0.037,0.565-0.037,0.866
c0,1.608,0.65,2.876,1.953,3.806c1.303,0.929,3.383,1.645,6.24,2.146c4.459,0.754,7.768,1.81,9.922,3.164
c2.154,1.355,3.232,3.516,3.232,6.479c0,4.118-1.729,7.245-5.186,9.38c-3.459,2.133-7.268,3.201-11.428,3.201
c-0.801,0-1.604-0.025-2.404-0.075c-0.803-0.05-1.58-0.15-2.33-0.301c-0.953-0.101-1.906-0.276-2.857-0.526
c-0.953-0.251-1.93-0.525-2.932-0.826c0.301-1.303,0.5-2.656,0.602-4.06c0.1-1.403,0.174-2.831,0.225-4.285
c0-0.351,0-0.701,0-1.052s0-0.701,0-1.053V197.474z"/>
</g>
<g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 49 KiB

requests/__init__.py
+14 -8
View File
@@ -6,14 +6,14 @@
# /
"""
requests HTTP library
Requests HTTP library
~~~~~~~~~~~~~~~~~~~~~
Requests is an HTTP library, written in Python, for human beings. Basic GET
usage:
>>> import requests
>>> r = requests.get('http://python.org')
>>> r = requests.get('https://www.python.org')
>>> r.status_code
200
>>> 'Python is a programming language' in r.content
@@ -22,7 +22,7 @@ usage:
... or POST:
>>> payload = dict(key1='value1', key2='value2')
>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> r = requests.post('http://httpbin.org/post', data=payload)
>>> print(r.text)
{
...
@@ -36,17 +36,17 @@ usage:
The other HTTP methods are supported - see `requests.api`. Full documentation
is at <http://python-requests.org>.
:copyright: (c) 2014 by Kenneth Reitz.
:copyright: (c) 2015 by Kenneth Reitz.
:license: Apache 2.0, see LICENSE for more details.
"""
__title__ = 'requests'
__version__ = '2.2.0'
__build__ = 0x020200
__version__ = '2.9.0'
__build__ = 0x020900
__author__ = 'Kenneth Reitz'
__license__ = 'Apache 2.0'
__copyright__ = 'Copyright 2014 Kenneth Reitz'
__copyright__ = 'Copyright 2015 Kenneth Reitz'
# Attempt to enable urllib3's SNI support, if possible
try:
@@ -62,7 +62,8 @@ from .sessions import session, Session
from .status_codes import codes
from .exceptions import (
RequestException, Timeout, URLRequired,
TooManyRedirects, HTTPError, ConnectionError
TooManyRedirects, HTTPError, ConnectionError,
FileModeWarning,
)
# Set default logging handler to avoid "No handler found" warnings.
@@ -75,3 +76,8 @@ except ImportError:
pass
logging.getLogger(__name__).addHandler(NullHandler())
import warnings
# FileModeWarnings go off per the default.
warnings.simplefilter('default', FileModeWarning, append=True)
requests/adapters.py
+112 -59
View File
@@ -8,28 +8,38 @@ This module contains the transport adapters that Requests uses to define
and maintain connections.
"""
import os.path
import socket
from .models import Response
from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
from .packages.urllib3.response import HTTPResponse
from .packages.urllib3.util import Timeout as TimeoutSauce
from .compat import urlparse, basestring, urldefrag, unquote
from .packages.urllib3.util.retry import Retry
from .compat import urlparse, basestring
from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,
except_on_missing_scheme, get_auth_from_url)
prepend_scheme_if_needed, get_auth_from_url, urldefragauth,
select_proxy)
from .structures import CaseInsensitiveDict
from .packages.urllib3.exceptions import MaxRetryError
from .packages.urllib3.exceptions import TimeoutError
from .packages.urllib3.exceptions import SSLError as _SSLError
from .packages.urllib3.exceptions import ClosedPoolError
from .packages.urllib3.exceptions import ConnectTimeoutError
from .packages.urllib3.exceptions import HTTPError as _HTTPError
from .packages.urllib3.exceptions import MaxRetryError
from .packages.urllib3.exceptions import NewConnectionError
from .packages.urllib3.exceptions import ProxyError as _ProxyError
from .packages.urllib3.exceptions import ProtocolError
from .packages.urllib3.exceptions import ReadTimeoutError
from .packages.urllib3.exceptions import SSLError as _SSLError
from .packages.urllib3.exceptions import ResponseError
from .cookies import extract_cookies_to_jar
from .exceptions import ConnectionError, Timeout, SSLError, ProxyError
from .exceptions import (ConnectionError, ConnectTimeout, ReadTimeout, SSLError,
ProxyError, RetryError)
from .auth import _basic_auth_str
DEFAULT_POOLBLOCK = False
DEFAULT_POOLSIZE = 10
DEFAULT_RETRIES = 0
DEFAULT_POOL_TIMEOUT = None
class BaseAdapter(object):
@@ -56,8 +66,12 @@ class HTTPAdapter(BaseAdapter):
:param pool_connections: The number of urllib3 connection pools to cache.
:param pool_maxsize: The maximum number of connections to save in the pool.
:param int max_retries: The maximum number of retries each connection
should attempt. Note, this applies only to failed connections and
timeouts, never to requests where the server returns a response.
should attempt. Note, this applies only to failed DNS lookups, socket
connections and connection timeouts, never to requests where data has
made it to the server. By default, Requests does not retry failed
connections. If you need granular control over the conditions under
which we retry a request, import urllib3's ``Retry`` class and pass
that instead.
:param pool_block: Whether the connection pool should block for connections.
Usage::
@@ -73,7 +87,10 @@ class HTTPAdapter(BaseAdapter):
def __init__(self, pool_connections=DEFAULT_POOLSIZE,
pool_maxsize=DEFAULT_POOLSIZE, max_retries=DEFAULT_RETRIES,
pool_block=DEFAULT_POOLBLOCK):
self.max_retries = max_retries
if max_retries == DEFAULT_RETRIES:
self.max_retries = Retry(0, read=False)
else:
self.max_retries = Retry.from_int(max_retries)
self.config = {}
self.proxy_manager = {}
@@ -91,7 +108,7 @@ class HTTPAdapter(BaseAdapter):
def __setstate__(self, state):
# Can't handle by adding 'proxy_manager' to self.__attrs__ because
# because self.poolmanager uses a lambda function, which isn't pickleable.
# self.poolmanager uses a lambda function, which isn't pickleable.
self.proxy_manager = {}
self.config = {}
@@ -101,14 +118,17 @@ class HTTPAdapter(BaseAdapter):
self.init_poolmanager(self._pool_connections, self._pool_maxsize,
block=self._pool_block)
def init_poolmanager(self, connections, maxsize, block=DEFAULT_POOLBLOCK):
"""Initializes a urllib3 PoolManager. This method should not be called
from user code, and is only exposed for use when subclassing the
def init_poolmanager(self, connections, maxsize, block=DEFAULT_POOLBLOCK, **pool_kwargs):
"""Initializes a urllib3 PoolManager.
This method should not be called from user code, and is only
exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param connections: The number of urllib3 connection pools to cache.
:param maxsize: The maximum number of connections to save in the pool.
:param block: Block when no free connections are available.
:param pool_kwargs: Extra keyword arguments used to initialize the Pool Manager.
"""
# save these values for pickling
self._pool_connections = connections
@@ -116,7 +136,30 @@ class HTTPAdapter(BaseAdapter):
self._pool_block = block
self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
block=block)
block=block, strict=True, **pool_kwargs)
def proxy_manager_for(self, proxy, **proxy_kwargs):
"""Return urllib3 ProxyManager for the given proxy.
This method should not be called from user code, and is only
exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param proxy: The proxy to return a urllib3 ProxyManager for.
:param proxy_kwargs: Extra keyword arguments used to configure the Proxy Manager.
:returns: ProxyManager
"""
if not proxy in self.proxy_manager:
proxy_headers = self.proxy_headers(proxy)
self.proxy_manager[proxy] = proxy_from_url(
proxy,
proxy_headers=proxy_headers,
num_pools=self._pool_connections,
maxsize=self._pool_maxsize,
block=self._pool_block,
**proxy_kwargs)
return self.proxy_manager[proxy]
def cert_verify(self, conn, url, verify, cert):
"""Verify a SSL certificate. This method should not be called from user
@@ -143,10 +186,15 @@ class HTTPAdapter(BaseAdapter):
raise Exception("Could not find a suitable SSL CA certificate bundle.")
conn.cert_reqs = 'CERT_REQUIRED'
conn.ca_certs = cert_loc
if not os.path.isdir(cert_loc):
conn.ca_certs = cert_loc
else:
conn.ca_cert_dir = cert_loc
else:
conn.cert_reqs = 'CERT_NONE'
conn.ca_certs = None
conn.ca_cert_dir = None
if cert:
if not isinstance(cert, basestring):
@@ -199,22 +247,12 @@ class HTTPAdapter(BaseAdapter):
:param url: The URL to connect to.
:param proxies: (optional) A Requests-style dictionary of proxies used on this request.
"""
proxies = proxies or {}
proxy = proxies.get(urlparse(url.lower()).scheme)
proxy = select_proxy(url, proxies)
if proxy:
except_on_missing_scheme(proxy)
proxy_headers = self.proxy_headers(proxy)
if not proxy in self.proxy_manager:
self.proxy_manager[proxy] = proxy_from_url(
proxy,
proxy_headers=proxy_headers,
num_pools=self._pool_connections,
maxsize=self._pool_maxsize,
block=self._pool_block)
conn = self.proxy_manager[proxy].connection_from_url(url)
proxy = prepend_scheme_if_needed(proxy, 'http')
proxy_manager = self.proxy_manager_for(proxy)
conn = proxy_manager.connection_from_url(url)
else:
# Only scheme should be lower case
parsed = urlparse(url)
@@ -242,14 +280,12 @@ class HTTPAdapter(BaseAdapter):
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param proxies: A dictionary of schemes to proxy URLs.
:param proxies: A dictionary of schemes or schemes and hosts to proxy URLs.
"""
proxies = proxies or {}
proxy = select_proxy(request.url, proxies)
scheme = urlparse(request.url).scheme
proxy = proxies.get(scheme)
if proxy and scheme != 'https':
url, _ = urldefrag(request.url)
url = urldefragauth(request.url)
else:
url = request.path_url
@@ -280,16 +316,11 @@ class HTTPAdapter(BaseAdapter):
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param proxies: The url of the proxy being used for this request.
:param kwargs: Optional additional keyword arguments.
"""
headers = {}
username, password = get_auth_from_url(proxy)
if username and password:
# Proxy auth usernames and passwords will be urlencoded, we need
# to decode them.
username = unquote(username)
password = unquote(password)
headers['Proxy-Authorization'] = _basic_auth_str(username,
password)
@@ -300,7 +331,10 @@ class HTTPAdapter(BaseAdapter):
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param stream: (optional) Whether to stream the request content.
:param timeout: (optional) The timeout on the request.
:param timeout: (optional) How long to wait for the server to send
data before giving up, as a float, or a :ref:`(connect timeout,
read timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param verify: (optional) Whether to verify SSL certificates.
:param cert: (optional) Any user-provided SSL certificate to be trusted.
:param proxies: (optional) The proxies dictionary to apply to the request.
@@ -314,8 +348,16 @@ class HTTPAdapter(BaseAdapter):
chunked = not (request.body is None or 'Content-Length' in request.headers)
if stream:
timeout = TimeoutSauce(connect=timeout)
if isinstance(timeout, tuple):
try:
connect, read = timeout
timeout = TimeoutSauce(connect=connect, read=read)
except ValueError as e:
# this may raise a string formatting error.
err = ("Invalid timeout {0}. Pass a (connect, read) "
"timeout tuple, or a single float to set "
"both timeouts to the same value".format(timeout))
raise ValueError(err)
else:
timeout = TimeoutSauce(connect=timeout, read=timeout)
@@ -339,7 +381,7 @@ class HTTPAdapter(BaseAdapter):
if hasattr(conn, 'proxy_pool'):
conn = conn.proxy_pool
low_conn = conn._get_conn(timeout=timeout)
low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)
try:
low_conn.putrequest(request.method,
@@ -358,7 +400,15 @@ class HTTPAdapter(BaseAdapter):
low_conn.send(b'\r\n')
low_conn.send(b'0\r\n\r\n')
r = low_conn.getresponse()
# Receive the response from the server
try:
# For Python 2.7+ versions, use buffering of HTTP
# responses
r = low_conn.getresponse(buffering=True)
except TypeError:
# For compatibility with Python 2.6 versions and back
r = low_conn.getresponse()
resp = HTTPResponse.from_httplib(
r,
pool=conn,
@@ -371,30 +421,33 @@ class HTTPAdapter(BaseAdapter):
# Then, reraise so that we can handle the actual exception.
low_conn.close()
raise
else:
# All is well, return the connection to the pool.
conn._put_conn(low_conn)
except socket.error as sockerr:
raise ConnectionError(sockerr)
except (ProtocolError, socket.error) as err:
raise ConnectionError(err, request=request)
except MaxRetryError as e:
raise ConnectionError(e)
if isinstance(e.reason, ConnectTimeoutError):
# TODO: Remove this in 3.0.0: see #2811
if not isinstance(e.reason, NewConnectionError):
raise ConnectTimeout(e, request=request)
if isinstance(e.reason, ResponseError):
raise RetryError(e, request=request)
raise ConnectionError(e, request=request)
except ClosedPoolError as e:
raise ConnectionError(e, request=request)
except _ProxyError as e:
raise ProxyError(e)
except (_SSLError, _HTTPError) as e:
if isinstance(e, _SSLError):
raise SSLError(e)
elif isinstance(e, TimeoutError):
raise Timeout(e)
raise SSLError(e, request=request)
elif isinstance(e, ReadTimeoutError):
raise ReadTimeout(e, request=request)
else:
raise
r = self.build_response(request, resp)
if not stream:
r.content
return r
return self.build_response(request, resp)
requests/api.py
+42 -17
View File
@@ -16,22 +16,28 @@ from . import sessions
def request(method, url, **kwargs):
"""Constructs and sends a :class:`Request <Request>`.
Returns :class:`Response <Response>` object.
:param method: method for the new :class:`Request` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param json: (optional) json data to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
:param files: (optional) Dictionary of 'name': file-like-objects (or {'name': ('filename', fileobj)}) for multipart encoding upload.
:param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
:param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
:param timeout: (optional) Float describing the timeout of the request.
:param timeout: (optional) How long to wait for the server to send data
before giving up, as a float, or a :ref:`(connect timeout, read
timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
:type allow_redirects: bool
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
:param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.
:param stream: (optional) if ``False``, the response content will be immediately downloaded.
:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
:return: :class:`Response <Response>` object
:rtype: requests.Response
Usage::
@@ -40,26 +46,34 @@ def request(method, url, **kwargs):
<Response [200]>
"""
session = sessions.Session()
return session.request(method=method, url=url, **kwargs)
# By using the 'with' statement we are sure the session is closed, thus we
# avoid leaving sockets open which can trigger a ResourceWarning in some
# cases, and look like a memory leak in others.
with sessions.Session() as session:
return session.request(method=method, url=url, **kwargs)
def get(url, **kwargs):
"""Sends a GET request. Returns :class:`Response` object.
def get(url, params=None, **kwargs):
"""Sends a GET request.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', True)
return request('get', url, **kwargs)
return request('get', url, params=params, **kwargs)
def options(url, **kwargs):
"""Sends a OPTIONS request. Returns :class:`Response` object.
"""Sends a OPTIONS request.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', True)
@@ -67,54 +81,65 @@ def options(url, **kwargs):
def head(url, **kwargs):
"""Sends a HEAD request. Returns :class:`Response` object.
"""Sends a HEAD request.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', False)
return request('head', url, **kwargs)
def post(url, data=None, **kwargs):
"""Sends a POST request. Returns :class:`Response` object.
def post(url, data=None, json=None, **kwargs):
"""Sends a POST request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param json: (optional) json data to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request('post', url, data=data, **kwargs)
return request('post', url, data=data, json=json, **kwargs)
def put(url, data=None, **kwargs):
"""Sends a PUT request. Returns :class:`Response` object.
"""Sends a PUT request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request('put', url, data=data, **kwargs)
def patch(url, data=None, **kwargs):
"""Sends a PATCH request. Returns :class:`Response` object.
"""Sends a PATCH request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request('patch', url, data=data, **kwargs)
def delete(url, **kwargs):
"""Sends a DELETE request. Returns :class:`Response` object.
"""Sends a DELETE request.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request('delete', url, **kwargs)
requests/auth.py
+61 -34
View File
@@ -11,15 +11,14 @@ import os
import re
import time
import hashlib
import logging
import threading
from base64 import b64encode
from .compat import urlparse, str
from .cookies import extract_cookies_to_jar
from .utils import parse_dict_header
log = logging.getLogger(__name__)
from .utils import parse_dict_header, to_native_string
from .status_codes import codes
CONTENT_TYPE_FORM_URLENCODED = 'application/x-www-form-urlencoded'
CONTENT_TYPE_MULTI_PART = 'multipart/form-data'
@@ -28,7 +27,11 @@ CONTENT_TYPE_MULTI_PART = 'multipart/form-data'
def _basic_auth_str(username, password):
"""Returns a Basic Auth string."""
return 'Basic ' + b64encode(('%s:%s' % (username, password)).encode('latin1')).strip().decode('latin1')
authstr = 'Basic ' + to_native_string(
b64encode(('%s:%s' % (username, password)).encode('latin1')).strip()
)
return authstr
class AuthBase(object):
@@ -61,18 +64,26 @@ class HTTPDigestAuth(AuthBase):
def __init__(self, username, password):
self.username = username
self.password = password
self.last_nonce = ''
self.nonce_count = 0
self.chal = {}
self.pos = None
# Keep state in per-thread local storage
self._thread_local = threading.local()
def init_per_thread_state(self):
# Ensure state is initialized just once per-thread
if not hasattr(self._thread_local, 'init'):
self._thread_local.init = True
self._thread_local.last_nonce = ''
self._thread_local.nonce_count = 0
self._thread_local.chal = {}
self._thread_local.pos = None
self._thread_local.num_401_calls = None
def build_digest_header(self, method, url):
realm = self.chal['realm']
nonce = self.chal['nonce']
qop = self.chal.get('qop')
algorithm = self.chal.get('algorithm')
opaque = self.chal.get('opaque')
realm = self._thread_local.chal['realm']
nonce = self._thread_local.chal['nonce']
qop = self._thread_local.chal.get('qop')
algorithm = self._thread_local.chal.get('algorithm')
opaque = self._thread_local.chal.get('opaque')
if algorithm is None:
_algorithm = 'MD5'
@@ -100,7 +111,8 @@ class HTTPDigestAuth(AuthBase):
# XXX not implemented yet
entdig = None
p_parsed = urlparse(url)
path = p_parsed.path
#: path is request-uri defined in RFC 2616 which should not be empty
path = p_parsed.path or "/"
if p_parsed.query:
path += '?' + p_parsed.query
@@ -110,30 +122,32 @@ class HTTPDigestAuth(AuthBase):
HA1 = hash_utf8(A1)
HA2 = hash_utf8(A2)
if nonce == self.last_nonce:
self.nonce_count += 1
if nonce == self._thread_local.last_nonce:
self._thread_local.nonce_count += 1
else:
self.nonce_count = 1
ncvalue = '%08x' % self.nonce_count
s = str(self.nonce_count).encode('utf-8')
self._thread_local.nonce_count = 1
ncvalue = '%08x' % self._thread_local.nonce_count
s = str(self._thread_local.nonce_count).encode('utf-8')
s += nonce.encode('utf-8')
s += time.ctime().encode('utf-8')
s += os.urandom(8)
cnonce = (hashlib.sha1(s).hexdigest()[:16])
noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, HA2)
if _algorithm == 'MD5-SESS':
HA1 = hash_utf8('%s:%s:%s' % (HA1, nonce, cnonce))
if qop is None:
if not qop:
respdig = KD(HA1, "%s:%s" % (nonce, HA2))
elif qop == 'auth' or 'auth' in qop.split(','):
noncebit = "%s:%s:%s:%s:%s" % (
nonce, ncvalue, cnonce, 'auth', HA2
)
respdig = KD(HA1, noncebit)
else:
# XXX handle auth-int.
return None
self.last_nonce = nonce
self._thread_local.last_nonce = nonce
# XXX should the partial digests be encoded too?
base = 'username="%s", realm="%s", nonce="%s", uri="%s", ' \
@@ -149,26 +163,30 @@ class HTTPDigestAuth(AuthBase):
return 'Digest %s' % (base)
def handle_redirect(self, r, **kwargs):
"""Reset num_401_calls counter on redirects."""
if r.is_redirect:
self._thread_local.num_401_calls = 1
def handle_401(self, r, **kwargs):
"""Takes the given response and tries digest-auth, if needed."""
if self.pos is not None:
if self._thread_local.pos is not None:
# Rewind the file position indicator of the body to where
# it was to resend the request.
r.request.body.seek(self.pos)
num_401_calls = getattr(self, 'num_401_calls', 1)
r.request.body.seek(self._thread_local.pos)
s_auth = r.headers.get('www-authenticate', '')
if 'digest' in s_auth.lower() and num_401_calls < 2:
if 'digest' in s_auth.lower() and self._thread_local.num_401_calls < 2:
setattr(self, 'num_401_calls', num_401_calls + 1)
self._thread_local.num_401_calls += 1
pat = re.compile(r'digest ', flags=re.IGNORECASE)
self.chal = parse_dict_header(pat.sub('', s_auth, count=1))
self._thread_local.chal = parse_dict_header(pat.sub('', s_auth, count=1))
# Consume content and release the original connection
# to allow our new request to reuse the same one.
r.content
r.raw.release_conn()
r.close()
prep = r.request.copy()
extract_cookies_to_jar(prep._cookies, r.request, r.raw)
prep.prepare_cookies(prep._cookies)
@@ -181,16 +199,25 @@ class HTTPDigestAuth(AuthBase):
return _r
setattr(self, 'num_401_calls', 1)
self._thread_local.num_401_calls = 1
return r
def __call__(self, r):
# Initialize per-thread state, if needed
self.init_per_thread_state()
# If we have a saved nonce, skip the 401
if self.last_nonce:
if self._thread_local.last_nonce:
r.headers['Authorization'] = self.build_digest_header(r.method, r.url)
try:
self.pos = r.body.tell()
self._thread_local.pos = r.body.tell()
except AttributeError:
pass
# In the case of HTTPDigestAuth being reused and the body of
# the previous request was a file-like object, pos has the
# file position of the previous body. Ensure it's set to
# None.
self._thread_local.pos = None
r.register_hook('response', self.handle_401)
r.register_hook('response', self.handle_redirect)
self._thread_local.num_401_calls = 1
return r
requests/cacert.pem
+1385 -795
View File
File diff suppressed because it is too large Load Diff
requests/certs.py
+7 -6
View File
@@ -11,14 +11,15 @@ If you are packaging Requests, e.g., for a Linux distribution or a managed
environment, you can change the definition of where() to return a separately
packaged CA bundle.
"""
import os.path
def where():
"""Return the preferred certificate bundle."""
# vendored bundle inside Requests
return os.path.join(os.path.dirname(__file__), 'cacert.pem')
try:
from certifi import where
except ImportError:
def where():
"""Return the preferred certificate bundle."""
# vendored bundle inside Requests
return os.path.join(os.path.dirname(__file__), 'cacert.pem')
if __name__ == '__main__':
print(where())
requests/compat.py
+3 -56
View File
@@ -21,61 +21,11 @@ is_py2 = (_ver[0] == 2)
#: Python 3.x?
is_py3 = (_ver[0] == 3)
#: Python 3.0.x
is_py30 = (is_py3 and _ver[1] == 0)
#: Python 3.1.x
is_py31 = (is_py3 and _ver[1] == 1)
#: Python 3.2.x
is_py32 = (is_py3 and _ver[1] == 2)
#: Python 3.3.x
is_py33 = (is_py3 and _ver[1] == 3)
#: Python 3.4.x
is_py34 = (is_py3 and _ver[1] == 4)
#: Python 2.7.x
is_py27 = (is_py2 and _ver[1] == 7)
#: Python 2.6.x
is_py26 = (is_py2 and _ver[1] == 6)
#: Python 2.5.x
is_py25 = (is_py2 and _ver[1] == 5)
#: Python 2.4.x
is_py24 = (is_py2 and _ver[1] == 4) # I'm assuming this is not by choice.
# ---------
# Platforms
# ---------
# Syntax sugar.
_ver = sys.version.lower()
is_pypy = ('pypy' in _ver)
is_jython = ('jython' in _ver)
is_ironpython = ('iron' in _ver)
# Assume CPython, if nothing else.
is_cpython = not any((is_pypy, is_jython, is_ironpython))
# Windows-based system.
is_windows = 'win32' in str(sys.platform).lower()
# Standard Linux 2+ system.
is_linux = ('linux' in str(sys.platform).lower())
is_osx = ('darwin' in str(sys.platform).lower())
is_hpux = ('hpux' in str(sys.platform).lower()) # Complete guess.
is_solaris = ('solar==' in str(sys.platform).lower()) # Complete guess.
try:
import simplejson as json
except ImportError:
except (ImportError, SyntaxError):
# simplejson does not support Python 3.2, it throws a SyntaxError
# because of u'...' Unicode literals.
import json
# ---------
@@ -90,7 +40,6 @@ if is_py2:
from Cookie import Morsel
from StringIO import StringIO
from .packages.urllib3.packages.ordered_dict import OrderedDict
from httplib import IncompleteRead
builtin_str = str
bytes = str
@@ -98,7 +47,6 @@ if is_py2:
basestring = basestring
numeric_types = (int, long, float)
elif is_py3:
from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
from urllib.request import parse_http_list, getproxies, proxy_bypass
@@ -106,7 +54,6 @@ elif is_py3:
from http.cookies import Morsel
from io import StringIO
from collections import OrderedDict
from http.client import IncompleteRead
builtin_str = str
str = str
requests/cookies.py
+80 -48
View File
@@ -6,6 +6,7 @@ Compatibility code to be able to use `cookielib.CookieJar` with requests.
requests.utils imports from here, so be careful with imports.
"""
import copy
import time
import calendar
import collections
@@ -143,10 +144,13 @@ def remove_cookie_by_name(cookiejar, name, domain=None, path=None):
"""
clearables = []
for cookie in cookiejar:
if cookie.name == name:
if domain is None or domain == cookie.domain:
if path is None or path == cookie.path:
clearables.append((cookie.domain, cookie.path, cookie.name))
if cookie.name != name:
continue
if domain is not None and domain != cookie.domain:
continue
if path is not None and path != cookie.path:
continue
clearables.append((cookie.domain, cookie.path, cookie.name))
for domain, path, name in clearables:
cookiejar.clear(domain, path, name)
@@ -158,26 +162,28 @@ class CookieConflictError(RuntimeError):
class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
"""Compatibility class; is a cookielib.CookieJar, but exposes a dict interface.
"""Compatibility class; is a cookielib.CookieJar, but exposes a dict
interface.
This is the CookieJar we create by default for requests and sessions that
don't specify one, since some clients may expect response.cookies and
session.cookies to support dict operations.
Don't use the dict interface internally; it's just for compatibility with
with external client code. All `requests` code should work out of the box
with externally provided instances of CookieJar, e.g., LWPCookieJar and
FileCookieJar.
Caution: dictionary operations that are normally O(1) may be O(n).
Requests does not use the dict interface internally; it's just for
compatibility with external client code. All requests code should work
out of the box with externally provided instances of ``CookieJar``, e.g.
``LWPCookieJar`` and ``FileCookieJar``.
Unlike a regular CookieJar, this class is pickleable.
"""
.. warning:: dictionary operations that are normally O(1) may be O(n).
"""
def get(self, name, default=None, domain=None, path=None):
"""Dict-like get() that also supports optional domain and path args in
order to resolve naming collisions from using one cookie jar over
multiple domains. Caution: operation is O(n), not O(1)."""
multiple domains.
.. warning:: operation is O(n), not O(1)."""
try:
return self._find_no_duplicates(name, domain, path)
except KeyError:
@@ -200,37 +206,38 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
return c
def iterkeys(self):
"""Dict-like iterkeys() that returns an iterator of names of cookies from the jar.
See itervalues() and iteritems()."""
"""Dict-like iterkeys() that returns an iterator of names of cookies
from the jar. See itervalues() and iteritems()."""
for cookie in iter(self):
yield cookie.name
def keys(self):
"""Dict-like keys() that returns a list of names of cookies from the jar.
See values() and items()."""
"""Dict-like keys() that returns a list of names of cookies from the
jar. See values() and items()."""
return list(self.iterkeys())
def itervalues(self):
"""Dict-like itervalues() that returns an iterator of values of cookies from the jar.
See iterkeys() and iteritems()."""
"""Dict-like itervalues() that returns an iterator of values of cookies
from the jar. See iterkeys() and iteritems()."""
for cookie in iter(self):
yield cookie.value
def values(self):
"""Dict-like values() that returns a list of values of cookies from the jar.
See keys() and items()."""
"""Dict-like values() that returns a list of values of cookies from the
jar. See keys() and items()."""
return list(self.itervalues())
def iteritems(self):
"""Dict-like iteritems() that returns an iterator of name-value tuples from the jar.
See iterkeys() and itervalues()."""
"""Dict-like iteritems() that returns an iterator of name-value tuples
from the jar. See iterkeys() and itervalues()."""
for cookie in iter(self):
yield cookie.name, cookie.value
def items(self):
"""Dict-like items() that returns a list of name-value tuples from the jar.
See keys() and values(). Allows client-code to call "dict(RequestsCookieJar)
and get a vanilla python dict of key value pairs."""
"""Dict-like items() that returns a list of name-value tuples from the
jar. See keys() and values(). Allows client-code to call
``dict(RequestsCookieJar)`` and get a vanilla python dict of key value
pairs."""
return list(self.iteritems())
def list_domains(self):
@@ -260,8 +267,9 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
return False # there is only one domain in jar
def get_dict(self, domain=None, path=None):
"""Takes as an argument an optional domain and path and returns a plain old
Python dict of name-value pairs of cookies that meet the requirements."""
"""Takes as an argument an optional domain and path and returns a plain
old Python dict of name-value pairs of cookies that meet the
requirements."""
dictionary = {}
for cookie in iter(self):
if (domain is None or cookie.domain == domain) and (path is None
@@ -270,21 +278,24 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
return dictionary
def __getitem__(self, name):
"""Dict-like __getitem__() for compatibility with client code. Throws exception
if there are more than one cookie with name. In that case, use the more
explicit get() method instead. Caution: operation is O(n), not O(1)."""
"""Dict-like __getitem__() for compatibility with client code. Throws
exception if there are more than one cookie with name. In that case,
use the more explicit get() method instead.
.. warning:: operation is O(n), not O(1)."""
return self._find_no_duplicates(name)
def __setitem__(self, name, value):
"""Dict-like __setitem__ for compatibility with client code. Throws exception
if there is already a cookie of that name in the jar. In that case, use the more
explicit set() method instead."""
"""Dict-like __setitem__ for compatibility with client code. Throws
exception if there is already a cookie of that name in the jar. In that
case, use the more explicit set() method instead."""
self.set(name, value)
def __delitem__(self, name):
"""Deletes a cookie given a name. Wraps cookielib.CookieJar's remove_cookie_by_name()."""
"""Deletes a cookie given a name. Wraps ``cookielib.CookieJar``'s
``remove_cookie_by_name()``."""
remove_cookie_by_name(self, name)
def set_cookie(self, cookie, *args, **kwargs):
@@ -296,15 +307,16 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
"""Updates this jar with cookies from another CookieJar or dict-like"""
if isinstance(other, cookielib.CookieJar):
for cookie in other:
self.set_cookie(cookie)
self.set_cookie(copy.copy(cookie))
else:
super(RequestsCookieJar, self).update(other)
def _find(self, name, domain=None, path=None):
"""Requests uses this method internally to get cookie values. Takes as args name
and optional domain and path. Returns a cookie.value. If there are conflicting cookies,
_find arbitrarily chooses one. See _find_no_duplicates if you want an exception thrown
if there are conflicting cookies."""
"""Requests uses this method internally to get cookie values. Takes as
args name and optional domain and path. Returns a cookie.value. If
there are conflicting cookies, _find arbitrarily chooses one. See
_find_no_duplicates if you want an exception thrown if there are
conflicting cookies."""
for cookie in iter(self):
if cookie.name == name:
if domain is None or cookie.domain == domain:
@@ -314,10 +326,11 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
def _find_no_duplicates(self, name, domain=None, path=None):
"""__get_item__ and get call _find_no_duplicates -- never used in Requests internally.
Takes as args name and optional domain and path. Returns a cookie.value.
Throws KeyError if cookie is not found and CookieConflictError if there are
multiple cookies that match name and optionally domain and path."""
"""Both ``__get_item__`` and ``get`` call this function: it's never
used elsewhere in Requests. Takes as args name and optional domain and
path. Returns a cookie.value. Throws KeyError if cookie is not found
and CookieConflictError if there are multiple cookies that match name
and optionally domain and path."""
toReturn = None
for cookie in iter(self):
if cookie.name == name:
@@ -351,6 +364,21 @@ class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
return new_cj
def _copy_cookie_jar(jar):
if jar is None:
return None
if hasattr(jar, 'copy'):
# We're dealing with an instance of RequestsCookieJar
return jar.copy()
# We're dealing with a generic CookieJar instance
new_jar = copy.copy(jar)
new_jar.clear()
for cookie in jar:
new_jar.set_cookie(copy.copy(cookie))
return new_jar
def create_cookie(name, value, **kwargs):
"""Make a cookie from underspecified parameters.
@@ -391,11 +419,15 @@ def morsel_to_cookie(morsel):
expires = None
if morsel['max-age']:
expires = time.time() + morsel['max-age']
try:
expires = int(time.time() + int(morsel['max-age']))
except ValueError:
raise TypeError('max-age: %s must be integer' % morsel['max-age'])
elif morsel['expires']:
time_template = '%a, %d-%b-%Y %H:%M:%S GMT'
expires = calendar.timegm(time.strptime(morsel['expires'],
time_template))
expires = calendar.timegm(
time.strptime(morsel['expires'], time_template)
)
return create_cookie(
comment=morsel['comment'],
comment_url=bool(morsel['comment']),
@@ -441,7 +473,7 @@ def merge_cookies(cookiejar, cookies):
"""
if not isinstance(cookiejar, cookielib.CookieJar):
raise ValueError('You can only merge into CookieJar')
if isinstance(cookies, dict):
cookiejar = cookiejar_from_dict(
cookies, cookiejar=cookiejar, overwrite=False)
requests/exceptions.py
+52 -6
View File
@@ -14,15 +14,22 @@ class RequestException(IOError):
"""There was an ambiguous exception that occurred while handling your
request."""
def __init__(self, *args, **kwargs):
"""
Initialize RequestException with `request` and `response` objects.
"""
response = kwargs.pop('response', None)
self.response = response
self.request = kwargs.pop('request', None)
if (response is not None and not self.request and
hasattr(response, 'request')):
self.request = self.response.request
super(RequestException, self).__init__(*args, **kwargs)
class HTTPError(RequestException):
"""An HTTP error occurred."""
def __init__(self, *args, **kwargs):
""" Initializes HTTPError with optional `response` object. """
self.response = kwargs.pop('response', None)
super(HTTPError, self).__init__(*args, **kwargs)
class ConnectionError(RequestException):
"""A Connection error occurred."""
@@ -37,7 +44,23 @@ class SSLError(ConnectionError):
class Timeout(RequestException):
"""The request timed out."""
"""The request timed out.
Catching this error will catch both
:exc:`~requests.exceptions.ConnectTimeout` and
:exc:`~requests.exceptions.ReadTimeout` errors.
"""
class ConnectTimeout(ConnectionError, Timeout):
"""The request timed out while trying to connect to the remote server.
Requests that produced this error are safe to retry.
"""
class ReadTimeout(Timeout):
"""The server did not send any data in the allotted amount of time."""
class URLRequired(RequestException):
@@ -66,3 +89,26 @@ class ChunkedEncodingError(RequestException):
class ContentDecodingError(RequestException, BaseHTTPError):
"""Failed to decode response content"""
class StreamConsumedError(RequestException, TypeError):
"""The content for this response was already consumed"""
class RetryError(RequestException):
"""Custom retries logic failed"""
# Warnings
class RequestsWarning(Warning):
"""Base warning for Requests."""
pass
class FileModeWarning(RequestsWarning, DeprecationWarning):
"""
A file was opened in text mode, but Requests determined its binary length.
"""
pass
requests/hooks.py
+3 -14
View File
@@ -12,34 +12,23 @@ Available hooks:
The response generated from a Request.
"""
HOOKS = ['response']
def default_hooks():
hooks = {}
for event in HOOKS:
hooks[event] = []
return hooks
return dict((event, []) for event in HOOKS)
# TODO: response is the only one
def dispatch_hook(key, hooks, hook_data, **kwargs):
"""Dispatches a hook dictionary on a given piece of data."""
hooks = hooks or dict()
if key in hooks:
hooks = hooks.get(key)
hooks = hooks.get(key)
if hooks:
if hasattr(hooks, '__call__'):
hooks = [hooks]
for hook in hooks:
_hook_data = hook(hook_data, **kwargs)
if _hook_data is not None:
hook_data = _hook_data
return hook_data
requests/models.py
+158 -88
View File
@@ -8,7 +8,6 @@ This module contains the primary objects that power Requests.
"""
import collections
import logging
import datetime
from io import BytesIO, UnsupportedOperation
@@ -16,27 +15,39 @@ from .hooks import default_hooks
from .structures import CaseInsensitiveDict
from .auth import HTTPBasicAuth
from .cookies import cookiejar_from_dict, get_cookie_header
from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
from .packages.urllib3.fields import RequestField
from .packages.urllib3.filepost import encode_multipart_formdata
from .packages.urllib3.util import parse_url
from .packages.urllib3.exceptions import DecodeError
from .packages.urllib3.exceptions import (
DecodeError, ReadTimeoutError, ProtocolError, LocationParseError)
from .exceptions import (
HTTPError, RequestException, MissingSchema, InvalidURL,
ChunkedEncodingError, ContentDecodingError)
HTTPError, MissingSchema, InvalidURL, ChunkedEncodingError,
ContentDecodingError, ConnectionError, StreamConsumedError)
from .utils import (
guess_filename, get_auth_from_url, requote_uri,
stream_decode_response_unicode, to_key_val_list, parse_header_links,
iter_slices, guess_json_utf, super_len, to_native_string)
from .compat import (
cookielib, urlunparse, urlsplit, urlencode, str, bytes, StringIO,
is_py2, chardet, json, builtin_str, basestring, IncompleteRead)
is_py2, chardet, builtin_str, basestring)
from .compat import json as complexjson
from .status_codes import codes
#: The set of HTTP status codes that indicate an automatically
#: processable redirect.
REDIRECT_STATI = (
codes.moved, # 301
codes.found, # 302
codes.other, # 303
codes.temporary_redirect, # 307
codes.permanent_redirect, # 308
)
DEFAULT_REDIRECT_LIMIT = 30
CONTENT_CHUNK_SIZE = 10 * 1024
ITER_CHUNK_SIZE = 512
log = logging.getLogger(__name__)
class RequestEncodingMixin(object):
@property
@@ -132,13 +143,13 @@ class RequestEncodingMixin(object):
else:
fn = guess_filename(v) or k
fp = v
if isinstance(fp, str):
fp = StringIO(fp)
if isinstance(fp, bytes):
fp = BytesIO(fp)
rf = RequestField(name=k, data=fp.read(),
filename=fn, headers=fh)
if isinstance(fp, (str, bytes, bytearray)):
fdata = fp
else:
fdata = fp.read()
rf = RequestField(name=k, data=fdata, filename=fn, headers=fh)
rf.make_multipart(content_type=ft)
new_fields.append(rf)
@@ -180,7 +191,8 @@ class Request(RequestHooksMixin):
:param url: URL to send.
:param headers: dictionary of headers to send.
:param files: dictionary of {filename: fileobject} files to multipart upload.
:param data: the body to attach the request. If a dictionary is provided, form-encoding will take place.
:param data: the body to attach to the request. If a dictionary is provided, form-encoding will take place.
:param json: json for the body to attach to the request (if files or data is not specified).
:param params: dictionary of URL parameters to append to the URL.
:param auth: Auth handler or (user, pass) tuple.
:param cookies: dictionary or CookieJar of cookies to attach to this request.
@@ -194,16 +206,8 @@ class Request(RequestHooksMixin):
<PreparedRequest [GET]>
"""
def __init__(self,
method=None,
url=None,
headers=None,
files=None,
data=None,
params=None,
auth=None,
cookies=None,
hooks=None):
def __init__(self, method=None, url=None, headers=None, files=None,
data=None, params=None, auth=None, cookies=None, hooks=None, json=None):
# Default empty dicts for dict params.
data = [] if data is None else data
@@ -221,6 +225,7 @@ class Request(RequestHooksMixin):
self.headers = headers
self.files = files
self.data = data
self.json = json
self.params = params
self.auth = auth
self.cookies = cookies
@@ -237,6 +242,7 @@ class Request(RequestHooksMixin):
headers=self.headers,
files=self.files,
data=self.data,
json=self.json,
params=self.params,
auth=self.auth,
cookies=self.cookies,
@@ -280,15 +286,16 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
self.hooks = default_hooks()
def prepare(self, method=None, url=None, headers=None, files=None,
data=None, params=None, auth=None, cookies=None, hooks=None):
data=None, params=None, auth=None, cookies=None, hooks=None, json=None):
"""Prepares the entire request with the given parameters."""
self.prepare_method(method)
self.prepare_url(url, params)
self.prepare_headers(headers)
self.prepare_cookies(cookies)
self.prepare_body(data, files)
self.prepare_body(data, files, json)
self.prepare_auth(auth, url)
# Note that prepare_auth must be last to enable authentication schemes
# such as OAuth to work on a fully prepared request.
@@ -302,8 +309,8 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
p = PreparedRequest()
p.method = self.method
p.url = self.url
p.headers = self.headers.copy()
p._cookies = self._cookies.copy()
p.headers = self.headers.copy() if self.headers is not None else None
p._cookies = _copy_cookie_jar(self._cookies)
p.body = self.body
p.hooks = self.hooks
return p
@@ -312,30 +319,38 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
"""Prepares the given HTTP method."""
self.method = method
if self.method is not None:
self.method = self.method.upper()
self.method = to_native_string(self.method.upper())
def prepare_url(self, url, params):
"""Prepares the given HTTP URL."""
#: Accept objects that have string representations.
try:
url = unicode(url)
except NameError:
# We're on Python 3.
url = str(url)
except UnicodeDecodeError:
pass
#: We're unable to blindly call unicode/str functions
#: as this will include the bytestring indicator (b'')
#: on python 3.x.
#: https://github.com/kennethreitz/requests/pull/2238
if isinstance(url, bytes):
url = url.decode('utf8')
else:
url = unicode(url) if is_py2 else str(url)
# Don't do any URL preparation for oddball schemes
# Don't do any URL preparation for non-HTTP schemes like `mailto`,
# `data` etc to work around exceptions from `url_parse`, which
# handles RFC 3986 only.
if ':' in url and not url.lower().startswith('http'):
self.url = url
return
# Support for unicode domain names and paths.
scheme, auth, host, port, path, query, fragment = parse_url(url)
try:
scheme, auth, host, port, path, query, fragment = parse_url(url)
except LocationParseError as e:
raise InvalidURL(*e.args)
if not scheme:
raise MissingSchema("Invalid URL {0!r}: No schema supplied. "
"Perhaps you meant http://{0}?".format(url))
error = ("Invalid URL {0!r}: No schema supplied. Perhaps you meant http://{0}?")
error = error.format(to_native_string(url, 'utf8'))
raise MissingSchema(error)
if not host:
raise InvalidURL("Invalid URL %r: No host supplied" % url)
@@ -370,6 +385,9 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
if isinstance(fragment, str):
fragment = fragment.encode('utf-8')
if isinstance(params, (str, bytes)):
params = to_native_string(params)
enc_params = self._encode_params(params)
if enc_params:
if query:
@@ -388,7 +406,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
else:
self.headers = CaseInsensitiveDict()
def prepare_body(self, data, files):
def prepare_body(self, data, files, json=None):
"""Prepares the given HTTP body data."""
# Check if file, fo, generator, iterator.
@@ -399,11 +417,13 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
content_type = None
length = None
if not data and json is not None:
content_type = 'application/json'
body = complexjson.dumps(json)
is_stream = all([
hasattr(data, '__iter__'),
not isinstance(data, basestring),
not isinstance(data, list),
not isinstance(data, dict)
not isinstance(data, (basestring, list, tuple, dict))
])
try:
@@ -417,7 +437,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
if files:
raise NotImplementedError('Streamed bodies and files are mutually exclusive.')
if length is not None:
if length:
self.headers['Content-Length'] = builtin_str(length)
else:
self.headers['Transfer-Encoding'] = 'chunked'
@@ -428,7 +448,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
else:
if data:
body = self._encode_params(data)
if isinstance(data, str) or isinstance(data, builtin_str) or hasattr(data, 'read'):
if isinstance(data, basestring) or hasattr(data, 'read'):
content_type = None
else:
content_type = 'application/x-www-form-urlencoded'
@@ -436,7 +456,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
self.prepare_content_length(body)
# Add content-type if it wasn't explicitly provided.
if (content_type) and (not 'content-type' in self.headers):
if content_type and ('content-type' not in self.headers):
self.headers['Content-Type'] = content_type
self.body = body
@@ -450,7 +470,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
l = super_len(body)
if l:
self.headers['Content-Length'] = builtin_str(l)
elif self.method not in ('GET', 'HEAD'):
elif (self.method not in ('GET', 'HEAD')) and (self.headers.get('Content-Length') is None):
self.headers['Content-Length'] = '0'
def prepare_auth(self, auth, url=''):
@@ -476,7 +496,15 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
self.prepare_content_length(self.body)
def prepare_cookies(self, cookies):
"""Prepares the given HTTP cookie data."""
"""Prepares the given HTTP cookie data.
This function eventually generates a ``Cookie`` header from the
given cookies using cookielib. Due to cookielib's design, the header
will not be regenerated if it already exists, meaning this function
can only be called once for the life of the
:class:`PreparedRequest <PreparedRequest>` object. Any subsequent calls
to ``prepare_cookies`` will have no actual effect, unless the "Cookie"
header is removed beforehand."""
if isinstance(cookies, cookielib.CookieJar):
self._cookies = cookies
@@ -489,6 +517,10 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
def prepare_hooks(self, hooks):
"""Prepares the given hooks."""
# hooks can be passed as None to the prepare method and to this
# method. To prevent iterating over None, simply use an empty list
# if hooks is False-y
hooks = hooks or []
for event in hooks:
self.register_hook(event, hooks[event])
@@ -499,16 +531,8 @@ class Response(object):
"""
__attrs__ = [
'_content',
'status_code',
'headers',
'url',
'history',
'encoding',
'reason',
'cookies',
'elapsed',
'request',
'_content', 'status_code', 'headers', 'url', 'history',
'encoding', 'reason', 'cookies', 'elapsed', 'request'
]
def __init__(self):
@@ -517,7 +541,7 @@ class Response(object):
self._content = False
self._content_consumed = False
#: Integer Code of responded HTTP Status.
#: Integer Code of responded HTTP Status, e.g. 404 or 200.
self.status_code = None
#: Case-insensitive Dictionary of Response Headers.
@@ -541,15 +565,24 @@ class Response(object):
#: up here. The list is sorted from the oldest to the most recent request.
self.history = []
#: Textual reason of responded HTTP Status, e.g. "Not Found" or "OK".
self.reason = None
#: A CookieJar of Cookies the server sent back.
self.cookies = cookiejar_from_dict({})
#: The amount of time elapsed between sending the request
#: and the arrival of the response (as a timedelta)
#: and the arrival of the response (as a timedelta).
#: This property specifically measures the time taken between sending
#: the first byte of the request and finishing parsing the headers. It
#: is therefore unaffected by consuming the response content or the
#: value of the ``stream`` keyword argument.
self.elapsed = datetime.timedelta(0)
#: The :class:`PreparedRequest <PreparedRequest>` object to which this
#: is a response.
self.request = None
def __getstate__(self):
# Consume everything; accessing the content attribute makes
# sure the content has been fully read.
@@ -567,6 +600,7 @@ class Response(object):
# pickled objects do not have .raw
setattr(self, '_content_consumed', True)
setattr(self, 'raw', None)
def __repr__(self):
return '<Response [%s]>' % (self.status_code)
@@ -587,14 +621,25 @@ class Response(object):
def ok(self):
try:
self.raise_for_status()
except RequestException:
except HTTPError:
return False
return True
@property
def is_redirect(self):
"""True if this Response is a well-formed HTTP redirect that could have
been processed automatically (by :meth:`Session.resolve_redirects`).
"""
return ('location' in self.headers and self.status_code in REDIRECT_STATI)
@property
def is_permanent_redirect(self):
"""True if this Response one of the permanent versions of redirect"""
return ('location' in self.headers and self.status_code in (codes.moved_permanently, codes.permanent_redirect))
@property
def apparent_encoding(self):
"""The apparent encoding, provided by the lovely Charade library
(Thanks, Ian!)."""
"""The apparent encoding, provided by the chardet library"""
return chardet.detect(self.content)['encoding']
def iter_content(self, chunk_size=1, decode_unicode=False):
@@ -603,23 +648,24 @@ class Response(object):
large responses. The chunk size is the number of bytes it should
read into memory. This is not necessarily the length of each item
returned as decoding can take place.
If decode_unicode is True, content will be decoded using the best
available encoding based on the response.
"""
if self._content_consumed:
# simulate reading small chunks of the content
return iter_slices(self._content, chunk_size)
def generate():
try:
# Special case for urllib3.
# Special case for urllib3.
if hasattr(self.raw, 'stream'):
try:
for chunk in self.raw.stream(chunk_size,
decode_content=True):
for chunk in self.raw.stream(chunk_size, decode_content=True):
yield chunk
except IncompleteRead as e:
except ProtocolError as e:
raise ChunkedEncodingError(e)
except DecodeError as e:
raise ContentDecodingError(e)
except AttributeError:
except ReadTimeoutError as e:
raise ConnectionError(e)
else:
# Standard file-like object.
while True:
chunk = self.raw.read(chunk_size)
@@ -629,27 +675,39 @@ class Response(object):
self._content_consumed = True
gen = generate()
if self._content_consumed and isinstance(self._content, bool):
raise StreamConsumedError()
# simulate reading small chunks of the content
reused_chunks = iter_slices(self._content, chunk_size)
stream_chunks = generate()
chunks = reused_chunks if self._content_consumed else stream_chunks
if decode_unicode:
gen = stream_decode_response_unicode(gen, self)
chunks = stream_decode_response_unicode(chunks, self)
return gen
return chunks
def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None):
def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None, delimiter=None):
"""Iterates over the response data, one line at a time. When
stream=True is set on the request, this avoids reading the
content at once into memory for large responses.
.. note:: This method is not reentrant safe.
"""
pending = None
for chunk in self.iter_content(chunk_size=chunk_size,
decode_unicode=decode_unicode):
for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
if pending is not None:
chunk = pending + chunk
lines = chunk.splitlines()
if delimiter:
lines = chunk.split(delimiter)
else:
lines = chunk.splitlines()
if lines and lines[-1] and chunk and lines[-1][-1] == chunk[-1]:
pending = lines.pop()
@@ -693,7 +751,7 @@ class Response(object):
If Response.encoding is None, encoding will be guessed using
``chardet``.
The encoding of the response content is determined based soley on HTTP
The encoding of the response content is determined based solely on HTTP
headers, following RFC 2616 to the letter. If you can take advantage of
non-HTTP knowledge to make a better guess at the encoding, you should
set ``r.encoding`` appropriately before accessing this property.
@@ -737,8 +795,17 @@ class Response(object):
# a best guess).
encoding = guess_json_utf(self.content)
if encoding is not None:
return json.loads(self.content.decode(encoding), **kwargs)
return json.loads(self.text, **kwargs)
try:
return complexjson.loads(
self.content.decode(encoding), **kwargs
)
except UnicodeDecodeError:
# Wrong UTF codec detected; usually because it's not UTF-8
# but some other 8-bit codec. This is an RFC violation,
# and the server didn't bother to tell us what codec *was*
# used.
pass
return complexjson.loads(self.text, **kwargs)
@property
def links(self):
@@ -764,18 +831,21 @@ class Response(object):
http_error_msg = ''
if 400 <= self.status_code < 500:
http_error_msg = '%s Client Error: %s' % (self.status_code, self.reason)
http_error_msg = '%s Client Error: %s for url: %s' % (self.status_code, self.reason, self.url)
elif 500 <= self.status_code < 600:
http_error_msg = '%s Server Error: %s' % (self.status_code, self.reason)
http_error_msg = '%s Server Error: %s for url: %s' % (self.status_code, self.reason, self.url)
if http_error_msg:
raise HTTPError(http_error_msg, response=self)
def close(self):
"""Closes the underlying file descriptor and releases the connection
back to the pool.
"""Releases the connection back to the pool. Once this method has been
called the underlying ``raw`` object must not be accessed again.
*Note: Should not normally need to be called explicitly.*
"""
if not self._content_consumed:
return self.raw.close()
return self.raw.release_conn()
requests/packages/README.rst
+5 -2
View File
@@ -1,8 +1,11 @@
If you are planning to submit a pull request to requests with any changes in
this library do not go any further. These are independent libraries which we
vendor into requests. Any changes necessary to these libraries must be made in
this library do not go any further. These are independent libraries which we
vendor into requests. Any changes necessary to these libraries must be made in
them and submitted as separate pull requests to those libraries.
urllib3 pull requests go here: https://github.com/shazow/urllib3
chardet pull requests go here: https://github.com/chardet/chardet
See https://github.com/kennethreitz/requests/pull/1812#issuecomment-30854316
for the reasoning behind this.
requests/packages/__init__.py
+35 -2
View File
@@ -1,3 +1,36 @@
from __future__ import absolute_import
'''
Debian and other distributions "unbundle" requests' vendored dependencies, and
rewrite all imports to use the global versions of ``urllib3`` and ``chardet``.
The problem with this is that not only requests itself imports those
dependencies, but third-party code outside of the distros' control too.
from . import urllib3
In reaction to these problems, the distro maintainers replaced
``requests.packages`` with a magical "stub module" that imports the correct
modules. The implementations were varying in quality and all had severe
problems. For example, a symlink (or hardlink) that links the correct modules
into place introduces problems regarding object identity, since you now have
two modules in `sys.modules` with the same API, but different identities::
requests.packages.urllib3 is not urllib3
With version ``2.5.2``, requests started to maintain its own stub, so that
distro-specific breakage would be reduced to a minimum, even though the whole
issue is not requests' fault in the first place. See
https://github.com/kennethreitz/requests/pull/2375 for the corresponding pull
request.
'''
from __future__ import absolute_import
import sys
try:
from . import urllib3
except ImportError:
import urllib3
sys.modules['%s.urllib3' % __name__] = urllib3
try:
from . import chardet
except ImportError:
import chardet
sys.modules['%s.chardet' % __name__] = chardet
requests/packages/chardet/__init__.py
+1 -1
View File
@@ -15,7 +15,7 @@
# 02110-1301 USA
######################### END LICENSE BLOCK #########################
__version__ = "2.2.1"
__version__ = "2.3.0"
from sys import version_info
requests/packages/chardet/chardetect.py
+50 -16
View File
@@ -12,34 +12,68 @@ Example::
If no paths are provided, it takes its input from stdin.
"""
from io import open
from sys import argv, stdin
from __future__ import absolute_import, print_function, unicode_literals
import argparse
import sys
from io import open
from chardet import __version__
from chardet.universaldetector import UniversalDetector
def description_of(file, name='stdin'):
"""Return a string describing the probable encoding of a file."""
def description_of(lines, name='stdin'):
"""
Return a string describing the probable encoding of a file or
list of strings.
:param lines: The lines to get the encoding of.
:type lines: Iterable of bytes
:param name: Name of file or collection of lines
:type name: str
"""
u = UniversalDetector()
for line in file:
for line in lines:
u.feed(line)
u.close()
result = u.result
if result['encoding']:
return '%s: %s with confidence %s' % (name,
result['encoding'],
result['confidence'])
return '{0}: {1} with confidence {2}'.format(name, result['encoding'],
result['confidence'])
else:
return '%s: no result' % name
return '{0}: no result'.format(name)
def main():
if len(argv) <= 1:
print(description_of(stdin))
else:
for path in argv[1:]:
with open(path, 'rb') as f:
print(description_of(f, path))
def main(argv=None):
'''
Handles command line arguments and gets things started.
:param argv: List of arguments, as if specified on the command-line.
If None, ``sys.argv[1:]`` is used instead.
:type argv: list of str
'''
# Get command line arguments
parser = argparse.ArgumentParser(
description="Takes one or more file paths and reports their detected \
encodings",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
conflict_handler='resolve')
parser.add_argument('input',
help='File whose encoding we would like to determine.',
type=argparse.FileType('rb'), nargs='*',
default=[sys.stdin])
parser.add_argument('--version', action='version',
version='%(prog)s {0}'.format(__version__))
args = parser.parse_args(argv)
for f in args.input:
if f.isatty():
print("You are running chardetect interactively. Press " +
"CTRL-D twice at the start of a blank line to signal the " +
"end of your input. If you want help, run chardetect " +
"--help\n", file=sys.stderr)
print(description_of(f, f.name))
if __name__ == '__main__':
requests/packages/chardet/jpcntx.py
+8
View File
@@ -177,6 +177,12 @@ class JapaneseContextAnalysis:
return -1, 1
class SJISContextAnalysis(JapaneseContextAnalysis):
def __init__(self):
self.charset_name = "SHIFT_JIS"
def get_charset_name(self):
return self.charset_name
def get_order(self, aBuf):
if not aBuf:
return -1, 1
@@ -184,6 +190,8 @@ class SJISContextAnalysis(JapaneseContextAnalysis):
first_char = wrap_ord(aBuf[0])
if ((0x81 <= first_char <= 0x9F) or (0xE0 <= first_char <= 0xFC)):
charLen = 2
if (first_char == 0x87) or (0xFA <= first_char <= 0xFC):
self.charset_name = "CP932"
else:
charLen = 1
requests/packages/chardet/latin1prober.py
+3 -3
View File
@@ -129,11 +129,11 @@ class Latin1Prober(CharSetProber):
if total < 0.01:
confidence = 0.0
else:
confidence = ((self._mFreqCounter[3] / total)
- (self._mFreqCounter[1] * 20.0 / total))
confidence = ((self._mFreqCounter[3] - self._mFreqCounter[1] * 20.0)
/ total)
if confidence < 0.0:
confidence = 0.0
# lower the confidence of latin1 so that other more accurate
# detector can take priority.
confidence = confidence * 0.5
confidence = confidence * 0.73
return confidence
requests/packages/chardet/mbcssm.py
+3 -6
View File
@@ -353,7 +353,7 @@ SJIS_cls = (
2,2,2,2,2,2,2,2, # 68 - 6f
2,2,2,2,2,2,2,2, # 70 - 77
2,2,2,2,2,2,2,1, # 78 - 7f
3,3,3,3,3,3,3,3, # 80 - 87
3,3,3,3,3,2,2,3, # 80 - 87
3,3,3,3,3,3,3,3, # 88 - 8f
3,3,3,3,3,3,3,3, # 90 - 97
3,3,3,3,3,3,3,3, # 98 - 9f
@@ -369,9 +369,8 @@ SJIS_cls = (
2,2,2,2,2,2,2,2, # d8 - df
3,3,3,3,3,3,3,3, # e0 - e7
3,3,3,3,3,4,4,4, # e8 - ef
4,4,4,4,4,4,4,4, # f0 - f7
4,4,4,4,4,0,0,0 # f8 - ff
)
3,3,3,3,3,3,3,3, # f0 - f7
3,3,3,3,3,0,0,0) # f8 - ff
SJIS_st = (
@@ -571,5 +570,3 @@ UTF8SMModel = {'classTable': UTF8_cls,
'stateTable': UTF8_st,
'charLenTable': UTF8CharLenTable,
'name': 'UTF-8'}
# flake8: noqa
requests/packages/chardet/sjisprober.py
+1 -1
View File
@@ -47,7 +47,7 @@ class SJISProber(MultiByteCharSetProber):
self._mContextAnalyzer.reset()
def get_charset_name(self):
return "SHIFT_JIS"
return self._mContextAnalyzer.get_charset_name()
def feed(self, aBuf):
aLen = len(aBuf)
requests/packages/chardet/universaldetector.py
+2 -2
View File
@@ -71,9 +71,9 @@ class UniversalDetector:
if not self._mGotData:
# If the data starts with BOM, we know it is UTF
if aBuf[:3] == codecs.BOM:
if aBuf[:3] == codecs.BOM_UTF8:
# EF BB BF UTF-8 with BOM
self.result = {'encoding': "UTF-8", 'confidence': 1.0}
self.result = {'encoding': "UTF-8-SIG", 'confidence': 1.0}
elif aBuf[:4] == codecs.BOM_UTF32_LE:
# FF FE 00 00 UTF-32, little-endian BOM
self.result = {'encoding': "UTF-32LE", 'confidence': 1.0}
requests/packages/urllib3/__init__.py
+47 -12
View File
@@ -1,17 +1,9 @@
# urllib3/__init__.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
"""
urllib3 - Thread-safe connection pooling and re-using.
"""
__author__ = 'Andrey Petrov (andrey.petrov@shazow.net)'
__license__ = 'MIT'
__version__ = 'dev'
from __future__ import absolute_import
import warnings
from .connectionpool import (
HTTPConnectionPool,
@@ -23,7 +15,10 @@ from . import exceptions
from .filepost import encode_multipart_formdata
from .poolmanager import PoolManager, ProxyManager, proxy_from_url
from .response import HTTPResponse
from .util import make_headers, get_host, Timeout
from .util.request import make_headers
from .util.url import get_host
from .util.timeout import Timeout
from .util.retry import Retry
# Set default logging handler to avoid "No handler found" warnings.
@@ -35,8 +30,30 @@ except ImportError:
def emit(self, record):
pass
__author__ = 'Andrey Petrov (andrey.petrov@shazow.net)'
__license__ = 'MIT'
__version__ = '1.13'
__all__ = (
'HTTPConnectionPool',
'HTTPSConnectionPool',
'PoolManager',
'ProxyManager',
'HTTPResponse',
'Retry',
'Timeout',
'add_stderr_logger',
'connection_from_url',
'disable_warnings',
'encode_multipart_formdata',
'get_host',
'make_headers',
'proxy_from_url',
)
logging.getLogger(__name__).addHandler(NullHandler())
def add_stderr_logger(level=logging.DEBUG):
"""
Helper for quickly adding a StreamHandler to the logger. Useful for
@@ -51,8 +68,26 @@ def add_stderr_logger(level=logging.DEBUG):
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
logger.addHandler(handler)
logger.setLevel(level)
logger.debug('Added an stderr logging handler to logger: %s' % __name__)
logger.debug('Added a stderr logging handler to logger: %s' % __name__)
return handler
# ... Clean up.
del NullHandler
# SecurityWarning's always go off by default.
warnings.simplefilter('always', exceptions.SecurityWarning, append=True)
# SubjectAltNameWarning's should go off once per host
warnings.simplefilter('default', exceptions.SubjectAltNameWarning)
# InsecurePlatformWarning's don't vary between requests, so we keep it default.
warnings.simplefilter('default', exceptions.InsecurePlatformWarning,
append=True)
# SNIMissingWarnings should go off only once.
warnings.simplefilter('default', exceptions.SNIMissingWarning)
def disable_warnings(category=exceptions.HTTPWarning):
"""
Helper for quickly disabling all urllib3 warnings.
"""
warnings.simplefilter('ignore', category)
requests/packages/urllib3/_collections.py
+234 -13
View File
@@ -1,13 +1,8 @@
# urllib3/_collections.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from collections import MutableMapping
from __future__ import absolute_import
from collections import Mapping, MutableMapping
try:
from threading import RLock
except ImportError: # Platform-specific: No threads available
except ImportError: # Platform-specific: No threads available
class RLock:
def __enter__(self):
pass
@@ -16,13 +11,14 @@ except ImportError: # Platform-specific: No threads available
pass
try: # Python 2.7+
try: # Python 2.7+
from collections import OrderedDict
except ImportError:
from .packages.ordered_dict import OrderedDict
from .packages.six import iterkeys, itervalues, PY3
__all__ = ['RecentlyUsedContainer']
__all__ = ['RecentlyUsedContainer', 'HTTPHeaderDict']
_Null = object()
@@ -90,8 +86,7 @@ class RecentlyUsedContainer(MutableMapping):
def clear(self):
with self.lock:
# Copy pointers to all values, then wipe the mapping
# under Python 2, this copies the list of values twice :-|
values = list(self._container.values())
values = list(itervalues(self._container))
self._container.clear()
if self.dispose_func:
@@ -100,4 +95,230 @@ class RecentlyUsedContainer(MutableMapping):
def keys(self):
with self.lock:
return self._container.keys()
return list(iterkeys(self._container))
class HTTPHeaderDict(MutableMapping):
"""
:param headers:
An iterable of field-value pairs. Must not contain multiple field names
when compared case-insensitively.
:param kwargs:
Additional field-value pairs to pass in to ``dict.update``.
A ``dict`` like container for storing HTTP Headers.
Field names are stored and compared case-insensitively in compliance with
RFC 7230. Iteration provides the first case-sensitive key seen for each
case-insensitive pair.
Using ``__setitem__`` syntax overwrites fields that compare equal
case-insensitively in order to maintain ``dict``'s api. For fields that
compare equal, instead create a new ``HTTPHeaderDict`` and use ``.add``
in a loop.
If multiple fields that are equal case-insensitively are passed to the
constructor or ``.update``, the behavior is undefined and some will be
lost.
>>> headers = HTTPHeaderDict()
>>> headers.add('Set-Cookie', 'foo=bar')
>>> headers.add('set-cookie', 'baz=quxx')
>>> headers['content-length'] = '7'
>>> headers['SET-cookie']
'foo=bar, baz=quxx'
>>> headers['Content-Length']
'7'
"""
def __init__(self, headers=None, **kwargs):
super(HTTPHeaderDict, self).__init__()
self._container = {}
if headers is not None:
if isinstance(headers, HTTPHeaderDict):
self._copy_from(headers)
else:
self.extend(headers)
if kwargs:
self.extend(kwargs)
def __setitem__(self, key, val):
self._container[key.lower()] = (key, val)
return self._container[key.lower()]
def __getitem__(self, key):
val = self._container[key.lower()]
return ', '.join(val[1:])
def __delitem__(self, key):
del self._container[key.lower()]
def __contains__(self, key):
return key.lower() in self._container
def __eq__(self, other):
if not isinstance(other, Mapping) and not hasattr(other, 'keys'):
return False
if not isinstance(other, type(self)):
other = type(self)(other)
return (dict((k.lower(), v) for k, v in self.itermerged()) ==
dict((k.lower(), v) for k, v in other.itermerged()))
def __ne__(self, other):
return not self.__eq__(other)
if not PY3: # Python 2
iterkeys = MutableMapping.iterkeys
itervalues = MutableMapping.itervalues
__marker = object()
def __len__(self):
return len(self._container)
def __iter__(self):
# Only provide the originally cased names
for vals in self._container.values():
yield vals[0]
def pop(self, key, default=__marker):
'''D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
If key is not found, d is returned if given, otherwise KeyError is raised.
'''
# Using the MutableMapping function directly fails due to the private marker.
# Using ordinary dict.pop would expose the internal structures.
# So let's reinvent the wheel.
try:
value = self[key]
except KeyError:
if default is self.__marker:
raise
return default
else:
del self[key]
return value
def discard(self, key):
try:
del self[key]
except KeyError:
pass
def add(self, key, val):
"""Adds a (name, value) pair, doesn't overwrite the value if it already
exists.
>>> headers = HTTPHeaderDict(foo='bar')
>>> headers.add('Foo', 'baz')
>>> headers['foo']
'bar, baz'
"""
key_lower = key.lower()
new_vals = key, val
# Keep the common case aka no item present as fast as possible
vals = self._container.setdefault(key_lower, new_vals)
if new_vals is not vals:
# new_vals was not inserted, as there was a previous one
if isinstance(vals, list):
# If already several items got inserted, we have a list
vals.append(val)
else:
# vals should be a tuple then, i.e. only one item so far
# Need to convert the tuple to list for further extension
self._container[key_lower] = [vals[0], vals[1], val]
def extend(self, *args, **kwargs):
"""Generic import function for any type of header-like object.
Adapted version of MutableMapping.update in order to insert items
with self.add instead of self.__setitem__
"""
if len(args) > 1:
raise TypeError("extend() takes at most 1 positional "
"arguments ({0} given)".format(len(args)))
other = args[0] if len(args) >= 1 else ()
if isinstance(other, HTTPHeaderDict):
for key, val in other.iteritems():
self.add(key, val)
elif isinstance(other, Mapping):
for key in other:
self.add(key, other[key])
elif hasattr(other, "keys"):
for key in other.keys():
self.add(key, other[key])
else:
for key, value in other:
self.add(key, value)
for key, value in kwargs.items():
self.add(key, value)
def getlist(self, key):
"""Returns a list of all the values for the named field. Returns an
empty list if the key doesn't exist."""
try:
vals = self._container[key.lower()]
except KeyError:
return []
else:
if isinstance(vals, tuple):
return [vals[1]]
else:
return vals[1:]
# Backwards compatibility for httplib
getheaders = getlist
getallmatchingheaders = getlist
iget = getlist
def __repr__(self):
return "%s(%s)" % (type(self).__name__, dict(self.itermerged()))
def _copy_from(self, other):
for key in other:
val = other.getlist(key)
if isinstance(val, list):
# Don't need to convert tuples
val = list(val)
self._container[key.lower()] = [key] + val
def copy(self):
clone = type(self)()
clone._copy_from(self)
return clone
def iteritems(self):
"""Iterate over all header lines, including duplicate ones."""
for key in self:
vals = self._container[key.lower()]
for val in vals[1:]:
yield vals[0], val
def itermerged(self):
"""Iterate over all headers, merging duplicate ones together."""
for key in self:
val = self._container[key.lower()]
yield val[0], ', '.join(val[1:])
def items(self):
return list(self.iteritems())
@classmethod
def from_httplib(cls, message): # Python 2
"""Read headers from a Python 2 httplib message object."""
# python2.7 does not expose a proper API for exporting multiheaders
# efficiently. This function re-reads raw lines from the message
# object and extracts the multiheaders properly.
headers = []
for line in message.headers:
if line.startswith((' ', '\t')):
key, value = headers[-1]
headers[-1] = (key, value + '\r\n' + line.rstrip())
continue
key, value = line.split(':', 1)
headers.append((key, value.strip()))
return cls(headers)
requests/packages/urllib3/connection.py
+219 -50
View File
@@ -1,50 +1,190 @@
# urllib3/connection.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from __future__ import absolute_import
import datetime
import os
import sys
import socket
from socket import timeout as SocketTimeout
from socket import error as SocketError, timeout as SocketTimeout
import warnings
from .packages import six
try: # Python 3
from http.client import HTTPConnection, HTTPException
try: # Python 3
from http.client import HTTPConnection as _HTTPConnection
from http.client import HTTPException # noqa: unused in this module
except ImportError:
from httplib import HTTPConnection, HTTPException
from httplib import HTTPConnection as _HTTPConnection
from httplib import HTTPException # noqa: unused in this module
class DummyConnection(object):
"Used to detect a failed ConnectionCls import."
pass
try: # Compiled with SSL?
try: # Compiled with SSL?
import ssl
BaseSSLError = ssl.SSLError
except (ImportError, AttributeError): # Platform-specific: No SSL.
ssl = None
HTTPSConnection = DummyConnection
class BaseSSLError(BaseException):
pass
try: # Python 3
from http.client import HTTPSConnection
except ImportError:
from httplib import HTTPSConnection
import ssl
BaseSSLError = ssl.SSLError
try: # Python 3:
# Not a no-op, we're adding this to the namespace so it can be imported.
ConnectionError = ConnectionError
except NameError: # Python 2:
class ConnectionError(Exception):
pass
except (ImportError, AttributeError): # Platform-specific: No SSL.
pass
from .exceptions import (
NewConnectionError,
ConnectTimeoutError,
SubjectAltNameWarning,
SystemTimeWarning,
)
from .packages.ssl_match_hostname import match_hostname
from .util import (
assert_fingerprint,
from .util.ssl_ import (
resolve_cert_reqs,
resolve_ssl_version,
ssl_wrap_socket,
assert_fingerprint,
)
from .util import connection
port_by_scheme = {
'http': 80,
'https': 443,
}
RECENT_DATE = datetime.date(2014, 1, 1)
class DummyConnection(object):
"""Used to detect a failed ConnectionCls import."""
pass
class HTTPConnection(_HTTPConnection, object):
"""
Based on httplib.HTTPConnection but provides an extra constructor
backwards-compatibility layer between older and newer Pythons.
Additional keyword parameters are used to configure attributes of the connection.
Accepted parameters include:
- ``strict``: See the documentation on :class:`urllib3.connectionpool.HTTPConnectionPool`
- ``source_address``: Set the source address for the current connection.
.. note:: This is ignored for Python 2.6. It is only applied for 2.7 and 3.x
- ``socket_options``: Set specific options on the underlying socket. If not specified, then
defaults are loaded from ``HTTPConnection.default_socket_options`` which includes disabling
Nagle's algorithm (sets TCP_NODELAY to 1) unless the connection is behind a proxy.
For example, if you wish to enable TCP Keep Alive in addition to the defaults,
you might pass::
HTTPConnection.default_socket_options + [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
]
Or you may want to disable the defaults by passing an empty list (e.g., ``[]``).
"""
default_port = port_by_scheme['http']
#: Disable Nagle's algorithm by default.
#: ``[(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)]``
default_socket_options = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)]
#: Whether this connection verifies the host's certificate.
is_verified = False
def __init__(self, *args, **kw):
if six.PY3: # Python 3
kw.pop('strict', None)
# Pre-set source_address in case we have an older Python like 2.6.
self.source_address = kw.get('source_address')
if sys.version_info < (2, 7): # Python 2.6
# _HTTPConnection on Python 2.6 will balk at this keyword arg, but
# not newer versions. We can still use it when creating a
# connection though, so we pop it *after* we have saved it as
# self.source_address.
kw.pop('source_address', None)
#: The socket options provided by the user. If no options are
#: provided, we use the default options.
self.socket_options = kw.pop('socket_options', self.default_socket_options)
# Superclass also sets self.source_address in Python 2.7+.
_HTTPConnection.__init__(self, *args, **kw)
def _new_conn(self):
""" Establish a socket connection and set nodelay settings on it.
:return: New socket connection.
"""
extra_kw = {}
if self.source_address:
extra_kw['source_address'] = self.source_address
if self.socket_options:
extra_kw['socket_options'] = self.socket_options
try:
conn = connection.create_connection(
(self.host, self.port), self.timeout, **extra_kw)
except SocketTimeout as e:
raise ConnectTimeoutError(
self, "Connection to %s timed out. (connect timeout=%s)" %
(self.host, self.timeout))
except SocketError as e:
raise NewConnectionError(
self, "Failed to establish a new connection: %s" % e)
return conn
def _prepare_conn(self, conn):
self.sock = conn
# the _tunnel_host attribute was added in python 2.6.3 (via
# http://hg.python.org/cpython/rev/0f57b30a152f) so pythons 2.6(0-2) do
# not have them.
if getattr(self, '_tunnel_host', None):
# TODO: Fix tunnel so it doesn't depend on self.sock state.
self._tunnel()
# Mark this connection as not reusable
self.auto_open = 0
def connect(self):
conn = self._new_conn()
self._prepare_conn(conn)
class HTTPSConnection(HTTPConnection):
default_port = port_by_scheme['https']
def __init__(self, host, port=None, key_file=None, cert_file=None,
strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, **kw):
HTTPConnection.__init__(self, host, port, strict=strict,
timeout=timeout, **kw)
self.key_file = key_file
self.cert_file = cert_file
# Required property for Google AppEngine 1.9.0 which otherwise causes
# HTTPS requests to go out as HTTP. (See Issue #356)
self._protocol = 'https'
def connect(self):
conn = self._new_conn()
self._prepare_conn(conn)
self.sock = ssl.wrap_socket(conn, self.key_file, self.cert_file)
class VerifiedHTTPSConnection(HTTPSConnection):
"""
Based on httplib.HTTPSConnection but wraps the socket with
@@ -52,59 +192,88 @@ class VerifiedHTTPSConnection(HTTPSConnection):
"""
cert_reqs = None
ca_certs = None
ca_cert_dir = None
ssl_version = None
assert_fingerprint = None
def set_cert(self, key_file=None, cert_file=None,
cert_reqs=None, ca_certs=None,
assert_hostname=None, assert_fingerprint=None):
assert_hostname=None, assert_fingerprint=None,
ca_cert_dir=None):
if (ca_certs or ca_cert_dir) and cert_reqs is None:
cert_reqs = 'CERT_REQUIRED'
self.key_file = key_file
self.cert_file = cert_file
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
self.assert_hostname = assert_hostname
self.assert_fingerprint = assert_fingerprint
self.ca_certs = ca_certs and os.path.expanduser(ca_certs)
self.ca_cert_dir = ca_cert_dir and os.path.expanduser(ca_cert_dir)
def connect(self):
# Add certificate verification
try:
sock = socket.create_connection(
address=(self.host, self.port),
timeout=self.timeout,
)
except SocketTimeout:
raise ConnectTimeoutError(
self, "Connection to %s timed out. (connect timeout=%s)" %
(self.host, self.timeout))
conn = self._new_conn()
resolved_cert_reqs = resolve_cert_reqs(self.cert_reqs)
resolved_ssl_version = resolve_ssl_version(self.ssl_version)
# the _tunnel_host attribute was added in python 2.6.3 (via
# http://hg.python.org/cpython/rev/0f57b30a152f) so pythons 2.6(0-2) do
# not have them.
hostname = self.host
if getattr(self, '_tunnel_host', None):
self.sock = sock
# _tunnel_host was added in Python 2.6.3
# (See: http://hg.python.org/cpython/rev/0f57b30a152f)
self.sock = conn
# Calls self._set_hostport(), so self.host is
# self._tunnel_host below.
self._tunnel()
# Mark this connection as not reusable
self.auto_open = 0
# Override the host with the one we're requesting data from.
hostname = self._tunnel_host
is_time_off = datetime.date.today() < RECENT_DATE
if is_time_off:
warnings.warn((
'System time is way off (before {0}). This will probably '
'lead to SSL verification errors').format(RECENT_DATE),
SystemTimeWarning
)
# Wrap socket using verification with the root certs in
# trusted_root_certs
self.sock = ssl_wrap_socket(sock, self.key_file, self.cert_file,
self.sock = ssl_wrap_socket(conn, self.key_file, self.cert_file,
cert_reqs=resolved_cert_reqs,
ca_certs=self.ca_certs,
server_hostname=self.host,
ca_cert_dir=self.ca_cert_dir,
server_hostname=hostname,
ssl_version=resolved_ssl_version)
if resolved_cert_reqs != ssl.CERT_NONE:
if self.assert_fingerprint:
assert_fingerprint(self.sock.getpeercert(binary_form=True),
self.assert_fingerprint)
elif self.assert_hostname is not False:
match_hostname(self.sock.getpeercert(),
self.assert_hostname or self.host)
if self.assert_fingerprint:
assert_fingerprint(self.sock.getpeercert(binary_form=True),
self.assert_fingerprint)
elif resolved_cert_reqs != ssl.CERT_NONE \
and self.assert_hostname is not False:
cert = self.sock.getpeercert()
if not cert.get('subjectAltName', ()):
warnings.warn((
'Certificate for {0} has no `subjectAltName`, falling back to check for a '
'`commonName` for now. This feature is being removed by major browsers and '
'deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 '
'for details.)'.format(hostname)),
SubjectAltNameWarning
)
match_hostname(cert, self.assert_hostname or hostname)
self.is_verified = (resolved_cert_reqs == ssl.CERT_REQUIRED or
self.assert_fingerprint is not None)
if ssl:
# Make a copy for testing.
UnverifiedHTTPSConnection = HTTPSConnection
HTTPSConnection = VerifiedHTTPSConnection
else:
HTTPSConnection = DummyConnection
requests/packages/urllib3/connectionpool.py
+283 -148
View File
@@ -1,48 +1,51 @@
# urllib3/connectionpool.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from __future__ import absolute_import
import errno
import logging
import sys
import warnings
from socket import error as SocketError, timeout as SocketTimeout
import socket
try: # Python 3
try: # Python 3
from queue import LifoQueue, Empty, Full
except ImportError:
from Queue import LifoQueue, Empty, Full
import Queue as _ # Platform-specific: Windows
# Queue is imported for side effects on MS Windows
import Queue as _unused_module_Queue # noqa: unused
from .exceptions import (
ClosedPoolError,
ConnectTimeoutError,
ProtocolError,
EmptyPoolError,
HeaderParsingError,
HostChangedError,
LocationValueError,
MaxRetryError,
ProxyError,
ReadTimeoutError,
SSLError,
TimeoutError,
ReadTimeoutError,
ProxyError,
InsecureRequestWarning,
NewConnectionError,
)
from .packages.ssl_match_hostname import CertificateError
from .packages import six
from .connection import (
port_by_scheme,
DummyConnection,
HTTPConnection, HTTPSConnection, VerifiedHTTPSConnection,
HTTPException, BaseSSLError,
)
from .request import RequestMethods
from .response import HTTPResponse
from .util import (
assert_fingerprint,
get_host,
is_connection_dropped,
Timeout,
)
from .util.connection import is_connection_dropped
from .util.response import assert_header_parsing
from .util.retry import Retry
from .util.timeout import Timeout
from .util.url import get_host, Url
xrange = six.moves.xrange
@@ -51,14 +54,8 @@ log = logging.getLogger(__name__)
_Default = object()
port_by_scheme = {
'http': 80,
'https': 443,
}
## Pool objects
# Pool objects
class ConnectionPool(object):
"""
Base class for all connection pools, such as
@@ -69,8 +66,8 @@ class ConnectionPool(object):
QueueCls = LifoQueue
def __init__(self, host, port=None):
# httplib doesn't like it when we include brackets in ipv6 addresses
host = host.strip('[]')
if not host:
raise LocationValueError("No host specified.")
self.host = host
self.port = port
@@ -79,9 +76,25 @@ class ConnectionPool(object):
return '%s(host=%r, port=%r)' % (type(self).__name__,
self.host, self.port)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
# Return False to re-raise any potential exceptions
return False
def close():
"""
Close all pooled connections and disable the pool.
"""
pass
# This is taken from http://hg.python.org/cpython/file/7aaba721ebc0/Lib/socket.py#l252
_blocking_errnos = set([errno.EAGAIN, errno.EWOULDBLOCK])
class HTTPConnectionPool(ConnectionPool, RequestMethods):
"""
Thread-safe connection pool for one host.
@@ -111,7 +124,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
:param maxsize:
Number of connections to save that can be reused. More than 1 is useful
in multithreaded situations. If ``block`` is set to false, more
in multithreaded situations. If ``block`` is set to False, more
connections will be created but they will not be saved once they've
been used.
@@ -126,6 +139,9 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
Headers to include with all requests, unless other headers are given
explicitly.
:param retries:
Retry configuration to use by default with requests in this pool.
:param _proxy:
Parsed proxy URL, should not be used directly, instead, see
:class:`urllib3.connectionpool.ProxyManager`"
@@ -133,6 +149,10 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
:param _proxy_headers:
A dictionary with proxy headers, should not be used directly,
instead, see :class:`urllib3.connectionpool.ProxyManager`"
:param \**conn_kw:
Additional parameters are used to create fresh :class:`urllib3.connection.HTTPConnection`,
:class:`urllib3.connection.HTTPSConnection` instances.
"""
scheme = 'http'
@@ -140,18 +160,22 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
def __init__(self, host, port=None, strict=False,
timeout=Timeout.DEFAULT_TIMEOUT, maxsize=1, block=False,
headers=None, _proxy=None, _proxy_headers=None):
headers=None, retries=None,
_proxy=None, _proxy_headers=None,
**conn_kw):
ConnectionPool.__init__(self, host, port)
RequestMethods.__init__(self, headers)
self.strict = strict
# This is for backwards compatibility and can be removed once a timeout
# can only be set to a Timeout object
if not isinstance(timeout, Timeout):
timeout = Timeout.from_float(timeout)
if retries is None:
retries = Retry.DEFAULT
self.timeout = timeout
self.retries = retries
self.pool = self.QueueCls(maxsize)
self.block = block
@@ -166,22 +190,26 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
# These are mostly for testing and debugging purposes.
self.num_connections = 0
self.num_requests = 0
self.conn_kw = conn_kw
if self.proxy:
# Enable Nagle's algorithm for proxies, to avoid packet fragmentation.
# We cannot know if the user has added default socket options, so we cannot replace the
# list.
self.conn_kw.setdefault('socket_options', [])
def _new_conn(self):
"""
Return a fresh :class:`httplib.HTTPConnection`.
Return a fresh :class:`HTTPConnection`.
"""
self.num_connections += 1
log.info("Starting new HTTP connection (%d): %s" %
(self.num_connections, self.host))
extra_params = {}
if not six.PY3: # Python 2
extra_params['strict'] = self.strict
return self.ConnectionCls(host=self.host, port=self.port,
conn = self.ConnectionCls(host=self.host, port=self.port,
timeout=self.timeout.connect_timeout,
**extra_params)
strict=self.strict, **self.conn_kw)
return conn
def _get_conn(self, timeout=None):
"""
@@ -199,7 +227,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
try:
conn = self.pool.get(block=self.block, timeout=timeout)
except AttributeError: # self.pool is None
except AttributeError: # self.pool is None
raise ClosedPoolError(self, "Pool is closed.")
except Empty:
@@ -213,6 +241,11 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
if conn and is_connection_dropped(conn):
log.info("Resetting dropped connection: %s" % self.host)
conn.close()
if getattr(conn, 'auto_open', 1) == 0:
# This is a proxied connection that has been mutated by
# httplib._tunnel() and cannot be reused (since it would
# attempt to bypass the proxy)
conn = None
return conn or self._new_conn()
@@ -232,19 +265,30 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
"""
try:
self.pool.put(conn, block=False)
return # Everything is dandy, done.
return # Everything is dandy, done.
except AttributeError:
# self.pool is None.
pass
except Full:
# This should never happen if self.block == True
log.warning("HttpConnectionPool is full, discarding connection: %s"
% self.host)
log.warning(
"Connection pool is full, discarding connection: %s" %
self.host)
# Connection never got put back into the pool, close it.
if conn:
conn.close()
def _validate_conn(self, conn):
"""
Called right before a request is made, after the socket is created.
"""
pass
def _prepare_proxy(self, conn):
# Nothing to do for HTTP connections.
pass
def _get_timeout(self, timeout):
""" Helper that always returns a :class:`urllib3.util.Timeout` """
if timeout is _Default:
@@ -257,10 +301,27 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
# can be removed later
return Timeout.from_float(timeout)
def _raise_timeout(self, err, url, timeout_value):
"""Is the error actually a timeout? Will raise a ReadTimeout or pass"""
if isinstance(err, SocketTimeout):
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
# See the above comment about EAGAIN in Python 3. In Python 2 we have
# to specifically catch it and throw the timeout error
if hasattr(err, 'errno') and err.errno in _blocking_errnos:
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
# Catch possible read timeouts thrown as SSL errors. If not the
# case, rethrow the original. We need to do this because of:
# http://bugs.python.org/issue10272
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
def _make_request(self, conn, method, url, timeout=_Default,
**httplib_request_kw):
"""
Perform a request on a given httplib connection object taken from our
Perform a request on a given urllib connection object taken from our
pool.
:param conn:
@@ -276,23 +337,26 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
self.num_requests += 1
timeout_obj = self._get_timeout(timeout)
timeout_obj.start_connect()
conn.timeout = timeout_obj.connect_timeout
# Trigger any extra validation we need to do.
try:
timeout_obj.start_connect()
conn.timeout = timeout_obj.connect_timeout
# conn.request() calls httplib.*.request, not the method in
# urllib3.request. It also calls makefile (recv) on the socket.
conn.request(method, url, **httplib_request_kw)
except SocketTimeout:
raise ConnectTimeoutError(
self, "Connection to %s timed out. (connect timeout=%s)" %
(self.host, timeout_obj.connect_timeout))
self._validate_conn(conn)
except (SocketTimeout, BaseSSLError) as e:
# Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
raise
# conn.request() calls httplib.*.request, not the method in
# urllib3.request. It also calls makefile (recv) on the socket.
conn.request(method, url, **httplib_request_kw)
# Reset the timeout for the recv() on the socket
read_timeout = timeout_obj.read_timeout
# App Engine doesn't have a sock attr
if hasattr(conn, 'sock'):
if getattr(conn, 'sock', None):
# In Python 3 socket.py will catch EAGAIN and return None when you
# try and read into the file pointer created by http.client, which
# instead raises a BadStatusLine exception. Instead of catching
@@ -300,41 +364,20 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
# timeouts, check for a zero timeout before making the request.
if read_timeout == 0:
raise ReadTimeoutError(
self, url,
"Read timed out. (read timeout=%s)" % read_timeout)
self, url, "Read timed out. (read timeout=%s)" % read_timeout)
if read_timeout is Timeout.DEFAULT_TIMEOUT:
conn.sock.settimeout(socket.getdefaulttimeout())
else: # None or a value
else: # None or a value
conn.sock.settimeout(read_timeout)
# Receive the response from the server
try:
try: # Python 2.7+, use buffering of HTTP responses
try: # Python 2.7, use buffering of HTTP responses
httplib_response = conn.getresponse(buffering=True)
except TypeError: # Python 2.6 and older
except TypeError: # Python 2.6 and older
httplib_response = conn.getresponse()
except SocketTimeout:
raise ReadTimeoutError(
self, url, "Read timed out. (read timeout=%s)" % read_timeout)
except BaseSSLError as e:
# Catch possible read timeouts thrown as SSL errors. If not the
# case, rethrow the original. We need to do this because of:
# http://bugs.python.org/issue10272
if 'timed out' in str(e) or \
'did not complete (read)' in str(e): # Python 2.6
raise ReadTimeoutError(self, url, "Read timed out.")
raise
except SocketError as e: # Platform-specific: Python 2
# See the above comment about EAGAIN in Python 3. In Python 2 we
# have to specifically catch it and throw the timeout error
if e.errno in _blocking_errnos:
raise ReadTimeoutError(
self, url,
"Read timed out. (read timeout=%s)" % read_timeout)
except (SocketTimeout, BaseSSLError, SocketError) as e:
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
raise
# AppEngine doesn't have a version attr.
@@ -342,8 +385,19 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
log.debug("\"%s %s %s\" %s %s" % (method, url, http_version,
httplib_response.status,
httplib_response.length))
try:
assert_header_parsing(httplib_response.msg)
except HeaderParsingError as hpe: # Platform-specific: Python 3
log.warning(
'Failed to parse headers (url=%s): %s',
self._absolute_url(url), hpe, exc_info=True)
return httplib_response
def _absolute_url(self, path):
return Url(scheme=self.scheme, host=self.host, port=self.port, path=path).url
def close(self):
"""
Close all pooled connections and disable the pool.
@@ -358,7 +412,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
conn.close()
except Empty:
pass # Done.
pass # Done.
def is_same_host(self, url):
"""
@@ -379,7 +433,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
return (scheme, host, port) == (self.scheme, self.host, self.port)
def urlopen(self, method, url, body=None, headers=None, retries=3,
def urlopen(self, method, url, body=None, headers=None, retries=None,
redirect=True, assert_same_host=True, timeout=_Default,
pool_timeout=None, release_conn=None, **response_kw):
"""
@@ -413,11 +467,25 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
these headers completely replace any pool-specific headers.
:param retries:
Number of retries to allow before raising a MaxRetryError exception.
Configure the number of retries to allow before raising a
:class:`~urllib3.exceptions.MaxRetryError` exception.
Pass ``None`` to retry until you receive a response. Pass a
:class:`~urllib3.util.retry.Retry` object for fine-grained control
over different types of retries.
Pass an integer number to retry connection errors that many times,
but no other types of errors. Pass zero to never retry.
If ``False``, then retries are disabled and any exception is raised
immediately. Also, instead of raising a MaxRetryError on redirects,
the redirect response will be returned.
:type retries: :class:`~urllib3.util.retry.Retry`, False, or an int.
:param redirect:
If True, automatically handle redirects (status codes 301, 302,
303, 307, 308). Each redirect counts as a retry.
303, 307, 308). Each redirect counts as a retry. Disabling retries
will disable redirect, too.
:param assert_same_host:
If ``True``, will make sure that the host of the pool requests is
@@ -451,15 +519,15 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
if headers is None:
headers = self.headers
if retries < 0:
raise MaxRetryError(self, url)
if not isinstance(retries, Retry):
retries = Retry.from_int(retries, redirect=redirect, default=self.retries)
if release_conn is None:
release_conn = response_kw.get('preload_content', True)
# Check host
if assert_same_host and not self.is_same_host(url):
raise HostChangedError(self, url, retries - 1)
raise HostChangedError(self, url, retries)
conn = None
@@ -470,13 +538,24 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
headers = headers.copy()
headers.update(self.proxy_headers)
# Must keep the exception bound to a separate variable or else Python 3
# complains about UnboundLocalError.
err = None
try:
# Request a connection from the queue
# Request a connection from the queue.
timeout_obj = self._get_timeout(timeout)
conn = self._get_conn(timeout=pool_timeout)
# Make the request on the httplib connection object
conn.timeout = timeout_obj.connect_timeout
is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None)
if is_new_proxy_conn:
self._prepare_proxy(conn)
# Make the request on the httplib connection object.
httplib_response = self._make_request(conn, method, url,
timeout=timeout,
timeout=timeout_obj,
body=body, headers=headers)
# If we're going to release the connection in ``finally:``, then
@@ -497,38 +576,42 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
# ``response.read()``)
except Empty:
# Timed out by queue
# Timed out by queue.
raise EmptyPoolError(self, "No pool connections are available.")
except BaseSSLError as e:
except (BaseSSLError, CertificateError) as e:
# Close the connection. If a connection is reused on which there
# was a Certificate error, the next request will certainly raise
# another Certificate error.
conn = conn and conn.close()
release_conn = True
raise SSLError(e)
except CertificateError as e:
# Name mismatch
raise SSLError(e)
except SSLError:
# Treat SSLError separately from BaseSSLError to preserve
# traceback.
conn = conn and conn.close()
release_conn = True
raise
except TimeoutError as e:
# Connection broken, discard.
conn = None
# Save the error off for retry logic.
except (TimeoutError, HTTPException, SocketError, ProtocolError) as e:
# Discard the connection for these exceptions. It will be
# be replaced during the next _get_conn() call.
conn = conn and conn.close()
release_conn = True
if isinstance(e, (SocketError, NewConnectionError)) and self.proxy:
e = ProxyError('Cannot connect to proxy.', e)
elif isinstance(e, (SocketError, HTTPException)):
e = ProtocolError('Connection aborted.', e)
retries = retries.increment(method, url, error=e, _pool=self,
_stacktrace=sys.exc_info()[2])
retries.sleep()
# Keep track of the error for the retry warning.
err = e
if retries == 0:
raise
except (HTTPException, SocketError) as e:
if isinstance(e, SocketError) and self.proxy is not None:
raise ProxyError('Cannot connect to proxy. '
'Socket error: %s.' % e)
# Connection broken, discard. It will be replaced next _get_conn().
conn = None
# This is necessary so we can access e below
err = e
if retries == 0:
raise MaxRetryError(self, url, e)
finally:
if release_conn:
# Put the connection back to be reused. If the connection is
@@ -538,9 +621,9 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
if not conn:
# Try again
log.warn("Retrying (%d attempts remain) after connection "
"broken by '%r': %s" % (retries, err, url))
return self.urlopen(method, url, body, headers, retries - 1,
log.warning("Retrying (%r) after connection "
"broken by '%r': %s" % (retries, err, url))
return self.urlopen(method, url, body, headers, retries,
redirect, assert_same_host,
timeout=timeout, pool_timeout=pool_timeout,
release_conn=release_conn, **response_kw)
@@ -550,11 +633,36 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
if redirect_location:
if response.status == 303:
method = 'GET'
try:
retries = retries.increment(method, url, response=response, _pool=self)
except MaxRetryError:
if retries.raise_on_redirect:
# Release the connection for this response, since we're not
# returning it to be released manually.
response.release_conn()
raise
return response
log.info("Redirecting %s -> %s" % (url, redirect_location))
return self.urlopen(method, redirect_location, body, headers,
retries - 1, redirect, assert_same_host,
timeout=timeout, pool_timeout=pool_timeout,
release_conn=release_conn, **response_kw)
return self.urlopen(
method, redirect_location, body, headers,
retries=retries, redirect=redirect,
assert_same_host=assert_same_host,
timeout=timeout, pool_timeout=pool_timeout,
release_conn=release_conn, **response_kw)
# Check if we should retry the HTTP response.
if retries.is_forced_retry(method, status_code=response.status):
retries = retries.increment(method, url, response=response, _pool=self)
retries.sleep()
log.info("Forced retry: %s" % url)
return self.urlopen(
method, url, body, headers,
retries=retries, redirect=redirect,
assert_same_host=assert_same_host,
timeout=timeout, pool_timeout=pool_timeout,
release_conn=release_conn, **response_kw)
return response
@@ -565,35 +673,42 @@ class HTTPSConnectionPool(HTTPConnectionPool):
When Python is compiled with the :mod:`ssl` module, then
:class:`.VerifiedHTTPSConnection` is used, which *can* verify certificates,
instead of :class:`httplib.HTTPSConnection`.
instead of :class:`.HTTPSConnection`.
:class:`.VerifiedHTTPSConnection` uses one of ``assert_fingerprint``,
``assert_hostname`` and ``host`` in this order to verify connections.
If ``assert_hostname`` is False, no verification is done.
The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs`` and
``ssl_version`` are only used if :mod:`ssl` is available and are fed into
:meth:`urllib3.util.ssl_wrap_socket` to upgrade the connection socket
into an SSL socket.
The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs``,
``ca_cert_dir``, and ``ssl_version`` are only used if :mod:`ssl` is
available and are fed into :meth:`urllib3.util.ssl_wrap_socket` to upgrade
the connection socket into an SSL socket.
"""
scheme = 'https'
ConnectionCls = HTTPSConnection
def __init__(self, host, port=None,
strict=False, timeout=None, maxsize=1,
block=False, headers=None,
strict=False, timeout=Timeout.DEFAULT_TIMEOUT, maxsize=1,
block=False, headers=None, retries=None,
_proxy=None, _proxy_headers=None,
key_file=None, cert_file=None, cert_reqs=None,
ca_certs=None, ssl_version=None,
assert_hostname=None, assert_fingerprint=None):
assert_hostname=None, assert_fingerprint=None,
ca_cert_dir=None, **conn_kw):
HTTPConnectionPool.__init__(self, host, port, strict, timeout, maxsize,
block, headers, _proxy, _proxy_headers)
block, headers, retries, _proxy, _proxy_headers,
**conn_kw)
if ca_certs and cert_reqs is None:
cert_reqs = 'CERT_REQUIRED'
self.key_file = key_file
self.cert_file = cert_file
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
self.ca_cert_dir = ca_cert_dir
self.ssl_version = ssl_version
self.assert_hostname = assert_hostname
self.assert_fingerprint = assert_fingerprint
@@ -609,23 +724,31 @@ class HTTPSConnectionPool(HTTPConnectionPool):
cert_file=self.cert_file,
cert_reqs=self.cert_reqs,
ca_certs=self.ca_certs,
ca_cert_dir=self.ca_cert_dir,
assert_hostname=self.assert_hostname,
assert_fingerprint=self.assert_fingerprint)
conn.ssl_version = self.ssl_version
if self.proxy is not None:
# Python 2.7+
try:
set_tunnel = conn.set_tunnel
except AttributeError: # Platform-specific: Python 2.6
set_tunnel = conn._set_tunnel
set_tunnel(self.host, self.port, self.proxy_headers)
# Establish tunnel connection early, because otherwise httplib
# would improperly set Host: header to proxy's IP:port.
conn.connect()
return conn
def _prepare_proxy(self, conn):
"""
Establish tunnel connection early, because otherwise httplib
would improperly set Host: header to proxy's IP:port.
"""
# Python 2.7+
try:
set_tunnel = conn.set_tunnel
except AttributeError: # Platform-specific: Python 2.6
set_tunnel = conn._set_tunnel
if sys.version_info <= (2, 6, 4) and not self.proxy_headers: # Python 2.6.4 and older
set_tunnel(self.host, self.port)
else:
set_tunnel(self.host, self.port, self.proxy_headers)
conn.connect()
def _new_conn(self):
"""
Return a fresh :class:`httplib.HTTPSConnection`.
@@ -635,7 +758,6 @@ class HTTPSConnectionPool(HTTPConnectionPool):
% (self.num_connections, self.host))
if not self.ConnectionCls or self.ConnectionCls is DummyConnection:
# Platform-specific: Python without ssl
raise SSLError("Can't connect to HTTPS URL because the SSL "
"module is not available.")
@@ -645,16 +767,29 @@ class HTTPSConnectionPool(HTTPConnectionPool):
actual_host = self.proxy.host
actual_port = self.proxy.port
extra_params = {}
if not six.PY3: # Python 2
extra_params['strict'] = self.strict
conn = self.ConnectionCls(host=actual_host, port=actual_port,
timeout=self.timeout.connect_timeout,
**extra_params)
strict=self.strict, **self.conn_kw)
return self._prepare_conn(conn)
def _validate_conn(self, conn):
"""
Called right before a request is made, after the socket is created.
"""
super(HTTPSConnectionPool, self)._validate_conn(conn)
# Force connect early to allow us to validate the connection.
if not getattr(conn, 'sock', None): # AppEngine might not have `.sock`
conn.connect()
if not conn.is_verified:
warnings.warn((
'Unverified HTTPS request is being made. '
'Adding certificate verification is strongly advised. See: '
'https://urllib3.readthedocs.org/en/latest/security.html'),
InsecureRequestWarning)
def connection_from_url(url, **kw):
"""
@@ -671,7 +806,7 @@ def connection_from_url(url, **kw):
:class:`.ConnectionPool`. Useful for specifying things like
timeout, maxsize, headers, etc.
Example: ::
Example::
>>> conn = connection_from_url('http://google.com/')
>>> r = conn.request('GET', '/')
requests/packages/urllib3/contrib/appengine.py
+223
View File
@@ -0,0 +1,223 @@
from __future__ import absolute_import
import logging
import os
import warnings
from ..exceptions import (
HTTPError,
HTTPWarning,
MaxRetryError,
ProtocolError,
TimeoutError,
SSLError
)
from ..packages.six import BytesIO
from ..request import RequestMethods
from ..response import HTTPResponse
from ..util.timeout import Timeout
from ..util.retry import Retry
try:
from google.appengine.api import urlfetch
except ImportError:
urlfetch = None
log = logging.getLogger(__name__)
class AppEnginePlatformWarning(HTTPWarning):
pass
class AppEnginePlatformError(HTTPError):
pass
class AppEngineManager(RequestMethods):
"""
Connection manager for Google App Engine sandbox applications.
This manager uses the URLFetch service directly instead of using the
emulated httplib, and is subject to URLFetch limitations as described in
the App Engine documentation here:
https://cloud.google.com/appengine/docs/python/urlfetch
Notably it will raise an AppEnginePlatformError if:
* URLFetch is not available.
* If you attempt to use this on GAEv2 (Managed VMs), as full socket
support is available.
* If a request size is more than 10 megabytes.
* If a response size is more than 32 megabtyes.
* If you use an unsupported request method such as OPTIONS.
Beyond those cases, it will raise normal urllib3 errors.
"""
def __init__(self, headers=None, retries=None, validate_certificate=True):
if not urlfetch:
raise AppEnginePlatformError(
"URLFetch is not available in this environment.")
if is_prod_appengine_mvms():
raise AppEnginePlatformError(
"Use normal urllib3.PoolManager instead of AppEngineManager"
"on Managed VMs, as using URLFetch is not necessary in "
"this environment.")
warnings.warn(
"urllib3 is using URLFetch on Google App Engine sandbox instead "
"of sockets. To use sockets directly instead of URLFetch see "
"https://urllib3.readthedocs.org/en/latest/contrib.html.",
AppEnginePlatformWarning)
RequestMethods.__init__(self, headers)
self.validate_certificate = validate_certificate
self.retries = retries or Retry.DEFAULT
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
# Return False to re-raise any potential exceptions
return False
def urlopen(self, method, url, body=None, headers=None,
retries=None, redirect=True, timeout=Timeout.DEFAULT_TIMEOUT,
**response_kw):
retries = self._get_retries(retries, redirect)
try:
response = urlfetch.fetch(
url,
payload=body,
method=method,
headers=headers or {},
allow_truncated=False,
follow_redirects=(
redirect and
retries.redirect != 0 and
retries.total),
deadline=self._get_absolute_timeout(timeout),
validate_certificate=self.validate_certificate,
)
except urlfetch.DeadlineExceededError as e:
raise TimeoutError(self, e)
except urlfetch.InvalidURLError as e:
if 'too large' in str(e):
raise AppEnginePlatformError(
"URLFetch request too large, URLFetch only "
"supports requests up to 10mb in size.", e)
raise ProtocolError(e)
except urlfetch.DownloadError as e:
if 'Too many redirects' in str(e):
raise MaxRetryError(self, url, reason=e)
raise ProtocolError(e)
except urlfetch.ResponseTooLargeError as e:
raise AppEnginePlatformError(
"URLFetch response too large, URLFetch only supports"
"responses up to 32mb in size.", e)
except urlfetch.SSLCertificateError as e:
raise SSLError(e)
except urlfetch.InvalidMethodError as e:
raise AppEnginePlatformError(
"URLFetch does not support method: %s" % method, e)
http_response = self._urlfetch_response_to_http_response(
response, **response_kw)
# Check for redirect response
if (http_response.get_redirect_location() and
retries.raise_on_redirect and redirect):
raise MaxRetryError(self, url, "too many redirects")
# Check if we should retry the HTTP response.
if retries.is_forced_retry(method, status_code=http_response.status):
retries = retries.increment(
method, url, response=http_response, _pool=self)
log.info("Forced retry: %s" % url)
retries.sleep()
return self.urlopen(
method, url,
body=body, headers=headers,
retries=retries, redirect=redirect,
timeout=timeout, **response_kw)
return http_response
def _urlfetch_response_to_http_response(self, urlfetch_resp, **response_kw):
if is_prod_appengine():
# Production GAE handles deflate encoding automatically, but does
# not remove the encoding header.
content_encoding = urlfetch_resp.headers.get('content-encoding')
if content_encoding == 'deflate':
del urlfetch_resp.headers['content-encoding']
return HTTPResponse(
# In order for decoding to work, we must present the content as
# a file-like object.
body=BytesIO(urlfetch_resp.content),
headers=urlfetch_resp.headers,
status=urlfetch_resp.status_code,
**response_kw
)
def _get_absolute_timeout(self, timeout):
if timeout is Timeout.DEFAULT_TIMEOUT:
return 5 # 5s is the default timeout for URLFetch.
if isinstance(timeout, Timeout):
if timeout.read is not timeout.connect:
warnings.warn(
"URLFetch does not support granular timeout settings, "
"reverting to total timeout.", AppEnginePlatformWarning)
return timeout.total
return timeout
def _get_retries(self, retries, redirect):
if not isinstance(retries, Retry):
retries = Retry.from_int(
retries, redirect=redirect, default=self.retries)
if retries.connect or retries.read or retries.redirect:
warnings.warn(
"URLFetch only supports total retries and does not "
"recognize connect, read, or redirect retry parameters.",
AppEnginePlatformWarning)
return retries
def is_appengine():
return (is_local_appengine() or
is_prod_appengine() or
is_prod_appengine_mvms())
def is_appengine_sandbox():
return is_appengine() and not is_prod_appengine_mvms()
def is_local_appengine():
return ('APPENGINE_RUNTIME' in os.environ and
'Development/' in os.environ['SERVER_SOFTWARE'])
def is_prod_appengine():
return ('APPENGINE_RUNTIME' in os.environ and
'Google App Engine/' in os.environ['SERVER_SOFTWARE'] and
not is_prod_appengine_mvms())
def is_prod_appengine_mvms():
return os.environ.get('GAE_VM', False) == 'true'
requests/packages/urllib3/contrib/ntlmpool.py
+1 -6
View File
@@ -1,14 +1,9 @@
# urllib3/contrib/ntlmpool.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
"""
NTLM authenticating pool, contributed by erikcederstran
Issue #10, see: http://code.google.com/p/urllib3/issues/detail?id=10
"""
from __future__ import absolute_import
try:
from http.client import HTTPSConnection
requests/packages/urllib3/contrib/pyopenssl.py
+153 -189
View File
@@ -1,4 +1,7 @@
'''SSL with SNI-support for Python 2.
'''SSL with SNI_-support for Python 2. Follow these instructions if you would
like to verify SSL certificates in Python 2. Note, the default libraries do
*not* do certificate checking; you need to do additional work to validate
certificates yourself.
This needs the following packages installed:
@@ -6,9 +9,15 @@ This needs the following packages installed:
* ndg-httpsclient (tested with 0.3.2)
* pyasn1 (tested with 0.1.6)
To activate it call :func:`~urllib3.contrib.pyopenssl.inject_into_urllib3`.
This can be done in a ``sitecustomize`` module, or at any other time before
your application begins using ``urllib3``, like this::
You can install them with the following command:
pip install pyopenssl ndg-httpsclient pyasn1
To activate certificate checking, call
:func:`~urllib3.contrib.pyopenssl.inject_into_urllib3` from your Python code
before you begin making HTTP requests. This can be done in a ``sitecustomize``
module, or at any other time before your application begins using ``urllib3``,
like this::
try:
import urllib3.contrib.pyopenssl
@@ -18,16 +27,36 @@ your application begins using ``urllib3``, like this::
Now you can use :mod:`urllib3` as you normally would, and it will support SNI
when the required modules are installed.
'''
from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
from ndg.httpsclient.subj_alt_name import SubjectAltName
Activating this module also has the positive side effect of disabling SSL/TLS
compression in Python 2 (see `CRIME attack`_).
If you want to configure the default list of supported cipher suites, you can
set the ``urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST`` variable.
Module Variables
----------------
:var DEFAULT_SSL_CIPHER_LIST: The list of supported SSL/TLS cipher suites.
.. _sni: https://en.wikipedia.org/wiki/Server_Name_Indication
.. _crime attack: https://en.wikipedia.org/wiki/CRIME_(security_exploit)
'''
from __future__ import absolute_import
try:
from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
from ndg.httpsclient.subj_alt_name import SubjectAltName as BaseSubjectAltName
except SyntaxError as e:
raise ImportError(e)
import OpenSSL.SSL
from pyasn1.codec.der import decoder as der_decoder
from socket import _fileobject
from pyasn1.type import univ, constraint
from socket import _fileobject, timeout, error as SocketError
import ssl
import select
from cStringIO import StringIO
from .. import connection
from .. import util
@@ -40,16 +69,31 @@ HAS_SNI = SUBJ_ALT_NAME_SUPPORT
# Map from urllib3 to PyOpenSSL compatible parameter-values.
_openssl_versions = {
ssl.PROTOCOL_SSLv23: OpenSSL.SSL.SSLv23_METHOD,
ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
ssl.PROTOCOL_TLSv1: OpenSSL.SSL.TLSv1_METHOD,
}
if hasattr(ssl, 'PROTOCOL_TLSv1_1') and hasattr(OpenSSL.SSL, 'TLSv1_1_METHOD'):
_openssl_versions[ssl.PROTOCOL_TLSv1_1] = OpenSSL.SSL.TLSv1_1_METHOD
if hasattr(ssl, 'PROTOCOL_TLSv1_2') and hasattr(OpenSSL.SSL, 'TLSv1_2_METHOD'):
_openssl_versions[ssl.PROTOCOL_TLSv1_2] = OpenSSL.SSL.TLSv1_2_METHOD
try:
_openssl_versions.update({ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD})
except AttributeError:
pass
_openssl_verify = {
ssl.CERT_NONE: OpenSSL.SSL.VERIFY_NONE,
ssl.CERT_OPTIONAL: OpenSSL.SSL.VERIFY_PEER,
ssl.CERT_REQUIRED: OpenSSL.SSL.VERIFY_PEER
+ OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
ssl.CERT_REQUIRED:
OpenSSL.SSL.VERIFY_PEER + OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
}
DEFAULT_SSL_CIPHER_LIST = util.ssl_.DEFAULT_CIPHERS
# OpenSSL will only write 16K at a time
SSL_WRITE_BLOCKSIZE = 16384
orig_util_HAS_SNI = util.HAS_SNI
orig_connection_ssl_wrap_socket = connection.ssl_wrap_socket
@@ -69,7 +113,18 @@ def extract_from_urllib3():
util.HAS_SNI = orig_util_HAS_SNI
### Note: This is a slightly bug-fixed version of same from ndg-httpsclient.
# Note: This is a slightly bug-fixed version of same from ndg-httpsclient.
class SubjectAltName(BaseSubjectAltName):
'''ASN.1 implementation for subjectAltNames support'''
# There is no limit to how many SAN certificates a certificate may have,
# however this needs to have some limit so we'll set an arbitrarily high
# limit.
sizeSpec = univ.SequenceOf.sizeSpec + \
constraint.ValueSizeConstraint(1, 1024)
# Note: This is a slightly bug-fixed version of same from ndg-httpsclient.
def get_subj_alt_name(peer_cert):
# Search through extensions
dns_name = []
@@ -100,193 +155,81 @@ def get_subj_alt_name(peer_cert):
return dns_name
class fileobject(_fileobject):
def read(self, size=-1):
# Use max, disallow tiny reads in a loop as they are very inefficient.
# We never leave read() with any leftover data from a new recv() call
# in our internal buffer.
rbufsize = max(self._rbufsize, self.default_bufsize)
# Our use of StringIO rather than lists of string objects returned by
# recv() minimizes memory usage and fragmentation that occurs when
# rbufsize is large compared to the typical return value of recv().
buf = self._rbuf
buf.seek(0, 2) # seek end
if size < 0:
# Read until EOF
self._rbuf = StringIO() # reset _rbuf. we consume it via buf.
while True:
try:
data = self._sock.recv(rbufsize)
except OpenSSL.SSL.WantReadError:
continue
if not data:
break
buf.write(data)
return buf.getvalue()
else:
# Read until size bytes or EOF seen, whichever comes first
buf_len = buf.tell()
if buf_len >= size:
# Already have size bytes in our buffer? Extract and return.
buf.seek(0)
rv = buf.read(size)
self._rbuf = StringIO()
self._rbuf.write(buf.read())
return rv
self._rbuf = StringIO() # reset _rbuf. we consume it via buf.
while True:
left = size - buf_len
# recv() will malloc the amount of memory given as its
# parameter even though it often returns much less data
# than that. The returned data string is short lived
# as we copy it into a StringIO and free it. This avoids
# fragmentation issues on many platforms.
try:
data = self._sock.recv(left)
except OpenSSL.SSL.WantReadError:
continue
if not data:
break
n = len(data)
if n == size and not buf_len:
# Shortcut. Avoid buffer data copies when:
# - We have no data in our buffer.
# AND
# - Our call to recv returned exactly the
# number of bytes we were asked to read.
return data
if n == left:
buf.write(data)
del data # explicit free
break
assert n <= left, "recv(%d) returned %d bytes" % (left, n)
buf.write(data)
buf_len += n
del data # explicit free
#assert buf_len == buf.tell()
return buf.getvalue()
def readline(self, size=-1):
buf = self._rbuf
buf.seek(0, 2) # seek end
if buf.tell() > 0:
# check if we already have it in our buffer
buf.seek(0)
bline = buf.readline(size)
if bline.endswith('\n') or len(bline) == size:
self._rbuf = StringIO()
self._rbuf.write(buf.read())
return bline
del bline
if size < 0:
# Read until \n or EOF, whichever comes first
if self._rbufsize <= 1:
# Speed up unbuffered case
buf.seek(0)
buffers = [buf.read()]
self._rbuf = StringIO() # reset _rbuf. we consume it via buf.
data = None
recv = self._sock.recv
while True:
try:
while data != "\n":
data = recv(1)
if not data:
break
buffers.append(data)
except OpenSSL.SSL.WantReadError:
continue
break
return "".join(buffers)
buf.seek(0, 2) # seek end
self._rbuf = StringIO() # reset _rbuf. we consume it via buf.
while True:
try:
data = self._sock.recv(self._rbufsize)
except OpenSSL.SSL.WantReadError:
continue
if not data:
break
nl = data.find('\n')
if nl >= 0:
nl += 1
buf.write(data[:nl])
self._rbuf.write(data[nl:])
del data
break
buf.write(data)
return buf.getvalue()
else:
# Read until size bytes or \n or EOF seen, whichever comes first
buf.seek(0, 2) # seek end
buf_len = buf.tell()
if buf_len >= size:
buf.seek(0)
rv = buf.read(size)
self._rbuf = StringIO()
self._rbuf.write(buf.read())
return rv
self._rbuf = StringIO() # reset _rbuf. we consume it via buf.
while True:
try:
data = self._sock.recv(self._rbufsize)
except OpenSSL.SSL.WantReadError:
continue
if not data:
break
left = size - buf_len
# did we just receive a newline?
nl = data.find('\n', 0, left)
if nl >= 0:
nl += 1
# save the excess data to _rbuf
self._rbuf.write(data[nl:])
if buf_len:
buf.write(data[:nl])
break
else:
# Shortcut. Avoid data copy through buf when returning
# a substring of our first recv().
return data[:nl]
n = len(data)
if n == size and not buf_len:
# Shortcut. Avoid data copy through buf when
# returning exactly all of our first recv().
return data
if n >= left:
buf.write(data[:left])
self._rbuf.write(data[left:])
break
buf.write(data)
buf_len += n
#assert buf_len == buf.tell()
return buf.getvalue()
class WrappedSocket(object):
'''API-compatibility wrapper for Python OpenSSL's Connection-class.'''
'''API-compatibility wrapper for Python OpenSSL's Connection-class.
def __init__(self, connection, socket):
Note: _makefile_refs, _drop() and _reuse() are needed for the garbage
collector of pypy.
'''
def __init__(self, connection, socket, suppress_ragged_eofs=True):
self.connection = connection
self.socket = socket
self.suppress_ragged_eofs = suppress_ragged_eofs
self._makefile_refs = 0
def fileno(self):
return self.socket.fileno()
def makefile(self, mode, bufsize=-1):
return fileobject(self.connection, mode, bufsize)
self._makefile_refs += 1
return _fileobject(self, mode, bufsize, close=True)
def recv(self, *args, **kwargs):
try:
data = self.connection.recv(*args, **kwargs)
except OpenSSL.SSL.SysCallError as e:
if self.suppress_ragged_eofs and e.args == (-1, 'Unexpected EOF'):
return b''
else:
raise SocketError(e)
except OpenSSL.SSL.ZeroReturnError as e:
if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
return b''
else:
raise
except OpenSSL.SSL.WantReadError:
rd, wd, ed = select.select(
[self.socket], [], [], self.socket.gettimeout())
if not rd:
raise timeout('The read operation timed out')
else:
return self.recv(*args, **kwargs)
else:
return data
def settimeout(self, timeout):
return self.socket.settimeout(timeout)
def _send_until_done(self, data):
while True:
try:
return self.connection.send(data)
except OpenSSL.SSL.WantWriteError:
_, wlist, _ = select.select([], [self.socket], [],
self.socket.gettimeout())
if not wlist:
raise timeout()
continue
def sendall(self, data):
return self.connection.sendall(data)
total_sent = 0
while total_sent < len(data):
sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
total_sent += sent
def shutdown(self):
# FIXME rethrow compatible exceptions should we ever use this
self.connection.shutdown()
def close(self):
return self.connection.shutdown()
if self._makefile_refs < 1:
try:
return self.connection.close()
except OpenSSL.SSL.Error:
return
else:
self._makefile_refs -= 1
def getpeercert(self, binary_form=False):
x509 = self.connection.get_peer_certificate()
@@ -309,6 +252,15 @@ class WrappedSocket(object):
]
}
def _reuse(self):
self._makefile_refs += 1
def _drop(self):
if self._makefile_refs < 1:
self.close()
else:
self._makefile_refs -= 1
def _verify_callback(cnx, x509, err_no, err_depth, return_code):
return err_no == 0
@@ -316,19 +268,29 @@ def _verify_callback(cnx, x509, err_no, err_depth, return_code):
def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
ca_certs=None, server_hostname=None,
ssl_version=None):
ssl_version=None, ca_cert_dir=None):
ctx = OpenSSL.SSL.Context(_openssl_versions[ssl_version])
if certfile:
keyfile = keyfile or certfile # Match behaviour of the normal python ssl library
ctx.use_certificate_file(certfile)
if keyfile:
ctx.use_privatekey_file(keyfile)
if cert_reqs != ssl.CERT_NONE:
ctx.set_verify(_openssl_verify[cert_reqs], _verify_callback)
if ca_certs:
if ca_certs or ca_cert_dir:
try:
ctx.load_verify_locations(ca_certs, None)
ctx.load_verify_locations(ca_certs, ca_cert_dir)
except OpenSSL.SSL.Error as e:
raise ssl.SSLError('bad ca_certs: %r' % ca_certs, e)
else:
ctx.set_default_verify_paths()
# Disable TLS compression to migitate CRIME attack (issue #309)
OP_NO_COMPRESSION = 0x20000
ctx.set_options(OP_NO_COMPRESSION)
# Set list of supported ciphersuites.
ctx.set_cipher_list(DEFAULT_SSL_CIPHER_LIST)
cnx = OpenSSL.SSL.Connection(ctx, sock)
cnx.set_tlsext_host_name(server_hostname)
@@ -337,10 +299,12 @@ def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
try:
cnx.do_handshake()
except OpenSSL.SSL.WantReadError:
select.select([sock], [], [])
rd, _, _ = select.select([sock], [], [], sock.gettimeout())
if not rd:
raise timeout('select timed out')
continue
except OpenSSL.SSL.Error as e:
raise ssl.SSLError('bad handshake', e)
raise ssl.SSLError('bad handshake: %r' % e)
break
return WrappedSocket(cnx, sock)
requests/packages/urllib3/exceptions.py
+95 -15
View File
@@ -1,17 +1,17 @@
# urllib3/exceptions.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from __future__ import absolute_import
# Base Exceptions
## Base Exceptions
class HTTPError(Exception):
"Base exception used by this module."
pass
class HTTPWarning(Warning):
"Base warning used by this module."
pass
class PoolError(HTTPError):
"Base exception for errors caused within a pool."
def __init__(self, pool, message):
@@ -49,19 +49,32 @@ class DecodeError(HTTPError):
pass
## Leaf Exceptions
class ProtocolError(HTTPError):
"Raised when something unexpected happens mid-request/response."
pass
#: Renamed to ProtocolError but aliased for backwards compatibility.
ConnectionError = ProtocolError
# Leaf Exceptions
class MaxRetryError(RequestError):
"Raised when the maximum number of retries is exceeded."
"""Raised when the maximum number of retries is exceeded.
:param pool: The connection pool
:type pool: :class:`~urllib3.connectionpool.HTTPConnectionPool`
:param string url: The requested Url
:param exceptions.Exception reason: The underlying error
"""
def __init__(self, pool, url, reason=None):
self.reason = reason
message = "Max retries exceeded with url: %s" % url
if reason:
message += " (Caused by %s: %s)" % (type(reason), reason)
else:
message += " (Caused by redirect)"
message = "Max retries exceeded with url: %s (Caused by %r)" % (
url, reason)
RequestError.__init__(self, pool, url, message)
@@ -101,6 +114,11 @@ class ConnectTimeoutError(TimeoutError):
pass
class NewConnectionError(ConnectTimeoutError, PoolError):
"Raised when we fail to establish a new connection. Usually ECONNREFUSED."
pass
class EmptyPoolError(PoolError):
"Raised when a pool runs out of connections and no more are allowed."
pass
@@ -111,7 +129,12 @@ class ClosedPoolError(PoolError):
pass
class LocationParseError(ValueError, HTTPError):
class LocationValueError(ValueError, HTTPError):
"Raised when there is something wrong with a given URL input."
pass
class LocationParseError(LocationValueError):
"Raised when get_host or similar fails to parse the URL input."
def __init__(self, location):
@@ -119,3 +142,60 @@ class LocationParseError(ValueError, HTTPError):
HTTPError.__init__(self, message)
self.location = location
class ResponseError(HTTPError):
"Used as a container for an error reason supplied in a MaxRetryError."
GENERIC_ERROR = 'too many error responses'
SPECIFIC_ERROR = 'too many {status_code} error responses'
class SecurityWarning(HTTPWarning):
"Warned when perfoming security reducing actions"
pass
class SubjectAltNameWarning(SecurityWarning):
"Warned when connecting to a host with a certificate missing a SAN."
pass
class InsecureRequestWarning(SecurityWarning):
"Warned when making an unverified HTTPS request."
pass
class SystemTimeWarning(SecurityWarning):
"Warned when system time is suspected to be wrong"
pass
class InsecurePlatformWarning(SecurityWarning):
"Warned when certain SSL configuration is not available on a platform."
pass
class SNIMissingWarning(HTTPWarning):
"Warned when making a HTTPS request without SNI available."
pass
class ResponseNotChunked(ProtocolError, ValueError):
"Response needs to be chunked in order to read it as chunks."
pass
class ProxySchemeUnknown(AssertionError, ValueError):
"ProxyManager does not support the supplied scheme"
# TODO(t-8ch): Stop inheriting from AssertionError in v2.0.
def __init__(self, scheme):
message = "Not supported proxy scheme %s" % scheme
super(ProxySchemeUnknown, self).__init__(message)
class HeaderParsingError(HTTPError):
"Raised by assert_header_parsing, but we convert it to a log.warning statement."
def __init__(self, defects, unparsed_data):
message = '%s, unparsed data: %r' % (defects or 'Unknown', unparsed_data)
super(HeaderParsingError, self).__init__(message)
requests/packages/urllib3/fields.py
+15 -14
View File
@@ -1,9 +1,4 @@
# urllib3/fields.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from __future__ import absolute_import
import email.utils
import mimetypes
@@ -15,7 +10,7 @@ def guess_content_type(filename, default='application/octet-stream'):
Guess the "Content-Type" of a file.
:param filename:
The filename to guess the "Content-Type" of using :mod:`mimetimes`.
The filename to guess the "Content-Type" of using :mod:`mimetypes`.
:param default:
If no "Content-Type" can be guessed, default to `default`.
"""
@@ -78,9 +73,10 @@ class RequestField(object):
"""
A :class:`~urllib3.fields.RequestField` factory from old-style tuple parameters.
Supports constructing :class:`~urllib3.fields.RequestField` from parameter
of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type)
tuple where the MIME type is optional. For example: ::
Supports constructing :class:`~urllib3.fields.RequestField` from
parameter of key/value strings AND key/filetuple. A filetuple is a
(filename, data, MIME type) tuple where the MIME type is optional.
For example::
'foo': 'bar',
'fakefile': ('foofile.txt', 'contents of foofile'),
@@ -125,8 +121,8 @@ class RequestField(object):
'Content-Disposition' fields.
:param header_parts:
A sequence of (k, v) typles or a :class:`dict` of (k, v) to format as
`k1="v1"; k2="v2"; ...`.
A sequence of (k, v) typles or a :class:`dict` of (k, v) to format
as `k1="v1"; k2="v2"; ...`.
"""
parts = []
iterable = header_parts
@@ -158,7 +154,8 @@ class RequestField(object):
lines.append('\r\n')
return '\r\n'.join(lines)
def make_multipart(self, content_disposition=None, content_type=None, content_location=None):
def make_multipart(self, content_disposition=None, content_type=None,
content_location=None):
"""
Makes this request field into a multipart request field.
@@ -172,6 +169,10 @@ class RequestField(object):
"""
self.headers['Content-Disposition'] = content_disposition or 'form-data'
self.headers['Content-Disposition'] += '; '.join(['', self._render_parts((('name', self._name), ('filename', self._filename)))])
self.headers['Content-Disposition'] += '; '.join([
'', self._render_parts(
(('name', self._name), ('filename', self._filename))
)
])
self.headers['Content-Type'] = content_type
self.headers['Content-Location'] = content_location
requests/packages/urllib3/filepost.py
+10 -17
View File
@@ -1,11 +1,5 @@
# urllib3/filepost.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from __future__ import absolute_import
import codecs
import mimetypes
from uuid import uuid4
from io import BytesIO
@@ -38,24 +32,23 @@ def iter_field_objects(fields):
i = iter(fields)
for field in i:
if isinstance(field, RequestField):
yield field
else:
yield RequestField.from_tuples(*field)
if isinstance(field, RequestField):
yield field
else:
yield RequestField.from_tuples(*field)
def iter_fields(fields):
"""
.. deprecated:: 1.6
Iterate over fields.
.. deprecated ::
The addition of `~urllib3.fields.RequestField` makes this function
obsolete. Instead, use :func:`iter_field_objects`, which returns
`~urllib3.fields.RequestField` objects, instead.
The addition of :class:`~urllib3.fields.RequestField` makes this function
obsolete. Instead, use :func:`iter_field_objects`, which returns
:class:`~urllib3.fields.RequestField` objects.
Supports list of (k, v) tuples and dicts.
"""
if isinstance(fields, dict):
return ((k, v) for k, v in six.iteritems(fields))
requests/packages/urllib3/packages/__init__.py
+1
View File
@@ -2,3 +2,4 @@ from __future__ import absolute_import
from . import ssl_match_hostname
__all__ = ('ssl_match_hostname', )
requests/packages/urllib3/packages/ordered_dict.py
-1
View File
@@ -2,7 +2,6 @@
# Passes Python2.7's test suite and incorporates all the latest updates.
# Copyright 2009 Raymond Hettinger, released under the MIT License.
# http://code.activestate.com/recipes/576693/
try:
from thread import get_ident as _get_ident
except ImportError:
requests/packages/urllib3/packages/ssl_match_hostname/.gitignore
+1
View File
@@ -0,0 +1 @@
env
requests/packages/urllib3/packages/ssl_match_hostname/__init__.py
+1 -1
View File
@@ -7,7 +7,7 @@ except ImportError:
from backports.ssl_match_hostname import CertificateError, match_hostname
except ImportError:
# Our vendored copy
from _implementation import CertificateError, match_hostname
from ._implementation import CertificateError, match_hostname
# Not needed, but documenting what we provide.
__all__ = ('CertificateError', 'match_hostname')
requests/packages/urllib3/poolmanager.py
+43 -20
View File
@@ -1,9 +1,4 @@
# urllib3/poolmanager.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from __future__ import absolute_import
import logging
try: # Python 3
@@ -14,8 +9,10 @@ except ImportError:
from ._collections import RecentlyUsedContainer
from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool
from .connectionpool import port_by_scheme
from .exceptions import LocationValueError, MaxRetryError, ProxySchemeUnknown
from .request import RequestMethods
from .util import parse_url
from .util.url import parse_url
from .util.retry import Retry
__all__ = ['PoolManager', 'ProxyManager', 'proxy_from_url']
@@ -29,7 +26,7 @@ pool_classes_by_scheme = {
log = logging.getLogger(__name__)
SSL_KEYWORDS = ('key_file', 'cert_file', 'cert_reqs', 'ca_certs',
'ssl_version')
'ssl_version', 'ca_cert_dir')
class PoolManager(RequestMethods):
@@ -49,7 +46,7 @@ class PoolManager(RequestMethods):
Additional parameters are used to create fresh
:class:`urllib3.connectionpool.ConnectionPool` instances.
Example: ::
Example::
>>> manager = PoolManager(num_pools=2)
>>> r = manager.request('GET', 'http://google.com/')
@@ -68,6 +65,14 @@ class PoolManager(RequestMethods):
self.pools = RecentlyUsedContainer(num_pools,
dispose_func=lambda p: p.close())
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.clear()
# Return False to re-raise any potential exceptions
return False
def _new_pool(self, scheme, host, port):
"""
Create a new :class:`ConnectionPool` based on host, port and scheme.
@@ -102,10 +107,11 @@ class PoolManager(RequestMethods):
``urllib3.connectionpool.port_by_scheme``.
"""
if not host:
raise LocationValueError("No host specified.")
scheme = scheme or 'http'
port = port or port_by_scheme.get(scheme, 80)
pool_key = (scheme, host, port)
with self.pools.lock:
@@ -118,6 +124,7 @@ class PoolManager(RequestMethods):
# Make a fresh ConnectionPool of the desired type
pool = self._new_pool(scheme, host, port)
self.pools[pool_key] = pool
return pool
def connection_from_url(self, url):
@@ -161,13 +168,25 @@ class PoolManager(RequestMethods):
# Support relative URLs for redirecting.
redirect_location = urljoin(url, redirect_location)
# RFC 2616, Section 10.3.4
# RFC 7231, Section 6.4.4
if response.status == 303:
method = 'GET'
log.info("Redirecting %s -> %s" % (url, redirect_location))
kw['retries'] = kw.get('retries', 3) - 1 # Persist retries countdown
retries = kw.get('retries')
if not isinstance(retries, Retry):
retries = Retry.from_int(retries, redirect=redirect)
try:
retries = retries.increment(method, url, response=response, _pool=conn)
except MaxRetryError:
if retries.raise_on_redirect:
raise
return response
kw['retries'] = retries
kw['redirect'] = redirect
log.info("Redirecting %s -> %s" % (url, redirect_location))
return self.urlopen(method, redirect_location, **kw)
@@ -176,7 +195,7 @@ class ProxyManager(PoolManager):
Behaves just like :class:`PoolManager`, but sends all requests through
the defined proxy, using the CONNECT method for HTTPS URLs.
:param poxy_url:
:param proxy_url:
The URL of the proxy to be used.
:param proxy_headers:
@@ -208,12 +227,16 @@ class ProxyManager(PoolManager):
if not proxy.port:
port = port_by_scheme.get(proxy.scheme, 80)
proxy = proxy._replace(port=port)
if proxy.scheme not in ("http", "https"):
raise ProxySchemeUnknown(proxy.scheme)
self.proxy = proxy
self.proxy_headers = proxy_headers or {}
assert self.proxy.scheme in ("http", "https"), \
'Not supported proxy scheme %s' % self.proxy.scheme
connection_pool_kw['_proxy'] = self.proxy
connection_pool_kw['_proxy_headers'] = self.proxy_headers
super(ProxyManager, self).__init__(
num_pools, headers, **connection_pool_kw)
@@ -248,10 +271,10 @@ class ProxyManager(PoolManager):
# For proxied HTTPS requests, httplib sets the necessary headers
# on the CONNECT to the proxy. For HTTP, we'll definitely
# need to set 'Host' at the very least.
kw['headers'] = self._set_proxy_headers(url, kw.get('headers',
self.headers))
headers = kw.get('headers', self.headers)
kw['headers'] = self._set_proxy_headers(url, headers)
return super(ProxyManager, self).urlopen(method, url, redirect, **kw)
return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
def proxy_from_url(url, **kw):
requests/packages/urllib3/request.py
+45 -35
View File
@@ -1,9 +1,4 @@
# urllib3/request.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from __future__ import absolute_import
try:
from urllib.parse import urlencode
except ImportError:
@@ -26,8 +21,8 @@ class RequestMethods(object):
Specifically,
:meth:`.request_encode_url` is for sending requests whose fields are encoded
in the URL (such as GET, HEAD, DELETE).
:meth:`.request_encode_url` is for sending requests whose fields are
encoded in the URL (such as GET, HEAD, DELETE).
:meth:`.request_encode_body` is for sending requests whose fields are
encoded in the *body* of the request using multipart or www-form-urlencoded
@@ -51,7 +46,7 @@ class RequestMethods(object):
def urlopen(self, method, url, body=None, headers=None,
encode_multipart=True, multipart_boundary=None,
**kw): # Abstract
**kw): # Abstract
raise NotImplemented("Classes extending RequestMethods must implement "
"their own ``urlopen`` method.")
@@ -61,8 +56,8 @@ class RequestMethods(object):
``fields`` based on the ``method`` used.
This is a convenience method that requires the least amount of manual
effort. It can be used in most situations, while still having the option
to drop down to more specific methods when necessary, such as
effort. It can be used in most situations, while still having the
option to drop down to more specific methods when necessary, such as
:meth:`request_encode_url`, :meth:`request_encode_body`,
or even the lowest level :meth:`urlopen`.
"""
@@ -70,21 +65,29 @@ class RequestMethods(object):
if method in self._encode_url_methods:
return self.request_encode_url(method, url, fields=fields,
headers=headers,
**urlopen_kw)
headers=headers,
**urlopen_kw)
else:
return self.request_encode_body(method, url, fields=fields,
headers=headers,
**urlopen_kw)
headers=headers,
**urlopen_kw)
def request_encode_url(self, method, url, fields=None, **urlopen_kw):
def request_encode_url(self, method, url, fields=None, headers=None,
**urlopen_kw):
"""
Make a request using :meth:`urlopen` with the ``fields`` encoded in
the url. This is useful for request methods like GET, HEAD, DELETE, etc.
"""
if headers is None:
headers = self.headers
extra_kw = {'headers': headers}
extra_kw.update(urlopen_kw)
if fields:
url += '?' + urlencode(fields)
return self.urlopen(method, url, **urlopen_kw)
return self.urlopen(method, url, **extra_kw)
def request_encode_body(self, method, url, fields=None, headers=None,
encode_multipart=True, multipart_boundary=None,
@@ -94,18 +97,18 @@ class RequestMethods(object):
the body. This is useful for request methods like POST, PUT, PATCH, etc.
When ``encode_multipart=True`` (default), then
:meth:`urllib3.filepost.encode_multipart_formdata` is used to encode the
payload with the appropriate content type. Otherwise
:meth:`urllib3.filepost.encode_multipart_formdata` is used to encode
the payload with the appropriate content type. Otherwise
:meth:`urllib.urlencode` is used with the
'application/x-www-form-urlencoded' content type.
Multipart encoding must be used when posting files, and it's reasonably
safe to use it in other times too. However, it may break request signing,
such as with OAuth.
safe to use it in other times too. However, it may break request
signing, such as with OAuth.
Supports an optional ``fields`` parameter of key/value strings AND
key/filetuple. A filetuple is a (filename, data, MIME type) tuple where
the MIME type is optional. For example: ::
the MIME type is optional. For example::
fields = {
'foo': 'bar',
@@ -119,23 +122,30 @@ class RequestMethods(object):
When uploading a file, providing a filename (the first parameter of the
tuple) is optional but recommended to best mimick behavior of browsers.
Note that if ``headers`` are supplied, the 'Content-Type' header will be
overwritten because it depends on the dynamic random boundary string
Note that if ``headers`` are supplied, the 'Content-Type' header will
be overwritten because it depends on the dynamic random boundary string
which is used to compose the body of the request. The random boundary
string can be explicitly set with the ``multipart_boundary`` parameter.
"""
if encode_multipart:
body, content_type = encode_multipart_formdata(fields or {},
boundary=multipart_boundary)
else:
body, content_type = (urlencode(fields or {}),
'application/x-www-form-urlencoded')
if headers is None:
headers = self.headers
headers_ = {'Content-Type': content_type}
headers_.update(headers)
extra_kw = {'headers': {}}
return self.urlopen(method, url, body=body, headers=headers_,
**urlopen_kw)
if fields:
if 'body' in urlopen_kw:
raise TypeError(
"request got values for both 'fields' and 'body', can only specify one.")
if encode_multipart:
body, content_type = encode_multipart_formdata(fields, boundary=multipart_boundary)
else:
body, content_type = urlencode(fields), 'application/x-www-form-urlencoded'
extra_kw['body'] = body
extra_kw['headers'] = {'Content-Type': content_type}
extra_kw['headers'].update(headers)
extra_kw.update(urlopen_kw)
return self.urlopen(method, url, **extra_kw)
requests/packages/urllib3/response.py
+265 -63
View File
@@ -1,20 +1,18 @@
# urllib3/response.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
import logging
from __future__ import absolute_import
from contextlib import contextmanager
import zlib
import io
from socket import timeout as SocketTimeout
from socket import error as SocketError
from .exceptions import DecodeError
from .packages.six import string_types as basestring, binary_type
from .util import is_fp_closed
log = logging.getLogger(__name__)
from ._collections import HTTPHeaderDict
from .exceptions import (
ProtocolError, DecodeError, ReadTimeoutError, ResponseNotChunked
)
from .packages.six import string_types as basestring, binary_type, PY3
from .packages.six.moves import http_client as httplib
from .connection import HTTPException, BaseSSLError
from .util.response import is_fp_closed, is_response_to_head
class DeflateDecoder(object):
@@ -28,6 +26,9 @@ class DeflateDecoder(object):
return getattr(self._obj, name)
def decompress(self, data):
if not data:
return data
if not self._first_try:
return self._obj.decompress(data)
@@ -43,9 +44,23 @@ class DeflateDecoder(object):
self._data = None
class GzipDecoder(object):
def __init__(self):
self._obj = zlib.decompressobj(16 + zlib.MAX_WBITS)
def __getattr__(self, name):
return getattr(self._obj, name)
def decompress(self, data):
if not data:
return data
return self._obj.decompress(data)
def _get_decoder(mode):
if mode == 'gzip':
return zlib.decompressobj(16 + zlib.MAX_WBITS)
return GzipDecoder()
return DeflateDecoder()
@@ -55,7 +70,10 @@ class HTTPResponse(io.IOBase):
HTTP Response container.
Backwards-compatible to httplib's HTTPResponse but the response ``body`` is
loaded and decoded on-demand when the ``data`` property is accessed.
loaded and decoded on-demand when the ``data`` property is accessed. This
class is also compatible with the Python standard library's :mod:`io`
module, and can hence be treated as a readable object in the context of that
framework.
Extra parameters for behaviour not present in httplib.HTTPResponse:
@@ -79,7 +97,11 @@ class HTTPResponse(io.IOBase):
def __init__(self, body='', headers=None, status=0, version=0, reason=None,
strict=0, preload_content=True, decode_content=True,
original_response=None, pool=None, connection=None):
self.headers = headers or {}
if isinstance(headers, HTTPHeaderDict):
self.headers = headers
else:
self.headers = HTTPHeaderDict(headers)
self.status = status
self.version = version
self.reason = reason
@@ -87,17 +109,30 @@ class HTTPResponse(io.IOBase):
self.decode_content = decode_content
self._decoder = None
self._body = body if body and isinstance(body, basestring) else None
self._body = None
self._fp = None
self._original_response = original_response
self._fp_bytes_read = 0
if body and isinstance(body, (basestring, binary_type)):
self._body = body
self._pool = pool
self._connection = connection
if hasattr(body, 'read'):
self._fp = body
# Are we using the chunked-style of transfer encoding?
self.chunked = False
self.chunk_left = None
tr_enc = self.headers.get('transfer-encoding', '').lower()
# Don't incur the penalty of creating a list and then discarding it
encodings = (enc.strip() for enc in tr_enc.split(","))
if "chunked" in encodings:
self.chunked = True
# If requested, preload the body.
if preload_content and not self._body:
self._body = self.read(decode_content=decode_content)
@@ -138,6 +173,93 @@ class HTTPResponse(io.IOBase):
"""
return self._fp_bytes_read
def _init_decoder(self):
"""
Set-up the _decoder attribute if necessar.
"""
# Note: content-encoding value should be case-insensitive, per RFC 7230
# Section 3.2
content_encoding = self.headers.get('content-encoding', '').lower()
if self._decoder is None and content_encoding in self.CONTENT_DECODERS:
self._decoder = _get_decoder(content_encoding)
def _decode(self, data, decode_content, flush_decoder):
"""
Decode the data passed in and potentially flush the decoder.
"""
try:
if decode_content and self._decoder:
data = self._decoder.decompress(data)
except (IOError, zlib.error) as e:
content_encoding = self.headers.get('content-encoding', '').lower()
raise DecodeError(
"Received response with content-encoding: %s, but "
"failed to decode it." % content_encoding, e)
if flush_decoder and decode_content:
data += self._flush_decoder()
return data
def _flush_decoder(self):
"""
Flushes the decoder. Should only be called if the decoder is actually
being used.
"""
if self._decoder:
buf = self._decoder.decompress(b'')
return buf + self._decoder.flush()
return b''
@contextmanager
def _error_catcher(self):
"""
Catch low-level python exceptions, instead re-raising urllib3
variants, so that low-level exceptions are not leaked in the
high-level api.
On exit, release the connection back to the pool.
"""
try:
try:
yield
except SocketTimeout:
# FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
# there is yet no clean way to get at it from this context.
raise ReadTimeoutError(self._pool, None, 'Read timed out.')
except BaseSSLError as e:
# FIXME: Is there a better way to differentiate between SSLErrors?
if 'read operation timed out' not in str(e): # Defensive:
# This shouldn't happen but just in case we're missing an edge
# case, let's avoid swallowing SSL errors.
raise
raise ReadTimeoutError(self._pool, None, 'Read timed out.')
except (HTTPException, SocketError) as e:
# This includes IncompleteRead.
raise ProtocolError('Connection broken: %r' % e, e)
except Exception:
# The response may not be closed but we're not going to use it anymore
# so close it now to ensure that the connection is released back to the pool.
if self._original_response and not self._original_response.isclosed():
self._original_response.close()
# Closing the response may not actually be sufficient to close
# everything, so if we have a hold of the connection close that
# too.
if self._connection is not None:
self._connection.close()
raise
finally:
if self._original_response and self._original_response.isclosed():
self.release_conn()
def read(self, amt=None, decode_content=None, cache_content=False):
"""
Similar to :meth:`httplib.HTTPResponse.read`, but with two additional
@@ -159,12 +281,7 @@ class HTTPResponse(io.IOBase):
after having ``.read()`` the file object. (Overridden if ``amt`` is
set.)
"""
# Note: content-encoding value should be case-insensitive, per RFC 2616
# Section 3.5
content_encoding = self.headers.get('content-encoding', '').lower()
if self._decoder is None:
if content_encoding in self.CONTENT_DECODERS:
self._decoder = _get_decoder(content_encoding)
self._init_decoder()
if decode_content is None:
decode_content = self.decode_content
@@ -172,8 +289,9 @@ class HTTPResponse(io.IOBase):
return
flush_decoder = False
data = None
try:
with self._error_catcher():
if amt is None:
# cStringIO doesn't like amt=None
data = self._fp.read()
@@ -186,35 +304,21 @@ class HTTPResponse(io.IOBase):
#
# This is redundant to what httplib/http.client _should_
# already do. However, versions of python released before
# December 15, 2012 (http://bugs.python.org/issue16298) do not
# properly close the connection in all cases. There is no harm
# in redundantly calling close.
# December 15, 2012 (http://bugs.python.org/issue16298) do
# not properly close the connection in all cases. There is
# no harm in redundantly calling close.
self._fp.close()
flush_decoder = True
if data:
self._fp_bytes_read += len(data)
try:
if decode_content and self._decoder:
data = self._decoder.decompress(data)
except (IOError, zlib.error) as e:
raise DecodeError(
"Received response with content-encoding: %s, but "
"failed to decode it." % content_encoding,
e)
if flush_decoder and decode_content and self._decoder:
buf = self._decoder.decompress(binary_type())
data += buf + self._decoder.flush()
data = self._decode(data, decode_content, flush_decoder)
if cache_content:
self._body = data
return data
finally:
if self._original_response and self._original_response.isclosed():
self.release_conn()
return data
def stream(self, amt=2**16, decode_content=None):
"""
@@ -232,12 +336,15 @@ class HTTPResponse(io.IOBase):
If True, will attempt to decode the body based on the
'content-encoding' header.
"""
while not is_fp_closed(self._fp):
data = self.read(amt=amt, decode_content=decode_content)
if data:
yield data
if self.chunked:
for line in self.read_chunked(amt, decode_content=decode_content):
yield line
else:
while not is_fp_closed(self._fp):
data = self.read(amt=amt, decode_content=decode_content)
if data:
yield data
@classmethod
def from_httplib(ResponseCls, r, **response_kw):
@@ -248,22 +355,17 @@ class HTTPResponse(io.IOBase):
Remaining parameters are passed to the HTTPResponse constructor, along
with ``original_response=r``.
"""
headers = r.msg
# Normalize headers between different versions of Python
headers = {}
for k, v in r.getheaders():
# Python 3: Header keys are returned capitalised
k = k.lower()
has_value = headers.get(k)
if has_value: # Python 3: Repeating header keys are unmerged.
v = ', '.join([has_value, v])
headers[k] = v
if not isinstance(headers, HTTPHeaderDict):
if PY3: # Python 3
headers = HTTPHeaderDict(headers.items())
else: # Python 2
headers = HTTPHeaderDict.from_httplib(headers)
# HTTPResponse objects in Python 3 don't have a .strict attribute
strict = getattr(r, 'strict', 0)
return ResponseCls(body=r,
resp = ResponseCls(body=r,
headers=headers,
status=r.status,
version=r.version,
@@ -271,6 +373,7 @@ class HTTPResponse(io.IOBase):
strict=strict,
original_response=r,
**response_kw)
return resp
# Backwards-compatibility methods for httplib.HTTPResponse
def getheaders(self):
@@ -301,7 +404,7 @@ class HTTPResponse(io.IOBase):
elif hasattr(self._fp, "fileno"):
return self._fp.fileno()
else:
raise IOError("The file-like object this HTTPResponse is wrapped "
raise IOError("The file-like object this HTTPResponse is wrapped "
"around has no file descriptor")
def flush(self):
@@ -309,4 +412,103 @@ class HTTPResponse(io.IOBase):
return self._fp.flush()
def readable(self):
# This method is required for `io` module compatibility.
return True
def readinto(self, b):
# This method is required for `io` module compatibility.
temp = self.read(len(b))
if len(temp) == 0:
return 0
else:
b[:len(temp)] = temp
return len(temp)
def _update_chunk_length(self):
# First, we'll figure out length of a chunk and then
# we'll try to read it from socket.
if self.chunk_left is not None:
return
line = self._fp.fp.readline()
line = line.split(b';', 1)[0]
try:
self.chunk_left = int(line, 16)
except ValueError:
# Invalid chunked protocol response, abort.
self.close()
raise httplib.IncompleteRead(line)
def _handle_chunk(self, amt):
returned_chunk = None
if amt is None:
chunk = self._fp._safe_read(self.chunk_left)
returned_chunk = chunk
self._fp._safe_read(2) # Toss the CRLF at the end of the chunk.
self.chunk_left = None
elif amt < self.chunk_left:
value = self._fp._safe_read(amt)
self.chunk_left = self.chunk_left - amt
returned_chunk = value
elif amt == self.chunk_left:
value = self._fp._safe_read(amt)
self._fp._safe_read(2) # Toss the CRLF at the end of the chunk.
self.chunk_left = None
returned_chunk = value
else: # amt > self.chunk_left
returned_chunk = self._fp._safe_read(self.chunk_left)
self._fp._safe_read(2) # Toss the CRLF at the end of the chunk.
self.chunk_left = None
return returned_chunk
def read_chunked(self, amt=None, decode_content=None):
"""
Similar to :meth:`HTTPResponse.read`, but with an additional
parameter: ``decode_content``.
:param decode_content:
If True, will attempt to decode the body based on the
'content-encoding' header.
"""
self._init_decoder()
# FIXME: Rewrite this method and make it a class with a better structured logic.
if not self.chunked:
raise ResponseNotChunked(
"Response is not chunked. "
"Header 'transfer-encoding: chunked' is missing.")
# Don't bother reading the body of a HEAD request.
if self._original_response and is_response_to_head(self._original_response):
self._original_response.close()
return
with self._error_catcher():
while True:
self._update_chunk_length()
if self.chunk_left == 0:
break
chunk = self._handle_chunk(amt)
decoded = self._decode(chunk, decode_content=decode_content,
flush_decoder=False)
if decoded:
yield decoded
if decode_content:
# On CPython and PyPy, we should never need to flush the
# decoder. However, on Jython we *might* need to, so
# lets defensively do it anyway.
decoded = self._flush_decoder()
if decoded: # Platform-specific: Jython.
yield decoded
# Chunk content ends with \r\n: discard it.
while True:
line = self._fp.fp.readline()
if not line:
# Some sites may not end with '\r\n'.
break
if line == b'\r\n':
break
# We read everything; close the "file".
if self._original_response:
self._original_response.close()
requests/packages/urllib3/util.py
-643
View File
@@ -1,643 +0,0 @@
# urllib3/util.py
# Copyright 2008-2013 Andrey Petrov and contributors (see CONTRIBUTORS.txt)
#
# This module is part of urllib3 and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
from base64 import b64encode
from binascii import hexlify, unhexlify
from collections import namedtuple
from hashlib import md5, sha1
from socket import error as SocketError, _GLOBAL_DEFAULT_TIMEOUT
import time
try:
from select import poll, POLLIN
except ImportError: # `poll` doesn't exist on OSX and other platforms
poll = False
try:
from select import select
except ImportError: # `select` doesn't exist on AppEngine.
select = False
try: # Test for SSL features
SSLContext = None
HAS_SNI = False
import ssl
from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23
from ssl import SSLContext # Modern SSL?
from ssl import HAS_SNI # Has SNI?
except ImportError:
pass
from .packages import six
from .exceptions import LocationParseError, SSLError, TimeoutStateError
_Default = object()
# The default timeout to use for socket connections. This is the attribute used
# by httplib to define the default timeout
def current_time():
"""
Retrieve the current time, this function is mocked out in unit testing.
"""
return time.time()
class Timeout(object):
"""
Utility object for storing timeout values.
Example usage:
.. code-block:: python
timeout = urllib3.util.Timeout(connect=2.0, read=7.0)
pool = HTTPConnectionPool('www.google.com', 80, timeout=timeout)
pool.request(...) # Etc, etc
:param connect:
The maximum amount of time to wait for a connection attempt to a server
to succeed. Omitting the parameter will default the connect timeout to
the system default, probably `the global default timeout in socket.py
<http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
None will set an infinite timeout for connection attempts.
:type connect: integer, float, or None
:param read:
The maximum amount of time to wait between consecutive
read operations for a response from the server. Omitting
the parameter will default the read timeout to the system
default, probably `the global default timeout in socket.py
<http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
None will set an infinite timeout.
:type read: integer, float, or None
:param total:
This combines the connect and read timeouts into one; the read timeout
will be set to the time leftover from the connect attempt. In the
event that both a connect timeout and a total are specified, or a read
timeout and a total are specified, the shorter timeout will be applied.
Defaults to None.
:type total: integer, float, or None
.. note::
Many factors can affect the total amount of time for urllib3 to return
an HTTP response. Specifically, Python's DNS resolver does not obey the
timeout specified on the socket. Other factors that can affect total
request time include high CPU load, high swap, the program running at a
low priority level, or other behaviors. The observed running time for
urllib3 to return a response may be greater than the value passed to
`total`.
In addition, the read and total timeouts only measure the time between
read operations on the socket connecting the client and the server,
not the total amount of time for the request to return a complete
response. For most requests, the timeout is raised because the server
has not sent the first byte in the specified time. This is not always
the case; if a server streams one byte every fifteen seconds, a timeout
of 20 seconds will not ever trigger, even though the request will
take several minutes to complete.
If your goal is to cut off any request after a set amount of wall clock
time, consider having a second "watcher" thread to cut off a slow
request.
"""
#: A sentinel object representing the default timeout value
DEFAULT_TIMEOUT = _GLOBAL_DEFAULT_TIMEOUT
def __init__(self, total=None, connect=_Default, read=_Default):
self._connect = self._validate_timeout(connect, 'connect')
self._read = self._validate_timeout(read, 'read')
self.total = self._validate_timeout(total, 'total')
self._start_connect = None
def __str__(self):
return '%s(connect=%r, read=%r, total=%r)' % (
type(self).__name__, self._connect, self._read, self.total)
@classmethod
def _validate_timeout(cls, value, name):
""" Check that a timeout attribute is valid
:param value: The timeout value to validate
:param name: The name of the timeout attribute to validate. This is used
for clear error messages
:return: the value
:raises ValueError: if the type is not an integer or a float, or if it
is a numeric value less than zero
"""
if value is _Default:
return cls.DEFAULT_TIMEOUT
if value is None or value is cls.DEFAULT_TIMEOUT:
return value
try:
float(value)
except (TypeError, ValueError):
raise ValueError("Timeout value %s was %s, but it must be an "
"int or float." % (name, value))
try:
if value < 0:
raise ValueError("Attempted to set %s timeout to %s, but the "
"timeout cannot be set to a value less "
"than 0." % (name, value))
except TypeError: # Python 3
raise ValueError("Timeout value %s was %s, but it must be an "
"int or float." % (name, value))
return value
@classmethod
def from_float(cls, timeout):
""" Create a new Timeout from a legacy timeout value.
The timeout value used by httplib.py sets the same timeout on the
connect(), and recv() socket requests. This creates a :class:`Timeout`
object that sets the individual timeouts to the ``timeout`` value passed
to this function.
:param timeout: The legacy timeout value
:type timeout: integer, float, sentinel default object, or None
:return: a Timeout object
:rtype: :class:`Timeout`
"""
return Timeout(read=timeout, connect=timeout)
def clone(self):
""" Create a copy of the timeout object
Timeout properties are stored per-pool but each request needs a fresh
Timeout object to ensure each one has its own start/stop configured.
:return: a copy of the timeout object
:rtype: :class:`Timeout`
"""
# We can't use copy.deepcopy because that will also create a new object
# for _GLOBAL_DEFAULT_TIMEOUT, which socket.py uses as a sentinel to
# detect the user default.
return Timeout(connect=self._connect, read=self._read,
total=self.total)
def start_connect(self):
""" Start the timeout clock, used during a connect() attempt
:raises urllib3.exceptions.TimeoutStateError: if you attempt
to start a timer that has been started already.
"""
if self._start_connect is not None:
raise TimeoutStateError("Timeout timer has already been started.")
self._start_connect = current_time()
return self._start_connect
def get_connect_duration(self):
""" Gets the time elapsed since the call to :meth:`start_connect`.
:return: the elapsed time
:rtype: float
:raises urllib3.exceptions.TimeoutStateError: if you attempt
to get duration for a timer that hasn't been started.
"""
if self._start_connect is None:
raise TimeoutStateError("Can't get connect duration for timer "
"that has not started.")
return current_time() - self._start_connect
@property
def connect_timeout(self):
""" Get the value to use when setting a connection timeout.
This will be a positive float or integer, the value None
(never timeout), or the default system timeout.
:return: the connect timeout
:rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None
"""
if self.total is None:
return self._connect
if self._connect is None or self._connect is self.DEFAULT_TIMEOUT:
return self.total
return min(self._connect, self.total)
@property
def read_timeout(self):
""" Get the value for the read timeout.
This assumes some time has elapsed in the connection timeout and
computes the read timeout appropriately.
If self.total is set, the read timeout is dependent on the amount of
time taken by the connect timeout. If the connection time has not been
established, a :exc:`~urllib3.exceptions.TimeoutStateError` will be
raised.
:return: the value to use for the read timeout
:rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None
:raises urllib3.exceptions.TimeoutStateError: If :meth:`start_connect`
has not yet been called on this object.
"""
if (self.total is not None and
self.total is not self.DEFAULT_TIMEOUT and
self._read is not None and
self._read is not self.DEFAULT_TIMEOUT):
# in case the connect timeout has not yet been established.
if self._start_connect is None:
return self._read
return max(0, min(self.total - self.get_connect_duration(),
self._read))
elif self.total is not None and self.total is not self.DEFAULT_TIMEOUT:
return max(0, self.total - self.get_connect_duration())
else:
return self._read
class Url(namedtuple('Url', ['scheme', 'auth', 'host', 'port', 'path', 'query', 'fragment'])):
"""
Datastructure for representing an HTTP URL. Used as a return value for
:func:`parse_url`.
"""
slots = ()
def __new__(cls, scheme=None, auth=None, host=None, port=None, path=None, query=None, fragment=None):
return super(Url, cls).__new__(cls, scheme, auth, host, port, path, query, fragment)
@property
def hostname(self):
"""For backwards-compatibility with urlparse. We're nice like that."""
return self.host
@property
def request_uri(self):
"""Absolute path including the query string."""
uri = self.path or '/'
if self.query is not None:
uri += '?' + self.query
return uri
@property
def netloc(self):
"""Network location including host and port"""
if self.port:
return '%s:%d' % (self.host, self.port)
return self.host
def split_first(s, delims):
"""
Given a string and an iterable of delimiters, split on the first found
delimiter. Return two split parts and the matched delimiter.
If not found, then the first part is the full input string.
Example: ::
>>> split_first('foo/bar?baz', '?/=')
('foo', 'bar?baz', '/')
>>> split_first('foo/bar?baz', '123')
('foo/bar?baz', '', None)
Scales linearly with number of delims. Not ideal for large number of delims.
"""
min_idx = None
min_delim = None
for d in delims:
idx = s.find(d)
if idx < 0:
continue
if min_idx is None or idx < min_idx:
min_idx = idx
min_delim = d
if min_idx is None or min_idx < 0:
return s, '', None
return s[:min_idx], s[min_idx+1:], min_delim
def parse_url(url):
"""
Given a url, return a parsed :class:`.Url` namedtuple. Best-effort is
performed to parse incomplete urls. Fields not provided will be None.
Partly backwards-compatible with :mod:`urlparse`.
Example: ::
>>> parse_url('http://google.com/mail/')
Url(scheme='http', host='google.com', port=None, path='/', ...)
>>> parse_url('google.com:80')
Url(scheme=None, host='google.com', port=80, path=None, ...)
>>> parse_url('/foo?bar')
Url(scheme=None, host=None, port=None, path='/foo', query='bar', ...)
"""
# While this code has overlap with stdlib's urlparse, it is much
# simplified for our needs and less annoying.
# Additionally, this implementations does silly things to be optimal
# on CPython.
scheme = None
auth = None
host = None
port = None
path = None
fragment = None
query = None
# Scheme
if '://' in url:
scheme, url = url.split('://', 1)
# Find the earliest Authority Terminator
# (http://tools.ietf.org/html/rfc3986#section-3.2)
url, path_, delim = split_first(url, ['/', '?', '#'])
if delim:
# Reassemble the path
path = delim + path_
# Auth
if '@' in url:
# Last '@' denotes end of auth part
auth, url = url.rsplit('@', 1)
# IPv6
if url and url[0] == '[':
host, url = url.split(']', 1)
host += ']'
# Port
if ':' in url:
_host, port = url.split(':', 1)
if not host:
host = _host
if port:
# If given, ports must be integers.
if not port.isdigit():
raise LocationParseError("Failed to parse: %s" % url)
port = int(port)
else:
# Blank ports are cool, too. (rfc3986#section-3.2.3)
port = None
elif not host and url:
host = url
if not path:
return Url(scheme, auth, host, port, path, query, fragment)
# Fragment
if '#' in path:
path, fragment = path.split('#', 1)
# Query
if '?' in path:
path, query = path.split('?', 1)
return Url(scheme, auth, host, port, path, query, fragment)
def get_host(url):
"""
Deprecated. Use :func:`.parse_url` instead.
"""
p = parse_url(url)
return p.scheme or 'http', p.hostname, p.port
def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
basic_auth=None, proxy_basic_auth=None):
"""
Shortcuts for generating request headers.
:param keep_alive:
If ``True``, adds 'connection: keep-alive' header.
:param accept_encoding:
Can be a boolean, list, or string.
``True`` translates to 'gzip,deflate'.
List will get joined by comma.
String will be used as provided.
:param user_agent:
String representing the user-agent you want, such as
"python-urllib3/0.6"
:param basic_auth:
Colon-separated username:password string for 'authorization: basic ...'
auth header.
:param proxy_basic_auth:
Colon-separated username:password string for 'proxy-authorization: basic ...'
auth header.
Example: ::
>>> make_headers(keep_alive=True, user_agent="Batman/1.0")
{'connection': 'keep-alive', 'user-agent': 'Batman/1.0'}
>>> make_headers(accept_encoding=True)
{'accept-encoding': 'gzip,deflate'}
"""
headers = {}
if accept_encoding:
if isinstance(accept_encoding, str):
pass
elif isinstance(accept_encoding, list):
accept_encoding = ','.join(accept_encoding)
else:
accept_encoding = 'gzip,deflate'
headers['accept-encoding'] = accept_encoding
if user_agent:
headers['user-agent'] = user_agent
if keep_alive:
headers['connection'] = 'keep-alive'
if basic_auth:
headers['authorization'] = 'Basic ' + \
b64encode(six.b(basic_auth)).decode('utf-8')
if proxy_basic_auth:
headers['proxy-authorization'] = 'Basic ' + \
b64encode(six.b(proxy_basic_auth)).decode('utf-8')
return headers
def is_connection_dropped(conn): # Platform-specific
"""
Returns True if the connection is dropped and should be closed.
:param conn:
:class:`httplib.HTTPConnection` object.
Note: For platforms like AppEngine, this will always return ``False`` to
let the platform handle connection recycling transparently for us.
"""
sock = getattr(conn, 'sock', False)
if not sock: # Platform-specific: AppEngine
return False
if not poll:
if not select: # Platform-specific: AppEngine
return False
try:
return select([sock], [], [], 0.0)[0]
except SocketError:
return True
# This version is better on platforms that support it.
p = poll()
p.register(sock, POLLIN)
for (fno, ev) in p.poll(0.0):
if fno == sock.fileno():
# Either data is buffered (bad), or the connection is dropped.
return True
def resolve_cert_reqs(candidate):
"""
Resolves the argument to a numeric constant, which can be passed to
the wrap_socket function/method from the ssl module.
Defaults to :data:`ssl.CERT_NONE`.
If given a string it is assumed to be the name of the constant in the
:mod:`ssl` module or its abbrevation.
(So you can specify `REQUIRED` instead of `CERT_REQUIRED`.
If it's neither `None` nor a string we assume it is already the numeric
constant which can directly be passed to wrap_socket.
"""
if candidate is None:
return CERT_NONE
if isinstance(candidate, str):
res = getattr(ssl, candidate, None)
if res is None:
res = getattr(ssl, 'CERT_' + candidate)
return res
return candidate
def resolve_ssl_version(candidate):
"""
like resolve_cert_reqs
"""
if candidate is None:
return PROTOCOL_SSLv23
if isinstance(candidate, str):
res = getattr(ssl, candidate, None)
if res is None:
res = getattr(ssl, 'PROTOCOL_' + candidate)
return res
return candidate
def assert_fingerprint(cert, fingerprint):
"""
Checks if given fingerprint matches the supplied certificate.
:param cert:
Certificate as bytes object.
:param fingerprint:
Fingerprint as string of hexdigits, can be interspersed by colons.
"""
# Maps the length of a digest to a possible hash function producing
# this digest.
hashfunc_map = {
16: md5,
20: sha1
}
fingerprint = fingerprint.replace(':', '').lower()
digest_length, rest = divmod(len(fingerprint), 2)
if rest or digest_length not in hashfunc_map:
raise SSLError('Fingerprint is of invalid length.')
# We need encode() here for py32; works on py2 and p33.
fingerprint_bytes = unhexlify(fingerprint.encode())
hashfunc = hashfunc_map[digest_length]
cert_digest = hashfunc(cert).digest()
if not cert_digest == fingerprint_bytes:
raise SSLError('Fingerprints did not match. Expected "{0}", got "{1}".'
.format(hexlify(fingerprint_bytes),
hexlify(cert_digest)))
def is_fp_closed(obj):
"""
Checks whether a given file-like object is closed.
:param obj:
The file-like object to check.
"""
if hasattr(obj, 'fp'):
# Object is a container for another file-like object that gets released
# on exhaustion (e.g. HTTPResponse)
return obj.fp is None
return obj.closed
if SSLContext is not None: # Python 3.2+
def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
ca_certs=None, server_hostname=None,
ssl_version=None):
"""
All arguments except `server_hostname` have the same meaning as for
:func:`ssl.wrap_socket`
:param server_hostname:
Hostname of the expected certificate
"""
context = SSLContext(ssl_version)
context.verify_mode = cert_reqs
if ca_certs:
try:
context.load_verify_locations(ca_certs)
# Py32 raises IOError
# Py33 raises FileNotFoundError
except Exception as e: # Reraise as SSLError
raise SSLError(e)
if certfile:
# FIXME: This block needs a test.
context.load_cert_chain(certfile, keyfile)
if HAS_SNI: # Platform-specific: OpenSSL with enabled SNI
return context.wrap_socket(sock, server_hostname=server_hostname)
return context.wrap_socket(sock)
else: # Python 3.1 and earlier
def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
ca_certs=None, server_hostname=None,
ssl_version=None):
return wrap_socket(sock, keyfile=keyfile, certfile=certfile,
ca_certs=ca_certs, cert_reqs=cert_reqs,
ssl_version=ssl_version)
requests/packages/urllib3/util/__init__.py
+44
View File
@@ -0,0 +1,44 @@
from __future__ import absolute_import
# For backwards compatibility, provide imports that used to be here.
from .connection import is_connection_dropped
from .request import make_headers
from .response import is_fp_closed
from .ssl_ import (
SSLContext,
HAS_SNI,
assert_fingerprint,
resolve_cert_reqs,
resolve_ssl_version,
ssl_wrap_socket,
)
from .timeout import (
current_time,
Timeout,
)
from .retry import Retry
from .url import (
get_host,
parse_url,
split_first,
Url,
)
__all__ = (
'HAS_SNI',
'SSLContext',
'Retry',
'Timeout',
'Url',
'assert_fingerprint',
'current_time',
'is_connection_dropped',
'is_fp_closed',
'get_host',
'parse_url',
'make_headers',
'resolve_cert_reqs',
'resolve_ssl_version',
'split_first',
'ssl_wrap_socket',
)
requests/packages/urllib3/util/connection.py
+101
View File
@@ -0,0 +1,101 @@
from __future__ import absolute_import
import socket
try:
from select import poll, POLLIN
except ImportError: # `poll` doesn't exist on OSX and other platforms
poll = False
try:
from select import select
except ImportError: # `select` doesn't exist on AppEngine.
select = False
def is_connection_dropped(conn): # Platform-specific
"""
Returns True if the connection is dropped and should be closed.
:param conn:
:class:`httplib.HTTPConnection` object.
Note: For platforms like AppEngine, this will always return ``False`` to
let the platform handle connection recycling transparently for us.
"""
sock = getattr(conn, 'sock', False)
if sock is False: # Platform-specific: AppEngine
return False
if sock is None: # Connection already closed (such as by httplib).
return True
if not poll:
if not select: # Platform-specific: AppEngine
return False
try:
return select([sock], [], [], 0.0)[0]
except socket.error:
return True
# This version is better on platforms that support it.
p = poll()
p.register(sock, POLLIN)
for (fno, ev) in p.poll(0.0):
if fno == sock.fileno():
# Either data is buffered (bad), or the connection is dropped.
return True
# This function is copied from socket.py in the Python 2.7 standard
# library test suite. Added to its signature is only `socket_options`.
def create_connection(address, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
source_address=None, socket_options=None):
"""Connect to *address* and return the socket object.
Convenience function. Connect to *address* (a 2-tuple ``(host,
port)``) and return the socket object. Passing the optional
*timeout* parameter will set the timeout on the socket instance
before attempting to connect. If no *timeout* is supplied, the
global default timeout setting returned by :func:`getdefaulttimeout`
is used. If *source_address* is set it must be a tuple of (host, port)
for the socket to bind as a source address before making the connection.
An host of '' or port 0 tells the OS to use the default.
"""
host, port = address
if host.startswith('['):
host = host.strip('[]')
err = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = socket.socket(af, socktype, proto)
# If provided, set socket level options before connecting.
# This is the only addition urllib3 makes to this function.
_set_socket_options(sock, socket_options)
if timeout is not socket._GLOBAL_DEFAULT_TIMEOUT:
sock.settimeout(timeout)
if source_address:
sock.bind(source_address)
sock.connect(sa)
return sock
except socket.error as e:
err = e
if sock is not None:
sock.close()
sock = None
if err is not None:
raise err
raise socket.error("getaddrinfo returns an empty list")
def _set_socket_options(sock, options):
if options is None:
return
for opt in options:
sock.setsockopt(*opt)
requests/packages/urllib3/util/request.py
+72
View File
@@ -0,0 +1,72 @@
from __future__ import absolute_import
from base64 import b64encode
from ..packages.six import b
ACCEPT_ENCODING = 'gzip,deflate'
def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
basic_auth=None, proxy_basic_auth=None, disable_cache=None):
"""
Shortcuts for generating request headers.
:param keep_alive:
If ``True``, adds 'connection: keep-alive' header.
:param accept_encoding:
Can be a boolean, list, or string.
``True`` translates to 'gzip,deflate'.
List will get joined by comma.
String will be used as provided.
:param user_agent:
String representing the user-agent you want, such as
"python-urllib3/0.6"
:param basic_auth:
Colon-separated username:password string for 'authorization: basic ...'
auth header.
:param proxy_basic_auth:
Colon-separated username:password string for 'proxy-authorization: basic ...'
auth header.
:param disable_cache:
If ``True``, adds 'cache-control: no-cache' header.
Example::
>>> make_headers(keep_alive=True, user_agent="Batman/1.0")
{'connection': 'keep-alive', 'user-agent': 'Batman/1.0'}
>>> make_headers(accept_encoding=True)
{'accept-encoding': 'gzip,deflate'}
"""
headers = {}
if accept_encoding:
if isinstance(accept_encoding, str):
pass
elif isinstance(accept_encoding, list):
accept_encoding = ','.join(accept_encoding)
else:
accept_encoding = ACCEPT_ENCODING
headers['accept-encoding'] = accept_encoding
if user_agent:
headers['user-agent'] = user_agent
if keep_alive:
headers['connection'] = 'keep-alive'
if basic_auth:
headers['authorization'] = 'Basic ' + \
b64encode(b(basic_auth)).decode('utf-8')
if proxy_basic_auth:
headers['proxy-authorization'] = 'Basic ' + \
b64encode(b(proxy_basic_auth)).decode('utf-8')
if disable_cache:
headers['cache-control'] = 'no-cache'
return headers
requests/packages/urllib3/util/response.py
+74
View File
@@ -0,0 +1,74 @@
from __future__ import absolute_import
from ..packages.six.moves import http_client as httplib
from ..exceptions import HeaderParsingError
def is_fp_closed(obj):
"""
Checks whether a given file-like object is closed.
:param obj:
The file-like object to check.
"""
try:
# Check via the official file-like-object way.
return obj.closed
except AttributeError:
pass
try:
# Check if the object is a container for another file-like object that
# gets released on exhaustion (e.g. HTTPResponse).
return obj.fp is None
except AttributeError:
pass
raise ValueError("Unable to determine whether fp is closed.")
def assert_header_parsing(headers):
"""
Asserts whether all headers have been successfully parsed.
Extracts encountered errors from the result of parsing headers.
Only works on Python 3.
:param headers: Headers to verify.
:type headers: `httplib.HTTPMessage`.
:raises urllib3.exceptions.HeaderParsingError:
If parsing errors are found.
"""
# This will fail silently if we pass in the wrong kind of parameter.
# To make debugging easier add an explicit check.
if not isinstance(headers, httplib.HTTPMessage):
raise TypeError('expected httplib.Message, got {0}.'.format(
type(headers)))
defects = getattr(headers, 'defects', None)
get_payload = getattr(headers, 'get_payload', None)
unparsed_data = None
if get_payload: # Platform-specific: Python 3.
unparsed_data = get_payload()
if defects or unparsed_data:
raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
def is_response_to_head(response):
"""
Checks, wether a the request of a response has been a HEAD-request.
Handles the quirks of AppEngine.
:param conn:
:type conn: :class:`httplib.HTTPResponse`
"""
# FIXME: Can we do this somehow without accessing private httplib _method?
method = response._method
if isinstance(method, int): # Platform-specific: Appengine
return method == 3
return method.upper() == 'HEAD'
requests/packages/urllib3/util/retry.py
+286
View File
@@ -0,0 +1,286 @@
from __future__ import absolute_import
import time
import logging
from ..exceptions import (
ConnectTimeoutError,
MaxRetryError,
ProtocolError,
ReadTimeoutError,
ResponseError,
)
from ..packages import six
log = logging.getLogger(__name__)
class Retry(object):
""" Retry configuration.
Each retry attempt will create a new Retry object with updated values, so
they can be safely reused.
Retries can be defined as a default for a pool::
retries = Retry(connect=5, read=2, redirect=5)
http = PoolManager(retries=retries)
response = http.request('GET', 'http://example.com/')
Or per-request (which overrides the default for the pool)::
response = http.request('GET', 'http://example.com/', retries=Retry(10))
Retries can be disabled by passing ``False``::
response = http.request('GET', 'http://example.com/', retries=False)
Errors will be wrapped in :class:`~urllib3.exceptions.MaxRetryError` unless
retries are disabled, in which case the causing exception will be raised.
:param int total:
Total number of retries to allow. Takes precedence over other counts.
Set to ``None`` to remove this constraint and fall back on other
counts. It's a good idea to set this to some sensibly-high value to
account for unexpected edge cases and avoid infinite retry loops.
Set to ``0`` to fail on the first retry.
Set to ``False`` to disable and imply ``raise_on_redirect=False``.
:param int connect:
How many connection-related errors to retry on.
These are errors raised before the request is sent to the remote server,
which we assume has not triggered the server to process the request.
Set to ``0`` to fail on the first retry of this type.
:param int read:
How many times to retry on read errors.
These errors are raised after the request was sent to the server, so the
request may have side-effects.
Set to ``0`` to fail on the first retry of this type.
:param int redirect:
How many redirects to perform. Limit this to avoid infinite redirect
loops.
A redirect is a HTTP response with a status code 301, 302, 303, 307 or
308.
Set to ``0`` to fail on the first retry of this type.
Set to ``False`` to disable and imply ``raise_on_redirect=False``.
:param iterable method_whitelist:
Set of uppercased HTTP method verbs that we should retry on.
By default, we only retry on methods which are considered to be
indempotent (multiple requests with the same parameters end with the
same state). See :attr:`Retry.DEFAULT_METHOD_WHITELIST`.
:param iterable status_forcelist:
A set of HTTP status codes that we should force a retry on.
By default, this is disabled with ``None``.
:param float backoff_factor:
A backoff factor to apply between attempts. urllib3 will sleep for::
{backoff factor} * (2 ^ ({number of total retries} - 1))
seconds. If the backoff_factor is 0.1, then :func:`.sleep` will sleep
for [0.1s, 0.2s, 0.4s, ...] between retries. It will never be longer
than :attr:`Retry.BACKOFF_MAX`.
By default, backoff is disabled (set to 0).
:param bool raise_on_redirect: Whether, if the number of redirects is
exhausted, to raise a MaxRetryError, or to return a response with a
response code in the 3xx range.
"""
DEFAULT_METHOD_WHITELIST = frozenset([
'HEAD', 'GET', 'PUT', 'DELETE', 'OPTIONS', 'TRACE'])
#: Maximum backoff time.
BACKOFF_MAX = 120
def __init__(self, total=10, connect=None, read=None, redirect=None,
method_whitelist=DEFAULT_METHOD_WHITELIST, status_forcelist=None,
backoff_factor=0, raise_on_redirect=True, _observed_errors=0):
self.total = total
self.connect = connect
self.read = read
if redirect is False or total is False:
redirect = 0
raise_on_redirect = False
self.redirect = redirect
self.status_forcelist = status_forcelist or set()
self.method_whitelist = method_whitelist
self.backoff_factor = backoff_factor
self.raise_on_redirect = raise_on_redirect
self._observed_errors = _observed_errors # TODO: use .history instead?
def new(self, **kw):
params = dict(
total=self.total,
connect=self.connect, read=self.read, redirect=self.redirect,
method_whitelist=self.method_whitelist,
status_forcelist=self.status_forcelist,
backoff_factor=self.backoff_factor,
raise_on_redirect=self.raise_on_redirect,
_observed_errors=self._observed_errors,
)
params.update(kw)
return type(self)(**params)
@classmethod
def from_int(cls, retries, redirect=True, default=None):
""" Backwards-compatibility for the old retries format."""
if retries is None:
retries = default if default is not None else cls.DEFAULT
if isinstance(retries, Retry):
return retries
redirect = bool(redirect) and None
new_retries = cls(retries, redirect=redirect)
log.debug("Converted retries value: %r -> %r" % (retries, new_retries))
return new_retries
def get_backoff_time(self):
""" Formula for computing the current backoff
:rtype: float
"""
if self._observed_errors <= 1:
return 0
backoff_value = self.backoff_factor * (2 ** (self._observed_errors - 1))
return min(self.BACKOFF_MAX, backoff_value)
def sleep(self):
""" Sleep between retry attempts using an exponential backoff.
By default, the backoff factor is 0 and this method will return
immediately.
"""
backoff = self.get_backoff_time()
if backoff <= 0:
return
time.sleep(backoff)
def _is_connection_error(self, err):
""" Errors when we're fairly sure that the server did not receive the
request, so it should be safe to retry.
"""
return isinstance(err, ConnectTimeoutError)
def _is_read_error(self, err):
""" Errors that occur after the request has been started, so we should
assume that the server began processing it.
"""
return isinstance(err, (ReadTimeoutError, ProtocolError))
def is_forced_retry(self, method, status_code):
""" Is this method/status code retryable? (Based on method/codes whitelists)
"""
if self.method_whitelist and method.upper() not in self.method_whitelist:
return False
return self.status_forcelist and status_code in self.status_forcelist
def is_exhausted(self):
""" Are we out of retries? """
retry_counts = (self.total, self.connect, self.read, self.redirect)
retry_counts = list(filter(None, retry_counts))
if not retry_counts:
return False
return min(retry_counts) < 0
def increment(self, method=None, url=None, response=None, error=None,
_pool=None, _stacktrace=None):
""" Return a new Retry object with incremented retry counters.
:param response: A response object, or None, if the server did not
return a response.
:type response: :class:`~urllib3.response.HTTPResponse`
:param Exception error: An error encountered during the request, or
None if the response was received successfully.
:return: A new ``Retry`` object.
"""
if self.total is False and error:
# Disabled, indicate to re-raise the error.
raise six.reraise(type(error), error, _stacktrace)
total = self.total
if total is not None:
total -= 1
_observed_errors = self._observed_errors
connect = self.connect
read = self.read
redirect = self.redirect
cause = 'unknown'
if error and self._is_connection_error(error):
# Connect retry?
if connect is False:
raise six.reraise(type(error), error, _stacktrace)
elif connect is not None:
connect -= 1
_observed_errors += 1
elif error and self._is_read_error(error):
# Read retry?
if read is False:
raise six.reraise(type(error), error, _stacktrace)
elif read is not None:
read -= 1
_observed_errors += 1
elif response and response.get_redirect_location():
# Redirect retry?
if redirect is not None:
redirect -= 1
cause = 'too many redirects'
else:
# Incrementing because of a server error like a 500 in
# status_forcelist and a the given method is in the whitelist
_observed_errors += 1
cause = ResponseError.GENERIC_ERROR
if response and response.status:
cause = ResponseError.SPECIFIC_ERROR.format(
status_code=response.status)
new_retry = self.new(
total=total,
connect=connect, read=read, redirect=redirect,
_observed_errors=_observed_errors)
if new_retry.is_exhausted():
raise MaxRetryError(_pool, url, error or ResponseError(cause))
log.debug("Incremented Retry for (url='%s'): %r" % (url, new_retry))
return new_retry
def __repr__(self):
return ('{cls.__name__}(total={self.total}, connect={self.connect}, '
'read={self.read}, redirect={self.redirect})').format(
cls=type(self), self=self)
# For backwards compatibility (equivalent to pre-v1.9):
Retry.DEFAULT = Retry(3)
requests/packages/urllib3/util/ssl_.py
+317
View File
@@ -0,0 +1,317 @@
from __future__ import absolute_import
import errno
import warnings
import hmac
from binascii import hexlify, unhexlify
from hashlib import md5, sha1, sha256
from ..exceptions import SSLError, InsecurePlatformWarning, SNIMissingWarning
SSLContext = None
HAS_SNI = False
create_default_context = None
# Maps the length of a digest to a possible hash function producing this digest
HASHFUNC_MAP = {
32: md5,
40: sha1,
64: sha256,
}
def _const_compare_digest_backport(a, b):
"""
Compare two digests of equal length in constant time.
The digests must be of type str/bytes.
Returns True if the digests match, and False otherwise.
"""
result = abs(len(a) - len(b))
for l, r in zip(bytearray(a), bytearray(b)):
result |= l ^ r
return result == 0
_const_compare_digest = getattr(hmac, 'compare_digest',
_const_compare_digest_backport)
try: # Test for SSL features
import ssl
from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23
from ssl import HAS_SNI # Has SNI?
except ImportError:
pass
try:
from ssl import OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_COMPRESSION
except ImportError:
OP_NO_SSLv2, OP_NO_SSLv3 = 0x1000000, 0x2000000
OP_NO_COMPRESSION = 0x20000
# A secure default.
# Sources for more information on TLS ciphers:
#
# - https://wiki.mozilla.org/Security/Server_Side_TLS
# - https://www.ssllabs.com/projects/best-practices/index.html
# - https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
#
# The general intent is:
# - Prefer cipher suites that offer perfect forward secrecy (DHE/ECDHE),
# - prefer ECDHE over DHE for better performance,
# - prefer any AES-GCM over any AES-CBC for better performance and security,
# - use 3DES as fallback which is secure but slow,
# - disable NULL authentication, MD5 MACs and DSS for security reasons.
DEFAULT_CIPHERS = (
'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
'!eNULL:!MD5'
)
try:
from ssl import SSLContext # Modern SSL?
except ImportError:
import sys
class SSLContext(object): # Platform-specific: Python 2 & 3.1
supports_set_ciphers = ((2, 7) <= sys.version_info < (3,) or
(3, 2) <= sys.version_info)
def __init__(self, protocol_version):
self.protocol = protocol_version
# Use default values from a real SSLContext
self.check_hostname = False
self.verify_mode = ssl.CERT_NONE
self.ca_certs = None
self.options = 0
self.certfile = None
self.keyfile = None
self.ciphers = None
def load_cert_chain(self, certfile, keyfile):
self.certfile = certfile
self.keyfile = keyfile
def load_verify_locations(self, cafile=None, capath=None):
self.ca_certs = cafile
if capath is not None:
raise SSLError("CA directories not supported in older Pythons")
def set_ciphers(self, cipher_suite):
if not self.supports_set_ciphers:
raise TypeError(
'Your version of Python does not support setting '
'a custom cipher suite. Please upgrade to Python '
'2.7, 3.2, or later if you need this functionality.'
)
self.ciphers = cipher_suite
def wrap_socket(self, socket, server_hostname=None):
warnings.warn(
'A true SSLContext object is not available. This prevents '
'urllib3 from configuring SSL appropriately and may cause '
'certain SSL connections to fail. For more information, see '
'https://urllib3.readthedocs.org/en/latest/security.html'
'#insecureplatformwarning.',
InsecurePlatformWarning
)
kwargs = {
'keyfile': self.keyfile,
'certfile': self.certfile,
'ca_certs': self.ca_certs,
'cert_reqs': self.verify_mode,
'ssl_version': self.protocol,
}
if self.supports_set_ciphers: # Platform-specific: Python 2.7+
return wrap_socket(socket, ciphers=self.ciphers, **kwargs)
else: # Platform-specific: Python 2.6
return wrap_socket(socket, **kwargs)
def assert_fingerprint(cert, fingerprint):
"""
Checks if given fingerprint matches the supplied certificate.
:param cert:
Certificate as bytes object.
:param fingerprint:
Fingerprint as string of hexdigits, can be interspersed by colons.
"""
fingerprint = fingerprint.replace(':', '').lower()
digest_length = len(fingerprint)
hashfunc = HASHFUNC_MAP.get(digest_length)
if not hashfunc:
raise SSLError(
'Fingerprint of invalid length: {0}'.format(fingerprint))
# We need encode() here for py32; works on py2 and p33.
fingerprint_bytes = unhexlify(fingerprint.encode())
cert_digest = hashfunc(cert).digest()
if not _const_compare_digest(cert_digest, fingerprint_bytes):
raise SSLError('Fingerprints did not match. Expected "{0}", got "{1}".'
.format(fingerprint, hexlify(cert_digest)))
def resolve_cert_reqs(candidate):
"""
Resolves the argument to a numeric constant, which can be passed to
the wrap_socket function/method from the ssl module.
Defaults to :data:`ssl.CERT_NONE`.
If given a string it is assumed to be the name of the constant in the
:mod:`ssl` module or its abbrevation.
(So you can specify `REQUIRED` instead of `CERT_REQUIRED`.
If it's neither `None` nor a string we assume it is already the numeric
constant which can directly be passed to wrap_socket.
"""
if candidate is None:
return CERT_NONE
if isinstance(candidate, str):
res = getattr(ssl, candidate, None)
if res is None:
res = getattr(ssl, 'CERT_' + candidate)
return res
return candidate
def resolve_ssl_version(candidate):
"""
like resolve_cert_reqs
"""
if candidate is None:
return PROTOCOL_SSLv23
if isinstance(candidate, str):
res = getattr(ssl, candidate, None)
if res is None:
res = getattr(ssl, 'PROTOCOL_' + candidate)
return res
return candidate
def create_urllib3_context(ssl_version=None, cert_reqs=None,
options=None, ciphers=None):
"""All arguments have the same meaning as ``ssl_wrap_socket``.
By default, this function does a lot of the same work that
``ssl.create_default_context`` does on Python 3.4+. It:
- Disables SSLv2, SSLv3, and compression
- Sets a restricted set of server ciphers
If you wish to enable SSLv3, you can do::
from urllib3.util import ssl_
context = ssl_.create_urllib3_context()
context.options &= ~ssl_.OP_NO_SSLv3
You can do the same to enable compression (substituting ``COMPRESSION``
for ``SSLv3`` in the last line above).
:param ssl_version:
The desired protocol version to use. This will default to
PROTOCOL_SSLv23 which will negotiate the highest protocol that both
the server and your installation of OpenSSL support.
:param cert_reqs:
Whether to require the certificate verification. This defaults to
``ssl.CERT_REQUIRED``.
:param options:
Specific OpenSSL options. These default to ``ssl.OP_NO_SSLv2``,
``ssl.OP_NO_SSLv3``, ``ssl.OP_NO_COMPRESSION``.
:param ciphers:
Which cipher suites to allow the server to select.
:returns:
Constructed SSLContext object with specified options
:rtype: SSLContext
"""
context = SSLContext(ssl_version or ssl.PROTOCOL_SSLv23)
# Setting the default here, as we may have no ssl module on import
cert_reqs = ssl.CERT_REQUIRED if cert_reqs is None else cert_reqs
if options is None:
options = 0
# SSLv2 is easily broken and is considered harmful and dangerous
options |= OP_NO_SSLv2
# SSLv3 has several problems and is now dangerous
options |= OP_NO_SSLv3
# Disable compression to prevent CRIME attacks for OpenSSL 1.0+
# (issue #309)
options |= OP_NO_COMPRESSION
context.options |= options
if getattr(context, 'supports_set_ciphers', True): # Platform-specific: Python 2.6
context.set_ciphers(ciphers or DEFAULT_CIPHERS)
context.verify_mode = cert_reqs
if getattr(context, 'check_hostname', None) is not None: # Platform-specific: Python 3.2
# We do our own verification, including fingerprints and alternative
# hostnames. So disable it here
context.check_hostname = False
return context
def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
ca_certs=None, server_hostname=None,
ssl_version=None, ciphers=None, ssl_context=None,
ca_cert_dir=None):
"""
All arguments except for server_hostname, ssl_context, and ca_cert_dir have
the same meaning as they do when using :func:`ssl.wrap_socket`.
:param server_hostname:
When SNI is supported, the expected hostname of the certificate
:param ssl_context:
A pre-made :class:`SSLContext` object. If none is provided, one will
be created using :func:`create_urllib3_context`.
:param ciphers:
A string of ciphers we wish the client to support. This is not
supported on Python 2.6 as the ssl module does not support it.
:param ca_cert_dir:
A directory containing CA certificates in multiple separate files, as
supported by OpenSSL's -CApath flag or the capath argument to
SSLContext.load_verify_locations().
"""
context = ssl_context
if context is None:
context = create_urllib3_context(ssl_version, cert_reqs,
ciphers=ciphers)
if ca_certs or ca_cert_dir:
try:
context.load_verify_locations(ca_certs, ca_cert_dir)
except IOError as e: # Platform-specific: Python 2.6, 2.7, 3.2
raise SSLError(e)
# Py33 raises FileNotFoundError which subclasses OSError
# These are not equivalent unless we check the errno attribute
except OSError as e: # Platform-specific: Python 3.3 and beyond
if e.errno == errno.ENOENT:
raise SSLError(e)
raise
if certfile:
context.load_cert_chain(certfile, keyfile)
if HAS_SNI: # Platform-specific: OpenSSL with enabled SNI
return context.wrap_socket(sock, server_hostname=server_hostname)
warnings.warn(
'An HTTPS request has been made, but the SNI (Subject Name '
'Indication) extension to TLS is not available on this platform. '
'This may cause the server to present an incorrect TLS '
'certificate, which can cause validation failures. For more '
'information, see '
'https://urllib3.readthedocs.org/en/latest/security.html'
'#snimissingwarning.',
SNIMissingWarning
)
return context.wrap_socket(sock)
requests/packages/urllib3/util/timeout.py
+242
View File
@@ -0,0 +1,242 @@
from __future__ import absolute_import
# The default socket timeout, used by httplib to indicate that no timeout was
# specified by the user
from socket import _GLOBAL_DEFAULT_TIMEOUT
import time
from ..exceptions import TimeoutStateError
# A sentinel value to indicate that no timeout was specified by the user in
# urllib3
_Default = object()
def current_time():
"""
Retrieve the current time. This function is mocked out in unit testing.
"""
return time.time()
class Timeout(object):
""" Timeout configuration.
Timeouts can be defined as a default for a pool::
timeout = Timeout(connect=2.0, read=7.0)
http = PoolManager(timeout=timeout)
response = http.request('GET', 'http://example.com/')
Or per-request (which overrides the default for the pool)::
response = http.request('GET', 'http://example.com/', timeout=Timeout(10))
Timeouts can be disabled by setting all the parameters to ``None``::
no_timeout = Timeout(connect=None, read=None)
response = http.request('GET', 'http://example.com/, timeout=no_timeout)
:param total:
This combines the connect and read timeouts into one; the read timeout
will be set to the time leftover from the connect attempt. In the
event that both a connect timeout and a total are specified, or a read
timeout and a total are specified, the shorter timeout will be applied.
Defaults to None.
:type total: integer, float, or None
:param connect:
The maximum amount of time to wait for a connection attempt to a server
to succeed. Omitting the parameter will default the connect timeout to
the system default, probably `the global default timeout in socket.py
<http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
None will set an infinite timeout for connection attempts.
:type connect: integer, float, or None
:param read:
The maximum amount of time to wait between consecutive
read operations for a response from the server. Omitting
the parameter will default the read timeout to the system
default, probably `the global default timeout in socket.py
<http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
None will set an infinite timeout.
:type read: integer, float, or None
.. note::
Many factors can affect the total amount of time for urllib3 to return
an HTTP response.
For example, Python's DNS resolver does not obey the timeout specified
on the socket. Other factors that can affect total request time include
high CPU load, high swap, the program running at a low priority level,
or other behaviors.
In addition, the read and total timeouts only measure the time between
read operations on the socket connecting the client and the server,
not the total amount of time for the request to return a complete
response. For most requests, the timeout is raised because the server
has not sent the first byte in the specified time. This is not always
the case; if a server streams one byte every fifteen seconds, a timeout
of 20 seconds will not trigger, even though the request will take
several minutes to complete.
If your goal is to cut off any request after a set amount of wall clock
time, consider having a second "watcher" thread to cut off a slow
request.
"""
#: A sentinel object representing the default timeout value
DEFAULT_TIMEOUT = _GLOBAL_DEFAULT_TIMEOUT
def __init__(self, total=None, connect=_Default, read=_Default):
self._connect = self._validate_timeout(connect, 'connect')
self._read = self._validate_timeout(read, 'read')
self.total = self._validate_timeout(total, 'total')
self._start_connect = None
def __str__(self):
return '%s(connect=%r, read=%r, total=%r)' % (
type(self).__name__, self._connect, self._read, self.total)
@classmethod
def _validate_timeout(cls, value, name):
""" Check that a timeout attribute is valid.
:param value: The timeout value to validate
:param name: The name of the timeout attribute to validate. This is
used to specify in error messages.
:return: The validated and casted version of the given value.
:raises ValueError: If the type is not an integer or a float, or if it
is a numeric value less than zero.
"""
if value is _Default:
return cls.DEFAULT_TIMEOUT
if value is None or value is cls.DEFAULT_TIMEOUT:
return value
try:
float(value)
except (TypeError, ValueError):
raise ValueError("Timeout value %s was %s, but it must be an "
"int or float." % (name, value))
try:
if value < 0:
raise ValueError("Attempted to set %s timeout to %s, but the "
"timeout cannot be set to a value less "
"than 0." % (name, value))
except TypeError: # Python 3
raise ValueError("Timeout value %s was %s, but it must be an "
"int or float." % (name, value))
return value
@classmethod
def from_float(cls, timeout):
""" Create a new Timeout from a legacy timeout value.
The timeout value used by httplib.py sets the same timeout on the
connect(), and recv() socket requests. This creates a :class:`Timeout`
object that sets the individual timeouts to the ``timeout`` value
passed to this function.
:param timeout: The legacy timeout value.
:type timeout: integer, float, sentinel default object, or None
:return: Timeout object
:rtype: :class:`Timeout`
"""
return Timeout(read=timeout, connect=timeout)
def clone(self):
""" Create a copy of the timeout object
Timeout properties are stored per-pool but each request needs a fresh
Timeout object to ensure each one has its own start/stop configured.
:return: a copy of the timeout object
:rtype: :class:`Timeout`
"""
# We can't use copy.deepcopy because that will also create a new object
# for _GLOBAL_DEFAULT_TIMEOUT, which socket.py uses as a sentinel to
# detect the user default.
return Timeout(connect=self._connect, read=self._read,
total=self.total)
def start_connect(self):
""" Start the timeout clock, used during a connect() attempt
:raises urllib3.exceptions.TimeoutStateError: if you attempt
to start a timer that has been started already.
"""
if self._start_connect is not None:
raise TimeoutStateError("Timeout timer has already been started.")
self._start_connect = current_time()
return self._start_connect
def get_connect_duration(self):
""" Gets the time elapsed since the call to :meth:`start_connect`.
:return: Elapsed time.
:rtype: float
:raises urllib3.exceptions.TimeoutStateError: if you attempt
to get duration for a timer that hasn't been started.
"""
if self._start_connect is None:
raise TimeoutStateError("Can't get connect duration for timer "
"that has not started.")
return current_time() - self._start_connect
@property
def connect_timeout(self):
""" Get the value to use when setting a connection timeout.
This will be a positive float or integer, the value None
(never timeout), or the default system timeout.
:return: Connect timeout.
:rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None
"""
if self.total is None:
return self._connect
if self._connect is None or self._connect is self.DEFAULT_TIMEOUT:
return self.total
return min(self._connect, self.total)
@property
def read_timeout(self):
""" Get the value for the read timeout.
This assumes some time has elapsed in the connection timeout and
computes the read timeout appropriately.
If self.total is set, the read timeout is dependent on the amount of
time taken by the connect timeout. If the connection time has not been
established, a :exc:`~urllib3.exceptions.TimeoutStateError` will be
raised.
:return: Value to use for the read timeout.
:rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None
:raises urllib3.exceptions.TimeoutStateError: If :meth:`start_connect`
has not yet been called on this object.
"""
if (self.total is not None and
self.total is not self.DEFAULT_TIMEOUT and
self._read is not None and
self._read is not self.DEFAULT_TIMEOUT):
# In case the connect timeout has not yet been established.
if self._start_connect is None:
return self._read
return max(0, min(self.total - self.get_connect_duration(),
self._read))
elif self.total is not None and self.total is not self.DEFAULT_TIMEOUT:
return max(0, self.total - self.get_connect_duration())
else:
return self._read
requests/packages/urllib3/util/url.py
+217
View File
@@ -0,0 +1,217 @@
from __future__ import absolute_import
from collections import namedtuple
from ..exceptions import LocationParseError
url_attrs = ['scheme', 'auth', 'host', 'port', 'path', 'query', 'fragment']
class Url(namedtuple('Url', url_attrs)):
"""
Datastructure for representing an HTTP URL. Used as a return value for
:func:`parse_url`.
"""
slots = ()
def __new__(cls, scheme=None, auth=None, host=None, port=None, path=None,
query=None, fragment=None):
if path and not path.startswith('/'):
path = '/' + path
return super(Url, cls).__new__(cls, scheme, auth, host, port, path,
query, fragment)
@property
def hostname(self):
"""For backwards-compatibility with urlparse. We're nice like that."""
return self.host
@property
def request_uri(self):
"""Absolute path including the query string."""
uri = self.path or '/'
if self.query is not None:
uri += '?' + self.query
return uri
@property
def netloc(self):
"""Network location including host and port"""
if self.port:
return '%s:%d' % (self.host, self.port)
return self.host
@property
def url(self):
"""
Convert self into a url
This function should more or less round-trip with :func:`.parse_url`. The
returned url may not be exactly the same as the url inputted to
:func:`.parse_url`, but it should be equivalent by the RFC (e.g., urls
with a blank port will have : removed).
Example: ::
>>> U = parse_url('http://google.com/mail/')
>>> U.url
'http://google.com/mail/'
>>> Url('http', 'username:password', 'host.com', 80,
... '/path', 'query', 'fragment').url
'http://username:password@host.com:80/path?query#fragment'
"""
scheme, auth, host, port, path, query, fragment = self
url = ''
# We use "is not None" we want things to happen with empty strings (or 0 port)
if scheme is not None:
url += scheme + '://'
if auth is not None:
url += auth + '@'
if host is not None:
url += host
if port is not None:
url += ':' + str(port)
if path is not None:
url += path
if query is not None:
url += '?' + query
if fragment is not None:
url += '#' + fragment
return url
def __str__(self):
return self.url
def split_first(s, delims):
"""
Given a string and an iterable of delimiters, split on the first found
delimiter. Return two split parts and the matched delimiter.
If not found, then the first part is the full input string.
Example::
>>> split_first('foo/bar?baz', '?/=')
('foo', 'bar?baz', '/')
>>> split_first('foo/bar?baz', '123')
('foo/bar?baz', '', None)
Scales linearly with number of delims. Not ideal for large number of delims.
"""
min_idx = None
min_delim = None
for d in delims:
idx = s.find(d)
if idx < 0:
continue
if min_idx is None or idx < min_idx:
min_idx = idx
min_delim = d
if min_idx is None or min_idx < 0:
return s, '', None
return s[:min_idx], s[min_idx + 1:], min_delim
def parse_url(url):
"""
Given a url, return a parsed :class:`.Url` namedtuple. Best-effort is
performed to parse incomplete urls. Fields not provided will be None.
Partly backwards-compatible with :mod:`urlparse`.
Example::
>>> parse_url('http://google.com/mail/')
Url(scheme='http', host='google.com', port=None, path='/mail/', ...)
>>> parse_url('google.com:80')
Url(scheme=None, host='google.com', port=80, path=None, ...)
>>> parse_url('/foo?bar')
Url(scheme=None, host=None, port=None, path='/foo', query='bar', ...)
"""
# While this code has overlap with stdlib's urlparse, it is much
# simplified for our needs and less annoying.
# Additionally, this implementations does silly things to be optimal
# on CPython.
if not url:
# Empty
return Url()
scheme = None
auth = None
host = None
port = None
path = None
fragment = None
query = None
# Scheme
if '://' in url:
scheme, url = url.split('://', 1)
# Find the earliest Authority Terminator
# (http://tools.ietf.org/html/rfc3986#section-3.2)
url, path_, delim = split_first(url, ['/', '?', '#'])
if delim:
# Reassemble the path
path = delim + path_
# Auth
if '@' in url:
# Last '@' denotes end of auth part
auth, url = url.rsplit('@', 1)
# IPv6
if url and url[0] == '[':
host, url = url.split(']', 1)
host += ']'
# Port
if ':' in url:
_host, port = url.split(':', 1)
if not host:
host = _host
if port:
# If given, ports must be integers.
if not port.isdigit():
raise LocationParseError(url)
port = int(port)
else:
# Blank ports are cool, too. (rfc3986#section-3.2.3)
port = None
elif not host and url:
host = url
if not path:
return Url(scheme, auth, host, port, path, query, fragment)
# Fragment
if '#' in path:
path, fragment = path.split('#', 1)
# Query
if '?' in path:
path, query = path.split('?', 1)
return Url(scheme, auth, host, port, path, query, fragment)
def get_host(url):
"""
Deprecated. Use :func:`.parse_url` instead.
"""
p = parse_url(url)
return p.scheme or 'http', p.hostname, p.port
requests/sessions.py
+214 -87
View File
@@ -12,27 +12,31 @@ import os
from collections import Mapping
from datetime import datetime
from .compat import cookielib, OrderedDict, urljoin, urlparse, builtin_str
from .auth import _basic_auth_str
from .compat import cookielib, OrderedDict, urljoin, urlparse
from .cookies import (
cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies)
from .models import Request, PreparedRequest
from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT
from .hooks import default_hooks, dispatch_hook
from .utils import to_key_val_list, default_headers
from .exceptions import TooManyRedirects, InvalidSchema
from .utils import to_key_val_list, default_headers, to_native_string
from .exceptions import (
TooManyRedirects, InvalidSchema, ChunkedEncodingError, ContentDecodingError)
from .packages.urllib3._collections import RecentlyUsedContainer
from .structures import CaseInsensitiveDict
from .adapters import HTTPAdapter
from .utils import requote_uri, get_environ_proxies, get_netrc_auth
from .utils import (
requote_uri, get_environ_proxies, get_netrc_auth, should_bypass_proxies,
get_auth_from_url
)
from .status_codes import codes
REDIRECT_STATI = (
codes.moved, # 301
codes.found, # 302
codes.other, # 303
codes.temporary_moved, # 307
)
DEFAULT_REDIRECT_LIMIT = 30
# formerly defined here, reexposed here for backward compatibility
from .models import REDIRECT_STATI
REDIRECT_CACHE_SIZE = 1000
def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
@@ -58,10 +62,11 @@ def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
merged_setting = dict_class(to_key_val_list(session_setting))
merged_setting.update(to_key_val_list(request_setting))
# Remove keys that are set to None.
for (k, v) in request_setting.items():
if v is None:
del merged_setting[k]
# Remove keys that are set to None. Extract keys first to avoid altering
# the dictionary during iteration.
none_keys = [k for (k, v) in merged_setting.items() if v is None]
for key in none_keys:
del merged_setting[key]
return merged_setting
@@ -84,16 +89,25 @@ def merge_hooks(request_hooks, session_hooks, dict_class=OrderedDict):
class SessionRedirectMixin(object):
def resolve_redirects(self, resp, req, stream=False, timeout=None,
verify=True, cert=None, proxies=None):
verify=True, cert=None, proxies=None, **adapter_kwargs):
"""Receives a Response. Returns a generator of Responses."""
i = 0
hist = [] # keep track of history
# ((resp.status_code is codes.see_other))
while ('location' in resp.headers and resp.status_code in REDIRECT_STATI):
while resp.is_redirect:
prepared_request = req.copy()
resp.content # Consume socket so it can be released
if i > 0:
# Update history and keep track of redirects.
hist.append(resp)
new_hist = list(hist)
resp.history = new_hist
try:
resp.content # Consume socket so it can be released
except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
resp.raw.read(decode_content=False)
if i >= self.max_redirects:
raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects)
@@ -113,17 +127,20 @@ class SessionRedirectMixin(object):
parsed = urlparse(url)
url = parsed.geturl()
# Facilitate non-RFC2616-compliant 'location' headers
# Facilitate relative 'location' headers, as allowed by RFC 7231.
# (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
# Compliant with RFC3986, we percent encode the url.
if not urlparse(url).netloc:
if not parsed.netloc:
url = urljoin(resp.url, requote_uri(url))
else:
url = requote_uri(url)
prepared_request.url = url
prepared_request.url = to_native_string(url)
# Cache the url, unless it redirects to itself.
if resp.is_permanent_redirect and req.url != prepared_request.url:
self.redirect_cache[req.url] = prepared_request.url
# http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4
# http://tools.ietf.org/html/rfc7231#section-6.4.4
if (resp.status_code == codes.see_other and
method != 'HEAD'):
method = 'GET'
@@ -141,7 +158,7 @@ class SessionRedirectMixin(object):
prepared_request.method = method
# https://github.com/kennethreitz/requests/issues/1084
if resp.status_code not in (codes.temporary, codes.resume):
if resp.status_code not in (codes.temporary_redirect, codes.permanent_redirect):
if 'Content-Length' in prepared_request.headers:
del prepared_request.headers['Content-Length']
@@ -153,19 +170,29 @@ class SessionRedirectMixin(object):
except KeyError:
pass
extract_cookies_to_jar(prepared_request._cookies,
prepared_request, resp.raw)
# Extract any cookies sent on the response to the cookiejar
# in the new request. Because we've mutated our copied prepared
# request, use the old one that we haven't yet touched.
extract_cookies_to_jar(prepared_request._cookies, req, resp.raw)
prepared_request._cookies.update(self.cookies)
prepared_request.prepare_cookies(prepared_request._cookies)
# Rebuild auth and proxy information.
proxies = self.rebuild_proxies(prepared_request, proxies)
self.rebuild_auth(prepared_request, resp)
# Override the original request.
req = prepared_request
resp = self.send(
prepared_request,
req,
stream=stream,
timeout=timeout,
verify=verify,
cert=cert,
proxies=proxies,
allow_redirects=False,
**adapter_kwargs
)
extract_cookies_to_jar(self.cookies, prepared_request, resp.raw)
@@ -173,6 +200,68 @@ class SessionRedirectMixin(object):
i += 1
yield resp
def rebuild_auth(self, prepared_request, response):
"""
When being redirected we may want to strip authentication from the
request to avoid leaking credentials. This method intelligently removes
and reapplies authentication where possible to avoid credential loss.
"""
headers = prepared_request.headers
url = prepared_request.url
if 'Authorization' in headers:
# If we get redirected to a new host, we should strip out any
# authentication headers.
original_parsed = urlparse(response.request.url)
redirect_parsed = urlparse(url)
if (original_parsed.hostname != redirect_parsed.hostname):
del headers['Authorization']
# .netrc might have more auth for us on our new host.
new_auth = get_netrc_auth(url) if self.trust_env else None
if new_auth is not None:
prepared_request.prepare_auth(new_auth)
return
def rebuild_proxies(self, prepared_request, proxies):
"""
This method re-evaluates the proxy configuration by considering the
environment variables. If we are redirected to a URL covered by
NO_PROXY, we strip the proxy configuration. Otherwise, we set missing
proxy keys for this URL (in case they were stripped by a previous
redirect).
This method also replaces the Proxy-Authorization header where
necessary.
"""
headers = prepared_request.headers
url = prepared_request.url
scheme = urlparse(url).scheme
new_proxies = proxies.copy() if proxies is not None else {}
if self.trust_env and not should_bypass_proxies(url):
environ_proxies = get_environ_proxies(url)
proxy = environ_proxies.get(scheme)
if proxy:
new_proxies.setdefault(scheme, environ_proxies[scheme])
if 'Proxy-Authorization' in headers:
del headers['Proxy-Authorization']
try:
username, password = get_auth_from_url(new_proxies[scheme])
except KeyError:
username, password = None, None
if username and password:
headers['Proxy-Authorization'] = _basic_auth_str(username, password)
return new_proxies
class Session(SessionRedirectMixin):
"""A Requests session.
@@ -184,13 +273,20 @@ class Session(SessionRedirectMixin):
>>> import requests
>>> s = requests.Session()
>>> s.get('http://httpbin.org/get')
200
<Response [200]>
Or as a context manager::
>>> with requests.Session() as s:
>>> s.get('http://httpbin.org/get')
<Response [200]>
"""
__attrs__ = [
'headers', 'cookies', 'auth', 'timeout', 'proxies', 'hooks',
'params', 'verify', 'cert', 'prefetch', 'adapters', 'stream',
'trust_env', 'max_redirects']
'headers', 'cookies', 'auth', 'proxies', 'hooks', 'params', 'verify',
'cert', 'prefetch', 'adapters', 'stream', 'trust_env',
'max_redirects',
]
def __init__(self):
@@ -203,9 +299,9 @@ class Session(SessionRedirectMixin):
#: :class:`Request <Request>`.
self.auth = None
#: Dictionary mapping protocol to the URL of the proxy (e.g.
#: {'http': 'foo.bar:3128'}) to be used on each
#: :class:`Request <Request>`.
#: Dictionary mapping protocol or protocol and host to the URL of the proxy
#: (e.g. {'http': 'foo.bar:3128', 'http://host.name': 'foo.bar:4012'}) to
#: be used on each :class:`Request <Request>`.
self.proxies = {}
#: Event-handling hooks.
@@ -229,7 +325,8 @@ class Session(SessionRedirectMixin):
#: limit, a :class:`TooManyRedirects` exception is raised.
self.max_redirects = DEFAULT_REDIRECT_LIMIT
#: Should we trust the environment?
#: Trust environment settings for proxy configuration, default
#: authentication and similar.
self.trust_env = True
#: A CookieJar containing all currently outstanding cookies set on this
@@ -243,6 +340,9 @@ class Session(SessionRedirectMixin):
self.mount('https://', HTTPAdapter())
self.mount('http://', HTTPAdapter())
# Only store 1000 redirects to prevent using infinite memory
self.redirect_cache = RecentlyUsedContainer(REDIRECT_CACHE_SIZE)
def __enter__(self):
return self
@@ -280,6 +380,7 @@ class Session(SessionRedirectMixin):
url=request.url,
files=request.files,
data=request.data,
json=request.json,
headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
params=merge_setting(request.params, self.params),
auth=merge_setting(auth, self.auth),
@@ -301,7 +402,8 @@ class Session(SessionRedirectMixin):
hooks=None,
stream=None,
verify=None,
cert=None):
cert=None,
json=None):
"""Constructs a :class:`Request <Request>`, prepares it and sends it.
Returns :class:`Response <Response>` object.
@@ -309,31 +411,33 @@ class Session(SessionRedirectMixin):
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary or bytes to be sent in the query
string for the :class:`Request`.
:param data: (optional) Dictionary or bytes to send in the body of the
:param data: (optional) Dictionary, bytes, or file-like object to send
in the body of the :class:`Request`.
:param json: (optional) json to send in the body of the
:class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the
:class:`Request`.
:param cookies: (optional) Dict or CookieJar object to send with the
:class:`Request`.
:param files: (optional) Dictionary of 'filename': file-like-objects
:param files: (optional) Dictionary of ``'filename': file-like-objects``
for multipart encoding upload.
:param auth: (optional) Auth tuple or callable to enable
Basic/Digest/Custom HTTP Auth.
:param timeout: (optional) Float describing the timeout of the
request.
:param allow_redirects: (optional) Boolean. Set to True by default.
:param proxies: (optional) Dictionary mapping protocol to the URL of
the proxy.
:param timeout: (optional) How long to wait for the server to send
data before giving up, as a float, or a :ref:`(connect timeout,
read timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param allow_redirects: (optional) Set to True by default.
:type allow_redirects: bool
:param proxies: (optional) Dictionary mapping protocol or protocol and
hostname to the URL of the proxy.
:param stream: (optional) whether to immediately download the response
content. Defaults to ``False``.
:param verify: (optional) if ``True``, the SSL cert will be verified.
A CA_BUNDLE path can also be provided.
:param verify: (optional) whether the SSL cert will be verified.
A CA_BUNDLE path can also be provided. Defaults to ``True``.
:param cert: (optional) if String, path to ssl client cert file (.pem).
If Tuple, ('cert', 'key') pair.
"""
method = builtin_str(method)
# Create the Request.
req = Request(
method = method.upper(),
@@ -341,6 +445,7 @@ class Session(SessionRedirectMixin):
headers = headers,
files = files,
data = data or {},
json = json,
params = params or {},
auth = auth,
cookies = cookies,
@@ -350,36 +455,16 @@ class Session(SessionRedirectMixin):
proxies = proxies or {}
# Gather clues from the surrounding environment.
if self.trust_env:
# Set environment's proxies.
env_proxies = get_environ_proxies(url) or {}
for (k, v) in env_proxies.items():
proxies.setdefault(k, v)
# Look for configuration.
if not verify and verify is not False:
verify = os.environ.get('REQUESTS_CA_BUNDLE')
# Curl compatibility.
if not verify and verify is not False:
verify = os.environ.get('CURL_CA_BUNDLE')
# Merge all the kwargs.
proxies = merge_setting(proxies, self.proxies)
stream = merge_setting(stream, self.stream)
verify = merge_setting(verify, self.verify)
cert = merge_setting(cert, self.cert)
settings = self.merge_environment_settings(
prep.url, proxies, stream, verify, cert
)
# Send the request.
send_kwargs = {
'stream': stream,
'timeout': timeout,
'verify': verify,
'cert': cert,
'proxies': proxies,
'allow_redirects': allow_redirects,
}
send_kwargs.update(settings)
resp = self.send(prep, **send_kwargs)
return resp
@@ -414,15 +499,16 @@ class Session(SessionRedirectMixin):
kwargs.setdefault('allow_redirects', False)
return self.request('HEAD', url, **kwargs)
def post(self, url, data=None, **kwargs):
def post(self, url, data=None, json=None, **kwargs):
"""Sends a POST request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param json: (optional) json to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
"""
return self.request('POST', url, data=data, **kwargs)
return self.request('POST', url, data=data, json=json, **kwargs)
def put(self, url, data=None, **kwargs):
"""Sends a PUT request. Returns :class:`Response` object.
@@ -467,14 +553,17 @@ class Session(SessionRedirectMixin):
if not isinstance(request, PreparedRequest):
raise ValueError('You can only send PreparedRequests.')
# Set up variables needed for resolve_redirects and dispatching of
# hooks
checked_urls = set()
while request.url in self.redirect_cache:
checked_urls.add(request.url)
new_url = self.redirect_cache.get(request.url)
if new_url in checked_urls:
break
request.url = new_url
# Set up variables needed for resolve_redirects and dispatching of hooks
allow_redirects = kwargs.pop('allow_redirects', True)
stream = kwargs.get('stream')
timeout = kwargs.get('timeout')
verify = kwargs.get('verify')
cert = kwargs.get('cert')
proxies = kwargs.get('proxies')
hooks = request.hooks
# Get the appropriate adapter to use
@@ -482,8 +571,10 @@ class Session(SessionRedirectMixin):
# Start time (approximately) of the request
start = datetime.utcnow()
# Send the request
r = adapter.send(request, **kwargs)
# Total elapsed time of the request (approximately)
r.elapsed = datetime.utcnow() - start
@@ -492,15 +583,15 @@ class Session(SessionRedirectMixin):
# Persist cookies
if r.history:
# If the hooks create history then we want those cookies too
for resp in r.history:
extract_cookies_to_jar(self.cookies, resp.request, resp.raw)
extract_cookies_to_jar(self.cookies, request, r.raw)
# Redirect resolving generator.
gen = self.resolve_redirects(r, request, stream=stream,
timeout=timeout, verify=verify, cert=cert,
proxies=proxies)
gen = self.resolve_redirects(r, request, **kwargs)
# Resolve redirects if allowed.
history = [resp for resp in gen] if allow_redirects else []
@@ -511,12 +602,39 @@ class Session(SessionRedirectMixin):
history.insert(0, r)
# Get the last request made
r = history.pop()
r.history = tuple(history)
r.history = history
if not stream:
r.content
return r
def merge_environment_settings(self, url, proxies, stream, verify, cert):
"""Check the environment and merge it with some settings."""
# Gather clues from the surrounding environment.
if self.trust_env:
# Set environment's proxies.
env_proxies = get_environ_proxies(url) or {}
for (k, v) in env_proxies.items():
proxies.setdefault(k, v)
# Look for requests environment configuration and be compatible
# with cURL.
if verify is True or verify is None:
verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
os.environ.get('CURL_CA_BUNDLE'))
# Merge all the kwargs.
proxies = merge_setting(proxies, self.proxies)
stream = merge_setting(stream, self.stream)
verify = merge_setting(verify, self.verify)
cert = merge_setting(cert, self.cert)
return {'verify': verify, 'proxies': proxies, 'stream': stream,
'cert': cert}
def get_adapter(self, url):
"""Returns the appropriate connnection adapter for the given URL."""
"""Returns the appropriate connection adapter for the given URL."""
for (prefix, adapter) in self.adapters.items():
if url.lower().startswith(prefix):
@@ -534,18 +652,27 @@ class Session(SessionRedirectMixin):
"""Registers a connection adapter to a prefix.
Adapters are sorted in descending order by key length."""
self.adapters[prefix] = adapter
keys_to_move = [k for k in self.adapters if len(k) < len(prefix)]
for key in keys_to_move:
self.adapters[key] = self.adapters.pop(key)
def __getstate__(self):
return dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)
state = dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)
state['redirect_cache'] = dict(self.redirect_cache)
return state
def __setstate__(self, state):
redirect_cache = state.pop('redirect_cache', {})
for attr, value in state.items():
setattr(self, attr, value)
self.redirect_cache = RecentlyUsedContainer(REDIRECT_CACHE_SIZE)
for redirect, to in redirect_cache.items():
self.redirect_cache[redirect] = to
def session():
"""Returns a :class:`Session` for context-management."""
requests/status_codes.py
+4 -2
View File
@@ -30,7 +30,8 @@ _codes = {
305: ('use_proxy',),
306: ('switch_proxy',),
307: ('temporary_redirect', 'temporary_moved', 'temporary'),
308: ('resume_incomplete', 'resume'),
308: ('permanent_redirect',
'resume_incomplete', 'resume',), # These 2 to be removed in 3.0
# Client Error.
400: ('bad_request', 'bad'),
@@ -77,11 +78,12 @@ _codes = {
507: ('insufficient_storage',),
509: ('bandwidth_limit_exceeded', 'bandwidth'),
510: ('not_extended',),
511: ('network_authentication_required', 'network_auth', 'network_authentication'),
}
codes = LookupDict(name='status_codes')
for (code, titles) in list(_codes.items()):
for code, titles in _codes.items():
for title in titles:
setattr(codes, title, code)
if not title.startswith('\\'):
requests/structures.py
+2 -26
View File
@@ -8,30 +8,7 @@ Data structures that power Requests.
"""
import os
import collections
from itertools import islice
class IteratorProxy(object):
"""docstring for IteratorProxy"""
def __init__(self, i):
self.i = i
# self.i = chain.from_iterable(i)
def __iter__(self):
return self.i
def __len__(self):
if hasattr(self.i, '__len__'):
return len(self.i)
if hasattr(self.i, 'len'):
return self.i.len
if hasattr(self.i, 'fileno'):
return os.fstat(self.i.fileno()).st_size
def read(self, n):
return "".join(islice(self.i, None, n))
class CaseInsensitiveDict(collections.MutableMapping):
@@ -46,7 +23,7 @@ class CaseInsensitiveDict(collections.MutableMapping):
case of the last key to be set, and ``iter(instance)``,
``keys()``, ``items()``, ``iterkeys()``, and ``iteritems()``
will contain case-sensitive keys. However, querying and contains
testing is case insensitive:
testing is case insensitive::
cid = CaseInsensitiveDict()
cid['Accept'] = 'application/json'
@@ -106,8 +83,7 @@ class CaseInsensitiveDict(collections.MutableMapping):
return CaseInsensitiveDict(self._store.values())
def __repr__(self):
return '%s(%r)' % (self.__class__.__name__, dict(self.items()))
return str(dict(self.items()))
class LookupDict(dict):
"""Dictionary lookup object."""
requests/utils.py
+130 -74
View File
@@ -19,15 +19,17 @@ import re
import sys
import socket
import struct
import warnings
from . import __version__
from . import certs
from .compat import parse_http_list as _parse_list_header
from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,
builtin_str, getproxies, proxy_bypass)
builtin_str, getproxies, proxy_bypass, urlunparse,
basestring)
from .cookies import RequestsCookieJar, cookiejar_from_dict
from .structures import CaseInsensitiveDict
from .exceptions import MissingSchema, InvalidURL
from .exceptions import InvalidURL, FileModeWarning
_hush_pyflakes = (RequestsCookieJar,)
@@ -46,26 +48,47 @@ def dict_to_sequence(d):
def super_len(o):
total_length = 0
current_position = 0
if hasattr(o, '__len__'):
return len(o)
total_length = len(o)
if hasattr(o, 'len'):
return o.len
elif hasattr(o, 'len'):
total_length = o.len
if hasattr(o, 'fileno'):
elif hasattr(o, 'getvalue'):
# e.g. BytesIO, cStringIO.StringIO
total_length = len(o.getvalue())
elif hasattr(o, 'fileno'):
try:
fileno = o.fileno()
except io.UnsupportedOperation:
pass
else:
return os.fstat(fileno).st_size
total_length = os.fstat(fileno).st_size
if hasattr(o, 'getvalue'):
# e.g. BytesIO, cStringIO.StringI
return len(o.getvalue())
# Having used fstat to determine the file length, we need to
# confirm that this file was opened up in binary mode.
if 'b' not in o.mode:
warnings.warn((
"Requests has determined the content-length for this "
"request using the binary size of the file: however, the "
"file has been opened in text mode (i.e. without the 'b' "
"flag in the mode). This may lead to an incorrect "
"content-length. In Requests 3.0, support will be removed "
"for files in text mode."),
FileModeWarning
)
if hasattr(o, 'tell'):
current_position = o.tell()
return max(0, total_length - current_position)
def get_netrc_auth(url):
def get_netrc_auth(url, raise_errors=False):
"""Returns the Requests tuple auth for a given url from netrc."""
try:
@@ -103,8 +126,9 @@ def get_netrc_auth(url):
return (_netrc[login_i], _netrc[2])
except (NetrcParseError, IOError):
# If there was a parsing error or a permissions issue reading the file,
# we'll just skip netrc auth
pass
# we'll just skip netrc auth unless explicitly asked to raise errors.
if raise_errors:
raise
# AppEngine hackiness.
except (ImportError, AttributeError):
@@ -114,7 +138,8 @@ def get_netrc_auth(url):
def guess_filename(obj):
"""Tries to guess the filename of the given object."""
name = getattr(obj, 'name', None)
if name and name[0] != '<' and name[-1] != '>':
if (name and isinstance(name, basestring) and name[0] != '<' and
name[-1] != '>'):
return os.path.basename(name)
@@ -287,6 +312,11 @@ def get_encodings_from_content(content):
:param content: bytestring to extract encodings from.
"""
warnings.warn((
'In requests 3.0, get_encodings_from_content will be removed. For '
'more information, please see the discussion on issue #2266. (This'
' warning should only appear once.)'),
DeprecationWarning)
charset_re = re.compile(r'<meta.*?charset=["\']*(.+?)["\'>]', flags=re.I)
pragma_re = re.compile(r'<meta.*?content=["\']*;?charset=(.+?)["\'>]', flags=re.I)
@@ -351,12 +381,14 @@ def get_unicode_from_response(r):
Tried:
1. charset from content-type
2. every encodings from ``<meta ... charset=XXX>``
3. fall back and replace all unicode characters
2. fall back and replace all unicode characters
"""
warnings.warn((
'In requests 3.0, get_unicode_from_response will be removed. For '
'more information, please see the discussion on issue #2266. (This'
' warning should only appear once.)'),
DeprecationWarning)
tried_encodings = []
@@ -410,10 +442,18 @@ def requote_uri(uri):
This function passes the given URI through an unquote/quote cycle to
ensure that it is fully and consistently quoted.
"""
# Unquote only the unreserved characters
# Then quote only illegal characters (do not quote reserved, unreserved,
# or '%')
return quote(unquote_unreserved(uri), safe="!#$%&'()*+,/:;=?@[]~")
safe_with_percent = "!#$%&'()*+,/:;=?@[]~"
safe_without_percent = "!#$&'()*+,/:;=?@[]~"
try:
# Unquote only the unreserved characters
# Then quote only illegal characters (do not quote reserved,
# unreserved, or '%')
return quote(unquote_unreserved(uri), safe=safe_with_percent)
except InvalidURL:
# We couldn't unquote the given URI, so let's try quoting it, but
# there may be unquoted '%'s in the URI. We need to make sure they're
# properly quoted so they do not cause issues elsewhere.
return quote(uri, safe=safe_without_percent)
def address_in_network(ip, net):
@@ -466,9 +506,10 @@ def is_valid_cidr(string_network):
return True
def get_environ_proxies(url):
"""Return a dict of environment proxies."""
def should_bypass_proxies(url):
"""
Returns whether we should bypass proxies or not.
"""
get_proxy = lambda k: os.environ.get(k) or os.environ.get(k.upper())
# First check whether no_proxy is defined. If it is, check that the URL
@@ -479,20 +520,22 @@ def get_environ_proxies(url):
if no_proxy:
# We need to check whether we match here. We need to see if we match
# the end of the netloc, both with and without the port.
no_proxy = no_proxy.replace(' ', '').split(',')
no_proxy = (
host for host in no_proxy.replace(' ', '').split(',') if host
)
ip = netloc.split(':')[0]
if is_ipv4_address(ip):
for proxy_ip in no_proxy:
if is_valid_cidr(proxy_ip):
if address_in_network(ip, proxy_ip):
return {}
return True
else:
for host in no_proxy:
if netloc.endswith(host) or netloc.split(':')[0].endswith(host):
# The URL does match something in no_proxy, so we don't want
# to apply the proxies on this URL.
return {}
return True
# If the system proxy settings indicate that this URL should be bypassed,
# don't proxy.
@@ -506,50 +549,41 @@ def get_environ_proxies(url):
bypass = False
if bypass:
return True
return False
def get_environ_proxies(url):
"""Return a dict of environment proxies."""
if should_bypass_proxies(url):
return {}
else:
return getproxies()
# If we get here, we either didn't have no_proxy set or we're not going
# anywhere that no_proxy applies to, and the system settings don't require
# bypassing the proxy for the current URL.
return getproxies()
def select_proxy(url, proxies):
"""Select a proxy for the url, if applicable.
:param url: The url being for the request
:param proxies: A dictionary of schemes or schemes and hosts to proxy URLs
"""
proxies = proxies or {}
urlparts = urlparse(url)
proxy = proxies.get(urlparts.scheme+'://'+urlparts.hostname)
if proxy is None:
proxy = proxies.get(urlparts.scheme)
return proxy
def default_user_agent(name="python-requests"):
"""Return a string representing the default user agent."""
_implementation = platform.python_implementation()
if _implementation == 'CPython':
_implementation_version = platform.python_version()
elif _implementation == 'PyPy':
_implementation_version = '%s.%s.%s' % (sys.pypy_version_info.major,
sys.pypy_version_info.minor,
sys.pypy_version_info.micro)
if sys.pypy_version_info.releaselevel != 'final':
_implementation_version = ''.join([_implementation_version, sys.pypy_version_info.releaselevel])
elif _implementation == 'Jython':
_implementation_version = platform.python_version() # Complete Guess
elif _implementation == 'IronPython':
_implementation_version = platform.python_version() # Complete Guess
else:
_implementation_version = 'Unknown'
try:
p_system = platform.system()
p_release = platform.release()
except IOError:
p_system = 'Unknown'
p_release = 'Unknown'
return " ".join(['%s/%s' % (name, __version__),
'%s/%s' % (_implementation, _implementation_version),
'%s/%s' % (p_system, p_release)])
return '%s/%s' % (name, __version__)
def default_headers():
return CaseInsensitiveDict({
'User-Agent': default_user_agent(),
'Accept-Encoding': ', '.join(('gzip', 'deflate', 'compress')),
'Accept': '*/*'
'Accept-Encoding': ', '.join(('gzip', 'deflate')),
'Accept': '*/*',
'Connection': 'keep-alive',
})
@@ -564,7 +598,7 @@ def parse_header_links(value):
replace_chars = " '\""
for val in value.split(","):
for val in re.split(", *<", value):
try:
url, params = val.split(";", 1)
except ValueError:
@@ -622,24 +656,31 @@ def guess_json_utf(data):
return None
def except_on_missing_scheme(url):
"""Given a URL, raise a MissingSchema exception if the scheme is missing.
"""
scheme, netloc, path, params, query, fragment = urlparse(url)
def prepend_scheme_if_needed(url, new_scheme):
'''Given a URL that may or may not have a scheme, prepend the given scheme.
Does not replace a present scheme with the one provided as an argument.'''
scheme, netloc, path, params, query, fragment = urlparse(url, new_scheme)
if not scheme:
raise MissingSchema('Proxy URLs must have explicit schemes.')
# urlparse is a finicky beast, and sometimes decides that there isn't a
# netloc present. Assume that it's being over-cautious, and switch netloc
# and path if urlparse decided there was no netloc.
if not netloc:
netloc, path = path, netloc
return urlunparse((scheme, netloc, path, params, query, fragment))
def get_auth_from_url(url):
"""Given a url with authentication components, extract them into a tuple of
username,password."""
if url:
url = unquote(url)
parsed = urlparse(url)
return (parsed.username, parsed.password)
else:
return ('', '')
parsed = urlparse(url)
try:
auth = (unquote(parsed.username), unquote(parsed.password))
except (AttributeError, TypeError):
auth = ('', '')
return auth
def to_native_string(string, encoding='ascii'):
@@ -659,3 +700,18 @@ def to_native_string(string, encoding='ascii'):
out = string.decode(encoding)
return out
def urldefragauth(url):
"""
Given a url remove the fragment and the authentication part
"""
scheme, netloc, path, params, query, fragment = urlparse(url)
# see func:`prepend_scheme_if_needed`
if not netloc:
netloc, path = path, netloc
netloc = netloc.rsplit('@', 1)[-1]
return urlunparse((scheme, netloc, path, params, query, ''))
requirements.txt
+5 -4
View File
@@ -1,5 +1,6 @@
py==1.4.12
pytest==2.3.4
pytest-cov==1.6
invoke==0.2.0
py==1.4.30
pytest==2.8.1
pytest-cov==2.1.0
pytest-httpbin==0.0.7
httpbin==0.4.0
wheel
setup.py
+21 -10
View File
@@ -1,9 +1,10 @@
#!/usr/bin/env python
import os
import re
import sys
import requests
from codecs import open
try:
from setuptools import setup
@@ -21,21 +22,28 @@ packages = [
'requests.packages.urllib3',
'requests.packages.urllib3.packages',
'requests.packages.urllib3.contrib',
'requests.packages.urllib3.packages.ssl_match_hostname'
'requests.packages.urllib3.util',
'requests.packages.urllib3.packages.ssl_match_hostname',
]
requires = []
with open('README.rst') as f:
version = ''
with open('requests/__init__.py', 'r') as fd:
version = re.search(r'^__version__\s*=\s*[\'"]([^\'"]*)[\'"]',
fd.read(), re.MULTILINE).group(1)
if not version:
raise RuntimeError('Cannot find version information')
with open('README.rst', 'r', 'utf-8') as f:
readme = f.read()
with open('HISTORY.rst') as f:
with open('HISTORY.rst', 'r', 'utf-8') as f:
history = f.read()
with open('LICENSE') as f:
license = f.read()
setup(
name='requests',
version=requests.__version__,
version=version,
description='Python HTTP for Humans.',
long_description=readme + '\n\n' + history,
author='Kenneth Reitz',
@@ -46,7 +54,7 @@ setup(
package_dir={'requests': 'requests'},
include_package_data=True,
install_requires=requires,
license=license,
license='Apache 2.0',
zip_safe=False,
classifiers=(
'Development Status :: 5 - Production/Stable',
@@ -54,10 +62,13 @@ setup(
'Natural Language :: English',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
),
extras_require={
'security': ['pyOpenSSL>=0.13', 'ndg-httpsclient', 'pyasn1'],
},
)
test_requests.py
+720 -130
View File
File diff suppressed because it is too large Load Diff