Pipenv is a production-ready tool that aims to bring the best of all packaging worlds to the Python world. It harnesses Pipfile, pip, and virtualenv into one single command.
It features very pretty terminal colors.
-
Stay Informed
Receive updates on new releases and upcoming projects.
Pipenv is a production-ready tool that aims to bring the best of all packaging worlds to the Python world. It harnesses Pipfile, pip, and virtualenv into one single command.
diff --git a/docs/basics.rst b/docs/basics.rst
index 00f4c7d9..2a37d7f9 100644
--- a/docs/basics.rst
+++ b/docs/basics.rst
@@ -127,6 +127,7 @@ Example Pipfile.lock
- Do not keep ``Pipfile.lock`` in version control if multiple versions of Python are being targeted.
- Specify your target Python version in your `Pipfile`'s ``[requires]`` section. Ideally, you should only have one target Python version, as this is a deployment tool.
- ``pipenv install`` is fully compatible with ``pip install`` syntax, for which the full documentation can be found `here `_.
+- Note that the ``Pipfile`` uses the `TOML Spec `_.
@@ -203,11 +204,11 @@ To make inclusive or exclusive version comparisons you can use: ::
$ pipenv install "requests<=2.13" # will install a version equal or lower than 2.13.0
$ pipenv install "requests>2.19" # will install 2.19.1 but not 2.19.0
-.. note:: The use of ``" "`` around the package and version specification is highly recommended
+.. note:: The use of double quotes around the package and version specification (i.e. ``"requests>2.19"``) is highly recommended
to avoid issues with `Input and output redirection `_
in Unix-based operating systems.
-The use of ``~=`` is preferred over the ``==`` identifier as the former prevents pipenv from updating the packages: ::
+The use of ``~=`` is preferred over the ``==`` identifier as the latter prevents pipenv from updating the packages: ::
$ pipenv install "requests~=2.2" # locks the major version of the package (this is equivalent to using ==2.*)
diff --git a/docs/index.rst b/docs/index.rst
index a832ee8e..0f3ae522 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -26,10 +26,12 @@ It automatically creates and manages a virtualenv for your projects, as well as
Pipenv is primarily meant to provide users and developers of applications with an easy method to setup a working environment. For the distinction between libraries and applications and the usage of ``setup.py`` vs ``Pipfile`` to define dependencies, see :ref:`pipfile-vs-setuppy`.
-.. raw:: html
-
-
-
+.. image:: https://s3.amazonaws.com/media.kennethreitz.com/pipenv.gif
+ :height: 341px
+ :width: 654px
+ :scale: 100 %
+ :alt: a short animation of pipenv at work
+
The problems that Pipenv seeks to solve are multi-faceted:
- You no longer need to use ``pip`` and ``virtualenv`` separately. They work together.
@@ -70,9 +72,6 @@ Otherwise, refer to the :ref:`installing-pipenv` chapter for instructions.
User Testimonials
-----------------
-**Jannis Leidel**, former pip maintainer—
- *Pipenv is the porcelain I always wanted to build for pip. It fits my brain and mostly replaces virtualenvwrapper and manual pip calls for me. Use it.*
-
**David Gang**—
*This package manager is really awesome. For the first time I know exactly what my dependencies are which I installed and what the transitive dependencies are. Combined with the fact that installs are deterministic, makes this package manager first class, like cargo*.
diff --git a/news/2553.behavior.rst b/news/2553.behavior.rst
new file mode 100644
index 00000000..d66edfa2
--- /dev/null
+++ b/news/2553.behavior.rst
@@ -0,0 +1 @@
+Make conservative checks of known exceptions when subprocess returns output, so user won't see the whole traceback - just the error.
\ No newline at end of file
diff --git a/news/2722.bugfix.rst b/news/2722.bugfix.rst
new file mode 100644
index 00000000..8c26df8d
--- /dev/null
+++ b/news/2722.bugfix.rst
@@ -0,0 +1 @@
+Fixed a bug which caused editable package resolution to sometimes fail with an unhelpful setuptools-related error message.
diff --git a/news/2783.bugfix.rst b/news/2783.bugfix.rst
new file mode 100644
index 00000000..7fa3cfd1
--- /dev/null
+++ b/news/2783.bugfix.rst
@@ -0,0 +1,2 @@
+Fixed an issue which caused errors due to reliance on the system utilities ``which`` and ``where`` which may not always exist on some systems.
+- Fixed a bug which caused periodic failures in python discovery when executables named ``python`` were not present on the target ``$PATH``.
diff --git a/news/3053.bugfix.rst b/news/3053.bugfix.rst
new file mode 100644
index 00000000..21134f59
--- /dev/null
+++ b/news/3053.bugfix.rst
@@ -0,0 +1 @@
+Dependency resolution now writes hashes for local and remote files to the lockfile.
diff --git a/news/3071.bugfix.rst b/news/3071.bugfix.rst
new file mode 100644
index 00000000..dd4145ea
--- /dev/null
+++ b/news/3071.bugfix.rst
@@ -0,0 +1 @@
+Fixed a bug which prevented ``pipenv graph`` from correctly showing all dependencies when running from within ``pipenv shell``.
diff --git a/news/3148.bugfix.rst b/news/3148.bugfix.rst
new file mode 100644
index 00000000..1f0f4a62
--- /dev/null
+++ b/news/3148.bugfix.rst
@@ -0,0 +1 @@
+Fixed resolution of direct-url dependencies in ``setup.py`` files to respect ``PEP-508`` style URL dependencies.
diff --git a/news/3148.feature.rst b/news/3148.feature.rst
new file mode 100644
index 00000000..e33434db
--- /dev/null
+++ b/news/3148.feature.rst
@@ -0,0 +1 @@
+Added support for resolution of direct-url dependencies in ``setup.py`` files to respect ``PEP-508`` style URL dependencies.
diff --git a/news/3292.trivial.rst b/news/3292.trivial.rst
new file mode 100644
index 00000000..9cab5de1
--- /dev/null
+++ b/news/3292.trivial.rst
@@ -0,0 +1 @@
+Update pytest-pypi documentation not to be pytest-httpbin documentation.
diff --git a/news/3298.bugfix.rst b/news/3298.bugfix.rst
new file mode 100644
index 00000000..aa378723
--- /dev/null
+++ b/news/3298.bugfix.rst
@@ -0,0 +1,3 @@
+Fixed a bug which caused failures in warning reporting when running pipenv inside a virtualenv under some circumstances.
+
+- Fixed a bug with package discovery when running ``pipenv clean``.
diff --git a/news/3298.feature.rst b/news/3298.feature.rst
new file mode 100644
index 00000000..65a49424
--- /dev/null
+++ b/news/3298.feature.rst
@@ -0,0 +1,5 @@
+Added full support for resolution of all dependency types including direct URLs, zip archives, tarballs, etc.
+
+- Improved error handling and formatting.
+
+- Introduced improved cross platform stream wrappers for better ``stdout`` and ``stderr`` consistency.
diff --git a/news/3298.vendor.rst b/news/3298.vendor.rst
new file mode 100644
index 00000000..cab9a50b
--- /dev/null
+++ b/news/3298.vendor.rst
@@ -0,0 +1,34 @@
+Updated vendored dependencies:
+
+ - **attrs**: ``18.2.0`` => ``19.1.0``
+ - **certifi**: ``2018.10.15`` => ``2019.3.9``
+ - **cached_property**: ``1.4.3`` => ``1.5.1``
+ - **cerberus**: ``1.2.0`` => ``1.3.1``
+ - **click-completion**: ``0.5.0`` => ``0.5.1``
+ - **colorama**: ``0.3.9`` => ``0.4.1``
+ - **distlib**: ``0.2.8`` => ``0.2.9``
+ - **idna**: ``2.7`` => ``2.8``
+ - **jinja2**: ``2.10.0`` => ``2.10.1``
+ - **markupsafe**: ``1.0`` => ``1.1.1``
+ - **orderedmultidict**: ``(new)`` => ``1.0``
+ - **packaging**: ``18.0`` => ``19.0``
+ - **parse**: ``1.9.0`` => ``1.12.0``
+ - **pathlib2**: ``2.3.2`` => ``2.3.3``
+ - **pep517**: ``(new)`` => ``0.5.0``
+ - **pexpect**: ``4.6.0`` => ``4.7.0``
+ - **pipdeptree**: ``0.13.0`` => ``0.13.2``
+ - **pyparsing**: ``2.2.2`` => ``2.3.1``
+ - **python-dotenv**: ``0.9.1`` => ``0.10.2``
+ - **pythonfinder**: ``1.1.10`` => ``1.2.1``
+ - **pytoml**: ``(new)`` => ``0.1.20``
+ - **requests**: ``2.20.1`` => ``2.21.0``
+ - **requirementslib**: ``1.3.3`` => ``1.5.0``
+ - **scandir**: ``1.9.0`` => ``1.10.0``
+ - **shellingham**: ``1.2.7`` => ``1.3.1``
+ - **six**: ``1.11.0`` => ``1.12.0``
+ - **tomlkit**: ``0.5.2`` => ``0.5.3``
+ - **urllib3**: ``1.24`` => ``1.25.2``
+ - **vistir**: ``0.3.0`` => ``0.4.1``
+ - **yaspin**: ``0.14.0`` => ``0.14.3``
+
+- Removed vendored dependency **cursor**.
diff --git a/news/3328.feature.rst b/news/3328.feature.rst
new file mode 100644
index 00000000..7e92d39f
--- /dev/null
+++ b/news/3328.feature.rst
@@ -0,0 +1 @@
+Pipenv will now successfully recursively lock VCS sub-dependencies.
diff --git a/news/3368.feature.rst b/news/3368.feature.rst
new file mode 100644
index 00000000..a998fce1
--- /dev/null
+++ b/news/3368.feature.rst
@@ -0,0 +1 @@
+Pipenv will now discover and resolve the intrinsic dependencies of **all** VCS dependencies, whether they are editable or not, to prevent resolution conflicts.
diff --git a/news/3386.behavior.rst b/news/3386.behavior.rst
new file mode 100644
index 00000000..8ddc27c6
--- /dev/null
+++ b/news/3386.behavior.rst
@@ -0,0 +1 @@
+Do not touch Pipfile early and rely on it so that one can do ``pipenv sync`` without a Pipfile.
diff --git a/news/3404.bugfix.rst b/news/3404.bugfix.rst
new file mode 100644
index 00000000..fa678d6a
--- /dev/null
+++ b/news/3404.bugfix.rst
@@ -0,0 +1 @@
+Fixed a keyerror which could occur when locking VCS dependencies in some cases.
diff --git a/news/3427.bugfix.rst b/news/3427.bugfix.rst
index 42f659f0..76aeb489 100644
--- a/news/3427.bugfix.rst
+++ b/news/3427.bugfix.rst
@@ -1 +1 @@
-Fix a bug that ``ValidationError`` is thrown when some fields are missing in source section.
+Fixed a bug that ``ValidationError`` is thrown when some fields are missing in source section.
diff --git a/news/3434.trivial.rst b/news/3434.trivial.rst
new file mode 100644
index 00000000..622b52db
--- /dev/null
+++ b/news/3434.trivial.rst
@@ -0,0 +1 @@
+Improve the error message when one tries to initialize a Pipenv project under ``/``.
diff --git a/news/3446.trivial.rst b/news/3446.trivial.rst
index 1eb98bbb..c3f6a000 100644
--- a/news/3446.trivial.rst
+++ b/news/3446.trivial.rst
@@ -1 +1 @@
-Fix the wrong order of old and new hashes in message.
+Fixed the wrong order of old and new hashes in message.
diff --git a/news/3449.bugfix.rst b/news/3449.bugfix.rst
index 72e8d228..4ed07046 100644
--- a/news/3449.bugfix.rst
+++ b/news/3449.bugfix.rst
@@ -1 +1 @@
-Update the index names in lock file when source name in Pipfile is changed.
+Updated the index names in lock file when source name in Pipfile is changed.
diff --git a/news/3499.doc.rst b/news/3499.doc.rst
new file mode 100644
index 00000000..d98b0f15
--- /dev/null
+++ b/news/3499.doc.rst
@@ -0,0 +1 @@
+Replace (non-existant) video on docs index.rst with equivalent gif.
diff --git a/news/3502.bugfix.rst b/news/3502.bugfix.rst
new file mode 100644
index 00000000..2700a740
--- /dev/null
+++ b/news/3502.bugfix.rst
@@ -0,0 +1 @@
+Fix bug causing ``[SSL: CERTIFICATE_VERIFY_FAILED]`` when Pipfile ``[[source]]`` has verify_ssl=false and url with custom port.
diff --git a/news/3522.doc.rst b/news/3522.doc.rst
new file mode 100644
index 00000000..3d71061f
--- /dev/null
+++ b/news/3522.doc.rst
@@ -0,0 +1 @@
+Clarify wording in Basic Usage example on using double quotes to escape shell redirection
diff --git a/news/3584.bugfix.rst b/news/3584.bugfix.rst
new file mode 100644
index 00000000..09684d1d
--- /dev/null
+++ b/news/3584.bugfix.rst
@@ -0,0 +1 @@
+Fix the issue that lock file can't be created when ``PIPENV_PIPFILE`` is not under working directory.
diff --git a/news/3595.feature.rst b/news/3595.feature.rst
new file mode 100644
index 00000000..30b755b9
--- /dev/null
+++ b/news/3595.feature.rst
@@ -0,0 +1 @@
+Added the ability for Windows users to enable emojis by setting ``PIPENV_HIDE_EMOJIS=0``.
diff --git a/news/3621.trivial.rst b/news/3621.trivial.rst
new file mode 100644
index 00000000..4d38a31e
--- /dev/null
+++ b/news/3621.trivial.rst
@@ -0,0 +1 @@
+Removed unused vendored package shutilwhich
diff --git a/news/3629.doc.rst b/news/3629.doc.rst
new file mode 100644
index 00000000..4d878c40
--- /dev/null
+++ b/news/3629.doc.rst
@@ -0,0 +1 @@
+Added a link to the TOML Spec under General Recommendations & Version Control to clarify how Pipfiles should be written.
diff --git a/news/3640.trivial.rst b/news/3640.trivial.rst
new file mode 100644
index 00000000..eb9b718d
--- /dev/null
+++ b/news/3640.trivial.rst
@@ -0,0 +1 @@
+Removed unused vendored package blindspin
diff --git a/news/3669.trivial.rst b/news/3669.trivial.rst
new file mode 100644
index 00000000..86ff9280
--- /dev/null
+++ b/news/3669.trivial.rst
@@ -0,0 +1 @@
+Allow KeyboardInterrupt to cancel test suite checks for working internet and ssh
diff --git a/news/3684.trivial.rst b/news/3684.trivial.rst
new file mode 100644
index 00000000..64561ec7
--- /dev/null
+++ b/news/3684.trivial.rst
@@ -0,0 +1 @@
+Cleaned up some conditional logic that would always evaluate ``True``.
diff --git a/news/3711.trivial.rst b/news/3711.trivial.rst
new file mode 100644
index 00000000..48c531b2
--- /dev/null
+++ b/news/3711.trivial.rst
@@ -0,0 +1 @@
+Add installation instructions for Debian Buster+ in README
diff --git a/news/3724.trivial.rst b/news/3724.trivial.rst
new file mode 100644
index 00000000..63a55013
--- /dev/null
+++ b/news/3724.trivial.rst
@@ -0,0 +1 @@
+Update pytest configuration to support pytest 4.
diff --git a/news/3745.bugfix.rst b/news/3745.bugfix.rst
new file mode 100644
index 00000000..229047a4
--- /dev/null
+++ b/news/3745.bugfix.rst
@@ -0,0 +1 @@
+Normalize the package names to lowercase when comparing used and in-Pipfile packages.
diff --git a/peeps/PEEP-0004.md b/peeps/PEEP-0004.md
new file mode 100644
index 00000000..aea8d00a
--- /dev/null
+++ b/peeps/PEEP-0004.md
@@ -0,0 +1,9 @@
+## PEEP-003: Subcommands
+
+NOT YET ACCEPTED
+
+Pipenv will automatically run commands like "pipenv deploy" if the "pipenv-deploy" binary is available on the path.
+
+These subcommands cannot overwrite built-in commands.
+
+These subcommands will receive environment variables with contextual information.
diff --git a/peeps/PEEP-005.md b/peeps/PEEP-005.md
new file mode 100644
index 00000000..2cc0279f
--- /dev/null
+++ b/peeps/PEEP-005.md
@@ -0,0 +1,65 @@
+# PEEP-005: Do Not Remove Entries from the Lockfile When Using `--keep-outdated`
+
+**PROPOSED**
+
+This PEEP describes a change that would retain entries in the Lockfile even if they were not returned during resolution when the user passes the `--keep-outdated` flag.
+
+☤
+
+The `--keep-outdated` flag is currently provided by Pipenv for the purpose of holding back outdated dependencies (i.e. dependencies that are not newly introduced). This proposal attempts to identify the reasoning behind the flag and identifies a need for a project-wide scoping. Finally, this proposal outlines the expected behavior of `--keep-outdated` under the specified circumstances, as well as the required changes to achieve full implementation.
+
+## Retaining Outdated Dependencies
+
+The purpose of retaining outdated dependencies is to allow the user to introduce a new package to their environment with a minimal impact on their existing environment. In an effort to achieve this, `keep_outdated` was proposed as both a flag and a Pipfile setting [in this issue](https://github.com/pypa/pipenv/issues/1255#issuecomment-354585775), originally described as follows:
+
+> pipenv lock --keep-outdated to request a minimal update that only adjusts the lock file to account for Pipfile changes (additions, removals, and changes to version constraints)... and pipenv install --keep-outdated needed to request only the minimal changes required to satisfy the installation request
+
+However, the current implementation always fully re-locks, rather than only locking the new dependencies. As a result, dependencies in the `Pipfile.lock` with markers for a python version different from that of the running interpreter will be removed, even if they have nothing to do with the current changeset. For instance, say you have the following dependency in your `Pipfile.lock`:
+
+```json
+{
+ "default": {
+ "backports.weakref": {
+ "hashes": [...],
+ "version": "==1.5",
+ "markers": "python_version<='3.4'"
+ }
+ }
+}
+```
+
+If this lockfile were to be re-generated with Python 3, even with `--keep-outdated`, this entry would be removed. This makes it very difficult to maintain lockfiles which are compatible across major python versions, yet all that would be required to correct this would be a tweak to the implementation of `keep-outdated`. I believe this was the goal to begin with, but I feel this behavior should be documented and clarified before moving forward.
+
+## Desired Behavior
+
+1. The only changes that should occur in `Pipfile.lock` when `--keep-outdated` is passed should be changes resulting from new packages added or pin changes in the project `Pipfile`;
+2. Existing packages in the project `Pipfile.lock` should remain in place, even if they are not returned during resolution;
+3. New dependencies should be written to the lockfile;
+4. Conflicts should be resolved as outlined below.
+
+## Conflict Resolution
+
+If a conflict should occur due to the presence in the `Pipfile.lock` of a dependency of a new package, the following steps should be undertaken before alerting the user:
+
+1. Determine whether the previously locked version of the dependency meets the constraints required of the new package; if so, pin that version;
+2. If the previously locked version is not present in the `Pipfile` and is not a dependency of any other dependencies (i.e. has no presence in `pipenv graph`, etc), update the lockfile with the new version;
+3. If there is a new or existing dependency which has a conflict with existing entries in the lockfile, perform an intermediate resolution step by checking:
+ a. If the new dependency can be satisfied by existing installs;
+ b. Whether conflicts can be upgraded without affecting locked dependencies;
+ c. If locked dependencies must be upgraded, whether those dependencies ultimately have any dependencies in the `Pipfile`;
+ d. If a traversal up the graph lands in the `Pipfile`, create _abstract dependencies_ from the `Pipfile` entries and determine whether they will still be satisfied by the new version;
+ e. If a new pin is required, ensure that any subdependencies of the newly pinned dependencies are therefore also re-pinned (simply prefer the updated lockfile instead of the cached version);
+
+4. Raise an Exception alerting the user that they either need to do a full lock or manually pin a version.
+
+## Necessary Changes
+
+In order to make these changes, we will need to modify the dependency resolution process. Overall, locking will require the following implementation changes:
+
+1. The ability to restore any entries that would otherwise be removed when the `--keep-outdated` flag is passed. The process already provides a caching mechanism, so we simply need to restore missing cache keys;
+2. Conflict resolution steps:
+ a. Check an abstract dependency/candidate against a lockfile entry;
+ b. Requirements mapping for each dependency in the environment to determine if a lockfile entry is a descendent of any other entries;
+
+
+Author: Dan Ryan
diff --git a/pipenv/__init__.py b/pipenv/__init__.py
index 7a12158a..695a4939 100644
--- a/pipenv/__init__.py
+++ b/pipenv/__init__.py
@@ -26,16 +26,6 @@ warnings.filterwarnings("ignore", category=DependencyWarning)
warnings.filterwarnings("ignore", category=ResourceWarning)
warnings.filterwarnings("ignore", category=UserWarning)
-if sys.version_info >= (3, 1) and sys.version_info <= (3, 6):
- if sys.stdout.isatty() and sys.stderr.isatty():
- import io
- import atexit
- stdout_wrapper = io.TextIOWrapper(sys.stdout.buffer, encoding='utf8')
- atexit.register(stdout_wrapper.close)
- stderr_wrapper = io.TextIOWrapper(sys.stderr.buffer, encoding='utf8')
- atexit.register(stderr_wrapper.close)
- sys.stdout = stdout_wrapper
- sys.stderr = stderr_wrapper
os.environ["PIP_DISABLE_PIP_VERSION_CHECK"] = fs_str("1")
@@ -46,6 +36,20 @@ try:
except Exception:
pass
+from pipenv.vendor.vistir.misc import get_text_stream
+stdout = get_text_stream("stdout")
+stderr = get_text_stream("stderr")
+
+if os.name == "nt":
+ from pipenv.vendor.vistir.misc import _can_use_color, _wrap_for_color
+ if _can_use_color(stdout):
+ stdout = _wrap_for_color(stdout)
+ if _can_use_color(stderr):
+ stderr = _wrap_for_color(stderr)
+
+sys.stdout = stdout
+sys.stderr = stderr
+
from .cli import cli
from . import resolver
diff --git a/pipenv/_compat.py b/pipenv/_compat.py
index ca3bbd0f..caba80fe 100644
--- a/pipenv/_compat.py
+++ b/pipenv/_compat.py
@@ -5,9 +5,6 @@ Exposes a standard API that enables compatibility across python versions,
operating systems, etc.
"""
-import functools
-import importlib
-import io
import os
import sys
import warnings
@@ -122,6 +119,12 @@ UNICODE_TO_ASCII_TRANSLATION_MAP = {
}
+def decode_for_output(output, target=sys.stdout):
+ return vistir.misc.decode_for_output(
+ output, sys.stdout, translation_map=UNICODE_TO_ASCII_TRANSLATION_MAP
+ )
+
+
def decode_output(output):
if not isinstance(output, six.string_types):
return output
diff --git a/pipenv/cli/command.py b/pipenv/cli/command.py
index ec1bef61..e51c98a6 100644
--- a/pipenv/cli/command.py
+++ b/pipenv/cli/command.py
@@ -12,8 +12,6 @@ import click_completion
import crayons
import delegator
-from click_didyoumean import DYMCommandCollection
-
from ..__version__ import __version__
from .options import (
CONTEXT_SETTINGS, PipenvGroup, code_option, common_options, deploy_option,
@@ -154,7 +152,11 @@ def cli(
# There is no virtualenv yet.
if not project.virtualenv_exists:
echo(
- crayons.red("No virtualenv has been created for this project yet!"),
+ "{}({}){}".format(
+ crayons.red("No virtualenv has been created for this project"),
+ crayons.white(project.project_directory, bold=True),
+ crayons.red(" yet!")
+ ),
err=True,
)
ctx.abort()
@@ -300,6 +302,7 @@ def uninstall(
if retcode:
sys.exit(retcode)
+
@cli.command(short_help="Generates Pipfile.lock.", context_settings=CONTEXT_SETTINGS)
@lock_options
@pass_state
@@ -400,7 +403,6 @@ def shell(
def run(state, command, args):
"""Spawns a command installed into the virtualenv."""
from ..core import do_run
-
do_run(
command=command, args=args, three=state.three, python=state.python, pypi_mirror=state.pypi_mirror
)
@@ -555,7 +557,7 @@ def run_open(state, module, *args, **kwargs):
EDITOR=atom pipenv open requests
"""
- from ..core import which, ensure_project
+ from ..core import which, ensure_project, inline_activate_virtual_environment
# Ensure that virtualenv is available.
ensure_project(
@@ -575,6 +577,7 @@ def run_open(state, module, *args, **kwargs):
else:
p = c.out.strip().rstrip("cdo")
echo(crayons.normal("Opening {0!r} in your EDITOR.".format(p), bold=True))
+ inline_activate_virtual_environment()
edit(filename=p)
return 0
@@ -629,11 +632,9 @@ def sync(
def clean(ctx, state, dry_run=False, bare=False, user=False):
"""Uninstalls all packages not specified in Pipfile.lock."""
from ..core import do_clean
- do_clean(ctx=ctx, three=state.three, python=state.python, dry_run=dry_run)
+ do_clean(ctx=ctx, three=state.three, python=state.python, dry_run=dry_run,
+ system=state.system)
-# Only invoke the "did you mean" when an argument wasn't passed (it breaks those).
-if "-" not in "".join(sys.argv) and len(sys.argv) > 1:
- cli = DYMCommandCollection(sources=[cli])
if __name__ == "__main__":
cli()
diff --git a/pipenv/cli/options.py b/pipenv/cli/options.py
index 745275dd..4233d5ee 100644
--- a/pipenv/cli/options.py
+++ b/pipenv/cli/options.py
@@ -8,6 +8,7 @@ import click.types
from click import (
BadParameter, Group, Option, argument, echo, make_pass_decorator, option
)
+from click_didyoumean import DYMMixin
from .. import environments
from ..utils import is_valid_url
@@ -19,7 +20,7 @@ CONTEXT_SETTINGS = {
}
-class PipenvGroup(Group):
+class PipenvGroup(DYMMixin, Group):
"""Custom Group class provides formatted main help"""
def get_help_option(self, ctx):
@@ -293,8 +294,9 @@ def code_option(f):
if value:
state.installstate.code = value
return value
- return option("--code", "-c", nargs=1, default=False, help="Import from codebase.",
- callback=callback, expose_value=False)(f)
+ return option("--code", "-c", nargs=1, default=False, help="Install packages "
+ "automatically discovered from import statements.", callback=callback,
+ expose_value=False)(f)
def deploy_option(f):
diff --git a/pipenv/cmdparse.py b/pipenv/cmdparse.py
index 21aba77e..2e550e22 100644
--- a/pipenv/cmdparse.py
+++ b/pipenv/cmdparse.py
@@ -10,7 +10,7 @@ class ScriptEmptyError(ValueError):
def _quote_if_contains(value, pattern):
- if next(re.finditer(pattern, value), None):
+ if next(iter(re.finditer(pattern, value)), None):
return '"{0}"'.format(re.sub(r'(\\*)"', r'\1\1\\"', value))
return value
diff --git a/pipenv/core.py b/pipenv/core.py
index 934f13b6..7f04ac49 100644
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -1,5 +1,5 @@
# -*- coding=utf-8 -*-
-
+from __future__ import absolute_import, print_function
import json as simplejson
import logging
import os
@@ -14,13 +14,13 @@ import urllib3.util as urllib3_util
import vistir
import click_completion
-import crayons
import delegator
import dotenv
import pipfile
+from .patched import crayons
from . import environments, exceptions, pep508checker, progress
-from ._compat import fix_utf8
+from ._compat import fix_utf8, decode_for_output
from .cmdparse import Script
from .environments import (
PIPENV_CACHE_DIR, PIPENV_COLORBLIND, PIPENV_DEFAULT_PYTHON_VERSION,
@@ -34,7 +34,8 @@ from .utils import (
escape_cmd, escape_grouped_arguments, find_windows_executable,
get_canonical_names, is_pinned, is_pypi_url, is_required_version, is_star,
is_valid_url, parse_indexes, pep423_name, prepare_pip_source_args,
- proper_case, python_version, venv_resolve_deps
+ proper_case, python_version, venv_resolve_deps, run_command,
+ is_python_command, find_python
)
@@ -86,16 +87,19 @@ def which(command, location=None, allow_global=False):
location = os.environ.get("VIRTUAL_ENV", None)
if not (location and os.path.exists(location)) and not allow_global:
raise RuntimeError("location not created nor specified")
+
+ version_str = "python{0}".format(".".join([str(v) for v in sys.version_info[:2]]))
+ is_python = command in ("python", os.path.basename(sys.executable), version_str)
if not allow_global:
if os.name == "nt":
p = find_windows_executable(os.path.join(location, "Scripts"), command)
else:
p = os.path.join(location, "bin", command)
else:
- if command == "python":
+ if is_python:
p = sys.executable
if not os.path.exists(p):
- if command == "python":
+ if is_python:
p = sys.executable or system_which("python")
else:
p = system_which(command)
@@ -238,7 +242,9 @@ def import_from_code(path="."):
rs = []
try:
- for r in pipreqs.get_all_imports(path):
+ for r in pipreqs.get_all_imports(
+ path, encoding="utf-8", extra_ignore_dirs=[".venv"]
+ ):
if r not in BAD_PACKAGES:
rs.append(r)
pkg_names = pipreqs.get_pkg_names(rs)
@@ -323,26 +329,16 @@ def find_a_system_python(line):
* Search for "python" and "pythonX.Y" executables in PATH to find a match.
* Nothing fits, return None.
"""
- if not line:
- return None
- if os.path.isabs(line):
- return line
- from .vendor.pythonfinder import Finder
+ from .vendor.pythonfinder import Finder
finder = Finder(system=False, global_search=True)
+ if not line:
+ return next(iter(finder.find_all_python_versions()), None)
+ # Use the windows finder executable
if (line.startswith("py ") or line.startswith("py.exe ")) and os.name == "nt":
line = line.split(" ", 1)[1].lstrip("-")
- elif line.startswith("py"):
- python_entry = finder.which(line)
- if python_entry:
- return python_entry.path.as_posix()
- return None
- python_entry = finder.find_python_version(line)
- if not python_entry:
- python_entry = finder.which("python{0}".format(line))
- if python_entry:
- return python_entry.path.as_posix()
- return None
+ python_entry = find_python(finder, line)
+ return python_entry
def ensure_python(three=None, python=None):
@@ -472,6 +468,8 @@ def ensure_virtualenv(three=None, python=None, site_packages=False, pypi_mirror=
ensure_environment()
# Ensure Python is available.
python = ensure_python(three=three, python=python)
+ if python is not None and not isinstance(python, six.string_types):
+ python = python.path.as_posix()
# Create the virtualenv.
# Abort if --system (or running in a virtualenv).
if PIPENV_USE_SYSTEM:
@@ -493,7 +491,10 @@ def ensure_virtualenv(three=None, python=None, site_packages=False, pypi_mirror=
elif (python) or (three is not None) or (site_packages is not False):
USING_DEFAULT_PYTHON = False
# Ensure python is installed before deleting existing virtual env
- ensure_python(three=three, python=python)
+ python = ensure_python(three=three, python=python)
+ if python is not None and not isinstance(python, six.string_types):
+ python = python.path.as_posix()
+
click.echo(crayons.red("Virtualenv already exists!"), err=True)
# If VIRTUAL_ENV is set, there is a possibility that we are
# going to remove the active virtualenv that the user cares
@@ -540,11 +541,16 @@ def ensure_project(
# Automatically use an activated virtualenv.
if PIPENV_USE_SYSTEM:
system = True
- if not project.pipfile_exists:
- if deploy is True:
- raise exceptions.PipfileNotFound
- else:
- project.touch_pipfile()
+ if not project.pipfile_exists and deploy:
+ raise exceptions.PipfileNotFound
+ # Fail if working under /
+ if not project.name:
+ click.echo(
+ "{0}: Pipenv is not intended to work under the root directory, "
+ "please choose another path.".format(crayons.red("ERROR")),
+ err=True
+ )
+ sys.exit(1)
# Skip virtualenv creation when --system was used.
if not system:
ensure_virtualenv(
@@ -607,24 +613,23 @@ def shorten_path(location, bold=False):
def do_where(virtualenv=False, bare=True):
"""Executes the where functionality."""
if not virtualenv:
- location = project.pipfile_location
- # Shorten the virtual display of the path to the virtualenv.
- if not bare:
- location = shorten_path(location)
- if not location:
+ if not project.pipfile_exists:
click.echo(
"No Pipfile present at project home. Consider running "
"{0} first to automatically generate a Pipfile for you."
"".format(crayons.green("`pipenv install`")),
err=True,
)
- elif not bare:
+ return
+ location = project.pipfile_location
+ # Shorten the virtual display of the path to the virtualenv.
+ if not bare:
+ location = shorten_path(location)
click.echo(
"Pipfile found at {0}.\n Considering this to be the project home."
"".format(crayons.green(location)),
err=True,
)
- pass
else:
click.echo(project.project_directory)
else:
@@ -675,10 +680,10 @@ def batch_install(deps_list, procs, failed_deps_queue,
requirements_dir, no_deps=False, ignore_hashes=False,
allow_global=False, blocking=False, pypi_mirror=None,
nprocs=PIPENV_MAX_SUBPROCESS, retry=True):
-
+ from .vendor.requirementslib.models.utils import strip_extras_markers_from_requirement
failed = (not retry)
if not failed:
- label = INSTALL_LABEL if os.name != "nt" else ""
+ label = INSTALL_LABEL if not PIPENV_HIDE_EMOJIS else ""
else:
label = INSTALL_LABEL2
@@ -690,6 +695,10 @@ def batch_install(deps_list, procs, failed_deps_queue,
trusted_hosts = []
# Install these because
for dep in deps_list_bar:
+ if dep.req.req:
+ dep.req.req = strip_extras_markers_from_requirement(dep.req.req)
+ if dep.markers:
+ dep.markers = str(strip_extras_markers_from_requirement(dep.get_markers()))
index = None
if dep.index:
index = project.find_source(dep.index)
@@ -698,10 +707,18 @@ def batch_install(deps_list, procs, failed_deps_queue,
trusted_hosts.append(urllib3_util.parse_url(index.get("url")).host)
# Install the module.
is_artifact = False
+ if no_deps:
+ link = getattr(dep.req, "link", None)
+ is_wheel = False
+ if link:
+ is_wheel = link.is_wheel
if dep.is_file_or_url and (dep.is_direct_url or any(
dep.req.uri.endswith(ext) for ext in ["zip", "tar.gz"]
)):
is_artifact = True
+ elif dep.is_vcs:
+ is_artifact = True
+ needs_deps = not no_deps if no_deps is True else is_artifact
extra_indexes = []
if not index and indexes:
@@ -714,27 +731,25 @@ def batch_install(deps_list, procs, failed_deps_queue,
os.environ["PIP_USER"] = vistir.compat.fs_str("0")
if "PYTHONHOME" in os.environ:
del os.environ["PYTHONHOME"]
- if no_deps:
+ if not needs_deps:
link = getattr(dep.req, "link", None)
is_wheel = False
if link:
is_wheel = link.is_wheel
- is_non_editable_vcs = (dep.is_vcs and not dep.editable)
- no_deps = not (dep.is_file_or_url and not (is_wheel or dep.editable))
+ needs_deps = dep.is_file_or_url and not (is_wheel or dep.editable)
c = pip_install(
dep,
ignore_hashes=any([ignore_hashes, dep.editable, dep.is_vcs]),
allow_global=allow_global,
- no_deps=no_deps,
+ no_deps=not needs_deps,
block=any([dep.editable, dep.is_vcs, blocking]),
index=index,
requirements_dir=requirements_dir,
pypi_mirror=pypi_mirror,
trusted_hosts=trusted_hosts,
- extra_indexes=extra_indexes
+ extra_indexes=extra_indexes,
+ use_pep517=not failed,
)
- if dep.is_vcs or dep.editable:
- c.block()
if procs.qsize() < nprocs:
c.dep = dep
procs.put(c)
@@ -814,6 +829,7 @@ def do_install_dependencies(
else:
install_kwargs["nprocs"] = 1
+ # with project.environment.activated():
batch_install(
deps_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
)
@@ -859,6 +875,7 @@ def convert_three_to_python(three, python):
def do_create_virtualenv(python=None, site_packages=False, pypi_mirror=None):
"""Creates a virtualenv."""
+
click.echo(
crayons.normal(fix_utf8("Creating a virtualenv for this project…"), bold=True), err=True
)
@@ -902,20 +919,21 @@ def do_create_virtualenv(python=None, site_packages=False, pypi_mirror=None):
pip_config = {}
# Actually create the virtualenv.
- nospin = environments.PIPENV_NOSPIN
- with create_spinner("Creating virtual environment...") as sp:
+ with create_spinner(u"Creating virtual environment...") as sp:
c = vistir.misc.run(
cmd, verbose=False, return_object=True, write_to_stdout=False,
combine_stderr=False, block=True, nospin=True, env=pip_config,
)
- click.echo(crayons.blue("{0}".format(c.out)), err=True)
+ click.echo(crayons.blue(u"{0}".format(c.out)), err=True)
if c.returncode != 0:
- sp.fail(environments.PIPENV_SPINNER_FAIL_TEXT.format("Failed creating virtual environment"))
+ sp.fail(environments.PIPENV_SPINNER_FAIL_TEXT.format(u"Failed creating virtual environment"))
+ error = c.err if environments.is_verbose() else exceptions.prettify_exc(c.err)
raise exceptions.VirtualenvCreationException(
- extra=[crayons.blue("{0}".format(c.err)),]
+ extra=crayons.red("{0}".format(error))
)
else:
- sp.green.ok(environments.PIPENV_SPINNER_OK_TEXT.format("Successfully created virtual environment!"))
+
+ sp.green.ok(environments.PIPENV_SPINNER_OK_TEXT.format(u"Successfully created virtual environment!"))
# Associate project directory with the environment.
# This mimics Pew's "setproject".
@@ -1016,9 +1034,12 @@ def do_lock(
dev_packages = overwrite_dev(project.packages, dev_packages)
# Resolve dev-package dependencies, with pip-tools.
for is_dev in [True, False]:
- pipfile_section = "dev_packages" if is_dev else "packages"
+ pipfile_section = "dev-packages" if is_dev else "packages"
lockfile_section = "develop" if is_dev else "default"
- packages = getattr(project, pipfile_section)
+ if project.pipfile_exists:
+ packages = project.parsed_pipfile.get(pipfile_section, {})
+ else:
+ packages = getattr(project, pipfile_section.replace("-", "_"))
if write:
# Alert the user of progress.
@@ -1031,12 +1052,9 @@ def do_lock(
err=True,
)
- deps = convert_deps_to_pip(
- packages, project, r=False, include_index=True
- )
# Mutates the lockfile
venv_resolve_deps(
- deps,
+ packages,
which=which,
project=project,
dev=is_dev,
@@ -1045,7 +1063,8 @@ def do_lock(
allow_global=system,
pypi_mirror=pypi_mirror,
pipfile=packages,
- lockfile=lockfile
+ lockfile=lockfile,
+ keep_outdated=keep_outdated
)
# Support for --keep-outdated…
@@ -1062,6 +1081,12 @@ def do_lock(
lockfile[section_name][canonical_name] = cached_lockfile[
section_name
][canonical_name].copy()
+ for key in ["default", "develop"]:
+ packages = set(cached_lockfile[key].keys())
+ new_lockfile = set(lockfile[key].keys())
+ missing = packages - new_lockfile
+ for missing_pkg in missing:
+ lockfile[key][missing_pkg] = cached_lockfile[key][missing_pkg].copy()
# Overwrite any develop packages with default packages.
lockfile["develop"].update(overwrite_dev(lockfile.get("default", {}), lockfile["develop"]))
if write:
@@ -1140,12 +1165,19 @@ def do_init(
pypi_mirror=None,
):
"""Executes the init functionality."""
- from .environments import PIPENV_VIRTUALENV
+ from .environments import (
+ PIPENV_VIRTUALENV, PIPENV_DEFAULT_PYTHON_VERSION, PIPENV_PYTHON, PIPENV_USE_SYSTEM
+ )
+ python = None
+ if PIPENV_PYTHON is not None:
+ python = PIPENV_PYTHON
+ elif PIPENV_DEFAULT_PYTHON_VERSION is not None:
+ python = PIPENV_DEFAULT_PYTHON_VERSION
- if not system:
+ if not system and not PIPENV_USE_SYSTEM:
if not project.virtualenv_exists:
try:
- do_create_virtualenv(pypi_mirror=pypi_mirror)
+ do_create_virtualenv(python=python, three=None, pypi_mirror=pypi_mirror)
except KeyboardInterrupt:
cleanup_virtualenv(bare=False)
sys.exit(1)
@@ -1257,12 +1289,12 @@ def pip_install(
requirements_dir=None,
extra_indexes=None,
pypi_mirror=None,
- trusted_hosts=None
+ trusted_hosts=None,
+ use_pep517=True
):
from pipenv.patched.notpip._internal import logger as piplogger
- from .utils import Mapping
+ from .vendor.vistir.compat import Mapping
from .vendor.urllib3.util import parse_url
-
src = []
write_to_tmpfile = False
if requirement:
@@ -1280,6 +1312,10 @@ def pip_install(
crayons.normal("Installing {0!r}".format(requirement.name), bold=True),
err=True,
)
+
+ if requirement:
+ ignore_hashes = True if not requirement.hashes else ignore_hashes
+
# Create files for hash mode.
if write_to_tmpfile:
if not requirements_dir:
@@ -1289,12 +1325,18 @@ def pip_install(
prefix="pipenv-", suffix="-requirement.txt", dir=requirements_dir,
delete=False
)
- f.write(vistir.misc.to_bytes(requirement.as_line()))
+ line = requirement.as_line(include_hashes=not ignore_hashes)
+ if environments.is_verbose():
+ click.echo(
+ "Writing requirement line to temporary file: {0!r}".format(line),
+ err=True
+ )
+ f.write(vistir.misc.to_bytes(line))
r = f.name
f.close()
- # Install dependencies when a package is a VCS dependency.
+
if requirement and requirement.vcs:
- no_deps = False
+ # Install dependencies when a package is a non-editable VCS dependency.
# Don't specify a source directory when using --system.
if not allow_global and ("PIP_SRC" not in os.environ):
src.extend(["--src", "{0}".format(project.virtualenv_src_location)])
@@ -1340,25 +1382,98 @@ def pip_install(
create_mirror_source(pypi_mirror) if is_pypi_url(source["url"]) else source
for source in sources
]
+
+ line_kwargs = {"as_list": True, "include_hashes": not ignore_hashes}
+
+ # Install dependencies when a package is a VCS dependency.
+ if requirement and requirement.vcs:
+ ignore_hashes = True
+ # Don't specify a source directory when using --system.
+ src_dir = None
+ if "PIP_SRC" in os.environ:
+ src_dir = os.environ["PIP_SRC"]
+ src = ["--src", os.environ["PIP_SRC"]]
+ if not requirement.editable:
+ no_deps = False
+
+ if src_dir is not None:
+ if environments.is_verbose():
+ click.echo("Using source directory: {0!r}".format(src_dir))
+ repo = requirement.req.get_vcs_repo(src_dir=src_dir)
+ else:
+ repo = requirement.req.get_vcs_repo()
+ write_to_tmpfile = True
+ line_kwargs["include_markers"] = False
+ line_kwargs["include_hashes"] = False
+ if not requirements_dir:
+ requirements_dir = vistir.path.create_tracked_tempdir(prefix="pipenv",
+ suffix="requirements")
+ f = vistir.compat.NamedTemporaryFile(
+ prefix="pipenv-", suffix="-requirement.txt", dir=requirements_dir,
+ delete=False
+ )
+ line = "-e" if requirement.editable else ""
+ if requirement.editable or requirement.name is not None:
+ name = requirement.name
+ if requirement.extras:
+ name = "{0}{1}".format(name, requirement.extras_as_pip)
+ line = "{0}{1}#egg={2}".format(
+ line, vistir.path.path_to_url(repo.checkout_directory), requirement.name
+ )
+ if repo.subdirectory:
+ line = "{0}&subdirectory={1}".format(line, repo.subdirectory)
+ else:
+ line = requirement.as_line(**line_kwargs)
+ if environments.is_verbose():
+ click.echo(
+ "Writing requirement line to temporary file: {0!r}".format(line),
+ err=True
+ )
+ f.write(vistir.misc.to_bytes(line))
+ r = f.name
+ f.close()
+
+ # Create files for hash mode.
+ if write_to_tmpfile and not r:
+ if not requirements_dir:
+ requirements_dir = vistir.path.create_tracked_tempdir(
+ prefix="pipenv", suffix="requirements")
+ f = vistir.compat.NamedTemporaryFile(
+ prefix="pipenv-", suffix="-requirement.txt", dir=requirements_dir,
+ delete=False
+ )
+ ignore_hashes = True if not requirement.hashes else ignore_hashes
+ line = requirement.as_line(include_hashes=not ignore_hashes)
+ line = "{0} {1}".format(line, " ".join(src))
+ if environments.is_verbose():
+ click.echo(
+ "Writing requirement line to temporary file: {0!r}".format(line),
+ err=True
+ )
+ f.write(vistir.misc.to_bytes(line))
+ r = f.name
+ f.close()
+
if (requirement and requirement.editable) and not r:
- line_kwargs = {"as_list": True}
- if requirement.markers:
- line_kwargs["include_markers"] = False
+ line_kwargs["include_markers"] = False
+ line_kwargs["include_hashes"] = False
install_reqs = requirement.as_line(**line_kwargs)
if requirement.editable and install_reqs[0].startswith("-e "):
req, install_reqs = install_reqs[0], install_reqs[1:]
+ possible_hashes = install_reqs[:]
editable_opt, req = req.split(" ", 1)
install_reqs = [editable_opt, req] + install_reqs
- if not all(item.startswith("--hash") for item in install_reqs):
- ignore_hashes = True
+
+ # hashes must be passed via a file
+ ignore_hashes = True
elif r:
install_reqs = ["-r", r]
with open(r) as f:
if "--hash" not in f.read():
ignore_hashes = True
else:
- ignore_hashes = True if not requirement.hashes else False
- install_reqs = requirement.as_line(as_list=True)
+ ignore_hashes = True if not requirement.hashes else ignore_hashes
+ install_reqs = requirement.as_line(as_list=True, include_hashes=not ignore_hashes)
if not requirement.markers:
install_reqs = [escape_cmd(r) for r in install_reqs]
elif len(install_reqs) > 1:
@@ -1379,7 +1494,11 @@ def pip_install(
pip_command.extend(prepare_pip_source_args(sources))
if not ignore_hashes:
pip_command.append("--require-hashes")
-
+ if not use_pep517:
+ from .vendor.packaging.version import parse as parse_version
+ pip_command.append("--no-build-isolation")
+ if project.environment.pip_version >= parse_version("19.0"):
+ pip_command.append("--no-use-pep517")
if environments.is_verbose():
click.echo("$ {0}".format(pip_command), err=True)
cache_dir = vistir.compat.Path(PIPENV_CACHE_DIR)
@@ -1398,6 +1517,8 @@ def pip_install(
)
cmd = Script.parse(pip_command)
pip_command = cmd.cmdify()
+ c = None
+ # with project.environment.activated():
c = delegator.run(pip_command, block=block, env=pip_config)
return c
@@ -1425,18 +1546,61 @@ def pip_download(package_name):
return c
+def fallback_which(command, location=None, allow_global=False, system=False):
+ """
+ A fallback implementation of the `which` utility command that relies exclusively on
+ searching the path for commands.
+
+ :param str command: The command to search for, optional
+ :param str location: The search location to prioritize (prepend to path), defaults to None
+ :param bool allow_global: Whether to search the global path, defaults to False
+ :param bool system: Whether to use the system python instead of pipenv's python, defaults to False
+ :raises ValueError: Raised if no command is provided
+ :raises TypeError: Raised if the command provided is not a string
+ :return: A path to the discovered command location
+ :rtype: str
+ """
+
+ from .vendor.pythonfinder import Finder
+ if not command:
+ raise ValueError("fallback_which: Must provide a command to search for...")
+ if not isinstance(command, six.string_types):
+ raise TypeError("Provided command must be a string, received {0!r}".format(command))
+ global_search = system or allow_global
+ if location is None:
+ global_search = True
+ finder = Finder(system=False, global_search=global_search, path=location)
+ if is_python_command(command):
+ result = find_python(finder, command)
+ if result:
+ return result
+ result = finder.which(command)
+ if result:
+ return result.path.as_posix()
+ return ""
+
+
def which_pip(allow_global=False):
"""Returns the location of virtualenv-installed pip."""
+
+ location = None
+ if "VIRTUAL_ENV" in os.environ:
+ location = os.environ["VIRTUAL_ENV"]
if allow_global:
- if "VIRTUAL_ENV" in os.environ:
- return which("pip", location=os.environ["VIRTUAL_ENV"])
+ if location:
+ pip = which("pip", location=location)
+ if pip:
+ return pip
for p in ("pip", "pip3", "pip2"):
where = system_which(p)
if where:
return where
- return which("pip")
+ pip = which("pip")
+ if not pip:
+ pip = fallback_which("pip", allow_global=allow_global, location=location)
+ return pip
def system_which(command, mult=False):
@@ -1446,6 +1610,7 @@ def system_which(command, mult=False):
vistir.compat.fs_str(k): vistir.compat.fs_str(val)
for k, val in os.environ.items()
}
+ result = None
try:
c = delegator.run("{0} {1}".format(_which, command))
try:
@@ -1460,21 +1625,20 @@ def system_which(command, mult=False):
)
assert c.return_code == 0
except AssertionError:
- return None if not mult else []
+ result = fallback_which(command, allow_global=True)
except TypeError:
- from .vendor.pythonfinder import Finder
- finder = Finder()
- result = finder.which(command)
- if result:
- return result.path.as_posix()
- return
+ if not result:
+ result = fallback_which(command, allow_global=True)
else:
- result = c.out.strip() or c.err.strip()
- if mult:
- return result.split("\n")
+ if not result:
+ result = next(iter([c.out, c.err]), "").split("\n")
+ result = next(iter(result)) if not mult else result
+ return result
+ if not result:
+ result = fallback_which(command, allow_global=True)
+ result = [result] if mult else result
+ return result
- else:
- return result.split("\n")[0]
def format_help(help):
@@ -1856,7 +2020,7 @@ def do_install(
# This is for if the user passed in dependencies, then we want to make sure we
else:
- from .vendor.requirementslib import Requirement
+ from .vendor.requirementslib.models.requirements import Requirement
# make a tuple of (display_name, entry)
pkg_list = packages + ["-e {0}".format(pkg) for pkg in editable_packages]
@@ -1872,6 +2036,7 @@ def do_install(
pypi_mirror=pypi_mirror,
skip_lock=skip_lock,
)
+ pip_shims_module = os.environ.pop("PIP_SHIMS_BASE_MODULE", None)
for pkg_line in pkg_list:
click.echo(
crayons.normal(
@@ -1907,6 +2072,15 @@ def do_install(
pypi_mirror=pypi_mirror,
)
if not c.ok:
+ sp.write_err(vistir.compat.fs_str(
+ "{0}: {1}".format(
+ crayons.red("WARNING"),
+ "Failed installing package {0}".format(pkg_line)
+ ),
+ ))
+ sp.write_err(vistir.compat.fs_str(
+ "Error text: {0}".format(c.out)
+ ))
raise RuntimeError(c.err)
except (ValueError, RuntimeError) as e:
sp.write_err(vistir.compat.fs_str(
@@ -1954,15 +2128,25 @@ def do_install(
crayons.normal(fix_utf8("…"), bold=True),
)
))
+ # Add the package to the Pipfile.
+ try:
+ project.add_package_to_pipfile(pkg_requirement, dev)
+ except ValueError as e:
+ import traceback
+ sp.write_err(
+ "{0} {1}".format(
+ crayons.red("Error:", bold=True), traceback.format_exc()
+ )
+ )
+ sp.fail(environments.PIPENV_SPINNER_FAIL_TEXT.format(
+ "Failed adding package to Pipfile"
+ ))
sp.ok(environments.PIPENV_SPINNER_OK_TEXT.format("Installation Succeeded"))
- # Add the package to the Pipfile.
- try:
- project.add_package_to_pipfile(pkg_requirement, dev)
- except ValueError as e:
- raise exceptions.PipfileException(e)
# Update project settings with pre preference.
if pre:
project.update_settings({"allow_prereleases": pre})
+ if pip_shims_module:
+ os.environ["PIP_SHIMS_BASE_MODULE"] = pip_shims_module
do_init(
dev=dev,
system=system,
@@ -2074,6 +2258,7 @@ def do_uninstall(
p for normalized, p in selected_pkg_map.items()
if normalized in (used_packages - bad_pkgs)
]
+ pip_path = None
for normalized, package_name in selected_pkg_map.items():
click.echo(
crayons.white(
@@ -2083,12 +2268,10 @@ def do_uninstall(
# Uninstall the package.
if package_name in packages_to_remove:
with project.environment.activated():
- cmd = "{0} uninstall {1} -y".format(
- escape_grouped_arguments(which_pip(allow_global=system)), package_name,
- )
- if environments.is_verbose():
- click.echo("$ {0}".format(cmd))
- c = delegator.run(cmd)
+ if pip_path is None:
+ pip_path = which_pip(allow_global=system)
+ cmd = [pip_path, "uninstall", package_name, "-y"]
+ c = run_command(cmd)
click.echo(crayons.blue(c.out))
if c.return_code != 0:
failure = True
@@ -2158,7 +2341,7 @@ def do_shell(three=None, python=False, fancy=False, shell_args=None, pypi_mirror
# Only set PIPENV_ACTIVE after finishing reading virtualenv_location
# otherwise its value will be changed
os.environ["PIPENV_ACTIVE"] = vistir.misc.fs_str("1")
-
+
os.environ.pop("PIP_SHIMS_BASE_MODULE", None)
if fancy:
@@ -2297,6 +2480,8 @@ def do_run(command, args, three=None, python=False, pypi_mirror=None):
load_dot_env()
+ previous_pip_shims_module = os.environ.pop("PIP_SHIMS_BASE_MODULE", None)
+
# Activate virtualenv under the current interpreter's environment
inline_activate_virtual_environment()
@@ -2304,10 +2489,9 @@ def do_run(command, args, three=None, python=False, pypi_mirror=None):
# Only set PIPENV_ACTIVE after finishing reading virtualenv_location
# such as in inline_activate_virtual_environment
# otherwise its value will be changed
+ previous_pipenv_active_value = os.environ.get("PIPENV_ACTIVE")
os.environ["PIPENV_ACTIVE"] = vistir.misc.fs_str("1")
- os.environ.pop("PIP_SHIMS_BASE_MODULE", None)
-
try:
script = project.build_script(command, args)
cmd_string = ' '.join([script.command] + script.args)
@@ -2315,10 +2499,21 @@ def do_run(command, args, three=None, python=False, pypi_mirror=None):
click.echo(crayons.normal("$ {0}".format(cmd_string)), err=True)
except ScriptEmptyError:
click.echo("Can't run script {0!r}-it's empty?", err=True)
+ run_args = [script]
+ run_kwargs = {}
if os.name == "nt":
- do_run_nt(script)
+ run_fn = do_run_nt
else:
- do_run_posix(script, command=command)
+ run_fn = do_run_posix
+ run_kwargs = {"command": command}
+ try:
+ run_fn(*run_args, **run_kwargs)
+ finally:
+ os.environ.pop("PIPENV_ACTIVE", None)
+ if previous_pipenv_active_value is not None:
+ os.environ["PIPENV_ACTIVE"] = previous_pipenv_active_value
+ if previous_pip_shims_module is not None:
+ os.environ["PIP_SHIMS_BASE_MODULE"] = previous_pip_shims_module
def do_check(
@@ -2330,6 +2525,8 @@ def do_check(
args=None,
pypi_mirror=None,
):
+ from .environments import is_verbose
+ from pipenv.vendor.vistir.compat import JSONDecodeError
if not system:
# Ensure that virtualenv is available.
ensure_project(
@@ -2342,8 +2539,8 @@ def do_check(
if not args:
args = []
if unused:
- deps_required = [k for k in project.packages.keys()]
- deps_needed = import_from_code(unused)
+ deps_required = [k.lower() for k in project.packages.keys()]
+ deps_needed = [k.lower() for k in import_from_code(unused)]
for dep in deps_needed:
try:
deps_required.remove(dep)
@@ -2360,18 +2557,29 @@ def do_check(
sys.exit(1)
else:
sys.exit(0)
- click.echo(crayons.normal(fix_utf8("Checking PEP 508 requirements…"), bold=True))
- if system:
- python = system_which("python")
- else:
- python = which("python")
- # Run the PEP 508 checker in the virtualenv.
- c = delegator.run(
- '"{0}" {1}'.format(
- python, escape_grouped_arguments(pep508checker.__file__.rstrip("cdo"))
- )
+ click.echo(crayons.normal(decode_for_output("Checking PEP 508 requirements…"), bold=True))
+ pep508checker_path = pep508checker.__file__.rstrip("cdo")
+ safety_path = os.path.join(
+ os.path.dirname(os.path.abspath(__file__)), "patched", "safety.zip"
)
- results = simplejson.loads(c.out)
+ if not system:
+ python = which("python")
+ else:
+ python = system_which("python")
+ _cmd = [vistir.compat.Path(python).as_posix()]
+ # Run the PEP 508 checker in the virtualenv.
+ cmd = _cmd + [vistir.compat.Path(pep508checker_path).as_posix()]
+ c = run_command(cmd)
+ if c.return_code is not None:
+ try:
+ results = simplejson.loads(c.out.strip())
+ except JSONDecodeError:
+ click.echo("{0}\n{1}\n{2}".format(
+ crayons.white(decode_for_output("Failed parsing pep508 results: "), bold=True),
+ c.out.strip(),
+ c.err.strip()
+ ))
+ sys.exit(1)
# Load the pipfile.
p = pipfile.Pipfile.load(project.pipfile_location)
failed = False
@@ -2396,15 +2604,13 @@ def do_check(
sys.exit(1)
else:
click.echo(crayons.green("Passed!"))
- click.echo(crayons.normal(fix_utf8("Checking installed package safety…"), bold=True))
- path = pep508checker.__file__.rstrip("cdo")
- path = os.sep.join(__file__.split(os.sep)[:-1] + ["patched", "safety.zip"])
- if not system:
- python = which("python")
- else:
- python = system_which("python")
+ click.echo(crayons.normal(
+ decode_for_output("Checking installed package safety…"), bold=True)
+ )
if ignore:
- ignored = "--ignore {0}".format(" --ignore ".join(ignore))
+ if not isinstance(ignore, (tuple, list)):
+ ignore = [ignore]
+ ignored = [["--ignore", cve] for cve in ignore]
click.echo(
crayons.normal(
"Notice: Ignoring CVE(s) {0}".format(crayons.yellow(", ".join(ignore)))
@@ -2413,17 +2619,21 @@ def do_check(
)
else:
ignored = ""
- c = delegator.run(
- '"{0}" {1} check --json --key={2} {3}'.format(
- python, escape_grouped_arguments(path), PIPENV_PYUP_API_KEY, ignored
- )
- )
+ key = "--key={0}".format(PIPENV_PYUP_API_KEY)
+ cmd = _cmd + [safety_path, "check", "--json", key]
+ if ignored:
+ for cve in ignored:
+ cmd += cve
+ c = run_command(cmd, catch_exceptions=False)
try:
results = simplejson.loads(c.out)
- except ValueError:
- click.echo("An error occurred:", err=True)
- click.echo(c.err if len(c.err) > 0 else c.out, err=True)
- sys.exit(1)
+ except (ValueError, JSONDecodeError):
+ raise exceptions.JSONParseError(c.out, c.err)
+ except Exception:
+ raise exceptions.PipenvCmdError(c.cmd, c.out, c.err, c.return_code)
+ if c.ok:
+ click.echo(crayons.green("All good!"))
+ sys.exit(0)
for (package, resolved, installed, description, vuln) in results:
click.echo(
"{0}: {1} {2} resolved ({3} installed)!".format(
@@ -2435,14 +2645,14 @@ def do_check(
)
click.echo("{0}".format(description))
click.echo()
- if not results:
- click.echo(crayons.green("All good!"))
else:
sys.exit(1)
def do_graph(bare=False, json=False, json_tree=False, reverse=False):
+ from pipenv.vendor.vistir.compat import JSONDecodeError
import pipdeptree
+ pipdeptree_path = pipdeptree.__file__.rstrip("cdo")
try:
python_path = which("python")
except AttributeError:
@@ -2457,6 +2667,9 @@ def do_graph(bare=False, json=False, json_tree=False, reverse=False):
sys.exit(1)
except RuntimeError:
pass
+ else:
+ python_path = vistir.compat.Path(python_path).as_posix()
+ pipdeptree_path = vistir.compat.Path(pipdeptree_path).as_posix()
if reverse and json:
click.echo(
@@ -2507,17 +2720,20 @@ def do_graph(bare=False, json=False, json_tree=False, reverse=False):
err=True,
)
sys.exit(1)
- cmd = '"{0}" {1} {2} -l'.format(
- python_path, escape_grouped_arguments(pipdeptree.__file__.rstrip("cdo")), flag
- )
+ cmd_args = [python_path, pipdeptree_path, flag, "-l"]
+ c = run_command(cmd_args)
# Run dep-tree.
- c = delegator.run(cmd)
if not bare:
if json:
data = []
- for d in simplejson.loads(c.out):
- if d["package"]["key"] not in BAD_PACKAGES:
- data.append(d)
+ try:
+ parsed = simplejson.loads(c.out.strip())
+ except JSONDecodeError:
+ raise exceptions.JSONParseError(c.out, c.err)
+ else:
+ for d in parsed:
+ if d["package"]["key"] not in BAD_PACKAGES:
+ data.append(d)
click.echo(simplejson.dumps(data, indent=4))
sys.exit(0)
elif json_tree:
@@ -2533,12 +2749,18 @@ def do_graph(bare=False, json=False, json_tree=False, reverse=False):
obj["dependencies"] = traverse(obj["dependencies"])
return obj
- data = traverse(simplejson.loads(c.out))
- click.echo(simplejson.dumps(data, indent=4))
- sys.exit(0)
+ try:
+ parsed = simplejson.loads(c.out.strip())
+ except JSONDecodeError:
+ raise exceptions.JSONParseError(c.out, c.err)
+ else:
+ data = traverse(parsed)
+ click.echo(simplejson.dumps(data, indent=4))
+ sys.exit(0)
else:
for line in c.out.strip().split("\n"):
# Ignore bad packages as top level.
+ # TODO: This should probably be a "==" in + line.partition
if line.split("==")[0] in BAD_PACKAGES and not reverse:
continue
@@ -2607,34 +2829,32 @@ def do_sync(
click.echo(crayons.green("All dependencies are now up-to-date!"))
-def do_clean(ctx, three=None, python=None, dry_run=False, bare=False, pypi_mirror=None):
+def do_clean(
+ ctx, three=None, python=None, dry_run=False, bare=False, pypi_mirror=None,
+ system=False
+):
# Ensure that virtualenv is available.
from packaging.utils import canonicalize_name
ensure_project(three=three, python=python, validate=False, pypi_mirror=pypi_mirror)
ensure_lockfile(pypi_mirror=pypi_mirror)
# Make sure that the virtualenv's site packages are configured correctly
# otherwise we may end up removing from the global site packages directory
- installed_package_names = [
- canonicalize_name(pkg.project_name) for pkg
- in project.environment.get_installed_packages()
- ]
+ installed_package_names = project.installed_package_names.copy()
# Remove known "bad packages" from the list.
for bad_package in BAD_PACKAGES:
if canonicalize_name(bad_package) in installed_package_names:
if environments.is_verbose():
click.echo("Ignoring {0}.".format(bad_package), err=True)
- del installed_package_names[installed_package_names.index(
- canonicalize_name(bad_package)
- )]
+ installed_package_names.remove(canonicalize_name(bad_package))
# Intelligently detect if --dev should be used or not.
- develop = [canonicalize_name(k) for k in project.lockfile_content["develop"].keys()]
- default = [canonicalize_name(k) for k in project.lockfile_content["default"].keys()]
- for used_package in set(develop + default):
+ locked_packages = {
+ canonicalize_name(pkg) for pkg in project.lockfile_package_names["combined"]
+ }
+ for used_package in locked_packages:
if used_package in installed_package_names:
- del installed_package_names[installed_package_names.index(
- canonicalize_name(used_package)
- )]
+ installed_package_names.remove(used_package)
failure = False
+ cmd = [which_pip(allow_global=system), "uninstall", "-y", "-qq"]
for apparent_bad_package in installed_package_names:
if dry_run and not bare:
click.echo(apparent_bad_package)
@@ -2646,9 +2866,8 @@ def do_clean(ctx, three=None, python=None, dry_run=False, bare=False, pypi_mirro
)
)
# Uninstall the package.
- c = delegator.run(
- "{0} uninstall {1} -y".format(which_pip(), apparent_bad_package)
- )
+ cmd = [which_pip(), "uninstall", apparent_bad_package, "-y"]
+ c = run_command(cmd)
if c.return_code != 0:
failure = True
sys.exit(int(failure))
diff --git a/pipenv/environment.py b/pipenv/environment.py
index 06757938..49cad91d 100644
--- a/pipenv/environment.py
+++ b/pipenv/environment.py
@@ -9,17 +9,18 @@ import site
import sys
from distutils.sysconfig import get_python_lib
-from sysconfig import get_paths
+from sysconfig import get_paths, get_python_version
+import itertools
import pkg_resources
import six
-import vistir
import pipenv
-from cached_property import cached_property
+from .vendor.cached_property import cached_property
+import vistir
-from .utils import normalize_path
+from .utils import normalize_path, make_posix
BASE_WORKING_SET = pkg_resources.WorkingSet(sys.path)
@@ -92,19 +93,27 @@ class Environment(object):
deps |= cls.resolve_dist(dist, working_set)
return deps
- def add_dist(self, dist_name):
- dist = pkg_resources.get_distribution(pkg_resources.Requirement(dist_name))
+ def extend_dists(self, dist):
extras = self.resolve_dist(dist, self.base_working_set)
+ self.extra_dists.append(dist)
if extras:
self.extra_dists.extend(extras)
+ def add_dist(self, dist_name):
+ dist = pkg_resources.get_distribution(pkg_resources.Requirement(dist_name))
+ self.extend_dists(dist)
+
@cached_property
def python_version(self):
with self.activated():
- from sysconfig import get_python_version
- py_version = get_python_version()
+ sysconfig = self.safe_import("sysconfig")
+ py_version = sysconfig.get_python_version()
return py_version
+ def find_libdir(self):
+ libdir = self.prefix / "lib"
+ return next(iter(list(libdir.iterdir())), None)
+
@property
def python_info(self):
include_dir = self.prefix / "include"
@@ -115,6 +124,16 @@ class Environment(object):
return {"py_version_short": py_version_short, "abiflags": abiflags}
return {}
+ def _replace_parent_version(self, path, replace_version):
+ if not os.path.exists(path):
+ base, leaf = os.path.split(path)
+ base, parent = os.path.split(base)
+ leaf = os.path.join(parent, leaf).replace(
+ replace_version, self.python_info.get("py_version_short", get_python_version())
+ )
+ return os.path.join(base, leaf)
+ return path
+
@cached_property
def base_paths(self):
"""
@@ -145,50 +164,61 @@ class Environment(object):
'stdlib': '/home/hawk/.pyenv/versions/3.7.1/lib/python3.7'}
"""
- prefix = self.prefix.as_posix()
+ prefix = make_posix(self.prefix.as_posix())
install_scheme = 'nt' if (os.name == 'nt') else 'posix_prefix'
paths = get_paths(install_scheme, vars={
'base': prefix,
'platbase': prefix,
})
+ current_version = get_python_version()
+ for k in list(paths.keys()):
+ if not os.path.exists(paths[k]):
+ paths[k] = self._replace_parent_version(paths[k], current_version)
+ if not os.path.exists(paths["purelib"]) and not os.path.exists(paths["platlib"]):
+ paths = self.get_paths()
paths["PATH"] = paths["scripts"] + os.pathsep + os.defpath
if "prefix" not in paths:
paths["prefix"] = prefix
- purelib = get_python_lib(plat_specific=0, prefix=prefix)
- platlib = get_python_lib(plat_specific=1, prefix=prefix)
+ purelib = paths["purelib"] = make_posix(paths["purelib"])
+ platlib = paths["platlib"] = make_posix(paths["platlib"])
if purelib == platlib:
lib_dirs = purelib
else:
lib_dirs = purelib + os.pathsep + platlib
paths["libdir"] = purelib
- paths["purelib"] = purelib
- paths["platlib"] = platlib
- paths['PYTHONPATH'] = lib_dirs
+ paths['PYTHONPATH'] = os.pathsep.join(["", ".", lib_dirs])
paths["libdirs"] = lib_dirs
return paths
@cached_property
def script_basedir(self):
"""Path to the environment scripts dir"""
- script_dir = self.base_paths["scripts"]
- return script_dir
+ prefix = make_posix(self.prefix.as_posix())
+ install_scheme = 'nt' if (os.name == 'nt') else 'posix_prefix'
+ paths = get_paths(install_scheme, vars={
+ 'base': prefix,
+ 'platbase': prefix,
+ })
+ return paths["scripts"]
@property
def python(self):
"""Path to the environment python"""
- py = vistir.compat.Path(self.base_paths["scripts"]).joinpath("python").as_posix()
+ py = vistir.compat.Path(self.script_basedir).joinpath("python").absolute().as_posix()
if not py:
return vistir.compat.Path(sys.executable).as_posix()
return py
@cached_property
def sys_path(self):
- """The system path inside the environment
+ """
+ The system path inside the environment
:return: The :data:`sys.path` from the environment
:rtype: list
"""
+ from .vendor.vistir.compat import JSONDecodeError
current_executable = vistir.compat.Path(sys.executable).as_posix()
if not self.python or self.python == current_executable:
return sys.path
@@ -196,18 +226,46 @@ class Environment(object):
return sys.path
cmd_args = [self.python, "-c", "import json, sys; print(json.dumps(sys.path))"]
path, _ = vistir.misc.run(cmd_args, return_object=False, nospin=True, block=True, combine_stderr=False, write_to_stdout=False)
- path = json.loads(path.strip())
+ try:
+ path = json.loads(path.strip())
+ except JSONDecodeError:
+ path = sys.path
return path
+ def get_paths(self):
+ """
+ Get the paths for the environment by running a subcommand
+
+ :return: The python paths for the environment
+ :rtype: Dict[str, str]
+ """
+ prefix = make_posix(self.prefix.as_posix())
+ install_scheme = 'nt' if (os.name == 'nt') else 'posix_prefix'
+ py_command = (
+ "import sysconfig, json, distutils.sysconfig;"
+ "paths = sysconfig.get_paths('{0}', vars={{'base': '{1}', 'platbase': '{1}'}}"
+ ");paths['purelib'] = distutils.sysconfig.get_python_lib(plat_specific=0, "
+ "prefix='{1}');paths['platlib'] = distutils.sysconfig.get_python_lib("
+ "plat_specific=1, prefix='{1}');print(json.dumps(paths))"
+ )
+ vistir.misc.echo("command: {0}".format(py_command.format(install_scheme, prefix)), fg="white", style="bold", err=True)
+ command = [self.python, "-c", py_command.format(install_scheme, prefix)]
+ c = vistir.misc.run(
+ command, return_object=True, block=True, nospin=True, write_to_stdout=False
+ )
+ paths = json.loads(vistir.misc.to_text(c.out.strip()))
+ return paths
+
@cached_property
def sys_prefix(self):
- """The prefix run inside the context of the environment
+ """
+ The prefix run inside the context of the environment
:return: The python prefix inside the environment
:rtype: :data:`sys.prefix`
"""
- command = [self.python, "-c" "import sys; print(sys.prefix)"]
+ command = [self.python, "-c", "import sys; print(sys.prefix)"]
c = vistir.misc.run(command, return_object=True, block=True, nospin=True, write_to_stdout=False)
sys_prefix = vistir.compat.Path(vistir.misc.to_text(c.out).strip()).as_posix()
return sys_prefix
@@ -236,15 +294,33 @@ class Environment(object):
return "purelib", purelib
return "platlib", self.paths["platlib"]
+ @property
+ def pip_version(self):
+ """
+ Get the pip version in the environment. Useful for knowing which args we can use
+ when installing.
+ """
+ from .vendor.packaging.version import parse as parse_version
+ pip = next(iter(
+ pkg for pkg in self.get_installed_packages() if pkg.key == "pip"
+ ), None)
+ if pip is not None:
+ return parse_version(pip.version)
+ return parse_version("18.0")
+
def get_distributions(self):
- """Retrives the distributions installed on the library path of the environment
+ """
+ Retrives the distributions installed on the library path of the environment
:return: A set of distributions found on the library path
:rtype: iterator
"""
pkg_resources = self.safe_import("pkg_resources")
- return pkg_resources.find_distributions(self.paths["PYTHONPATH"])
+ libdirs = self.base_paths["libdirs"].split(os.pathsep)
+ dists = (pkg_resources.find_distributions(libdir) for libdir in libdirs)
+ for dist in itertools.chain.from_iterable(dists):
+ yield dist
def find_egg(self, egg_dist):
"""Find an egg by name in the given environment"""
@@ -271,21 +347,28 @@ class Environment(object):
def dist_is_in_project(self, dist):
"""Determine whether the supplied distribution is in the environment."""
from .project import _normalized
- prefix = _normalized(self.base_paths["prefix"])
+ prefixes = [
+ _normalized(prefix) for prefix in self.base_paths["libdirs"].split(os.pathsep)
+ if _normalized(prefix).startswith(_normalized(self.prefix.as_posix()))
+ ]
location = self.locate_dist(dist)
if not location:
return False
- return _normalized(location).startswith(prefix)
+ location = _normalized(make_posix(location))
+ return any(location.startswith(prefix) for prefix in prefixes)
def get_installed_packages(self):
"""Returns all of the installed packages in a given environment"""
workingset = self.get_working_set()
- packages = [pkg for pkg in workingset if self.dist_is_in_project(pkg)]
+ packages = [
+ pkg for pkg in workingset
+ if self.dist_is_in_project(pkg) and pkg.key != "python"
+ ]
return packages
@contextlib.contextmanager
def get_finder(self, pre=False):
- from .vendor.pip_shims import Command, cmdoptions, index_group, PackageFinder
+ from .vendor.pip_shims.shims import Command, cmdoptions, index_group, PackageFinder
from .environments import PIPENV_CACHE_DIR
index_urls = [source.get("url") for source in self.sources]
@@ -353,33 +436,86 @@ class Environment(object):
if pkg.latest_version._version > pkg.parsed_version._version
]
- def get_package_requirements(self):
- from .vendor.pipdeptree import flatten, sorted_tree, build_dist_index, construct_tree
- dist_index = build_dist_index(self.get_installed_packages())
- tree = sorted_tree(construct_tree(dist_index))
- branch_keys = set(r.key for r in flatten(tree.values()))
- nodes = [p for p in tree.keys() if p.key not in branch_keys]
- key_tree = dict((k.key, v) for k, v in tree.items())
+ @classmethod
+ def _get_requirements_for_package(cls, node, key_tree, parent=None, chain=None):
+ if chain is None:
+ chain = [node.project_name]
+
+ d = node.as_dict()
+ if parent:
+ d['required_version'] = node.version_spec if node.version_spec else 'Any'
+ else:
+ d['required_version'] = d['installed_version']
+
get_children = lambda n: key_tree.get(n.key, [])
- def aux(node, parent=None, chain=None):
- if chain is None:
- chain = [node.project_name]
+ d['dependencies'] = [
+ cls._get_requirements_for_package(c, key_tree, parent=node,
+ chain=chain+[c.project_name])
+ for c in get_children(node)
+ if c.project_name not in chain
+ ]
- d = node.as_dict()
- if parent:
- d['required_version'] = node.version_spec if node.version_spec else 'Any'
- else:
- d['required_version'] = d['installed_version']
+ return d
- d['dependencies'] = [
- aux(c, parent=node, chain=chain+[c.project_name])
- for c in get_children(node)
- if c.project_name not in chain
- ]
+ def get_package_requirements(self, pkg=None):
+ from .vendor.pipdeptree import flatten, sorted_tree, build_dist_index, construct_tree
+ packages = self.get_installed_packages()
+ if pkg:
+ packages = [p for p in packages if p.key == pkg]
+ dist_index = build_dist_index(packages)
+ tree = sorted_tree(construct_tree(dist_index))
+ branch_keys = set(r.key for r in flatten(tree.values()))
+ if pkg is not None:
+ nodes = [p for p in tree.keys() if p.key == pkg]
+ else:
+ nodes = [p for p in tree.keys() if p.key not in branch_keys]
+ key_tree = dict((k.key, v) for k, v in tree.items())
- return d
- return [aux(p) for p in nodes]
+ return [self._get_requirements_for_package(p, key_tree) for p in nodes]
+
+ @classmethod
+ def reverse_dependency(cls, node):
+ new_node = {
+ "package_name": node["package_name"],
+ "installed_version": node["installed_version"],
+ "required_version": node["required_version"]
+ }
+ for dependency in node.get("dependencies", []):
+ for dep in cls.reverse_dependency(dependency):
+ new_dep = dep.copy()
+ new_dep["parent"] = (node["package_name"], node["installed_version"])
+ yield new_dep
+ yield new_node
+
+ def reverse_dependencies(self):
+ from vistir.misc import unnest, chunked
+ rdeps = {}
+ for req in self.get_package_requirements():
+ for d in self.reverse_dependency(req):
+ parents = None
+ name = d["package_name"]
+ pkg = {
+ name: {
+ "installed": d["installed_version"],
+ "required": d["required_version"]
+ }
+ }
+ parents = tuple(d.get("parent", ()))
+ pkg[name]["parents"] = parents
+ if rdeps.get(name):
+ if not (rdeps[name].get("required") or rdeps[name].get("installed")):
+ rdeps[name].update(pkg[name])
+ rdeps[name]["parents"] = rdeps[name].get("parents", ()) + parents
+ else:
+ rdeps[name] = pkg[name]
+ for k in list(rdeps.keys()):
+ entry = rdeps[k]
+ if entry.get("parents"):
+ rdeps[k]["parents"] = set([
+ p for p, version in chunked(2, unnest(entry["parents"]))
+ ])
+ return rdeps
def get_working_set(self):
"""Retrieve the working set of installed packages for the environment.
@@ -475,21 +611,34 @@ class Environment(object):
vendor_dir = parent_path.joinpath("vendor").as_posix()
patched_dir = parent_path.joinpath("patched").as_posix()
parent_path = parent_path.as_posix()
+ self.add_dist("pip")
prefix = self.prefix.as_posix()
with vistir.contextmanagers.temp_environ(), vistir.contextmanagers.temp_path():
os.environ["PATH"] = os.pathsep.join([
- vistir.compat.fs_str(self.scripts_dir),
+ vistir.compat.fs_str(self.script_basedir),
vistir.compat.fs_str(self.prefix.as_posix()),
os.environ.get("PATH", "")
])
os.environ["PYTHONIOENCODING"] = vistir.compat.fs_str("utf-8")
os.environ["PYTHONDONTWRITEBYTECODE"] = vistir.compat.fs_str("1")
- os.environ["PYTHONPATH"] = self.base_paths["PYTHONPATH"]
+ from .environments import PIPENV_USE_SYSTEM
if self.is_venv:
+ os.environ["PYTHONPATH"] = self.base_paths["PYTHONPATH"]
os.environ["VIRTUAL_ENV"] = vistir.compat.fs_str(prefix)
+ else:
+ if not PIPENV_USE_SYSTEM and not os.environ.get("VIRTUAL_ENV"):
+ os.environ["PYTHONPATH"] = self.base_paths["PYTHONPATH"]
+ os.environ.pop("PYTHONHOME", None)
sys.path = self.sys_path
sys.prefix = self.sys_prefix
site.addsitedir(self.base_paths["purelib"])
+ pip = self.safe_import("pip")
+ pip_vendor = self.safe_import("pip._vendor")
+ pep517_dir = os.path.join(os.path.dirname(pip_vendor.__file__), "pep517")
+ site.addsitedir(pep517_dir)
+ os.environ["PYTHONPATH"] = os.pathsep.join([
+ os.environ.get("PYTHONPATH", self.base_paths["PYTHONPATH"]), pep517_dir
+ ])
if include_extras:
site.addsitedir(parent_path)
sys.path.extend([parent_path, patched_dir, vendor_dir])
@@ -593,10 +742,7 @@ class Environment(object):
monkey_patch.activate()
pip_shims = self.safe_import("pip_shims")
pathset_base = pip_shims.UninstallPathSet
- import recursive_monkey_patch
- recursive_monkey_patch.monkey_patch(
- PatchedUninstaller, pathset_base
- )
+ pathset_base._permitted = PatchedUninstaller._permitted
dist = next(
iter(filter(lambda d: d.project_name == pkgname, self.get_working_set())),
None
diff --git a/pipenv/environments.py b/pipenv/environments.py
index 445e3b63..34aef2bc 100644
--- a/pipenv/environments.py
+++ b/pipenv/environments.py
@@ -3,16 +3,27 @@
import os
import sys
+from io import UnsupportedOperation
+
from appdirs import user_cache_dir
from ._compat import fix_utf8
-from .vendor.vistir.misc import fs_str
+from .vendor.vistir.misc import _isatty, fs_str
# HACK: avoid resolver.py uses the wrong byte code files.
# I hope I can remove this one day.
os.environ["PYTHONDONTWRITEBYTECODE"] = fs_str("1")
+
+def _is_env_truthy(name):
+ """An environment variable is truthy if it exists and isn't one of (0, false, no, off)
+ """
+ if name not in os.environ:
+ return False
+ return os.environ.get(name).lower() not in ("0", "false", "no", "off")
+
+
PIPENV_IS_CI = bool("CI" in os.environ or "TF_BUILD" in os.environ)
# HACK: Prevent invalid shebangs with Homebrew-installed Python:
@@ -70,13 +81,15 @@ Default is to detect emulators automatically. This should be set if your
emulator, e.g. Cmder, cannot be detected correctly.
"""
-PIPENV_HIDE_EMOJIS = bool(os.environ.get("PIPENV_HIDE_EMOJIS"))
+PIPENV_HIDE_EMOJIS = (
+ os.environ.get("PIPENV_HIDE_EMOJIS") is None
+ and (os.name == "nt" or PIPENV_IS_CI)
+ or _is_env_truthy("PIPENV_HIDE_EMOJIS")
+)
"""Disable emojis in output.
Default is to show emojis. This is automatically set on Windows.
"""
-if os.name == "nt" or PIPENV_IS_CI:
- PIPENV_HIDE_EMOJIS = True
PIPENV_IGNORE_VIRTUALENVS = bool(os.environ.get("PIPENV_IGNORE_VIRTUALENVS"))
"""If set, Pipenv will always assign a virtual environment for this project.
@@ -252,7 +265,10 @@ PIPENV_SHELL = (
)
# Internal, to tell whether the command line session is interactive.
-SESSION_IS_INTERACTIVE = bool(os.isatty(sys.stdout.fileno()))
+try:
+ SESSION_IS_INTERACTIVE = _isatty(sys.stdout.fileno())
+except UnsupportedOperation:
+ SESSION_IS_INTERACTIVE = _isatty(sys.stdout)
# Internal, consolidated verbosity representation as an integer. The default
@@ -280,13 +296,35 @@ def is_quiet(threshold=-1):
def is_in_virtualenv():
- pipenv_active = os.environ.get("PIPENV_ACTIVE")
- virtual_env = os.environ.get("VIRTUAL_ENV")
- return (PIPENV_USE_SYSTEM or virtual_env) and not (
- pipenv_active or PIPENV_IGNORE_VIRTUALENVS
- )
+ """
+ Check virtualenv membership dynamically
+
+ :return: True or false depending on whether we are in a regular virtualenv or not
+ :rtype: bool
+ """
+
+ pipenv_active = os.environ.get("PIPENV_ACTIVE", False)
+ virtual_env = None
+ use_system = False
+ ignore_virtualenvs = bool(os.environ.get("PIPENV_IGNORE_VIRTUALENVS", False))
+
+ if not pipenv_active and not ignore_virtualenvs:
+ virtual_env = os.environ.get("VIRTUAL_ENV")
+ use_system = bool(virtual_env)
+ return (use_system or virtual_env) and not (pipenv_active or ignore_virtualenvs)
PIPENV_SPINNER_FAIL_TEXT = fix_utf8(u"✘ {0}") if not PIPENV_HIDE_EMOJIS else ("{0}")
PIPENV_SPINNER_OK_TEXT = fix_utf8(u"✔ {0}") if not PIPENV_HIDE_EMOJIS else ("{0}")
+
+
+def is_type_checking():
+ try:
+ from typing import TYPE_CHECKING
+ except ImportError:
+ return False
+ return TYPE_CHECKING
+
+
+MYPY_RUNNING = is_type_checking()
diff --git a/pipenv/exceptions.py b/pipenv/exceptions.py
index 0da42fe7..c8ca0dc8 100644
--- a/pipenv/exceptions.py
+++ b/pipenv/exceptions.py
@@ -1,42 +1,67 @@
# -*- coding=utf-8 -*-
import itertools
+import re
import sys
-from pprint import pformat
-from traceback import format_exception
+from collections import namedtuple
+from traceback import format_tb
import six
from . import environments
-from ._compat import fix_utf8
+from ._compat import decode_for_output
from .patched import crayons
-from .vendor.click._compat import get_text_stderr
from .vendor.click.exceptions import (
- Abort, BadOptionUsage, BadParameter, ClickException, Exit, FileError,
- MissingParameter, UsageError
+ ClickException, FileError, UsageError
)
-from .vendor.click.types import Path
-from .vendor.click.utils import echo as click_echo
+from .vendor.vistir.misc import echo as click_echo
+import vistir
+
+ANSI_REMOVAL_RE = re.compile(r"\033\[((?:\d|;)*)([a-zA-Z])", re.MULTILINE)
+STRING_TYPES = (six.string_types, crayons.ColoredString)
+
+if sys.version_info[:2] >= (3, 7):
+ KnownException = namedtuple(
+ 'KnownException', ['exception_name', 'match_string', 'show_from_string', 'prefix'],
+ defaults=[None, None, None, ""]
+ )
+else:
+ KnownException = namedtuple(
+ 'KnownException', ['exception_name', 'match_string', 'show_from_string', 'prefix'],
+ )
+ KnownException.__new__.__defaults__ = (None, None, None, "")
+
+KNOWN_EXCEPTIONS = [
+ KnownException("PermissionError", prefix="Permission Denied:"),
+ KnownException(
+ "VirtualenvCreationException",
+ match_string="do_create_virtualenv",
+ show_from_string=None
+ )
+]
def handle_exception(exc_type, exception, traceback, hook=sys.excepthook):
if environments.is_verbose() or not issubclass(exc_type, ClickException):
hook(exc_type, exception, traceback)
else:
- exc = format_exception(exc_type, exception, traceback)
- lines = itertools.chain.from_iterable([l.splitlines() for l in exc])
- lines = list(lines)[-11:-1]
+ tb = format_tb(traceback, limit=-6)
+ lines = itertools.chain.from_iterable([frame.splitlines() for frame in tb])
+ formatted_lines = []
for line in lines:
line = line.strip("'").strip('"').strip("\n").strip()
if not line.startswith("File"):
line = " {0}".format(line)
else:
line = " {0}".format(line)
- line = "[pipenv.exceptions.{0!s}]: {1}".format(
+ line = "[{0!s}]: {1}".format(
exception.__class__.__name__, line
)
- click_echo(fix_utf8(line), err=True)
+ formatted_lines.append(line)
+ # use new exception prettification rules to format exceptions according to
+ # UX rules
+ click_echo(decode_for_output(prettify_exc("\n".join(formatted_lines))), err=True)
exception.show()
@@ -56,16 +81,65 @@ class PipenvException(ClickException):
def show(self, file=None):
if file is None:
- file = get_text_stderr()
+ file = vistir.misc.get_text_stderr()
if self.extra:
- if isinstance(self.extra, six.string_types):
+ if isinstance(self.extra, STRING_TYPES):
self.extra = [self.extra,]
for extra in self.extra:
extra = "[pipenv.exceptions.{0!s}]: {1}".format(
self.__class__.__name__, extra
)
+ extra = decode_for_output(extra, file)
click_echo(extra, file=file)
- click_echo(fix_utf8("{0}".format(self.message)), file=file)
+ click_echo(decode_for_output("{0}".format(self.message), file), file=file)
+
+
+class PipenvCmdError(PipenvException):
+ def __init__(self, cmd, out="", err="", exit_code=1):
+ self.cmd = cmd
+ self.out = out
+ self.err = err
+ self.exit_code = exit_code
+ message = "Error running command: {0}".format(cmd)
+ PipenvException.__init__(self, message)
+
+ def show(self, file=None):
+ if file is None:
+ file = vistir.misc.get_text_stderr()
+ click_echo("{0} {1}".format(
+ crayons.red("Error running command: "),
+ crayons.white(decode_for_output("$ {0}".format(self.cmd), file), bold=True)
+ ), err=True)
+ if self.out:
+ click_echo("{0} {1}".format(
+ crayons.white("OUTPUT: "),
+ decode_for_output(self.out, file)
+ ), err=True)
+ if self.err:
+ click_echo("{0} {1}".format(
+ crayons.white("STDERR: "),
+ decode_for_output(self.err, file)
+ ), err=True)
+
+
+class JSONParseError(PipenvException):
+ def __init__(self, contents="", error_text=""):
+ self.error_text = error_text
+ PipenvException.__init__(self, contents)
+
+ def show(self, file=None):
+ if file is None:
+ file = vistir.misc.get_text_stderr()
+ message = "{0}\n{1}".format(
+ crayons.white("Failed parsing JSON results:", bold=True),
+ decode_for_output(self.message.strip(), file)
+ )
+ click_echo(message, err=True)
+ if self.error_text:
+ click_echo("{0} {1}".format(
+ crayons.white("ERROR TEXT:", bold=True),
+ decode_for_output(self.error_text, file)
+ ), err=True)
class PipenvUsageError(UsageError):
@@ -78,22 +152,22 @@ class PipenvUsageError(UsageError):
message = formatted_message.format(msg_prefix, crayons.white(message, bold=True))
self.message = message
extra = kwargs.pop("extra", [])
- UsageError.__init__(self, fix_utf8(message), ctx)
+ UsageError.__init__(self, decode_for_output(message), ctx)
self.extra = extra
def show(self, file=None):
if file is None:
- file = get_text_stderr()
+ file = vistir.misc.get_text_stderr()
color = None
if self.ctx is not None:
color = self.ctx.color
if self.extra:
- if isinstance(self.extra, six.string_types):
+ if isinstance(self.extra, STRING_TYPES):
self.extra = [self.extra,]
for extra in self.extra:
if color:
extra = getattr(crayons, color, "blue")(extra)
- click_echo(fix_utf8(extra), file=file)
+ click_echo(decode_for_output(extra, file), file=file)
hint = ''
if (self.cmd is not None and
self.cmd.get_help_option(self.ctx) is not None):
@@ -117,30 +191,33 @@ class PipenvFileError(FileError):
crayons.white("{0} not found!".format(filename), bold=True),
message
)
- FileError.__init__(self, filename=filename, hint=fix_utf8(message), **kwargs)
+ FileError.__init__(self, filename=filename, hint=decode_for_output(message), **kwargs)
self.extra = extra
def show(self, file=None):
if file is None:
- file = get_text_stderr()
+ file = vistir.misc.get_text_stderr()
if self.extra:
- if isinstance(self.extra, six.string_types):
+ if isinstance(self.extra, STRING_TYPES):
self.extra = [self.extra,]
for extra in self.extra:
- click_echo(fix_utf8(extra), file=file)
+ click_echo(decode_for_output(extra, file), file=file)
click_echo(self.message, file=file)
class PipfileNotFound(PipenvFileError):
def __init__(self, filename="Pipfile", extra=None, **kwargs):
extra = kwargs.pop("extra", [])
- message = ("{0} {1}".format(
+ message = (
+ "{0} {1}".format(
crayons.red("Aborting!", bold=True),
- crayons.white("Please ensure that the file exists and is located in your"
- " project root directory.", bold=True)
+ crayons.white(
+ "Please ensure that the file exists and is located in your"
+ " project root directory.", bold=True
+ )
)
)
- super(PipfileNotFound, self).__init__(filename, message=fix_utf8(message), extra=extra, **kwargs)
+ super(PipfileNotFound, self).__init__(filename, message=message, extra=extra, **kwargs)
class LockfileNotFound(PipenvFileError):
@@ -151,7 +228,7 @@ class LockfileNotFound(PipenvFileError):
crayons.red("$ pipenv lock", bold=True),
crayons.white("before you can continue.", bold=True)
)
- super(LockfileNotFound, self).__init__(filename, message=fix_utf8(message), extra=extra, **kwargs)
+ super(LockfileNotFound, self).__init__(filename, message=message, extra=extra, **kwargs)
class DeployException(PipenvUsageError):
@@ -159,13 +236,13 @@ class DeployException(PipenvUsageError):
if not message:
message = crayons.normal("Aborting deploy", bold=True)
extra = kwargs.pop("extra", [])
- PipenvUsageError.__init__(self, message=fix_utf8(message), extra=extra, **kwargs)
+ PipenvUsageError.__init__(self, message=message, extra=extra, **kwargs)
class PipenvOptionsError(PipenvUsageError):
def __init__(self, option_name, message=None, ctx=None, **kwargs):
extra = kwargs.pop("extra", [])
- PipenvUsageError.__init__(self, message=fix_utf8(message), ctx=ctx, **kwargs)
+ PipenvUsageError.__init__(self, message=message, ctx=ctx, **kwargs)
self.extra = extra
self.option_name = option_name
@@ -179,7 +256,8 @@ class SystemUsageError(PipenvOptionsError):
crayons.red("Warning", bold=True)
),
]
- message = crayons.blue("See also: {0}".format(crayons.white("-deploy flag.")))
+ if message is None:
+ message = crayons.blue("See also: {0}".format(crayons.white("--deploy flag.")))
super(SystemUsageError, self).__init__(option_name, message=message, ctx=ctx, extra=extra, **kwargs)
@@ -191,7 +269,7 @@ class PipfileException(PipenvFileError):
hint = "{0} {1}".format(crayons.red("ERROR (PACKAGE NOT INSTALLED):"), hint)
filename = project.pipfile_location
extra = kwargs.pop("extra", [])
- PipenvFileError.__init__(self, filename, fix_utf8(hint), extra=extra, **kwargs)
+ PipenvFileError.__init__(self, filename, hint, extra=extra, **kwargs)
class SetupException(PipenvException):
@@ -207,7 +285,7 @@ class VirtualenvException(PipenvException):
"There was an unexpected error while activating your virtualenv. "
"Continuing anyway..."
)
- PipenvException.__init__(self, fix_utf8(message), **kwargs)
+ PipenvException.__init__(self, message, **kwargs)
class VirtualenvActivationException(VirtualenvException):
@@ -218,7 +296,7 @@ class VirtualenvActivationException(VirtualenvException):
"not activated. Continuing anyway…"
)
self.message = message
- VirtualenvException.__init__(self, fix_utf8(message), **kwargs)
+ VirtualenvException.__init__(self, message, **kwargs)
class VirtualenvCreationException(VirtualenvException):
@@ -226,45 +304,70 @@ class VirtualenvCreationException(VirtualenvException):
if not message:
message = "Failed to create virtual environment."
self.message = message
- VirtualenvException.__init__(self, fix_utf8(message), **kwargs)
+ extra = kwargs.pop("extra", None)
+ if extra is not None and isinstance(extra, STRING_TYPES):
+ # note we need the format interpolation because ``crayons.ColoredString``
+ # is not an actual string type but is only a preparation for interpolation
+ # so replacement or parsing requires this step
+ extra = ANSI_REMOVAL_RE.sub("", "{0}".format(extra))
+ if "KeyboardInterrupt" in extra:
+ extra = crayons.red("Virtualenv creation interrupted by user", bold=True)
+ self.extra = extra = [extra,]
+ VirtualenvException.__init__(self, message, extra=extra)
class UninstallError(PipenvException):
def __init__(self, package, command, return_values, return_code, **kwargs):
- extra = [crayons.blue("Attempted to run command: {0}".format(
- crayons.yellow("$ {0}".format(command), bold=True)
- )),]
+ extra = [
+ "{0} {1}".format(
+ crayons.blue("Attempted to run command: "),
+ crayons.yellow("$ {0!r}".format(command), bold=True
+ )
+ )]
extra.extend([crayons.blue(line.strip()) for line in return_values.splitlines()])
- if isinstance(package, (tuple, list)):
+ if isinstance(package, (tuple, list, set)):
package = " ".join(package)
- message = "{0} {1}...".format(
+ message = "{0!s} {1!s}...".format(
crayons.normal("Failed to uninstall package(s)"),
- crayons.yellow(package, bold=True)
+ crayons.yellow("{0}!s".format(package), bold=True)
)
self.exit_code = return_code
- PipenvException.__init__(self, message=fix_utf8(message), extra=extra)
+ PipenvException.__init__(self, message=message, extra=extra)
self.extra = extra
class InstallError(PipenvException):
def __init__(self, package, **kwargs):
+ package_message = ""
+ if package is not None:
+ package_message = "Couldn't install package: {0}\n".format(
+ crayons.white("{0!s}".format(package), bold=True)
+ )
message = "{0} {1}".format(
- crayons.red("ERROR:", bold=True),
+ "{0}".format(package_message),
crayons.yellow("Package installation failed...")
)
extra = kwargs.pop("extra", [])
- PipenvException.__init__(self, message=fix_utf8(message), extra=extra, **kwargs)
+ PipenvException.__init__(self, message=message, extra=extra, **kwargs)
class CacheError(PipenvException):
def __init__(self, path, **kwargs):
- message = "{0} {1} {2}\n{0}".format(
- crayons.red("ERROR:", bold=True),
+ message = "{0} {1}\n{2}".format(
crayons.blue("Corrupt cache file"),
- crayons.white(path),
+ crayons.white("{0!s}".format(path)),
crayons.white('Consider trying "pipenv lock --clear" to clear the cache.')
)
- super(PipenvException, self).__init__(message=fix_utf8(message))
+ PipenvException.__init__(self, message=message)
+
+
+class DependencyConflict(PipenvException):
+ def __init__(self, message):
+ extra = ["{0} {1}".format(
+ crayons.red("The operation failed...", bold=True),
+ crayons.red("A dependency conflict was detected and could not be resolved."),
+ )]
+ PipenvException.__init__(self, message, extra=extra)
class ResolutionFailure(PipenvException):
@@ -286,9 +389,7 @@ class ResolutionFailure(PipenvException):
)
if "no version found at all" in message:
no_version_found = True
- message = "{0} {1}".format(
- crayons.red("ERROR:", bold=True), crayons.yellow(message)
- )
+ message = crayons.yellow("{0}".format(message))
if no_version_found:
message = "{0}\n{1}".format(
message,
@@ -297,4 +398,67 @@ class ResolutionFailure(PipenvException):
"See PEP440 for more information."
)
)
- super(ResolutionFailure, self).__init__(fix_utf8(message), extra=extra)
+ PipenvException.__init__(self, message, extra=extra)
+
+
+class RequirementError(PipenvException):
+
+ def __init__(self, req=None):
+ from .utils import VCS_LIST
+ keys = ("name", "path",) + VCS_LIST + ("line", "uri", "url", "relpath")
+ if req is not None:
+ possible_display_values = [getattr(req, value, None) for value in keys]
+ req_value = next(iter(
+ val for val in possible_display_values if val is not None
+ ), None)
+ if not req_value:
+ getstate_fn = getattr(req, "__getstate__", None)
+ slots = getattr(req, "__slots__", None)
+ keys_fn = getattr(req, "keys", None)
+ if getstate_fn:
+ req_value = getstate_fn()
+ elif slots:
+ slot_vals = [
+ (k, getattr(req, k, None)) for k in slots
+ if getattr(req, k, None)
+ ]
+ req_value = "\n".join([
+ " {0}: {1}".format(k, v) for k, v in slot_vals
+ ])
+ elif keys_fn:
+ values = [(k, req.get(k)) for k in keys_fn() if req.get(k)]
+ req_value = "\n".join([
+ " {0}: {1}".format(k, v) for k, v in values
+ ])
+ else:
+ req_value = getattr(req.line_instance, "line", None)
+ message = "{0} {1}".format(
+ crayons.normal(decode_for_output("Failed creating requirement instance")),
+ crayons.white(decode_for_output("{0!r}".format(req_value)))
+ )
+ extra = [str(req)]
+ PipenvException.__init__(self, message, extra=extra)
+
+
+def prettify_exc(error):
+ """Catch known errors and prettify them instead of showing the
+ entire traceback, for better UX"""
+ errors = []
+ for exc in KNOWN_EXCEPTIONS:
+ search_string = exc.match_string if exc.match_string else exc.exception_name
+ split_string = exc.show_from_string if exc.show_from_string else exc.exception_name
+ if search_string in error:
+ # for known exceptions with no display rules and no prefix
+ # we should simply show nothing
+ if not exc.show_from_string and not exc.prefix:
+ errors.append("")
+ continue
+ elif exc.prefix and exc.prefix in error:
+ _, error, info = error.rpartition(exc.prefix)
+ else:
+ _, error, info = error.rpartition(split_string)
+ errors.append("{0} {1}".format(error, info))
+ if not errors:
+ return "{}".format(vistir.misc.decode_for_output(error))
+
+ return "\n".join(errors)
diff --git a/pipenv/patched/notpip/COPYING b/pipenv/patched/notpip/COPYING
new file mode 100644
index 00000000..f067af3a
--- /dev/null
+++ b/pipenv/patched/notpip/COPYING
@@ -0,0 +1,14 @@
+Copyright (C) 2008-2011 INADA Naoki
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
diff --git a/pipenv/patched/notpip/LICENSE.BSD b/pipenv/patched/notpip/LICENSE.BSD
new file mode 100644
index 00000000..42ce7b75
--- /dev/null
+++ b/pipenv/patched/notpip/LICENSE.BSD
@@ -0,0 +1,23 @@
+Copyright (c) Donald Stufft and individual contributors.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+
+ 2. Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/pipenv/patched/notpip/__init__.py b/pipenv/patched/notpip/__init__.py
index ae265fa7..f48c1ca6 100644
--- a/pipenv/patched/notpip/__init__.py
+++ b/pipenv/patched/notpip/__init__.py
@@ -1 +1 @@
-__version__ = "18.1"
+__version__ = "19.0.3"
diff --git a/pipenv/patched/notpip/_internal/build_env.py b/pipenv/patched/notpip/_internal/build_env.py
index 6d696fbd..d38adc49 100644
--- a/pipenv/patched/notpip/_internal/build_env.py
+++ b/pipenv/patched/notpip/_internal/build_env.py
@@ -4,98 +4,173 @@
import logging
import os
import sys
+import textwrap
+from collections import OrderedDict
from distutils.sysconfig import get_python_lib
from sysconfig import get_paths
from pipenv.patched.notpip._vendor.pkg_resources import Requirement, VersionConflict, WorkingSet
+from pipenv.patched.notpip import __file__ as pip_location
from pipenv.patched.notpip._internal.utils.misc import call_subprocess
from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.utils.ui import open_spinner
+if MYPY_CHECK_RUNNING:
+ from typing import Tuple, Set, Iterable, Optional, List # noqa: F401
+ from pipenv.patched.notpip._internal.index import PackageFinder # noqa: F401
+
logger = logging.getLogger(__name__)
+class _Prefix:
+
+ def __init__(self, path):
+ # type: (str) -> None
+ self.path = path
+ self.setup = False
+ self.bin_dir = get_paths(
+ 'nt' if os.name == 'nt' else 'posix_prefix',
+ vars={'base': path, 'platbase': path}
+ )['scripts']
+ # Note: prefer distutils' sysconfig to get the
+ # library paths so PyPy is correctly supported.
+ purelib = get_python_lib(plat_specific=False, prefix=path)
+ platlib = get_python_lib(plat_specific=True, prefix=path)
+ if purelib == platlib:
+ self.lib_dirs = [purelib]
+ else:
+ self.lib_dirs = [purelib, platlib]
+
+
class BuildEnvironment(object):
"""Creates and manages an isolated environment to install build deps
"""
def __init__(self):
+ # type: () -> None
self._temp_dir = TempDirectory(kind="build-env")
self._temp_dir.create()
- @property
- def path(self):
- return self._temp_dir.path
+ self._prefixes = OrderedDict((
+ (name, _Prefix(os.path.join(self._temp_dir.path, name)))
+ for name in ('normal', 'overlay')
+ ))
+
+ self._bin_dirs = [] # type: List[str]
+ self._lib_dirs = [] # type: List[str]
+ for prefix in reversed(list(self._prefixes.values())):
+ self._bin_dirs.append(prefix.bin_dir)
+ self._lib_dirs.extend(prefix.lib_dirs)
+
+ # Customize site to:
+ # - ensure .pth files are honored
+ # - prevent access to system site packages
+ system_sites = {
+ os.path.normcase(site) for site in (
+ get_python_lib(plat_specific=False),
+ get_python_lib(plat_specific=True),
+ )
+ }
+ self._site_dir = os.path.join(self._temp_dir.path, 'site')
+ if not os.path.exists(self._site_dir):
+ os.mkdir(self._site_dir)
+ with open(os.path.join(self._site_dir, 'sitecustomize.py'), 'w') as fp:
+ fp.write(textwrap.dedent(
+ '''
+ import os, site, sys
+
+ # First, drop system-sites related paths.
+ original_sys_path = sys.path[:]
+ known_paths = set()
+ for path in {system_sites!r}:
+ site.addsitedir(path, known_paths=known_paths)
+ system_paths = set(
+ os.path.normcase(path)
+ for path in sys.path[len(original_sys_path):]
+ )
+ original_sys_path = [
+ path for path in original_sys_path
+ if os.path.normcase(path) not in system_paths
+ ]
+ sys.path = original_sys_path
+
+ # Second, add lib directories.
+ # ensuring .pth file are processed.
+ for path in {lib_dirs!r}:
+ assert not path in sys.path
+ site.addsitedir(path)
+ '''
+ ).format(system_sites=system_sites, lib_dirs=self._lib_dirs))
def __enter__(self):
- self.save_path = os.environ.get('PATH', None)
- self.save_pythonpath = os.environ.get('PYTHONPATH', None)
- self.save_nousersite = os.environ.get('PYTHONNOUSERSITE', None)
+ self._save_env = {
+ name: os.environ.get(name, None)
+ for name in ('PATH', 'PYTHONNOUSERSITE', 'PYTHONPATH')
+ }
- install_scheme = 'nt' if (os.name == 'nt') else 'posix_prefix'
- install_dirs = get_paths(install_scheme, vars={
- 'base': self.path,
- 'platbase': self.path,
+ path = self._bin_dirs[:]
+ old_path = self._save_env['PATH']
+ if old_path:
+ path.extend(old_path.split(os.pathsep))
+
+ pythonpath = [self._site_dir]
+
+ os.environ.update({
+ 'PATH': os.pathsep.join(path),
+ 'PYTHONNOUSERSITE': '1',
+ 'PYTHONPATH': os.pathsep.join(pythonpath),
})
- scripts = install_dirs['scripts']
- if self.save_path:
- os.environ['PATH'] = scripts + os.pathsep + self.save_path
- else:
- os.environ['PATH'] = scripts + os.pathsep + os.defpath
-
- # Note: prefer distutils' sysconfig to get the
- # library paths so PyPy is correctly supported.
- purelib = get_python_lib(plat_specific=0, prefix=self.path)
- platlib = get_python_lib(plat_specific=1, prefix=self.path)
- if purelib == platlib:
- lib_dirs = purelib
- else:
- lib_dirs = purelib + os.pathsep + platlib
- if self.save_pythonpath:
- os.environ['PYTHONPATH'] = lib_dirs + os.pathsep + \
- self.save_pythonpath
- else:
- os.environ['PYTHONPATH'] = lib_dirs
-
- os.environ['PYTHONNOUSERSITE'] = '1'
-
- return self.path
-
def __exit__(self, exc_type, exc_val, exc_tb):
- def restore_var(varname, old_value):
+ for varname, old_value in self._save_env.items():
if old_value is None:
os.environ.pop(varname, None)
else:
os.environ[varname] = old_value
- restore_var('PATH', self.save_path)
- restore_var('PYTHONPATH', self.save_pythonpath)
- restore_var('PYTHONNOUSERSITE', self.save_nousersite)
-
def cleanup(self):
+ # type: () -> None
self._temp_dir.cleanup()
- def missing_requirements(self, reqs):
- """Return a list of the requirements from reqs that are not present
+ def check_requirements(self, reqs):
+ # type: (Iterable[str]) -> Tuple[Set[Tuple[str, str]], Set[str]]
+ """Return 2 sets:
+ - conflicting requirements: set of (installed, wanted) reqs tuples
+ - missing requirements: set of reqs
"""
- missing = []
- with self:
- ws = WorkingSet(os.environ["PYTHONPATH"].split(os.pathsep))
+ missing = set()
+ conflicting = set()
+ if reqs:
+ ws = WorkingSet(self._lib_dirs)
for req in reqs:
try:
if ws.find(Requirement.parse(req)) is None:
- missing.append(req)
- except VersionConflict:
- missing.append(req)
- return missing
+ missing.add(req)
+ except VersionConflict as e:
+ conflicting.add((str(e.args[0].as_requirement()),
+ str(e.args[1])))
+ return conflicting, missing
- def install_requirements(self, finder, requirements, message):
+ def install_requirements(
+ self,
+ finder, # type: PackageFinder
+ requirements, # type: Iterable[str]
+ prefix_as_string, # type: str
+ message # type: Optional[str]
+ ):
+ # type: (...) -> None
+ prefix = self._prefixes[prefix_as_string]
+ assert not prefix.setup
+ prefix.setup = True
+ if not requirements:
+ return
args = [
- sys.executable, '-m', 'pip', 'install', '--ignore-installed',
- '--no-user', '--prefix', self.path, '--no-warn-script-location',
- ]
+ sys.executable, os.path.dirname(pip_location), 'install',
+ '--ignore-installed', '--no-user', '--prefix', prefix.path,
+ '--no-warn-script-location',
+ ] # type: List[str]
if logger.getEffectiveLevel() <= logging.DEBUG:
args.append('-v')
for format_control in ('no_binary', 'only_binary'):
@@ -114,8 +189,6 @@ class BuildEnvironment(object):
args.extend(['--trusted-host', host])
if finder.allow_all_prereleases:
args.append('--pre')
- if finder.process_dependency_links:
- args.append('--process-dependency-links')
args.append('--')
args.extend(requirements)
with open_spinner(message) as spinner:
@@ -138,5 +211,5 @@ class NoOpBuildEnvironment(BuildEnvironment):
def cleanup(self):
pass
- def install_requirements(self, finder, requirements, message):
+ def install_requirements(self, finder, requirements, prefix, message):
raise NotImplementedError()
diff --git a/pipenv/patched/notpip/_internal/cache.py b/pipenv/patched/notpip/_internal/cache.py
index d91b8170..9f35e83d 100644
--- a/pipenv/patched/notpip/_internal/cache.py
+++ b/pipenv/patched/notpip/_internal/cache.py
@@ -12,8 +12,13 @@ from pipenv.patched.notpip._internal.download import path_to_url
from pipenv.patched.notpip._internal.models.link import Link
from pipenv.patched.notpip._internal.utils.compat import expanduser
from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.wheel import InvalidWheelFilename, Wheel
+if MYPY_CHECK_RUNNING:
+ from typing import Optional, Set, List, Any # noqa: F401
+ from pipenv.patched.notpip._internal.index import FormatControl # noqa: F401
+
logger = logging.getLogger(__name__)
@@ -29,6 +34,7 @@ class Cache(object):
"""
def __init__(self, cache_dir, format_control, allowed_formats):
+ # type: (str, FormatControl, Set[str]) -> None
super(Cache, self).__init__()
self.cache_dir = expanduser(cache_dir) if cache_dir else None
self.format_control = format_control
@@ -38,6 +44,7 @@ class Cache(object):
assert self.allowed_formats.union(_valid_formats) == _valid_formats
def _get_cache_path_parts(self, link):
+ # type: (Link) -> List[str]
"""Get parts of part that must be os.path.joined with cache_dir
"""
@@ -63,6 +70,7 @@ class Cache(object):
return parts
def _get_candidates(self, link, package_name):
+ # type: (Link, Optional[str]) -> List[Any]
can_not_cache = (
not self.cache_dir or
not package_name or
@@ -87,23 +95,27 @@ class Cache(object):
raise
def get_path_for_link(self, link):
+ # type: (Link) -> str
"""Return a directory to store cached items in for link.
"""
raise NotImplementedError()
def get(self, link, package_name):
+ # type: (Link, Optional[str]) -> Link
"""Returns a link to a cached item if it exists, otherwise returns the
passed link.
"""
raise NotImplementedError()
def _link_for_candidate(self, link, candidate):
+ # type: (Link, str) -> Link
root = self.get_path_for_link(link)
path = os.path.join(root, candidate)
return Link(path_to_url(path))
def cleanup(self):
+ # type: () -> None
pass
@@ -112,11 +124,13 @@ class SimpleWheelCache(Cache):
"""
def __init__(self, cache_dir, format_control):
+ # type: (str, FormatControl) -> None
super(SimpleWheelCache, self).__init__(
cache_dir, format_control, {"binary"}
)
def get_path_for_link(self, link):
+ # type: (Link) -> str
"""Return a directory to store cached wheels for link
Because there are M wheels for any one sdist, we provide a directory
@@ -137,6 +151,7 @@ class SimpleWheelCache(Cache):
return os.path.join(self.cache_dir, "wheels", *parts)
def get(self, link, package_name):
+ # type: (Link, Optional[str]) -> Link
candidates = []
for wheel_name in self._get_candidates(link, package_name):
@@ -160,6 +175,7 @@ class EphemWheelCache(SimpleWheelCache):
"""
def __init__(self, format_control):
+ # type: (FormatControl) -> None
self._temp_dir = TempDirectory(kind="ephem-wheel-cache")
self._temp_dir.create()
@@ -168,6 +184,7 @@ class EphemWheelCache(SimpleWheelCache):
)
def cleanup(self):
+ # type: () -> None
self._temp_dir.cleanup()
@@ -179,6 +196,7 @@ class WheelCache(Cache):
"""
def __init__(self, cache_dir, format_control):
+ # type: (str, FormatControl) -> None
super(WheelCache, self).__init__(
cache_dir, format_control, {'binary'}
)
@@ -186,17 +204,21 @@ class WheelCache(Cache):
self._ephem_cache = EphemWheelCache(format_control)
def get_path_for_link(self, link):
+ # type: (Link) -> str
return self._wheel_cache.get_path_for_link(link)
def get_ephem_path_for_link(self, link):
+ # type: (Link) -> str
return self._ephem_cache.get_path_for_link(link)
def get(self, link, package_name):
+ # type: (Link, Optional[str]) -> Link
retval = self._wheel_cache.get(link, package_name)
if retval is link:
retval = self._ephem_cache.get(link, package_name)
return retval
def cleanup(self):
+ # type: () -> None
self._wheel_cache.cleanup()
self._ephem_cache.cleanup()
diff --git a/pipenv/patched/notpip/_internal/cli/base_command.py b/pipenv/patched/notpip/_internal/cli/base_command.py
index 229831f2..4aa16da6 100644
--- a/pipenv/patched/notpip/_internal/cli/base_command.py
+++ b/pipenv/patched/notpip/_internal/cli/base_command.py
@@ -1,11 +1,13 @@
"""Base Command class, and related routines"""
-from __future__ import absolute_import
+from __future__ import absolute_import, print_function
import logging
import logging.config
import optparse
import os
+import platform
import sys
+import traceback
from pipenv.patched.notpip._internal.cli import cmdoptions
from pipenv.patched.notpip._internal.cli.parser import (
@@ -26,13 +28,19 @@ from pipenv.patched.notpip._internal.req.constructors import (
install_req_from_editable, install_req_from_line,
)
from pipenv.patched.notpip._internal.req.req_file import parse_requirements
-from pipenv.patched.notpip._internal.utils.logging import setup_logging
-from pipenv.patched.notpip._internal.utils.misc import get_prog, normalize_path
+from pipenv.patched.notpip._internal.utils.deprecation import deprecated
+from pipenv.patched.notpip._internal.utils.logging import BrokenStdoutLoggingError, setup_logging
+from pipenv.patched.notpip._internal.utils.misc import (
+ get_prog, normalize_path, redact_password_from_url,
+)
from pipenv.patched.notpip._internal.utils.outdated import pip_version_check
from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
if MYPY_CHECK_RUNNING:
- from typing import Optional # noqa: F401
+ from typing import Optional, List, Tuple, Any # noqa: F401
+ from optparse import Values # noqa: F401
+ from pipenv.patched.notpip._internal.cache import WheelCache # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_set import RequirementSet # noqa: F401
__all__ = ['Command']
@@ -46,6 +54,7 @@ class Command(object):
ignore_require_venv = False # type: bool
def __init__(self, isolated=False):
+ # type: (bool) -> None
parser_kw = {
'usage': self.usage,
'prog': '%s %s' % (get_prog(), self.name),
@@ -69,7 +78,12 @@ class Command(object):
)
self.parser.add_option_group(gen_opts)
+ def run(self, options, args):
+ # type: (Values, List[Any]) -> Any
+ raise NotImplementedError
+
def _build_session(self, options, retries=None, timeout=None):
+ # type: (Values, Optional[int], Optional[int]) -> PipSession
session = PipSession(
cache=(
normalize_path(os.path.join(options.cache_dir, "http"))
@@ -106,21 +120,43 @@ class Command(object):
return session
def parse_args(self, args):
+ # type: (List[str]) -> Tuple
# factored out for testability
return self.parser.parse_args(args)
def main(self, args):
+ # type: (List[str]) -> int
options, args = self.parse_args(args)
# Set verbosity so that it can be used elsewhere.
self.verbosity = options.verbose - options.quiet
- setup_logging(
+ level_number = setup_logging(
verbosity=self.verbosity,
no_color=options.no_color,
user_log_file=options.log,
)
+ if sys.version_info[:2] == (3, 4):
+ deprecated(
+ "Python 3.4 support has been deprecated. pip 19.1 will be the "
+ "last one supporting it. Please upgrade your Python as Python "
+ "3.4 won't be maintained after March 2019 (cf PEP 429).",
+ replacement=None,
+ gone_in='19.2',
+ )
+ elif sys.version_info[:2] == (2, 7):
+ message = (
+ "A future version of pip will drop support for Python 2.7."
+ )
+ if platform.python_implementation() == "CPython":
+ message = (
+ "Python 2.7 will reach the end of its life on January "
+ "1st, 2020. Please upgrade your Python as Python 2.7 "
+ "won't be maintained after that date. "
+ ) + message
+ deprecated(message, replacement=None, gone_in=None)
+
# TODO: Try to get these passing down from the command?
# without resorting to os.environ to hold these.
# This also affects isolated builds and it should.
@@ -159,6 +195,14 @@ class Command(object):
logger.critical('ERROR: %s', exc)
logger.debug('Exception information:', exc_info=True)
+ return ERROR
+ except BrokenStdoutLoggingError:
+ # Bypass our logger and write any remaining messages to stderr
+ # because stdout no longer works.
+ print('ERROR: Pipe to stdout was broken', file=sys.stderr)
+ if level_number <= logging.DEBUG:
+ traceback.print_exc(file=sys.stderr)
+
return ERROR
except KeyboardInterrupt:
logger.critical('Operation cancelled by user')
@@ -195,8 +239,15 @@ class Command(object):
class RequirementCommand(Command):
@staticmethod
- def populate_requirement_set(requirement_set, args, options, finder,
- session, name, wheel_cache):
+ def populate_requirement_set(requirement_set, # type: RequirementSet
+ args, # type: List[str]
+ options, # type: Values
+ finder, # type: PackageFinder
+ session, # type: PipSession
+ name, # type: str
+ wheel_cache # type: Optional[WheelCache]
+ ):
+ # type: (...) -> None
"""
Marshal cmd line args into a requirement set.
"""
@@ -214,6 +265,7 @@ class RequirementCommand(Command):
for req in args:
req_to_add = install_req_from_line(
req, None, isolated=options.isolated_mode,
+ use_pep517=options.use_pep517,
wheel_cache=wheel_cache
)
req_to_add.is_direct = True
@@ -223,6 +275,7 @@ class RequirementCommand(Command):
req_to_add = install_req_from_editable(
req,
isolated=options.isolated_mode,
+ use_pep517=options.use_pep517,
wheel_cache=wheel_cache
)
req_to_add.is_direct = True
@@ -232,7 +285,8 @@ class RequirementCommand(Command):
for req_to_add in parse_requirements(
filename,
finder=finder, options=options, session=session,
- wheel_cache=wheel_cache):
+ wheel_cache=wheel_cache,
+ use_pep517=options.use_pep517):
req_to_add.is_direct = True
requirement_set.add_requirement(req_to_add)
# If --require-hashes was a line in a requirements file, tell
@@ -251,15 +305,25 @@ class RequirementCommand(Command):
'You must give at least one requirement to %(name)s '
'(see "pip help %(name)s")' % opts)
- def _build_package_finder(self, options, session,
- platform=None, python_versions=None,
- abi=None, implementation=None):
+ def _build_package_finder(
+ self,
+ options, # type: Values
+ session, # type: PipSession
+ platform=None, # type: Optional[str]
+ python_versions=None, # type: Optional[List[str]]
+ abi=None, # type: Optional[str]
+ implementation=None # type: Optional[str]
+ ):
+ # type: (...) -> PackageFinder
"""
Create a package finder appropriate to this requirement command.
"""
index_urls = [options.index_url] + options.extra_index_urls
if options.no_index:
- logger.debug('Ignoring indexes: %s', ','.join(index_urls))
+ logger.debug(
+ 'Ignoring indexes: %s',
+ ','.join(redact_password_from_url(url) for url in index_urls),
+ )
index_urls = []
return PackageFinder(
@@ -268,7 +332,6 @@ class RequirementCommand(Command):
index_urls=index_urls,
trusted_hosts=options.trusted_hosts,
allow_all_prereleases=options.pre,
- process_dependency_links=options.process_dependency_links,
session=session,
platform=platform,
versions=python_versions,
diff --git a/pipenv/patched/notpip/_internal/cli/cmdoptions.py b/pipenv/patched/notpip/_internal/cli/cmdoptions.py
index a075a67e..3c5a7084 100644
--- a/pipenv/patched/notpip/_internal/cli/cmdoptions.py
+++ b/pipenv/patched/notpip/_internal/cli/cmdoptions.py
@@ -9,7 +9,9 @@ pass on state. To be consistent, all options will follow this design.
"""
from __future__ import absolute_import
+import textwrap
import warnings
+from distutils.util import strtobool
from functools import partial
from optparse import SUPPRESS_HELP, Option, OptionGroup
@@ -22,10 +24,27 @@ from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.utils.ui import BAR_TYPES
if MYPY_CHECK_RUNNING:
- from typing import Any # noqa: F401
+ from typing import Any, Callable, Dict, List, Optional, Union # noqa: F401
+ from optparse import OptionParser, Values # noqa: F401
+ from pipenv.patched.notpip._internal.cli.parser import ConfigOptionParser # noqa: F401
+
+
+def raise_option_error(parser, option, msg):
+ """
+ Raise an option parsing error using parser.error().
+
+ Args:
+ parser: an OptionParser instance.
+ option: an Option instance.
+ msg: the error text.
+ """
+ msg = '{} error: {}'.format(option, msg)
+ msg = textwrap.fill(' '.join(msg.split()))
+ parser.error(msg)
def make_option_group(group, parser):
+ # type: (Dict[str, Any], ConfigOptionParser) -> OptionGroup
"""
Return an OptionGroup object
group -- assumed to be dict with 'name' and 'options' keys
@@ -38,6 +57,7 @@ def make_option_group(group, parser):
def check_install_build_global(options, check_options=None):
+ # type: (Values, Optional[Values]) -> None
"""Disable wheels if per-setup.py call options are set.
:param options: The OptionParser options to update.
@@ -60,6 +80,7 @@ def check_install_build_global(options, check_options=None):
def check_dist_restriction(options, check_target=False):
+ # type: (Values, bool) -> None
"""Function for determining if custom platform options are allowed.
:param options: The OptionParser options.
@@ -108,7 +129,7 @@ help_ = partial(
dest='help',
action='help',
help='Show help.',
-) # type: Any
+) # type: Callable[..., Option]
isolated_mode = partial(
Option,
@@ -120,7 +141,7 @@ isolated_mode = partial(
"Run pip in an isolated mode, ignoring environment variables and user "
"configuration."
),
-)
+) # type: Callable[..., Option]
require_virtualenv = partial(
Option,
@@ -130,7 +151,7 @@ require_virtualenv = partial(
action='store_true',
default=False,
help=SUPPRESS_HELP
-) # type: Any
+) # type: Callable[..., Option]
verbose = partial(
Option,
@@ -139,7 +160,7 @@ verbose = partial(
action='count',
default=0,
help='Give more output. Option is additive, and can be used up to 3 times.'
-)
+) # type: Callable[..., Option]
no_color = partial(
Option,
@@ -148,7 +169,7 @@ no_color = partial(
action='store_true',
default=False,
help="Suppress colored output",
-)
+) # type: Callable[..., Option]
version = partial(
Option,
@@ -156,7 +177,7 @@ version = partial(
dest='version',
action='store_true',
help='Show version and exit.',
-) # type: Any
+) # type: Callable[..., Option]
quiet = partial(
Option,
@@ -169,7 +190,7 @@ quiet = partial(
' times (corresponding to WARNING, ERROR, and CRITICAL logging'
' levels).'
),
-) # type: Any
+) # type: Callable[..., Option]
progress_bar = partial(
Option,
@@ -182,7 +203,7 @@ progress_bar = partial(
'Specify type of progress to be displayed [' +
'|'.join(BAR_TYPES.keys()) + '] (default: %default)'
),
-) # type: Any
+) # type: Callable[..., Option]
log = partial(
Option,
@@ -190,7 +211,7 @@ log = partial(
dest="log",
metavar="path",
help="Path to a verbose appending log."
-) # type: Any
+) # type: Callable[..., Option]
no_input = partial(
Option,
@@ -200,7 +221,7 @@ no_input = partial(
action='store_true',
default=False,
help=SUPPRESS_HELP
-) # type: Any
+) # type: Callable[..., Option]
proxy = partial(
Option,
@@ -209,7 +230,7 @@ proxy = partial(
type='str',
default='',
help="Specify a proxy in the form [user:passwd@]proxy.server:port."
-) # type: Any
+) # type: Callable[..., Option]
retries = partial(
Option,
@@ -219,7 +240,7 @@ retries = partial(
default=5,
help="Maximum number of retries each connection should attempt "
"(default %default times).",
-) # type: Any
+) # type: Callable[..., Option]
timeout = partial(
Option,
@@ -229,7 +250,7 @@ timeout = partial(
type='float',
default=15,
help='Set the socket timeout (default %default seconds).',
-) # type: Any
+) # type: Callable[..., Option]
skip_requirements_regex = partial(
Option,
@@ -239,10 +260,11 @@ skip_requirements_regex = partial(
type='str',
default='',
help=SUPPRESS_HELP,
-) # type: Any
+) # type: Callable[..., Option]
def exists_action():
+ # type: () -> Option
return Option(
# Option when path already exist
'--exists-action',
@@ -264,7 +286,7 @@ cert = partial(
type='str',
metavar='path',
help="Path to alternate CA bundle.",
-) # type: Any
+) # type: Callable[..., Option]
client_cert = partial(
Option,
@@ -275,7 +297,7 @@ client_cert = partial(
metavar='path',
help="Path to SSL client certificate, a single file containing the "
"private key and the certificate in PEM format.",
-) # type: Any
+) # type: Callable[..., Option]
index_url = partial(
Option,
@@ -287,7 +309,7 @@ index_url = partial(
"This should point to a repository compliant with PEP 503 "
"(the simple repository API) or a local directory laid out "
"in the same format.",
-) # type: Any
+) # type: Callable[..., Option]
def extra_index_url():
@@ -310,10 +332,11 @@ no_index = partial(
action='store_true',
default=False,
help='Ignore package index (only looking at --find-links URLs instead).',
-) # type: Any
+) # type: Callable[..., Option]
def find_links():
+ # type: () -> Option
return Option(
'-f', '--find-links',
dest='find_links',
@@ -327,6 +350,7 @@ def find_links():
def trusted_host():
+ # type: () -> Option
return Option(
"--trusted-host",
dest="trusted_hosts",
@@ -338,18 +362,8 @@ def trusted_host():
)
-# Remove after 1.5
-process_dependency_links = partial(
- Option,
- "--process-dependency-links",
- dest="process_dependency_links",
- action="store_true",
- default=False,
- help="Enable the processing of dependency links.",
-) # type: Any
-
-
def constraints():
+ # type: () -> Option
return Option(
'-c', '--constraint',
dest='constraints',
@@ -362,6 +376,7 @@ def constraints():
def requirements():
+ # type: () -> Option
return Option(
'-r', '--requirement',
dest='requirements',
@@ -374,6 +389,7 @@ def requirements():
def editable():
+ # type: () -> Option
return Option(
'-e', '--editable',
dest='editables',
@@ -394,15 +410,17 @@ src = partial(
help='Directory to check out editable projects into. '
'The default in a virtualenv is "/src". '
'The default for global installs is "/src".'
-) # type: Any
+) # type: Callable[..., Option]
def _get_format_control(values, option):
+ # type: (Values, Option) -> Any
"""Get a format_control object."""
return getattr(values, option.dest)
def _handle_no_binary(option, opt_str, value, parser):
+ # type: (Option, str, str, OptionParser) -> None
existing = _get_format_control(parser.values, option)
FormatControl.handle_mutual_excludes(
value, existing.no_binary, existing.only_binary,
@@ -410,6 +428,7 @@ def _handle_no_binary(option, opt_str, value, parser):
def _handle_only_binary(option, opt_str, value, parser):
+ # type: (Option, str, str, OptionParser) -> None
existing = _get_format_control(parser.values, option)
FormatControl.handle_mutual_excludes(
value, existing.only_binary, existing.no_binary,
@@ -417,6 +436,7 @@ def _handle_only_binary(option, opt_str, value, parser):
def no_binary():
+ # type: () -> Option
format_control = FormatControl(set(), set())
return Option(
"--no-binary", dest="format_control", action="callback",
@@ -432,6 +452,7 @@ def no_binary():
def only_binary():
+ # type: () -> Option
format_control = FormatControl(set(), set())
return Option(
"--only-binary", dest="format_control", action="callback",
@@ -454,7 +475,7 @@ platform = partial(
default=None,
help=("Only use wheels compatible with . "
"Defaults to the platform of the running system."),
-)
+) # type: Callable[..., Option]
python_version = partial(
@@ -469,7 +490,7 @@ python_version = partial(
"version (e.g. '2') can be specified to match all "
"minor revs of that major version. A minor version "
"(e.g. '34') can also be specified."),
-)
+) # type: Callable[..., Option]
implementation = partial(
@@ -483,7 +504,7 @@ implementation = partial(
" or 'ip'. If not specified, then the current "
"interpreter implementation is used. Use 'py' to force "
"implementation-agnostic wheels."),
-)
+) # type: Callable[..., Option]
abi = partial(
@@ -498,10 +519,11 @@ abi = partial(
"you will need to specify --implementation, "
"--platform, and --python-version when using "
"this option."),
-)
+) # type: Callable[..., Option]
def prefer_binary():
+ # type: () -> Option
return Option(
"--prefer-binary",
dest="prefer_binary",
@@ -518,15 +540,44 @@ cache_dir = partial(
default=USER_CACHE_DIR,
metavar="dir",
help="Store the cache data in ."
-)
+) # type: Callable[..., Option]
+
+
+def no_cache_dir_callback(option, opt, value, parser):
+ """
+ Process a value provided for the --no-cache-dir option.
+
+ This is an optparse.Option callback for the --no-cache-dir option.
+ """
+ # The value argument will be None if --no-cache-dir is passed via the
+ # command-line, since the option doesn't accept arguments. However,
+ # the value can be non-None if the option is triggered e.g. by an
+ # environment variable, like PIP_NO_CACHE_DIR=true.
+ if value is not None:
+ # Then parse the string value to get argument error-checking.
+ try:
+ strtobool(value)
+ except ValueError as exc:
+ raise_option_error(parser, option=option, msg=str(exc))
+
+ # Originally, setting PIP_NO_CACHE_DIR to a value that strtobool()
+ # converted to 0 (like "false" or "no") caused cache_dir to be disabled
+ # rather than enabled (logic would say the latter). Thus, we disable
+ # the cache directory not just on values that parse to True, but (for
+ # backwards compatibility reasons) also on values that parse to False.
+ # In other words, always set it to False if the option is provided in
+ # some (valid) form.
+ parser.values.cache_dir = False
+
no_cache = partial(
Option,
"--no-cache-dir",
dest="cache_dir",
- action="store_false",
+ action="callback",
+ callback=no_cache_dir_callback,
help="Disable the cache.",
-)
+) # type: Callable[..., Option]
no_deps = partial(
Option,
@@ -535,7 +586,7 @@ no_deps = partial(
action='store_true',
default=False,
help="Don't install package dependencies.",
-) # type: Any
+) # type: Callable[..., Option]
build_dir = partial(
Option,
@@ -547,7 +598,7 @@ build_dir = partial(
'The location of temporary directories can be controlled by setting '
'the TMPDIR environment variable (TEMP on Windows) appropriately. '
'When passed, build directories are not cleaned in case of failures.'
-) # type: Any
+) # type: Callable[..., Option]
ignore_requires_python = partial(
Option,
@@ -555,7 +606,7 @@ ignore_requires_python = partial(
dest='ignore_requires_python',
action='store_true',
help='Ignore the Requires-Python information.'
-) # type: Any
+) # type: Callable[..., Option]
no_build_isolation = partial(
Option,
@@ -566,6 +617,50 @@ no_build_isolation = partial(
help='Disable isolation when building a modern source distribution. '
'Build dependencies specified by PEP 518 must be already installed '
'if this option is used.'
+) # type: Callable[..., Option]
+
+
+def no_use_pep517_callback(option, opt, value, parser):
+ """
+ Process a value provided for the --no-use-pep517 option.
+
+ This is an optparse.Option callback for the no_use_pep517 option.
+ """
+ # Since --no-use-pep517 doesn't accept arguments, the value argument
+ # will be None if --no-use-pep517 is passed via the command-line.
+ # However, the value can be non-None if the option is triggered e.g.
+ # by an environment variable, for example "PIP_NO_USE_PEP517=true".
+ if value is not None:
+ msg = """A value was passed for --no-use-pep517,
+ probably using either the PIP_NO_USE_PEP517 environment variable
+ or the "no-use-pep517" config file option. Use an appropriate value
+ of the PIP_USE_PEP517 environment variable or the "use-pep517"
+ config file option instead.
+ """
+ raise_option_error(parser, option=option, msg=msg)
+
+ # Otherwise, --no-use-pep517 was passed via the command-line.
+ parser.values.use_pep517 = False
+
+
+use_pep517 = partial(
+ Option,
+ '--use-pep517',
+ dest='use_pep517',
+ action='store_true',
+ default=None,
+ help='Use PEP 517 for building source distributions '
+ '(use --no-use-pep517 to force legacy behaviour).'
+) # type: Any
+
+no_use_pep517 = partial(
+ Option,
+ '--no-use-pep517',
+ dest='use_pep517',
+ action='callback',
+ callback=no_use_pep517_callback,
+ default=None,
+ help=SUPPRESS_HELP
) # type: Any
install_options = partial(
@@ -579,7 +674,7 @@ install_options = partial(
"bin\"). Use multiple --install-option options to pass multiple "
"options to setup.py install. If you are using an option with a "
"directory path, be sure to use absolute path.",
-) # type: Any
+) # type: Callable[..., Option]
global_options = partial(
Option,
@@ -589,7 +684,7 @@ global_options = partial(
metavar='options',
help="Extra global options to be supplied to the setup.py "
"call before the install command.",
-) # type: Any
+) # type: Callable[..., Option]
no_clean = partial(
Option,
@@ -597,7 +692,7 @@ no_clean = partial(
action='store_true',
default=False,
help="Don't clean up build directories."
-) # type: Any
+) # type: Callable[..., Option]
pre = partial(
Option,
@@ -606,7 +701,7 @@ pre = partial(
default=False,
help="Include pre-release and development versions. By default, "
"pip only finds stable versions.",
-) # type: Any
+) # type: Callable[..., Option]
disable_pip_version_check = partial(
Option,
@@ -616,7 +711,7 @@ disable_pip_version_check = partial(
default=False,
help="Don't periodically check PyPI to determine whether a new version "
"of pip is available for download. Implied with --no-index.",
-) # type: Any
+) # type: Callable[..., Option]
# Deprecated, Remove later
@@ -626,14 +721,15 @@ always_unzip = partial(
dest='always_unzip',
action='store_true',
help=SUPPRESS_HELP,
-) # type: Any
+) # type: Callable[..., Option]
def _merge_hash(option, opt_str, value, parser):
+ # type: (Option, str, str, OptionParser) -> None
"""Given a value spelled "algo:digest", append the digest to a list
pointed to in a dict by the algo name."""
if not parser.values.hashes:
- parser.values.hashes = {}
+ parser.values.hashes = {} # type: ignore
try:
algo, digest = value.split(':', 1)
except ValueError:
@@ -657,7 +753,7 @@ hash = partial(
type='string',
help="Verify that the package's archive matches this "
'hash before installing. Example: --hash=sha256:abcdef...',
-) # type: Any
+) # type: Callable[..., Option]
require_hashes = partial(
@@ -669,7 +765,7 @@ require_hashes = partial(
help='Require a hash to check each requirement against, for '
'repeatable installs. This option is implied when any package in a '
'requirements file has a --hash option.',
-) # type: Any
+) # type: Callable[..., Option]
##########
@@ -700,7 +796,7 @@ general_group = {
disable_pip_version_check,
no_color,
]
-}
+} # type: Dict[str, Any]
index_group = {
'name': 'Package Index Options',
@@ -709,6 +805,5 @@ index_group = {
extra_index_url,
no_index,
find_links,
- process_dependency_links,
]
-}
+} # type: Dict[str, Any]
diff --git a/pipenv/patched/notpip/_internal/cli/main_parser.py b/pipenv/patched/notpip/_internal/cli/main_parser.py
index abe2f69e..704bf404 100644
--- a/pipenv/patched/notpip/_internal/cli/main_parser.py
+++ b/pipenv/patched/notpip/_internal/cli/main_parser.py
@@ -14,11 +14,17 @@ from pipenv.patched.notpip._internal.commands import (
)
from pipenv.patched.notpip._internal.exceptions import CommandError
from pipenv.patched.notpip._internal.utils.misc import get_prog
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Tuple, List # noqa: F401
+
__all__ = ["create_main_parser", "parse_command"]
def create_main_parser():
+ # type: () -> ConfigOptionParser
"""Creates and returns the main parser for pip's CLI
"""
@@ -44,7 +50,8 @@ def create_main_parser():
gen_opts = cmdoptions.make_option_group(cmdoptions.general_group, parser)
parser.add_option_group(gen_opts)
- parser.main = True # so the help formatter knows
+ # so the help formatter knows
+ parser.main = True # type: ignore
# create command listing for description
command_summaries = get_summaries()
@@ -55,6 +62,7 @@ def create_main_parser():
def parse_command(args):
+ # type: (List[str]) -> Tuple[str, List[str]]
parser = create_main_parser()
# Note: parser calls disable_interspersed_args(), so the result of this
@@ -68,7 +76,7 @@ def parse_command(args):
# --version
if general_options.version:
- sys.stdout.write(parser.version)
+ sys.stdout.write(parser.version) # type: ignore
sys.stdout.write(os.linesep)
sys.exit()
diff --git a/pipenv/patched/notpip/_internal/commands/check.py b/pipenv/patched/notpip/_internal/commands/check.py
index adf4f5e7..cf84f5df 100644
--- a/pipenv/patched/notpip/_internal/commands/check.py
+++ b/pipenv/patched/notpip/_internal/commands/check.py
@@ -16,7 +16,7 @@ class CheckCommand(Command):
summary = 'Verify installed packages have compatible dependencies.'
def run(self, options, args):
- package_set = create_package_set_from_installed()
+ package_set, parsing_probs = create_package_set_from_installed()
missing, conflicting = check_package_set(package_set)
for project_name in missing:
@@ -35,7 +35,7 @@ class CheckCommand(Command):
project_name, version, req, dep_name, dep_version,
)
- if missing or conflicting:
+ if missing or conflicting or parsing_probs:
return 1
else:
logger.info("No broken requirements found.")
diff --git a/pipenv/patched/notpip/_internal/commands/download.py b/pipenv/patched/notpip/_internal/commands/download.py
index e5d87121..133ca135 100644
--- a/pipenv/patched/notpip/_internal/commands/download.py
+++ b/pipenv/patched/notpip/_internal/commands/download.py
@@ -58,6 +58,8 @@ class DownloadCommand(RequirementCommand):
cmd_opts.add_option(cmdoptions.require_hashes())
cmd_opts.add_option(cmdoptions.progress_bar())
cmd_opts.add_option(cmdoptions.no_build_isolation())
+ cmd_opts.add_option(cmdoptions.use_pep517())
+ cmd_opts.add_option(cmdoptions.no_use_pep517())
cmd_opts.add_option(
'-d', '--dest', '--destination-dir', '--destination-directory',
diff --git a/pipenv/patched/notpip/_internal/commands/install.py b/pipenv/patched/notpip/_internal/commands/install.py
index ddcb4759..68255c85 100644
--- a/pipenv/patched/notpip/_internal/commands/install.py
+++ b/pipenv/patched/notpip/_internal/commands/install.py
@@ -30,12 +30,6 @@ from pipenv.patched.notpip._internal.utils.misc import (
from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
from pipenv.patched.notpip._internal.wheel import WheelBuilder
-try:
- import wheel
-except ImportError:
- wheel = None
-
-
logger = logging.getLogger(__name__)
@@ -158,6 +152,8 @@ class InstallCommand(RequirementCommand):
cmd_opts.add_option(cmdoptions.ignore_requires_python())
cmd_opts.add_option(cmdoptions.no_build_isolation())
+ cmd_opts.add_option(cmdoptions.use_pep517())
+ cmd_opts.add_option(cmdoptions.no_use_pep517())
cmd_opts.add_option(cmdoptions.install_options())
cmd_opts.add_option(cmdoptions.global_options())
@@ -314,6 +310,7 @@ class InstallCommand(RequirementCommand):
ignore_requires_python=options.ignore_requires_python,
ignore_installed=options.ignore_installed,
isolated=options.isolated_mode,
+ use_pep517=options.use_pep517
)
resolver.resolve(requirement_set)
@@ -321,21 +318,51 @@ class InstallCommand(RequirementCommand):
modifying_pip=requirement_set.has_requirement("pip")
)
- # If caching is disabled or wheel is not installed don't
- # try to build wheels.
- if wheel and options.cache_dir:
- # build wheels before install.
- wb = WheelBuilder(
- finder, preparer, wheel_cache,
- build_options=[], global_options=[],
- )
- # Ignore the result: a failed wheel will be
- # installed from the sdist/vcs whatever.
+ # Consider legacy and PEP517-using requirements separately
+ legacy_requirements = []
+ pep517_requirements = []
+ for req in requirement_set.requirements.values():
+ if req.use_pep517:
+ pep517_requirements.append(req)
+ else:
+ legacy_requirements.append(req)
+
+ # We don't build wheels for legacy requirements if we
+ # don't have wheel installed or we don't have a cache dir
+ try:
+ import wheel # noqa: F401
+ build_legacy = bool(options.cache_dir)
+ except ImportError:
+ build_legacy = False
+
+ wb = WheelBuilder(
+ finder, preparer, wheel_cache,
+ build_options=[], global_options=[],
+ )
+
+ # Always build PEP 517 requirements
+ build_failures = wb.build(
+ pep517_requirements,
+ session=session, autobuilding=True
+ )
+
+ if build_legacy:
+ # We don't care about failures building legacy
+ # requirements, as we'll fall through to a direct
+ # install for those.
wb.build(
- requirement_set.requirements.values(),
+ legacy_requirements,
session=session, autobuilding=True
)
+ # If we're using PEP 517, we cannot do a direct install
+ # so we fail here.
+ if build_failures:
+ raise InstallationError(
+ "Could not build wheels for {} which use"
+ " PEP 517 and cannot be installed directly".format(
+ ", ".join(r.name for r in build_failures)))
+
to_install = resolver.get_installation_order(
requirement_set
)
@@ -472,7 +499,11 @@ class InstallCommand(RequirementCommand):
)
def _warn_about_conflicts(self, to_install):
- package_set, _dep_info = check_install_conflicts(to_install)
+ try:
+ package_set, _dep_info = check_install_conflicts(to_install)
+ except Exception:
+ logger.error("Error checking for conflicts.", exc_info=True)
+ return
missing, conflicting = _dep_info
# NOTE: There is some duplication here from pipenv.patched.notpip check
diff --git a/pipenv/patched/notpip/_internal/commands/list.py b/pipenv/patched/notpip/_internal/commands/list.py
index 577c0b5f..a2bd5be1 100644
--- a/pipenv/patched/notpip/_internal/commands/list.py
+++ b/pipenv/patched/notpip/_internal/commands/list.py
@@ -118,7 +118,6 @@ class ListCommand(Command):
index_urls=index_urls,
allow_all_prereleases=options.pre,
trusted_hosts=options.trusted_hosts,
- process_dependency_links=options.process_dependency_links,
session=session,
)
@@ -134,14 +133,18 @@ class ListCommand(Command):
include_editables=options.include_editable,
)
+ # get_not_required must be called firstly in order to find and
+ # filter out all dependencies correctly. Otherwise a package
+ # can't be identified as requirement because some parent packages
+ # could be filtered out before.
+ if options.not_required:
+ packages = self.get_not_required(packages, options)
+
if options.outdated:
packages = self.get_outdated(packages, options)
elif options.uptodate:
packages = self.get_uptodate(packages, options)
- if options.not_required:
- packages = self.get_not_required(packages, options)
-
self.output_package_listing(packages, options)
def get_outdated(self, packages, options):
@@ -168,16 +171,8 @@ class ListCommand(Command):
logger.debug('Ignoring indexes: %s', ','.join(index_urls))
index_urls = []
- dependency_links = []
- for dist in packages:
- if dist.has_metadata('dependency_links.txt'):
- dependency_links.extend(
- dist.get_metadata_lines('dependency_links.txt'),
- )
-
with self._build_session(options) as session:
finder = self._build_package_finder(options, index_urls, session)
- finder.add_dependency_links(dependency_links)
for dist in packages:
typ = 'unknown'
diff --git a/pipenv/patched/notpip/_internal/commands/wheel.py b/pipenv/patched/notpip/_internal/commands/wheel.py
index 08d695ab..801efff8 100644
--- a/pipenv/patched/notpip/_internal/commands/wheel.py
+++ b/pipenv/patched/notpip/_internal/commands/wheel.py
@@ -67,6 +67,8 @@ class WheelCommand(RequirementCommand):
help="Extra arguments to be supplied to 'setup.py bdist_wheel'.",
)
cmd_opts.add_option(cmdoptions.no_build_isolation())
+ cmd_opts.add_option(cmdoptions.use_pep517())
+ cmd_opts.add_option(cmdoptions.no_use_pep517())
cmd_opts.add_option(cmdoptions.constraints())
cmd_opts.add_option(cmdoptions.editable())
cmd_opts.add_option(cmdoptions.requirements())
@@ -157,6 +159,7 @@ class WheelCommand(RequirementCommand):
ignore_requires_python=options.ignore_requires_python,
ignore_installed=True,
isolated=options.isolated_mode,
+ use_pep517=options.use_pep517
)
resolver.resolve(requirement_set)
@@ -167,10 +170,10 @@ class WheelCommand(RequirementCommand):
global_options=options.global_options or [],
no_clean=options.no_clean,
)
- wheels_built_successfully = wb.build(
+ build_failures = wb.build(
requirement_set.requirements.values(), session=session,
)
- if not wheels_built_successfully:
+ if len(build_failures) != 0:
raise CommandError(
"Failed to build one or more wheels"
)
diff --git a/pipenv/patched/notpip/_internal/download.py b/pipenv/patched/notpip/_internal/download.py
index 8f4c38f5..f593c2f2 100644
--- a/pipenv/patched/notpip/_internal/download.py
+++ b/pipenv/patched/notpip/_internal/download.py
@@ -19,7 +19,6 @@ from pipenv.patched.notpip._vendor.lockfile import LockError
from pipenv.patched.notpip._vendor.requests.adapters import BaseAdapter, HTTPAdapter
from pipenv.patched.notpip._vendor.requests.auth import AuthBase, HTTPBasicAuth
from pipenv.patched.notpip._vendor.requests.models import CONTENT_CHUNK_SIZE, Response
-from pipenv.patched.notpip._vendor.requests.sessions import Session
from pipenv.patched.notpip._vendor.requests.structures import CaseInsensitiveDict
from pipenv.patched.notpip._vendor.requests.utils import get_netrc_auth
# NOTE: XMLRPC Client is not annotated in typeshed as on 2017-07-17, which is
@@ -27,7 +26,6 @@ from pipenv.patched.notpip._vendor.requests.utils import get_netrc_auth
from pipenv.patched.notpip._vendor.six.moves import xmlrpc_client # type: ignore
from pipenv.patched.notpip._vendor.six.moves.urllib import parse as urllib_parse
from pipenv.patched.notpip._vendor.six.moves.urllib import request as urllib_request
-from pipenv.patched.notpip._vendor.six.moves.urllib.parse import unquote as urllib_unquote
from pipenv.patched.notpip._vendor.urllib3.util import IS_PYOPENSSL
import pipenv.patched.notpip
@@ -40,14 +38,23 @@ from pipenv.patched.notpip._internal.utils.glibc import libc_ver
from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import (
ARCHIVE_EXTENSIONS, ask_path_exists, backup_dir, call_subprocess, consume,
- display_path, format_size, get_installed_version, rmtree, splitext,
- unpack_file,
+ display_path, format_size, get_installed_version, rmtree,
+ split_auth_from_netloc, splitext, unpack_file,
)
from pipenv.patched.notpip._internal.utils.setuptools_build import SETUPTOOLS_SHIM
from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.utils.ui import DownloadProgressProvider
from pipenv.patched.notpip._internal.vcs import vcs
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Optional, Tuple, Dict, IO, Text, Union
+ )
+ from pipenv.patched.notpip._internal.models.link import Link # noqa: F401
+ from pipenv.patched.notpip._internal.utils.hashes import Hashes # noqa: F401
+ from pipenv.patched.notpip._internal.vcs import AuthInfo # noqa: F401
+
try:
import ssl # noqa
except ImportError:
@@ -137,14 +144,15 @@ def user_agent():
class MultiDomainBasicAuth(AuthBase):
def __init__(self, prompting=True):
+ # type: (bool) -> None
self.prompting = prompting
- self.passwords = {}
+ self.passwords = {} # type: Dict[str, AuthInfo]
def __call__(self, req):
parsed = urllib_parse.urlparse(req.url)
- # Get the netloc without any embedded credentials
- netloc = parsed.netloc.rsplit("@", 1)[-1]
+ # Split the credentials from the netloc.
+ netloc, url_user_password = split_auth_from_netloc(parsed.netloc)
# Set the url of the request to the url without any credentials
req.url = urllib_parse.urlunparse(parsed[:1] + (netloc,) + parsed[2:])
@@ -152,9 +160,9 @@ class MultiDomainBasicAuth(AuthBase):
# Use any stored credentials that we have for this netloc
username, password = self.passwords.get(netloc, (None, None))
- # Extract credentials embedded in the url if we have none stored
+ # Use the credentials embedded in the url if we have none stored
if username is None:
- username, password = self.parse_credentials(parsed.netloc)
+ username, password = url_user_password
# Get creds from netrc if we still don't have them
if username is None and password is None:
@@ -200,6 +208,7 @@ class MultiDomainBasicAuth(AuthBase):
# Add our new username and password to the request
req = HTTPBasicAuth(username or "", password or "")(resp.request)
+ req.register_hook("response", self.warn_on_401)
# Send our new request
new_resp = resp.connection.send(req, **kwargs)
@@ -207,14 +216,11 @@ class MultiDomainBasicAuth(AuthBase):
return new_resp
- def parse_credentials(self, netloc):
- if "@" in netloc:
- userinfo = netloc.rsplit("@", 1)[0]
- if ":" in userinfo:
- user, pwd = userinfo.split(":", 1)
- return (urllib_unquote(user), urllib_unquote(pwd))
- return urllib_unquote(userinfo), None
- return None, None
+ def warn_on_401(self, resp, **kwargs):
+ # warn user that they provided incorrect credentials
+ if resp.status_code == 401:
+ logger.warning('401 Error, Credentials not correct for %s',
+ resp.request.url)
class LocalFSAdapter(BaseAdapter):
@@ -325,7 +331,7 @@ class InsecureHTTPAdapter(HTTPAdapter):
class PipSession(requests.Session):
- timeout = None
+ timeout = None # type: Optional[int]
def __init__(self, *args, **kwargs):
retries = kwargs.pop("retries", 0)
@@ -398,6 +404,7 @@ class PipSession(requests.Session):
def get_file_content(url, comes_from=None, session=None):
+ # type: (str, Optional[str], Optional[PipSession]) -> Tuple[str, Text]
"""Gets the content of a file; it may be a filename, file: URL, or
http: URL. Returns (location, content). Content is unicode.
@@ -448,6 +455,7 @@ _url_slash_drive_re = re.compile(r'/*([a-z])\|', re.I)
def is_url(name):
+ # type: (Union[str, Text]) -> bool
"""Returns true if the name looks like a URL"""
if ':' not in name:
return False
@@ -456,6 +464,7 @@ def is_url(name):
def url_to_path(url):
+ # type: (str) -> str
"""
Convert a file: URL to a path.
"""
@@ -473,6 +482,7 @@ def url_to_path(url):
def path_to_url(path):
+ # type: (Union[str, Text]) -> str
"""
Convert a path to a file: URL. The path will be made absolute and have
quoted path parts.
@@ -483,6 +493,7 @@ def path_to_url(path):
def is_archive_file(name):
+ # type: (str) -> bool
"""Return True if `name` is a considered as an archive file."""
ext = splitext(name)[1].lower()
if ext in ARCHIVE_EXTENSIONS:
@@ -503,14 +514,17 @@ def _get_used_vcs_backend(link):
def is_vcs_url(link):
+ # type: (Link) -> bool
return bool(_get_used_vcs_backend(link))
def is_file_url(link):
+ # type: (Link) -> bool
return link.url.lower().startswith('file:')
def is_dir_url(link):
+ # type: (Link) -> bool
"""Return whether a file:// Link points to a directory.
``link`` must not have any other scheme but file://. Call is_file_url()
@@ -525,7 +539,14 @@ def _progress_indicator(iterable, *args, **kwargs):
return iterable
-def _download_url(resp, link, content_file, hashes, progress_bar):
+def _download_url(
+ resp, # type: Response
+ link, # type: Link
+ content_file, # type: IO
+ hashes, # type: Hashes
+ progress_bar # type: str
+):
+ # type: (...) -> None
try:
total_length = int(resp.headers['content-length'])
except (ValueError, KeyError, TypeError):
@@ -647,8 +668,15 @@ def _copy_file(filename, location, link):
logger.info('Saved %s', display_path(download_location))
-def unpack_http_url(link, location, download_dir=None,
- session=None, hashes=None, progress_bar="on"):
+def unpack_http_url(
+ link, # type: Link
+ location, # type: str
+ download_dir=None, # type: Optional[str]
+ session=None, # type: Optional[PipSession]
+ hashes=None, # type: Optional[Hashes]
+ progress_bar="on" # type: str
+):
+ # type: (...) -> None
if session is None:
raise TypeError(
"unpack_http_url() missing 1 required keyword argument: 'session'"
@@ -685,7 +713,13 @@ def unpack_http_url(link, location, download_dir=None,
os.unlink(from_path)
-def unpack_file_url(link, location, download_dir=None, hashes=None):
+def unpack_file_url(
+ link, # type: Link
+ location, # type: str
+ download_dir=None, # type: Optional[str]
+ hashes=None # type: Optional[Hashes]
+):
+ # type: (...) -> None
"""Unpack link into location.
If download_dir is provided and link points to a file, make a copy
@@ -798,9 +832,16 @@ class PipXmlrpcTransport(xmlrpc_client.Transport):
raise
-def unpack_url(link, location, download_dir=None,
- only_download=False, session=None, hashes=None,
- progress_bar="on"):
+def unpack_url(
+ link, # type: Optional[Link]
+ location, # type: Optional[str]
+ download_dir=None, # type: Optional[str]
+ only_download=False, # type: bool
+ session=None, # type: Optional[PipSession]
+ hashes=None, # type: Optional[Hashes]
+ progress_bar="on" # type: str
+):
+ # type: (...) -> None
"""Unpack link.
If link is a VCS link:
if only_download, export into download_dir and ignore location
@@ -840,7 +881,14 @@ def unpack_url(link, location, download_dir=None,
write_delete_marker_file(location)
-def _download_http_url(link, session, temp_dir, hashes, progress_bar):
+def _download_http_url(
+ link, # type: Link
+ session, # type: PipSession
+ temp_dir, # type: str
+ hashes, # type: Hashes
+ progress_bar # type: str
+):
+ # type: (...) -> Tuple[str, str]
"""Download link url into temp_dir using provided session"""
target_url = link.url.split('#', 1)[0]
try:
@@ -900,6 +948,7 @@ def _download_http_url(link, session, temp_dir, hashes, progress_bar):
def _check_download_dir(link, download_dir, hashes):
+ # type: (Link, str, Hashes) -> Optional[str]
""" Check download_dir for previously downloaded file with correct hash
If a correct file is found return its path else None
"""
diff --git a/pipenv/patched/notpip/_internal/exceptions.py b/pipenv/patched/notpip/_internal/exceptions.py
index 2eadcf28..1342935d 100644
--- a/pipenv/patched/notpip/_internal/exceptions.py
+++ b/pipenv/patched/notpip/_internal/exceptions.py
@@ -5,6 +5,12 @@ from itertools import chain, groupby, repeat
from pipenv.patched.notpip._vendor.six import iteritems
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Optional # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_install import InstallRequirement # noqa: F401
+
class PipError(Exception):
"""Base pip exception"""
@@ -96,7 +102,7 @@ class HashError(InstallationError):
typically available earlier.
"""
- req = None
+ req = None # type: Optional[InstallRequirement]
head = ''
def body(self):
diff --git a/pipenv/patched/notpip/_internal/index.py b/pipenv/patched/notpip/_internal/index.py
index b4b02373..ad145fe4 100644
--- a/pipenv/patched/notpip/_internal/index.py
+++ b/pipenv/patched/notpip/_internal/index.py
@@ -16,7 +16,7 @@ from pipenv.patched.notpip._vendor.distlib.compat import unescape
from pipenv.patched.notpip._vendor.packaging import specifiers
from pipenv.patched.notpip._vendor.packaging.utils import canonicalize_name
from pipenv.patched.notpip._vendor.packaging.version import parse as parse_version
-from pipenv.patched.notpip._vendor.requests.exceptions import SSLError
+from pipenv.patched.notpip._vendor.requests.exceptions import RetryError, SSLError
from pipenv.patched.notpip._vendor.six.moves.urllib import parse as urllib_parse
from pipenv.patched.notpip._vendor.six.moves.urllib import request as urllib_request
@@ -31,14 +31,29 @@ from pipenv.patched.notpip._internal.models.index import PyPI
from pipenv.patched.notpip._internal.models.link import Link
from pipenv.patched.notpip._internal.pep425tags import get_supported
from pipenv.patched.notpip._internal.utils.compat import ipaddress
-from pipenv.patched.notpip._internal.utils.deprecation import deprecated
from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import (
- ARCHIVE_EXTENSIONS, SUPPORTED_EXTENSIONS, normalize_path,
- remove_auth_from_url,
+ ARCHIVE_EXTENSIONS, SUPPORTED_EXTENSIONS, WHEEL_EXTENSION, normalize_path,
+ redact_password_from_url,
)
from pipenv.patched.notpip._internal.utils.packaging import check_requires_python
-from pipenv.patched.notpip._internal.wheel import Wheel, wheel_ext
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+from pipenv.patched.notpip._internal.wheel import Wheel
+
+if MYPY_CHECK_RUNNING:
+ from logging import Logger # noqa: F401
+ from typing import ( # noqa: F401
+ Tuple, Optional, Any, List, Union, Callable, Set, Sequence,
+ Iterable, MutableMapping
+ )
+ from pipenv.patched.notpip._vendor.packaging.version import _BaseVersion # noqa: F401
+ from pipenv.patched.notpip._vendor.requests import Response # noqa: F401
+ from pipenv.patched.notpip._internal.req import InstallRequirement # noqa: F401
+ from pipenv.patched.notpip._internal.download import PipSession # noqa: F401
+
+ SecureOrigin = Tuple[str, str, Optional[str]]
+ BuildTag = Tuple[Any, ...] # either emply tuple or Tuple[int, str]
+ CandidateSortingKey = Tuple[int, _BaseVersion, BuildTag, Optional[int]]
__all__ = ['FormatControl', 'PackageFinder']
@@ -53,126 +68,190 @@ SECURE_ORIGINS = [
("file", "*", None),
# ssh is always secure.
("ssh", "*", "*"),
-]
+] # type: List[SecureOrigin]
logger = logging.getLogger(__name__)
-def _get_content_type(url, session):
- """Get the Content-Type of the given url, using a HEAD request"""
+def _match_vcs_scheme(url):
+ # type: (str) -> Optional[str]
+ """Look for VCS schemes in the URL.
+
+ Returns the matched VCS scheme, or None if there's no match.
+ """
+ from pipenv.patched.notpip._internal.vcs import VcsSupport
+ for scheme in VcsSupport.schemes:
+ if url.lower().startswith(scheme) and url[len(scheme)] in '+:':
+ return scheme
+ return None
+
+
+def _is_url_like_archive(url):
+ # type: (str) -> bool
+ """Return whether the URL looks like an archive.
+ """
+ filename = Link(url).filename
+ for bad_ext in ARCHIVE_EXTENSIONS:
+ if filename.endswith(bad_ext):
+ return True
+ return False
+
+
+class _NotHTML(Exception):
+ def __init__(self, content_type, request_desc):
+ # type: (str, str) -> None
+ super(_NotHTML, self).__init__(content_type, request_desc)
+ self.content_type = content_type
+ self.request_desc = request_desc
+
+
+def _ensure_html_header(response):
+ # type: (Response) -> None
+ """Check the Content-Type header to ensure the response contains HTML.
+
+ Raises `_NotHTML` if the content type is not text/html.
+ """
+ content_type = response.headers.get("Content-Type", "")
+ if not content_type.lower().startswith("text/html"):
+ raise _NotHTML(content_type, response.request.method)
+
+
+class _NotHTTP(Exception):
+ pass
+
+
+def _ensure_html_response(url, session):
+ # type: (str, PipSession) -> None
+ """Send a HEAD request to the URL, and ensure the response contains HTML.
+
+ Raises `_NotHTTP` if the URL is not available for a HEAD request, or
+ `_NotHTML` if the content type is not text/html.
+ """
scheme, netloc, path, query, fragment = urllib_parse.urlsplit(url)
if scheme not in {'http', 'https'}:
- # FIXME: some warning or something?
- # assertion error?
- return ''
+ raise _NotHTTP()
resp = session.head(url, allow_redirects=True)
resp.raise_for_status()
- return resp.headers.get("Content-Type", "")
+ _ensure_html_header(resp)
-def _handle_get_page_fail(link, reason, url, meth=None):
+def _get_html_response(url, session):
+ # type: (str, PipSession) -> Response
+ """Access an HTML page with GET, and return the response.
+
+ This consists of three parts:
+
+ 1. If the URL looks suspiciously like an archive, send a HEAD first to
+ check the Content-Type is HTML, to avoid downloading a large file.
+ Raise `_NotHTTP` if the content type cannot be determined, or
+ `_NotHTML` if it is not HTML.
+ 2. Actually perform the request. Raise HTTP exceptions on network failures.
+ 3. Check the Content-Type header to make sure we got HTML, and raise
+ `_NotHTML` otherwise.
+ """
+ if _is_url_like_archive(url):
+ _ensure_html_response(url, session=session)
+
+ logger.debug('Getting page %s', url)
+
+ resp = session.get(
+ url,
+ headers={
+ "Accept": "text/html",
+ # We don't want to blindly returned cached data for
+ # /simple/, because authors generally expecting that
+ # twine upload && pip install will function, but if
+ # they've done a pip install in the last ~10 minutes
+ # it won't. Thus by setting this to zero we will not
+ # blindly use any cached data, however the benefit of
+ # using max-age=0 instead of no-cache, is that we will
+ # still support conditional requests, so we will still
+ # minimize traffic sent in cases where the page hasn't
+ # changed at all, we will just always incur the round
+ # trip for the conditional GET now instead of only
+ # once per 10 minutes.
+ # For more information, please see pypa/pip#5670.
+ "Cache-Control": "max-age=0",
+ },
+ )
+ resp.raise_for_status()
+
+ # The check for archives above only works if the url ends with
+ # something that looks like an archive. However that is not a
+ # requirement of an url. Unless we issue a HEAD request on every
+ # url we cannot know ahead of time for sure if something is HTML
+ # or not. However we can check after we've downloaded it.
+ _ensure_html_header(resp)
+
+ return resp
+
+
+def _handle_get_page_fail(
+ link, # type: Link
+ reason, # type: Union[str, Exception]
+ meth=None # type: Optional[Callable[..., None]]
+):
+ # type: (...) -> None
if meth is None:
meth = logger.debug
meth("Could not fetch URL %s: %s - skipping", link, reason)
def _get_html_page(link, session=None):
+ # type: (Link, Optional[PipSession]) -> Optional[HTMLPage]
if session is None:
raise TypeError(
"_get_html_page() missing 1 required keyword argument: 'session'"
)
- url = link.url
- url = url.split('#', 1)[0]
+ url = link.url.split('#', 1)[0]
# Check for VCS schemes that do not support lookup as web pages.
- from pipenv.patched.notpip._internal.vcs import VcsSupport
- for scheme in VcsSupport.schemes:
- if url.lower().startswith(scheme) and url[len(scheme)] in '+:':
- logger.debug('Cannot look at %s URL %s', scheme, link)
- return None
+ vcs_scheme = _match_vcs_scheme(url)
+ if vcs_scheme:
+ logger.debug('Cannot look at %s URL %s', vcs_scheme, link)
+ return None
+
+ # Tack index.html onto file:// URLs that point to directories
+ scheme, _, path, _, _, _ = urllib_parse.urlparse(url)
+ if (scheme == 'file' and os.path.isdir(urllib_request.url2pathname(path))):
+ # add trailing slash if not present so urljoin doesn't trim
+ # final segment
+ if not url.endswith('/'):
+ url += '/'
+ url = urllib_parse.urljoin(url, 'index.html')
+ logger.debug(' file: URL is directory, getting %s', url)
try:
- filename = link.filename
- for bad_ext in ARCHIVE_EXTENSIONS:
- if filename.endswith(bad_ext):
- content_type = _get_content_type(url, session=session)
- if content_type.lower().startswith('text/html'):
- break
- else:
- logger.debug(
- 'Skipping page %s because of Content-Type: %s',
- link,
- content_type,
- )
- return
-
- logger.debug('Getting page %s', url)
-
- # Tack index.html onto file:// URLs that point to directories
- (scheme, netloc, path, params, query, fragment) = \
- urllib_parse.urlparse(url)
- if (scheme == 'file' and
- os.path.isdir(urllib_request.url2pathname(path))):
- # add trailing slash if not present so urljoin doesn't trim
- # final segment
- if not url.endswith('/'):
- url += '/'
- url = urllib_parse.urljoin(url, 'index.html')
- logger.debug(' file: URL is directory, getting %s', url)
-
- resp = session.get(
- url,
- headers={
- "Accept": "text/html",
- # We don't want to blindly returned cached data for
- # /simple/, because authors generally expecting that
- # twine upload && pip install will function, but if
- # they've done a pip install in the last ~10 minutes
- # it won't. Thus by setting this to zero we will not
- # blindly use any cached data, however the benefit of
- # using max-age=0 instead of no-cache, is that we will
- # still support conditional requests, so we will still
- # minimize traffic sent in cases where the page hasn't
- # changed at all, we will just always incur the round
- # trip for the conditional GET now instead of only
- # once per 10 minutes.
- # For more information, please see pypa/pip#5670.
- "Cache-Control": "max-age=0",
- },
+ resp = _get_html_response(url, session=session)
+ except _NotHTTP as exc:
+ logger.debug(
+ 'Skipping page %s because it looks like an archive, and cannot '
+ 'be checked by HEAD.', link,
+ )
+ except _NotHTML as exc:
+ logger.debug(
+ 'Skipping page %s because the %s request got Content-Type: %s',
+ link, exc.request_desc, exc.content_type,
)
- resp.raise_for_status()
-
- # The check for archives above only works if the url ends with
- # something that looks like an archive. However that is not a
- # requirement of an url. Unless we issue a HEAD request on every
- # url we cannot know ahead of time for sure if something is HTML
- # or not. However we can check after we've downloaded it.
- content_type = resp.headers.get('Content-Type', 'unknown')
- if not content_type.lower().startswith("text/html"):
- logger.debug(
- 'Skipping page %s because of Content-Type: %s',
- link,
- content_type,
- )
- return
-
- inst = HTMLPage(resp.content, resp.url, resp.headers)
except requests.HTTPError as exc:
- _handle_get_page_fail(link, exc, url)
+ _handle_get_page_fail(link, exc)
+ except RetryError as exc:
+ _handle_get_page_fail(link, exc)
except SSLError as exc:
reason = "There was a problem confirming the ssl certificate: "
reason += str(exc)
- _handle_get_page_fail(link, reason, url, meth=logger.info)
+ _handle_get_page_fail(link, reason, meth=logger.info)
except requests.ConnectionError as exc:
- _handle_get_page_fail(link, "connection error: %s" % exc, url)
+ _handle_get_page_fail(link, "connection error: %s" % exc)
except requests.Timeout:
- _handle_get_page_fail(link, "timed out", url)
+ _handle_get_page_fail(link, "timed out")
else:
- return inst
+ return HTMLPage(resp.content, resp.url, resp.headers)
+ return None
class PackageFinder(object):
@@ -182,11 +261,21 @@ class PackageFinder(object):
packages, by reading pages and looking for appropriate links.
"""
- def __init__(self, find_links, index_urls, allow_all_prereleases=False,
- trusted_hosts=None, process_dependency_links=False,
- session=None, format_control=None, platform=None,
- versions=None, abi=None, implementation=None,
- prefer_binary=False):
+ def __init__(
+ self,
+ find_links, # type: List[str]
+ index_urls, # type: List[str]
+ allow_all_prereleases=False, # type: bool
+ trusted_hosts=None, # type: Optional[Iterable[str]]
+ session=None, # type: Optional[PipSession]
+ format_control=None, # type: Optional[FormatControl]
+ platform=None, # type: Optional[str]
+ versions=None, # type: Optional[List[str]]
+ abi=None, # type: Optional[str]
+ implementation=None, # type: Optional[str]
+ prefer_binary=False # type: bool
+ ):
+ # type: (...) -> None
"""Create a PackageFinder.
:param format_control: A FormatControl object or None. Used to control
@@ -215,7 +304,7 @@ class PackageFinder(object):
# it and if it exists, use the normalized version.
# This is deliberately conservative - it might be fine just to
# blindly normalize anything starting with a ~...
- self.find_links = []
+ self.find_links = [] # type: List[str]
for link in find_links:
if link.startswith('~'):
new_link = normalize_path(link)
@@ -224,10 +313,9 @@ class PackageFinder(object):
self.find_links.append(link)
self.index_urls = index_urls
- self.dependency_links = []
# These are boring links that have already been logged somehow:
- self.logged_links = set()
+ self.logged_links = set() # type: Set[Link]
self.format_control = format_control or FormatControl(set(), set())
@@ -235,14 +323,11 @@ class PackageFinder(object):
self.secure_origins = [
("*", host, "*")
for host in (trusted_hosts if trusted_hosts else [])
- ]
+ ] # type: List[SecureOrigin]
# Do we want to allow _all_ pre-releases?
self.allow_all_prereleases = allow_all_prereleases
- # Do we process dependency links?
- self.process_dependency_links = process_dependency_links
-
# The Session we'll use to make requests
self.session = session
@@ -274,11 +359,12 @@ class PackageFinder(object):
break
def get_formatted_locations(self):
+ # type: () -> str
lines = []
if self.index_urls and self.index_urls != [PyPI.simple_url]:
lines.append(
"Looking in indexes: {}".format(", ".join(
- remove_auth_from_url(url) for url in self.index_urls))
+ redact_password_from_url(url) for url in self.index_urls))
)
if self.find_links:
lines.append(
@@ -286,21 +372,6 @@ class PackageFinder(object):
)
return "\n".join(lines)
- def add_dependency_links(self, links):
- # FIXME: this shouldn't be global list this, it should only
- # apply to requirements of the package that specifies the
- # dependency_links value
- # FIXME: also, we should track comes_from (i.e., use Link)
- if self.process_dependency_links:
- deprecated(
- "Dependency Links processing has been deprecated and will be "
- "removed in a future release.",
- replacement="PEP 508 URL dependencies",
- gone_in="18.2",
- issue=4187,
- )
- self.dependency_links.extend(links)
-
@staticmethod
def get_extras_links(links):
requires = []
@@ -316,12 +387,11 @@ class PackageFinder(object):
extras[link[1:-1]] = current_list
else:
current_list.append(link)
-
return extras
-
@staticmethod
def _sort_locations(locations, expand_dir=False):
+ # type: (Sequence[str], bool) -> Tuple[List[str], List[str]]
"""
Sort locations into "files" (archives) and "urls", and return
a pair of lists (files,urls)
@@ -354,6 +424,11 @@ class PackageFinder(object):
sort_path(os.path.join(path, item))
elif is_file_url:
urls.append(url)
+ else:
+ logger.warning(
+ "Path '{0}' is ignored: "
+ "it is a directory.".format(path),
+ )
elif os.path.isfile(path):
sort_path(path)
else:
@@ -373,6 +448,7 @@ class PackageFinder(object):
return files, urls
def _candidate_sort_key(self, candidate, ignore_compatibility=True):
+ # type: (InstallationCandidate, bool) -> CandidateSortingKey
"""
Function used to generate link sort key for link tuples.
The greater the return value, the more preferred it is.
@@ -387,7 +463,7 @@ class PackageFinder(object):
with the same version, would have to be considered equal
"""
support_num = len(self.valid_tags)
- build_tag = tuple()
+ build_tag = tuple() # type: BuildTag
binary_preference = 0
if candidate.location.is_wheel:
# can raise InvalidWheelFilename
@@ -404,7 +480,6 @@ class PackageFinder(object):
pri = -(wheel.support_index_min(tags=tags))
except TypeError:
pri = -(support_num)
-
if wheel.build_tag is not None:
match = re.match(r'^(\d+)(.*)$', wheel.build_tag)
build_tag_groups = match.groups()
@@ -414,6 +489,7 @@ class PackageFinder(object):
return (binary_preference, candidate.version, build_tag, pri)
def _validate_secure_origin(self, logger, location):
+ # type: (Logger, Link) -> bool
# Determine if this url used a secure transport mechanism
parsed = urllib_parse.urlparse(str(location))
origin = (parsed.scheme, parsed.hostname, parsed.port)
@@ -445,7 +521,9 @@ class PackageFinder(object):
network = ipaddress.ip_network(
secure_origin[1]
if isinstance(secure_origin[1], six.text_type)
- else secure_origin[1].decode("utf8")
+ # setting secure_origin[1] to proper Union[bytes, str]
+ # creates problems in other places
+ else secure_origin[1].decode("utf8") # type: ignore
)
except ValueError:
# We don't have both a valid address or a valid network, so
@@ -485,6 +563,7 @@ class PackageFinder(object):
return False
def _get_index_urls_locations(self, project_name):
+ # type: (str) -> List[str]
"""Returns the locations found via self.index_urls
Checks the url_name on the main (first in the list) index and
@@ -507,9 +586,10 @@ class PackageFinder(object):
return [mkurl_pypi_url(url) for url in self.index_urls]
def find_all_candidates(self, project_name):
+ # type: (str) -> List[Optional[InstallationCandidate]]
"""Find all available InstallationCandidate for project_name
- This checks index_urls, find_links and dependency_links.
+ This checks index_urls and find_links.
All versions found are returned as an InstallationCandidate list.
See _link_package_versions for details on which files are accepted
@@ -519,21 +599,18 @@ class PackageFinder(object):
fl_file_loc, fl_url_loc = self._sort_locations(
self.find_links, expand_dir=True,
)
- dep_file_loc, dep_url_loc = self._sort_locations(self.dependency_links)
file_locations = (Link(url) for url in itertools.chain(
- index_file_loc, fl_file_loc, dep_file_loc,
+ index_file_loc, fl_file_loc,
))
# We trust every url that the user has given us whether it was given
- # via --index-url or --find-links
- # We explicitly do not trust links that came from dependency_links
+ # via --index-url or --find-links.
# We want to filter out any thing which does not have a secure origin.
url_locations = [
link for link in itertools.chain(
(Link(url) for url in index_url_loc),
(Link(url) for url in fl_url_loc),
- (Link(url) for url in dep_url_loc),
)
if self._validate_secure_origin(logger, link)
]
@@ -564,17 +641,6 @@ class PackageFinder(object):
self._package_versions(page.iter_links(), search)
)
- dependency_versions = self._package_versions(
- (Link(url) for url in self.dependency_links), search
- )
- if dependency_versions:
- logger.debug(
- 'dependency_links found: %s',
- ', '.join([
- version.location.url for version in dependency_versions
- ])
- )
-
file_versions = self._package_versions(file_locations, search)
if file_versions:
file_versions.sort(reverse=True)
@@ -587,12 +653,10 @@ class PackageFinder(object):
)
# This is an intentional priority ordering
- return (
- file_versions + find_links_versions + page_versions +
- dependency_versions
- )
+ return file_versions + find_links_versions + page_versions
def find_requirement(self, req, upgrade, ignore_compatibility=False):
+ # type: (InstallRequirement, bool, bool) -> Optional[Link]
"""Try to find a Link matching req
Expects req, an InstallRequirement and upgrade, a boolean
@@ -692,20 +756,18 @@ class PackageFinder(object):
return best_candidate.location
def _get_pages(self, locations, project_name):
+ # type: (Iterable[Link], str) -> Iterable[HTMLPage]
"""
Yields (page, page_url) from the given locations, skipping
locations that have errors.
"""
- seen = set()
+ seen = set() # type: Set[Link]
for location in locations:
if location in seen:
continue
seen.add(location)
- try:
- page = self._get_page(location)
- except requests.HTTPError:
- continue
+ page = _get_html_page(location, session=self.session)
if page is None:
continue
@@ -714,12 +776,13 @@ class PackageFinder(object):
_py_version_re = re.compile(r'-py([123]\.?[0-9]?)$')
def _sort_links(self, links):
+ # type: (Iterable[Link]) -> List[Link]
"""
Returns elements of links in order, non-egg links first, egg links
second, while eliminating duplicates
"""
eggs, no_eggs = [], []
- seen = set()
+ seen = set() # type: Set[Link]
for link in links:
if link not in seen:
seen.add(link)
@@ -729,7 +792,12 @@ class PackageFinder(object):
no_eggs.append(link)
return no_eggs + eggs
- def _package_versions(self, links, search):
+ def _package_versions(
+ self,
+ links, # type: Iterable[Link]
+ search # type: Search
+ ):
+ # type: (...) -> List[Optional[InstallationCandidate]]
result = []
for link in self._sort_links(links):
v = self._link_package_versions(link, search)
@@ -738,11 +806,13 @@ class PackageFinder(object):
return result
def _log_skipped_link(self, link, reason):
+ # type: (Link, str) -> None
if link not in self.logged_links:
logger.debug('Skipping link %s; %s', link, reason)
self.logged_links.add(link)
def _link_package_versions(self, link, search, ignore_compatibility=True):
+ # type: (Link, Search, bool) -> Optional[InstallationCandidate]
"""Return an InstallationCandidate or None"""
version = None
if link.egg_fragment:
@@ -752,51 +822,51 @@ class PackageFinder(object):
egg_info, ext = link.splitext()
if not ext:
self._log_skipped_link(link, 'not a file')
- return
+ return None
if ext not in SUPPORTED_EXTENSIONS:
self._log_skipped_link(
link, 'unsupported archive format: %s' % ext,
)
- return
- if "binary" not in search.formats and ext == wheel_ext and not ignore_compatibility:
+ return None
+ if "binary" not in search.formats and ext == WHEEL_EXTENSION and not ignore_compatibility:
self._log_skipped_link(
link, 'No binaries permitted for %s' % search.supplied,
)
- return
+ return None
if "macosx10" in link.path and ext == '.zip' and not ignore_compatibility:
self._log_skipped_link(link, 'macosx10 one')
- return
- if ext == wheel_ext:
+ return None
+ if ext == WHEEL_EXTENSION:
try:
wheel = Wheel(link.filename)
except InvalidWheelFilename:
self._log_skipped_link(link, 'invalid wheel filename')
- return
+ return None
if canonicalize_name(wheel.name) != search.canonical:
self._log_skipped_link(
link, 'wrong project name (not %s)' % search.supplied)
- return
+ return None
if not wheel.supported(self.valid_tags) and not ignore_compatibility:
self._log_skipped_link(
link, 'it is not compatible with this Python')
- return
+ return None
version = wheel.version
# This should be up by the search.ok_binary check, but see issue 2700.
- if "source" not in search.formats and ext != wheel_ext:
+ if "source" not in search.formats and ext != WHEEL_EXTENSION:
self._log_skipped_link(
link, 'No sources permitted for %s' % search.supplied,
)
- return
+ return None
if not version:
- version = egg_info_matches(egg_info, search.supplied, link)
- if version is None:
+ version = _egg_info_matches(egg_info, search.canonical)
+ if not version:
self._log_skipped_link(
link, 'Missing project version for %s' % search.supplied)
- return
+ return None
match = self._py_version_re.search(version)
if match:
@@ -805,7 +875,7 @@ class PackageFinder(object):
if py_version != sys.version[:3]:
self._log_skipped_link(
link, 'Python version is incorrect')
- return
+ return None
try:
support_this_python = check_requires_python(link.requires_python)
except specifiers.InvalidSpecifier:
@@ -814,45 +884,57 @@ class PackageFinder(object):
support_this_python = True
if not support_this_python and not ignore_compatibility:
- logger.debug("The package %s is incompatible with the python"
- "version in use. Acceptable python versions are:%s",
+ logger.debug("The package %s is incompatible with the python "
+ "version in use. Acceptable python versions are: %s",
link, link.requires_python)
- return
+ return None
logger.debug('Found link %s, version: %s', link, version)
return InstallationCandidate(search.supplied, version, link, link.requires_python)
- def _get_page(self, link):
- return _get_html_page(link, session=self.session)
+
+def _find_name_version_sep(egg_info, canonical_name):
+ # type: (str, str) -> int
+ """Find the separator's index based on the package's canonical name.
+
+ `egg_info` must be an egg info string for the given package, and
+ `canonical_name` must be the package's canonical name.
+
+ This function is needed since the canonicalized name does not necessarily
+ have the same length as the egg info's name part. An example::
+
+ >>> egg_info = 'foo__bar-1.0'
+ >>> canonical_name = 'foo-bar'
+ >>> _find_name_version_sep(egg_info, canonical_name)
+ 8
+ """
+ # Project name and version must be separated by one single dash. Find all
+ # occurrences of dashes; if the string in front of it matches the canonical
+ # name, this is the one separating the name and version parts.
+ for i, c in enumerate(egg_info):
+ if c != "-":
+ continue
+ if canonicalize_name(egg_info[:i]) == canonical_name:
+ return i
+ raise ValueError("{} does not match {}".format(egg_info, canonical_name))
-def egg_info_matches(
- egg_info, search_name, link,
- _egg_info_re=re.compile(r'([a-z0-9_.]+)-([a-z0-9_.!+-]+)', re.I)):
+def _egg_info_matches(egg_info, canonical_name):
+ # type: (str, str) -> Optional[str]
"""Pull the version part out of a string.
:param egg_info: The string to parse. E.g. foo-2.1
- :param search_name: The name of the package this belongs to. None to
- infer the name. Note that this cannot unambiguously parse strings
- like foo-2-2 which might be foo, 2-2 or foo-2, 2.
- :param link: The link the string came from, for logging on failure.
+ :param canonical_name: The canonicalized name of the package this
+ belongs to.
"""
- match = _egg_info_re.search(egg_info)
- if not match:
- logger.debug('Could not parse version from link: %s', link)
+ try:
+ version_start = _find_name_version_sep(egg_info, canonical_name) + 1
+ except ValueError:
return None
- if search_name is None:
- full_match = match.group(0)
- return full_match.split('-', 1)[-1]
- name = match.group(0).lower()
- # To match the "safe" name that pkg_resources creates:
- name = name.replace('_', '-')
- # project name and version must be separated by a dash
- look_for = search_name.lower() + "-"
- if name.startswith(look_for):
- return match.group(0)[len(look_for):]
- else:
+ version = egg_info[version_start:]
+ if not version:
return None
+ return version
def _determine_base_url(document, page_url):
@@ -888,6 +970,7 @@ _CLEAN_LINK_RE = re.compile(r'[^a-z0-9$&+,/:;=?@.#%_\\|-]', re.I)
def _clean_link(url):
+ # type: (str) -> str
"""Makes sure a link is fully encoded. That is, if a ' ' shows up in
the link, it will be rewritten to %20 (while not over-quoting
% or other characters)."""
@@ -898,14 +981,16 @@ class HTMLPage(object):
"""Represents one page, along with its URL"""
def __init__(self, content, url, headers=None):
+ # type: (bytes, str, MutableMapping[str, str]) -> None
self.content = content
self.url = url
self.headers = headers
def __str__(self):
- return self.url
+ return redact_password_from_url(self.url)
def iter_links(self):
+ # type: () -> Iterable[Link]
"""Yields all links in the page"""
document = html5lib.parse(
self.content,
diff --git a/pipenv/patched/notpip/_internal/locations.py b/pipenv/patched/notpip/_internal/locations.py
index 3c7d5bd8..e8f5a268 100644
--- a/pipenv/patched/notpip/_internal/locations.py
+++ b/pipenv/patched/notpip/_internal/locations.py
@@ -12,6 +12,11 @@ from distutils.command.install import SCHEME_KEYS # type: ignore
from pipenv.patched.notpip._internal.utils import appdirs
from pipenv.patched.notpip._internal.utils.compat import WINDOWS, expanduser
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Any, Union, Dict, List, Optional # noqa: F401
+
# Application Directories
USER_CACHE_DIR = appdirs.user_cache_dir("pip")
@@ -28,6 +33,7 @@ PIP_DELETE_MARKER_FILENAME = 'pip-delete-this-directory.txt'
def write_delete_marker_file(directory):
+ # type: (str) -> None
"""
Write the pip delete marker file into this directory.
"""
@@ -37,6 +43,7 @@ def write_delete_marker_file(directory):
def running_under_virtualenv():
+ # type: () -> bool
"""
Return True if we're running inside a virtualenv, False otherwise.
@@ -50,6 +57,7 @@ def running_under_virtualenv():
def virtualenv_no_global():
+ # type: () -> bool
"""
Return True if in a venv and no system site packages.
"""
@@ -59,6 +67,8 @@ def virtualenv_no_global():
no_global_file = os.path.join(site_mod_dir, 'no-global-site-packages.txt')
if running_under_virtualenv() and os.path.isfile(no_global_file):
return True
+ else:
+ return False
if running_under_virtualenv():
@@ -80,7 +90,8 @@ src_prefix = os.path.abspath(src_prefix)
# FIXME doesn't account for venv linked to global site-packages
-site_packages = sysconfig.get_path("purelib")
+site_packages = sysconfig.get_path("purelib") # type: Optional[str]
+
# This is because of a bug in PyPy's sysconfig module, see
# https://bitbucket.org/pypy/pypy/issues/2506/sysconfig-returns-incorrect-paths
# for more information.
@@ -135,6 +146,7 @@ new_config_file = os.path.join(appdirs.user_config_dir("pip"), config_basename)
def distutils_scheme(dist_name, user=False, home=None, root=None,
isolated=False, prefix=None):
+ # type:(str, bool, str, str, bool, str) -> dict
"""
Return a distutils install scheme
"""
@@ -146,12 +158,15 @@ def distutils_scheme(dist_name, user=False, home=None, root=None,
extra_dist_args = {"script_args": ["--no-user-cfg"]}
else:
extra_dist_args = {}
- dist_args = {'name': dist_name}
+ dist_args = {'name': dist_name} # type: Dict[str, Union[str, List[str]]]
dist_args.update(extra_dist_args)
d = Distribution(dist_args)
+ # Ignoring, typeshed issue reported python/typeshed/issues/2567
d.parse_config_files()
- i = d.get_command_obj('install', create=True)
+ # NOTE: Ignoring type since mypy can't find attributes on 'Command'
+ i = d.get_command_obj('install', create=True) # type: Any
+ assert i is not None
# NOTE: setting user or home has the side-effect of creating the home dir
# or user base for installations during finalize_options()
# ideally, we'd prefer a scheme class that has no side-effects.
@@ -171,7 +186,9 @@ def distutils_scheme(dist_name, user=False, home=None, root=None,
# platlib). Note, i.install_lib is *always* set after
# finalize_options(); we only want to override here if the user
# has explicitly requested it hence going back to the config
- if 'install_lib' in d.get_option_dict('install'):
+
+ # Ignoring, typeshed issue reported python/typeshed/issues/2567
+ if 'install_lib' in d.get_option_dict('install'): # type: ignore
scheme.update(dict(purelib=i.install_lib, platlib=i.install_lib))
if running_under_virtualenv():
diff --git a/pipenv/patched/notpip/_internal/models/candidate.py b/pipenv/patched/notpip/_internal/models/candidate.py
index 9627589e..adc3550e 100644
--- a/pipenv/patched/notpip/_internal/models/candidate.py
+++ b/pipenv/patched/notpip/_internal/models/candidate.py
@@ -1,6 +1,12 @@
from pipenv.patched.notpip._vendor.packaging.version import parse as parse_version
from pipenv.patched.notpip._internal.utils.models import KeyBasedCompareMixin
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from pipenv.patched.notpip._vendor.packaging.version import _BaseVersion # noqa: F401
+ from pipenv.patched.notpip._internal.models.link import Link # noqa: F401
+ from typing import Any, Union # noqa: F401
class InstallationCandidate(KeyBasedCompareMixin):
@@ -8,8 +14,9 @@ class InstallationCandidate(KeyBasedCompareMixin):
"""
def __init__(self, project, version, location, requires_python=None):
+ # type: (Any, str, Link, Any) -> None
self.project = project
- self.version = parse_version(version)
+ self.version = parse_version(version) # type: _BaseVersion
self.location = location
self.requires_python = requires_python
@@ -19,6 +26,7 @@ class InstallationCandidate(KeyBasedCompareMixin):
)
def __repr__(self):
+ # type: () -> str
return "".format(
self.project, self.version, self.location,
)
diff --git a/pipenv/patched/notpip/_internal/models/format_control.py b/pipenv/patched/notpip/_internal/models/format_control.py
index caad3cba..7172ad9f 100644
--- a/pipenv/patched/notpip/_internal/models/format_control.py
+++ b/pipenv/patched/notpip/_internal/models/format_control.py
@@ -1,16 +1,24 @@
from pipenv.patched.notpip._vendor.packaging.utils import canonicalize_name
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Optional, Set, FrozenSet # noqa: F401
+
class FormatControl(object):
- """A helper class for controlling formats from which packages are installed.
- If a field is falsy, it isn't set. If it is {':all:'}, it should match all
- packages except those listed in the other field. Only one field can be set
- to {':all:'} at a time. The rest of the time exact package name matches
- are listed, with any given package only showing up in one field at a time.
+ """Helper for managing formats from which a package can be installed.
"""
+
def __init__(self, no_binary=None, only_binary=None):
- self.no_binary = set() if no_binary is None else no_binary
- self.only_binary = set() if only_binary is None else only_binary
+ # type: (Optional[Set], Optional[Set]) -> None
+ if no_binary is None:
+ no_binary = set()
+ if only_binary is None:
+ only_binary = set()
+
+ self.no_binary = no_binary
+ self.only_binary = only_binary
def __eq__(self, other):
return self.__dict__ == other.__dict__
@@ -27,6 +35,7 @@ class FormatControl(object):
@staticmethod
def handle_mutual_excludes(value, target, other):
+ # type: (str, Optional[Set], Optional[Set]) -> None
new = value.split(',')
while ':all:' in new:
other.clear()
@@ -45,6 +54,7 @@ class FormatControl(object):
target.add(name)
def get_allowed_formats(self, canonical_name):
+ # type: (str) -> FrozenSet
result = {"binary", "source"}
if canonical_name in self.only_binary:
result.discard('source')
@@ -57,6 +67,7 @@ class FormatControl(object):
return frozenset(result)
def disallow_binaries(self):
+ # type: () -> None
self.handle_mutual_excludes(
':all:', self.no_binary, self.only_binary,
)
diff --git a/pipenv/patched/notpip/_internal/models/index.py b/pipenv/patched/notpip/_internal/models/index.py
index 0983fc9c..b2895dab 100644
--- a/pipenv/patched/notpip/_internal/models/index.py
+++ b/pipenv/patched/notpip/_internal/models/index.py
@@ -6,6 +6,7 @@ class PackageIndex(object):
"""
def __init__(self, url, file_storage_domain):
+ # type: (str, str) -> None
super(PackageIndex, self).__init__()
self.url = url
self.netloc = urllib_parse.urlsplit(url).netloc
@@ -18,6 +19,7 @@ class PackageIndex(object):
self.file_storage_domain = file_storage_domain
def _url_for_path(self, path):
+ # type: (str) -> str
return urllib_parse.urljoin(self.url, path)
diff --git a/pipenv/patched/notpip/_internal/models/link.py b/pipenv/patched/notpip/_internal/models/link.py
index 686af1d0..ded4de43 100644
--- a/pipenv/patched/notpip/_internal/models/link.py
+++ b/pipenv/patched/notpip/_internal/models/link.py
@@ -4,9 +4,15 @@ import re
from pipenv.patched.notpip._vendor.six.moves.urllib import parse as urllib_parse
from pipenv.patched.notpip._internal.download import path_to_url
-from pipenv.patched.notpip._internal.utils.misc import splitext
+from pipenv.patched.notpip._internal.utils.misc import (
+ WHEEL_EXTENSION, redact_password_from_url, splitext,
+)
from pipenv.patched.notpip._internal.utils.models import KeyBasedCompareMixin
-from pipenv.patched.notpip._internal.wheel import wheel_ext
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Optional, Tuple, Union, Text # noqa: F401
+ from pipenv.patched.notpip._internal.index import HTMLPage # noqa: F401
class Link(KeyBasedCompareMixin):
@@ -14,6 +20,7 @@ class Link(KeyBasedCompareMixin):
"""
def __init__(self, url, comes_from=None, requires_python=None):
+ # type: (str, Optional[Union[str, HTMLPage]], Optional[str]) -> None
"""
url:
url of the resource pointed to (href of the link)
@@ -44,15 +51,17 @@ class Link(KeyBasedCompareMixin):
else:
rp = ''
if self.comes_from:
- return '%s (from %s)%s' % (self.url, self.comes_from, rp)
+ return '%s (from %s)%s' % (redact_password_from_url(self.url),
+ self.comes_from, rp)
else:
- return str(self.url)
+ return redact_password_from_url(str(self.url))
def __repr__(self):
return '' % self
@property
def filename(self):
+ # type: () -> str
_, netloc, path, _, _ = urllib_parse.urlsplit(self.url)
name = posixpath.basename(path.rstrip('/')) or netloc
name = urllib_parse.unquote(name)
@@ -61,25 +70,31 @@ class Link(KeyBasedCompareMixin):
@property
def scheme(self):
+ # type: () -> str
return urllib_parse.urlsplit(self.url)[0]
@property
def netloc(self):
+ # type: () -> str
return urllib_parse.urlsplit(self.url)[1]
@property
def path(self):
+ # type: () -> str
return urllib_parse.unquote(urllib_parse.urlsplit(self.url)[2])
def splitext(self):
+ # type: () -> Tuple[str, str]
return splitext(posixpath.basename(self.path.rstrip('/')))
@property
def ext(self):
+ # type: () -> str
return self.splitext()[1]
@property
def url_without_fragment(self):
+ # type: () -> str
scheme, netloc, path, query, fragment = urllib_parse.urlsplit(self.url)
return urllib_parse.urlunsplit((scheme, netloc, path, query, None))
@@ -87,6 +102,7 @@ class Link(KeyBasedCompareMixin):
@property
def egg_fragment(self):
+ # type: () -> Optional[str]
match = self._egg_fragment_re.search(self.url)
if not match:
return None
@@ -96,6 +112,7 @@ class Link(KeyBasedCompareMixin):
@property
def subdirectory_fragment(self):
+ # type: () -> Optional[str]
match = self._subdirectory_fragment_re.search(self.url)
if not match:
return None
@@ -107,6 +124,7 @@ class Link(KeyBasedCompareMixin):
@property
def hash(self):
+ # type: () -> Optional[str]
match = self._hash_re.search(self.url)
if match:
return match.group(2)
@@ -114,6 +132,7 @@ class Link(KeyBasedCompareMixin):
@property
def hash_name(self):
+ # type: () -> Optional[str]
match = self._hash_re.search(self.url)
if match:
return match.group(1)
@@ -121,14 +140,17 @@ class Link(KeyBasedCompareMixin):
@property
def show_url(self):
+ # type: () -> Optional[str]
return posixpath.basename(self.url.split('#', 1)[0].split('?', 1)[0])
@property
def is_wheel(self):
- return self.ext == wheel_ext
+ # type: () -> bool
+ return self.ext == WHEEL_EXTENSION
@property
def is_artifact(self):
+ # type: () -> bool
"""
Determines if this points to an actual artifact (e.g. a tarball) or if
it points to an "abstract" thing like a path or a VCS location.
diff --git a/pipenv/patched/notpip/_internal/operations/check.py b/pipenv/patched/notpip/_internal/operations/check.py
index 9c8ea08e..a73611d4 100644
--- a/pipenv/patched/notpip/_internal/operations/check.py
+++ b/pipenv/patched/notpip/_internal/operations/check.py
@@ -1,18 +1,22 @@
"""Validation of dependencies of packages
"""
+import logging
from collections import namedtuple
from pipenv.patched.notpip._vendor.packaging.utils import canonicalize_name
+from pipenv.patched.notpip._vendor.pkg_resources import RequirementParseError
from pipenv.patched.notpip._internal.operations.prepare import make_abstract_dist
from pipenv.patched.notpip._internal.utils.misc import get_installed_distributions
from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+logger = logging.getLogger(__name__)
+
if MYPY_CHECK_RUNNING:
from pipenv.patched.notpip._internal.req.req_install import InstallRequirement # noqa: F401
from typing import ( # noqa: F401
- Any, Callable, Dict, Iterator, Optional, Set, Tuple, List
+ Any, Callable, Dict, Optional, Set, Tuple, List
)
# Shorthands
@@ -28,7 +32,7 @@ PackageDetails = namedtuple('PackageDetails', ['version', 'requires'])
def create_package_set_from_installed(**kwargs):
- # type: (**Any) -> PackageSet
+ # type: (**Any) -> Tuple[PackageSet, bool]
"""Converts a list of distributions into a PackageSet.
"""
# Default to using all packages installed on the system
@@ -36,10 +40,16 @@ def create_package_set_from_installed(**kwargs):
kwargs = {"local_only": False, "skip": ()}
package_set = {}
+ problems = False
for dist in get_installed_distributions(**kwargs):
name = canonicalize_name(dist.project_name)
- package_set[name] = PackageDetails(dist.version, dist.requires())
- return package_set
+ try:
+ package_set[name] = PackageDetails(dist.version, dist.requires())
+ except RequirementParseError as e:
+ # Don't crash on broken metadata
+ logging.warning("Error parsing requirements for %s: %s", name, e)
+ problems = True
+ return package_set, problems
def check_package_set(package_set, should_ignore=None):
@@ -95,7 +105,7 @@ def check_install_conflicts(to_install):
installing given requirements
"""
# Start from the current state
- package_set = create_package_set_from_installed()
+ package_set, _ = create_package_set_from_installed()
# Install packages
would_be_installed = _simulate_installation_of(to_install, package_set)
@@ -110,9 +120,6 @@ def check_install_conflicts(to_install):
)
-# NOTE from @pradyunsg
-# This required a minor update in dependency link handling logic over at
-# operations.prepare.IsSDist.dist() to get it working
def _simulate_installation_of(to_install, package_set):
# type: (List[InstallRequirement], PackageSet) -> Set[str]
"""Computes the version of packages after installing to_install.
@@ -123,7 +130,7 @@ def _simulate_installation_of(to_install, package_set):
# Modify it as installing requirement_set would (assuming no errors)
for inst_req in to_install:
- dist = make_abstract_dist(inst_req).dist(finder=None)
+ dist = make_abstract_dist(inst_req).dist()
name = canonicalize_name(dist.key)
package_set[name] = PackageDetails(dist.version, dist.requires())
diff --git a/pipenv/patched/notpip/_internal/operations/freeze.py b/pipenv/patched/notpip/_internal/operations/freeze.py
index b18b98e4..8fd755e8 100644
--- a/pipenv/patched/notpip/_internal/operations/freeze.py
+++ b/pipenv/patched/notpip/_internal/operations/freeze.py
@@ -5,57 +5,61 @@ import logging
import os
import re
-from pipenv.patched.notpip._vendor import pkg_resources, six
+from pipenv.patched.notpip._vendor import six
from pipenv.patched.notpip._vendor.packaging.utils import canonicalize_name
from pipenv.patched.notpip._vendor.pkg_resources import RequirementParseError
-from pipenv.patched.notpip._internal.exceptions import InstallationError
+from pipenv.patched.notpip._internal.exceptions import BadCommand, InstallationError
from pipenv.patched.notpip._internal.req.constructors import (
install_req_from_editable, install_req_from_line,
)
from pipenv.patched.notpip._internal.req.req_file import COMMENT_RE
-from pipenv.patched.notpip._internal.utils.deprecation import deprecated
from pipenv.patched.notpip._internal.utils.misc import (
- dist_is_editable, get_installed_distributions, make_vcs_requirement_url,
+ dist_is_editable, get_installed_distributions,
)
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Iterator, Optional, List, Container, Set, Dict, Tuple, Iterable, Union
+ )
+ from pipenv.patched.notpip._internal.cache import WheelCache # noqa: F401
+ from pipenv.patched.notpip._vendor.pkg_resources import ( # noqa: F401
+ Distribution, Requirement
+ )
+
+ RequirementInfo = Tuple[Optional[Union[str, Requirement]], bool, List[str]]
+
logger = logging.getLogger(__name__)
def freeze(
- requirement=None,
- find_links=None, local_only=None, user_only=None, skip_regex=None,
- isolated=False,
- wheel_cache=None,
- exclude_editable=False,
- skip=()):
+ requirement=None, # type: Optional[List[str]]
+ find_links=None, # type: Optional[List[str]]
+ local_only=None, # type: Optional[bool]
+ user_only=None, # type: Optional[bool]
+ skip_regex=None, # type: Optional[str]
+ isolated=False, # type: bool
+ wheel_cache=None, # type: Optional[WheelCache]
+ exclude_editable=False, # type: bool
+ skip=() # type: Container[str]
+):
+ # type: (...) -> Iterator[str]
find_links = find_links or []
skip_match = None
if skip_regex:
skip_match = re.compile(skip_regex).search
- dependency_links = []
-
- for dist in pkg_resources.working_set:
- if dist.has_metadata('dependency_links.txt'):
- dependency_links.extend(
- dist.get_metadata_lines('dependency_links.txt')
- )
- for link in find_links:
- if '#egg=' in link:
- dependency_links.append(link)
for link in find_links:
yield '-f %s' % link
- installations = {}
+ installations = {} # type: Dict[str, FrozenRequirement]
for dist in get_installed_distributions(local_only=local_only,
skip=(),
user_only=user_only):
try:
- req = FrozenRequirement.from_dist(
- dist,
- dependency_links
- )
+ req = FrozenRequirement.from_dist(dist)
except RequirementParseError:
logger.warning(
"Could not parse requirement: %s",
@@ -71,10 +75,10 @@ def freeze(
# should only be emitted once, even if the same option is in multiple
# requirements files, so we need to keep track of what has been emitted
# so that we don't emit it again if it's seen again
- emitted_options = set()
+ emitted_options = set() # type: Set[str]
# keep track of which files a requirement is in so that we can
# give an accurate warning if a requirement appears multiple times.
- req_files = collections.defaultdict(list)
+ req_files = collections.defaultdict(list) # type: Dict[str, List[str]]
for req_file_path in requirement:
with open(req_file_path) as req_file:
for line in req_file:
@@ -128,10 +132,10 @@ def freeze(
# but has been processed already
if not req_files[line_req.name]:
logger.warning(
- "Requirement file [%s] contains %s, but that "
- "package is not installed",
+ "Requirement file [%s] contains %s, but "
+ "package %r is not installed",
req_file_path,
- COMMENT_RE.sub('', line).strip(),
+ COMMENT_RE.sub('', line).strip(), line_req.name
)
else:
req_files[line_req.name].append(req_file_path)
@@ -157,105 +161,84 @@ def freeze(
yield str(installation).rstrip()
+def get_requirement_info(dist):
+ # type: (Distribution) -> RequirementInfo
+ """
+ Compute and return values (req, editable, comments) for use in
+ FrozenRequirement.from_dist().
+ """
+ if not dist_is_editable(dist):
+ return (None, False, [])
+
+ location = os.path.normcase(os.path.abspath(dist.location))
+
+ from pipenv.patched.notpip._internal.vcs import vcs, RemoteNotFoundError
+ vc_type = vcs.get_backend_type(location)
+
+ if not vc_type:
+ req = dist.as_requirement()
+ logger.debug(
+ 'No VCS found for editable requirement {!r} in: {!r}', req,
+ location,
+ )
+ comments = [
+ '# Editable install with no version control ({})'.format(req)
+ ]
+ return (location, True, comments)
+
+ try:
+ req = vc_type.get_src_requirement(location, dist.project_name)
+ except RemoteNotFoundError:
+ req = dist.as_requirement()
+ comments = [
+ '# Editable {} install with no remote ({})'.format(
+ vc_type.__name__, req,
+ )
+ ]
+ return (location, True, comments)
+
+ except BadCommand:
+ logger.warning(
+ 'cannot determine version of editable source in %s '
+ '(%s command not found in path)',
+ location,
+ vc_type.name,
+ )
+ return (None, True, [])
+
+ except InstallationError as exc:
+ logger.warning(
+ "Error when trying to get requirement for VCS system %s, "
+ "falling back to uneditable format", exc
+ )
+ else:
+ if req is not None:
+ return (req, True, [])
+
+ logger.warning(
+ 'Could not determine repository location of %s', location
+ )
+ comments = ['## !! Could not determine repository location']
+
+ return (None, False, comments)
+
+
class FrozenRequirement(object):
def __init__(self, name, req, editable, comments=()):
+ # type: (str, Union[str, Requirement], bool, Iterable[str]) -> None
self.name = name
self.req = req
self.editable = editable
self.comments = comments
- _rev_re = re.compile(r'-r(\d+)$')
- _date_re = re.compile(r'-(20\d\d\d\d\d\d)$')
-
@classmethod
- def _init_args_from_dist(cls, dist, dependency_links):
- """
- Compute and return arguments (req, editable, comments) to pass to
- FrozenRequirement.__init__().
-
- This method is for use in FrozenRequirement.from_dist().
- """
- location = os.path.normcase(os.path.abspath(dist.location))
- comments = []
- from pipenv.patched.notpip._internal.vcs import vcs, get_src_requirement
- if dist_is_editable(dist) and vcs.get_backend_name(location):
- editable = True
- try:
- req = get_src_requirement(dist, location)
- except InstallationError as exc:
- logger.warning(
- "Error when trying to get requirement for VCS system %s, "
- "falling back to uneditable format", exc
- )
- req = None
- if req is None:
- logger.warning(
- 'Could not determine repository location of %s', location
- )
- comments.append(
- '## !! Could not determine repository location'
- )
- req = dist.as_requirement()
- editable = False
- else:
- editable = False
+ def from_dist(cls, dist):
+ # type: (Distribution) -> FrozenRequirement
+ req, editable, comments = get_requirement_info(dist)
+ if req is None:
req = dist.as_requirement()
- specs = req.specs
- assert len(specs) == 1 and specs[0][0] in ["==", "==="], \
- 'Expected 1 spec with == or ===; specs = %r; dist = %r' % \
- (specs, dist)
- version = specs[0][1]
- ver_match = cls._rev_re.search(version)
- date_match = cls._date_re.search(version)
- if ver_match or date_match:
- svn_backend = vcs.get_backend('svn')
- if svn_backend:
- svn_location = svn_backend().get_location(
- dist,
- dependency_links,
- )
- if not svn_location:
- logger.warning(
- 'Warning: cannot find svn location for %s', req,
- )
- comments.append(
- '## FIXME: could not find svn URL in dependency_links '
- 'for this package:'
- )
- else:
- deprecated(
- "SVN editable detection based on dependency links "
- "will be dropped in the future.",
- replacement=None,
- gone_in="18.2",
- issue=4187,
- )
- comments.append(
- '# Installing as editable to satisfy requirement %s:' %
- req
- )
- if ver_match:
- rev = ver_match.group(1)
- else:
- rev = '{%s}' % date_match.group(1)
- editable = True
- egg_name = cls.egg_name(dist)
- req = make_vcs_requirement_url(svn_location, rev, egg_name)
- return (req, editable, comments)
-
- @classmethod
- def from_dist(cls, dist, dependency_links):
- args = cls._init_args_from_dist(dist, dependency_links)
- return cls(dist.project_name, *args)
-
- @staticmethod
- def egg_name(dist):
- name = dist.egg_name()
- match = re.search(r'-py\d\.\d$', name)
- if match:
- name = name[:match.start()]
- return name
+ return cls(dist.project_name, req, editable, comments=comments)
def __str__(self):
req = self.req
diff --git a/pipenv/patched/notpip/_internal/operations/prepare.py b/pipenv/patched/notpip/_internal/operations/prepare.py
index d61270b4..018fca97 100644
--- a/pipenv/patched/notpip/_internal/operations/prepare.py
+++ b/pipenv/patched/notpip/_internal/operations/prepare.py
@@ -18,12 +18,21 @@ from pipenv.patched.notpip._internal.utils.compat import expanduser
from pipenv.patched.notpip._internal.utils.hashes import MissingHashes
from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import display_path, normalize_path, rmtree
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.vcs import vcs
+if MYPY_CHECK_RUNNING:
+ from typing import Any, Optional # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_install import InstallRequirement # noqa: F401
+ from pipenv.patched.notpip._internal.index import PackageFinder # noqa: F401
+ from pipenv.patched.notpip._internal.download import PipSession # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_tracker import RequirementTracker # noqa: F401
+
logger = logging.getLogger(__name__)
def make_abstract_dist(req):
+ # type: (InstallRequirement) -> DistAbstraction
"""Factory to make an abstract dist object.
Preconditions: Either an editable req with a source_dir, or satisfied_by or
@@ -59,40 +68,40 @@ class DistAbstraction(object):
"""
def __init__(self, req):
- self.req = req
+ # type: (InstallRequirement) -> None
+ self.req = req # type: InstallRequirement
- def dist(self, finder):
+ def dist(self):
+ # type: () -> Any
"""Return a setuptools Dist object."""
- raise NotImplementedError(self.dist)
+ raise NotImplementedError
def prep_for_dist(self, finder, build_isolation):
+ # type: (PackageFinder, bool) -> Any
"""Ensure that we can get a Dist for this requirement."""
- raise NotImplementedError(self.dist)
+ raise NotImplementedError
class IsWheel(DistAbstraction):
- def dist(self, finder):
+ def dist(self):
+ # type: () -> pkg_resources.Distribution
return list(pkg_resources.find_distributions(
self.req.source_dir))[0]
def prep_for_dist(self, finder, build_isolation):
+ # type: (PackageFinder, bool) -> Any
# FIXME:https://github.com/pypa/pip/issues/1112
pass
class IsSDist(DistAbstraction):
- def dist(self, finder):
- dist = self.req.get_dist()
- # FIXME: shouldn't be globally added.
- if finder and dist.has_metadata('dependency_links.txt'):
- finder.add_dependency_links(
- dist.get_metadata_lines('dependency_links.txt')
- )
- return dist
+ def dist(self):
+ return self.req.get_dist()
def prep_for_dist(self, finder, build_isolation):
+ # type: (PackageFinder, bool) -> None
# Prepare for building. We need to:
# 1. Load pyproject.toml (if it exists)
# 2. Set up the build environment
@@ -100,43 +109,64 @@ class IsSDist(DistAbstraction):
self.req.load_pyproject_toml()
should_isolate = self.req.use_pep517 and build_isolation
+ def _raise_conflicts(conflicting_with, conflicting_reqs):
+ raise InstallationError(
+ "Some build dependencies for %s conflict with %s: %s." % (
+ self.req, conflicting_with, ', '.join(
+ '%s is incompatible with %s' % (installed, wanted)
+ for installed, wanted in sorted(conflicting))))
+
if should_isolate:
# Isolate in a BuildEnvironment and install the build-time
# requirements.
self.req.build_env = BuildEnvironment()
self.req.build_env.install_requirements(
- finder, self.req.pyproject_requires,
+ finder, self.req.pyproject_requires, 'overlay',
"Installing build dependencies"
)
- missing = []
- if self.req.requirements_to_check:
- check = self.req.requirements_to_check
- missing = self.req.build_env.missing_requirements(check)
+ conflicting, missing = self.req.build_env.check_requirements(
+ self.req.requirements_to_check
+ )
+ if conflicting:
+ _raise_conflicts("PEP 517/518 supported requirements",
+ conflicting)
if missing:
logger.warning(
"Missing build requirements in pyproject.toml for %s.",
self.req,
)
logger.warning(
- "The project does not specify a build backend, and pip "
- "cannot fall back to setuptools without %s.",
+ "The project does not specify a build backend, and "
+ "pip cannot fall back to setuptools without %s.",
" and ".join(map(repr, sorted(missing)))
)
+ # Install any extra build dependencies that the backend requests.
+ # This must be done in a second pass, as the pyproject.toml
+ # dependencies must be installed before we can call the backend.
+ with self.req.build_env:
+ # We need to have the env active when calling the hook.
+ self.req.spin_message = "Getting requirements to build wheel"
+ reqs = self.req.pep517_backend.get_requires_for_build_wheel()
+ conflicting, missing = self.req.build_env.check_requirements(reqs)
+ if conflicting:
+ _raise_conflicts("the backend dependencies", conflicting)
+ self.req.build_env.install_requirements(
+ finder, missing, 'normal',
+ "Installing backend dependencies"
+ )
- try:
- self.req.run_egg_info()
- except (OSError, TypeError):
- self.req._correct_build_location()
- self.req.run_egg_info()
+ self.req.prepare_metadata()
self.req.assert_source_matches_version()
class Installed(DistAbstraction):
- def dist(self, finder):
+ def dist(self):
+ # type: () -> pkg_resources.Distribution
return self.req.satisfied_by
def prep_for_dist(self, finder, build_isolation):
+ # type: (PackageFinder, bool) -> Any
pass
@@ -144,8 +174,17 @@ class RequirementPreparer(object):
"""Prepares a Requirement
"""
- def __init__(self, build_dir, download_dir, src_dir, wheel_download_dir,
- progress_bar, build_isolation, req_tracker):
+ def __init__(
+ self,
+ build_dir, # type: str
+ download_dir, # type: Optional[str]
+ src_dir, # type: str
+ wheel_download_dir, # type: Optional[str]
+ progress_bar, # type: str
+ build_isolation, # type: bool
+ req_tracker # type: RequirementTracker
+ ):
+ # type: (...) -> None
super(RequirementPreparer, self).__init__()
self.src_dir = src_dir
@@ -175,6 +214,7 @@ class RequirementPreparer(object):
@property
def _download_should_save(self):
+ # type: () -> bool
# TODO: Modify to reduce indentation needed
if self.download_dir:
self.download_dir = expanduser(self.download_dir)
@@ -187,8 +227,15 @@ class RequirementPreparer(object):
% display_path(self.download_dir))
return False
- def prepare_linked_requirement(self, req, session, finder,
- upgrade_allowed, require_hashes):
+ def prepare_linked_requirement(
+ self,
+ req, # type: InstallRequirement
+ session, # type: PipSession
+ finder, # type: PackageFinder
+ upgrade_allowed, # type: bool
+ require_hashes # type: bool
+ ):
+ # type: (...) -> DistAbstraction
"""Prepare a requirement that would be obtained from req.link
"""
# TODO: Breakup into smaller functions
@@ -209,6 +256,7 @@ class RequirementPreparer(object):
# installation.
# FIXME: this won't upgrade when there's an existing
# package unpacked in `req.source_dir`
+ # package unpacked in `req.source_dir`
if os.path.exists(os.path.join(req.source_dir, 'setup.py')):
rmtree(req.source_dir)
req.populate_link(finder, upgrade_allowed, require_hashes)
@@ -298,8 +346,14 @@ class RequirementPreparer(object):
req.archive(self.download_dir)
return abstract_dist
- def prepare_editable_requirement(self, req, require_hashes, use_user_site,
- finder):
+ def prepare_editable_requirement(
+ self,
+ req, # type: InstallRequirement
+ require_hashes, # type: bool
+ use_user_site, # type: bool
+ finder # type: PackageFinder
+ ):
+ # type: (...) -> DistAbstraction
"""Prepare an editable requirement
"""
assert req.editable, "cannot prepare a non-editable req as editable"
@@ -327,6 +381,7 @@ class RequirementPreparer(object):
return abstract_dist
def prepare_installed_requirement(self, req, require_hashes, skip_reason):
+ # type: (InstallRequirement, bool, Optional[str]) -> DistAbstraction
"""Prepare an already-installed requirement
"""
assert req.satisfied_by, "req should have been satisfied but isn't"
diff --git a/pipenv/patched/notpip/_internal/pep425tags.py b/pipenv/patched/notpip/_internal/pep425tags.py
index 182c1c88..4b6eb2bc 100644
--- a/pipenv/patched/notpip/_internal/pep425tags.py
+++ b/pipenv/patched/notpip/_internal/pep425tags.py
@@ -14,8 +14,15 @@ try:
import pipenv.patched.notpip._internal.utils.glibc
except ImportError:
import pipenv.patched.notpip.utils.glibc
-
from pipenv.patched.notpip._internal.utils.compat import get_extension_suffixes
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Tuple, Callable, List, Optional, Union, Dict
+ )
+
+ Pep425Tag = Tuple[str, str, str]
logger = logging.getLogger(__name__)
@@ -23,6 +30,7 @@ _osx_arch_pat = re.compile(r'(.+)_(\d+)_(\d+)_(.+)')
def get_config_var(var):
+ # type: (str) -> Optional[str]
try:
return sysconfig.get_config_var(var)
except IOError as e: # Issue #1074
@@ -31,6 +39,7 @@ def get_config_var(var):
def get_abbr_impl():
+ # type: () -> str
"""Return abbreviated implementation name."""
if hasattr(sys, 'pypy_version_info'):
pyimpl = 'pp'
@@ -44,6 +53,7 @@ def get_abbr_impl():
def get_impl_ver():
+ # type: () -> str
"""Return implementation version."""
impl_ver = get_config_var("py_version_nodot")
if not impl_ver or get_abbr_impl() == 'pp':
@@ -52,17 +62,21 @@ def get_impl_ver():
def get_impl_version_info():
+ # type: () -> Tuple[int, ...]
"""Return sys.version_info-like tuple for use in decrementing the minor
version."""
if get_abbr_impl() == 'pp':
# as per https://github.com/pypa/pip/issues/2882
- return (sys.version_info[0], sys.pypy_version_info.major,
- sys.pypy_version_info.minor)
+ # attrs exist only on pypy
+ return (sys.version_info[0],
+ sys.pypy_version_info.major, # type: ignore
+ sys.pypy_version_info.minor) # type: ignore
else:
return sys.version_info[0], sys.version_info[1]
def get_impl_tag():
+ # type: () -> str
"""
Returns the Tag for this specific implementation.
"""
@@ -70,6 +84,7 @@ def get_impl_tag():
def get_flag(var, fallback, expected=True, warn=True):
+ # type: (str, Callable[..., bool], Union[bool, int], bool) -> bool
"""Use a fallback method for determining SOABI flags if the needed config
var is unset or unavailable."""
val = get_config_var(var)
@@ -82,6 +97,7 @@ def get_flag(var, fallback, expected=True, warn=True):
def get_abi_tag():
+ # type: () -> Optional[str]
"""Return the ABI tag based on SOABI (if available) or emulate SOABI
(CPython 2, PyPy)."""
soabi = get_config_var('SOABI')
@@ -116,10 +132,12 @@ def get_abi_tag():
def _is_running_32bit():
+ # type: () -> bool
return sys.maxsize == 2147483647
def get_platform():
+ # type: () -> str
"""Return our platform name 'win32', 'linux_x86_64'"""
if sys.platform == 'darwin':
# distutils.util.get_platform() returns the release based on the value
@@ -146,6 +164,7 @@ def get_platform():
def is_manylinux1_compatible():
+ # type: () -> bool
# Only Linux, and only x86-64 / i686
if get_platform() not in {"linux_x86_64", "linux_i686"}:
return False
@@ -162,13 +181,33 @@ def is_manylinux1_compatible():
return pipenv.patched.notpip._internal.utils.glibc.have_compatible_glibc(2, 5)
+def is_manylinux2010_compatible():
+ # type: () -> bool
+ # Only Linux, and only x86-64 / i686
+ if get_platform() not in {"linux_x86_64", "linux_i686"}:
+ return False
+
+ # Check for presence of _manylinux module
+ try:
+ import _manylinux
+ return bool(_manylinux.manylinux2010_compatible)
+ except (ImportError, AttributeError):
+ # Fall through to heuristic check below
+ pass
+
+ # Check glibc version. CentOS 6 uses glibc 2.12.
+ return pipenv.patched.notpip._internal.utils.glibc.have_compatible_glibc(2, 12)
+
+
def get_darwin_arches(major, minor, machine):
+ # type: (int, int, str) -> List[str]
"""Return a list of supported arches (including group arches) for
the given major, minor and machine architecture of an macOS machine.
"""
arches = []
def _supports_arch(major, minor, arch):
+ # type: (int, int, str) -> bool
# Looking at the application support for macOS versions in the chart
# provided by https://en.wikipedia.org/wiki/OS_X#Versions it appears
# our timeline looks roughly like:
@@ -209,7 +248,7 @@ def get_darwin_arches(major, minor, machine):
("intel", ("x86_64", "i386")),
("fat64", ("x86_64", "ppc64")),
("fat32", ("x86_64", "i386", "ppc")),
- ])
+ ]) # type: Dict[str, Tuple[str, ...]]
if _supports_arch(major, minor, machine):
arches.append(machine)
@@ -223,8 +262,24 @@ def get_darwin_arches(major, minor, machine):
return arches
-def get_supported(versions=None, noarch=False, platform=None,
- impl=None, abi=None):
+def get_all_minor_versions_as_strings(version_info):
+ # type: (Tuple[int, ...]) -> List[str]
+ versions = []
+ major = version_info[:-1]
+ # Support all previous minor Python versions.
+ for minor in range(version_info[-1], -1, -1):
+ versions.append(''.join(map(str, major + (minor,))))
+ return versions
+
+
+def get_supported(
+ versions=None, # type: Optional[List[str]]
+ noarch=False, # type: bool
+ platform=None, # type: Optional[str]
+ impl=None, # type: Optional[str]
+ abi=None # type: Optional[str]
+):
+ # type: (...) -> List[Pep425Tag]
"""Return a list of supported tags for each version specified in
`versions`.
@@ -241,16 +296,12 @@ def get_supported(versions=None, noarch=False, platform=None,
# Versions must be given with respect to the preference
if versions is None:
- versions = []
version_info = get_impl_version_info()
- major = version_info[:-1]
- # Support all previous minor Python versions.
- for minor in range(version_info[-1], -1, -1):
- versions.append(''.join(map(str, major + (minor,))))
+ versions = get_all_minor_versions_as_strings(version_info)
impl = impl or get_abbr_impl()
- abis = []
+ abis = [] # type: List[str]
abi = abi or get_abi_tag()
if abi:
@@ -267,6 +318,7 @@ def get_supported(versions=None, noarch=False, platform=None,
if not noarch:
arch = platform or get_platform()
+ arch_prefix, arch_sep, arch_suffix = arch.partition('_')
if arch.startswith('macosx'):
# support macosx-10.6-intel on macosx-10.9-x86_64
match = _osx_arch_pat.match(arch)
@@ -280,8 +332,19 @@ def get_supported(versions=None, noarch=False, platform=None,
else:
# arch pattern didn't match (?!)
arches = [arch]
- elif platform is None and is_manylinux1_compatible():
- arches = [arch.replace('linux', 'manylinux1'), arch]
+ elif arch_prefix == 'manylinux2010':
+ # manylinux1 wheels run on most manylinux2010 systems with the
+ # exception of wheels depending on ncurses. PEP 571 states
+ # manylinux1 wheels should be considered manylinux2010 wheels:
+ # https://www.python.org/dev/peps/pep-0571/#backwards-compatibility-with-manylinux1-wheels
+ arches = [arch, 'manylinux1' + arch_sep + arch_suffix]
+ elif platform is None:
+ arches = []
+ if is_manylinux2010_compatible():
+ arches.append('manylinux2010' + arch_sep + arch_suffix)
+ if is_manylinux1_compatible():
+ arches.append('manylinux1' + arch_sep + arch_suffix)
+ arches.append(arch)
else:
arches = [arch]
diff --git a/pipenv/patched/notpip/_internal/pyproject.py b/pipenv/patched/notpip/_internal/pyproject.py
index a47e0f05..8845b2dc 100644
--- a/pipenv/patched/notpip/_internal/pyproject.py
+++ b/pipenv/patched/notpip/_internal/pyproject.py
@@ -2,20 +2,43 @@ from __future__ import absolute_import
import io
import os
+import sys
from pipenv.patched.notpip._vendor import pytoml, six
from pipenv.patched.notpip._internal.exceptions import InstallationError
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Any, Tuple, Optional, List # noqa: F401
def _is_list_of_str(obj):
+ # type: (Any) -> bool
return (
isinstance(obj, list) and
all(isinstance(item, six.string_types) for item in obj)
)
-def load_pyproject_toml(use_pep517, pyproject_toml, setup_py, req_name):
+def make_pyproject_path(setup_py_dir):
+ # type: (str) -> str
+ path = os.path.join(setup_py_dir, 'pyproject.toml')
+
+ # Python2 __file__ should not be unicode
+ if six.PY2 and isinstance(path, six.text_type):
+ path = path.encode(sys.getfilesystemencoding())
+
+ return path
+
+
+def load_pyproject_toml(
+ use_pep517, # type: Optional[bool]
+ pyproject_toml, # type: str
+ setup_py, # type: str
+ req_name # type: str
+):
+ # type: (...) -> Optional[Tuple[List[str], str, List[str]]]
"""Load the pyproject.toml file.
Parameters:
@@ -46,17 +69,20 @@ def load_pyproject_toml(use_pep517, pyproject_toml, setup_py, req_name):
build_system = None
# The following cases must use PEP 517
- # We check for use_pep517 equalling False because that
- # means the user explicitly requested --no-use-pep517
+ # We check for use_pep517 being non-None and falsey because that means
+ # the user explicitly requested --no-use-pep517. The value 0 as
+ # opposed to False can occur when the value is provided via an
+ # environment variable or config file option (due to the quirk of
+ # strtobool() returning an integer in pip's configuration code).
if has_pyproject and not has_setup:
- if use_pep517 is False:
+ if use_pep517 is not None and not use_pep517:
raise InstallationError(
"Disabling PEP 517 processing is invalid: "
"project does not have a setup.py"
)
use_pep517 = True
elif build_system and "build-backend" in build_system:
- if use_pep517 is False:
+ if use_pep517 is not None and not use_pep517:
raise InstallationError(
"Disabling PEP 517 processing is invalid: "
"project specifies a build backend of {} "
@@ -85,11 +111,13 @@ def load_pyproject_toml(use_pep517, pyproject_toml, setup_py, req_name):
# section, or the user has no pyproject.toml, but has opted in
# explicitly via --use-pep517.
# In the absence of any explicit backend specification, we
- # assume the setuptools backend, and require wheel and a version
- # of setuptools that supports that backend.
+ # assume the setuptools backend that most closely emulates the
+ # traditional direct setup.py execution, and require wheel and
+ # a version of setuptools that supports that backend.
+
build_system = {
- "requires": ["setuptools>=38.2.5", "wheel"],
- "build-backend": "setuptools.build_meta",
+ "requires": ["setuptools>=40.8.0", "wheel"],
+ "build-backend": "setuptools.build_meta:__legacy__",
}
# If we're using PEP 517, we have build system information (either
@@ -123,22 +151,21 @@ def load_pyproject_toml(use_pep517, pyproject_toml, setup_py, req_name):
))
backend = build_system.get("build-backend")
- check = []
+ check = [] # type: List[str]
if backend is None:
# If the user didn't specify a backend, we assume they want to use
# the setuptools backend. But we can't be sure they have included
# a version of setuptools which supplies the backend, or wheel
- # (which is neede by the backend) in their requirements. So we
+ # (which is needed by the backend) in their requirements. So we
# make a note to check that those requirements are present once
# we have set up the environment.
- # TODO: Review this - it's quite a lot of work to check for a very
- # specific case. The problem is, that case is potentially quite
- # common - projects that adopted PEP 518 early for the ability to
- # specify requirements to execute setup.py, but never considered
- # needing to mention the build tools themselves. The original PEP
- # 518 code had a similar check (but implemented in a different
- # way).
- backend = "setuptools.build_meta"
- check = ["setuptools>=38.2.5", "wheel"]
+ # This is quite a lot of work to check for a very specific case. But
+ # the problem is, that case is potentially quite common - projects that
+ # adopted PEP 518 early for the ability to specify requirements to
+ # execute setup.py, but never considered needing to mention the build
+ # tools themselves. The original PEP 518 code had a similar check (but
+ # implemented in a different way).
+ backend = "setuptools.build_meta:__legacy__"
+ check = ["setuptools>=40.8.0", "wheel"]
return (requires, backend, check)
diff --git a/pipenv/patched/notpip/_internal/req/__init__.py b/pipenv/patched/notpip/_internal/req/__init__.py
index 72aecb7c..51606fec 100644
--- a/pipenv/patched/notpip/_internal/req/__init__.py
+++ b/pipenv/patched/notpip/_internal/req/__init__.py
@@ -6,7 +6,10 @@ from .req_install import InstallRequirement
from .req_set import RequirementSet
from .req_file import parse_requirements
from pipenv.patched.notpip._internal.utils.logging import indent_log
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+if MYPY_CHECK_RUNNING:
+ from typing import List, Sequence # noqa: F401
__all__ = [
"RequirementSet", "InstallRequirement",
@@ -16,8 +19,13 @@ __all__ = [
logger = logging.getLogger(__name__)
-def install_given_reqs(to_install, install_options, global_options=(),
- *args, **kwargs):
+def install_given_reqs(
+ to_install, # type: List[InstallRequirement]
+ install_options, # type: List[str]
+ global_options=(), # type: Sequence[str]
+ *args, **kwargs
+):
+ # type: (...) -> List[InstallRequirement]
"""
Install everything in the given list.
diff --git a/pipenv/patched/notpip/_internal/req/constructors.py b/pipenv/patched/notpip/_internal/req/constructors.py
index 9fe28d88..a9d8221a 100644
--- a/pipenv/patched/notpip/_internal/req/constructors.py
+++ b/pipenv/patched/notpip/_internal/req/constructors.py
@@ -11,7 +11,6 @@ InstallRequirement.
import logging
import os
import re
-import traceback
from pipenv.patched.notpip._vendor.packaging.markers import Marker
from pipenv.patched.notpip._vendor.packaging.requirements import InvalidRequirement, Requirement
@@ -24,11 +23,20 @@ from pipenv.patched.notpip._internal.download import (
from pipenv.patched.notpip._internal.exceptions import InstallationError
from pipenv.patched.notpip._internal.models.index import PyPI, TestPyPI
from pipenv.patched.notpip._internal.models.link import Link
+from pipenv.patched.notpip._internal.pyproject import make_pyproject_path
from pipenv.patched.notpip._internal.req.req_install import InstallRequirement
from pipenv.patched.notpip._internal.utils.misc import is_installable_dir
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.vcs import vcs
from pipenv.patched.notpip._internal.wheel import Wheel
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Optional, Tuple, Set, Any, Union, Text, Dict,
+ )
+ from pipenv.patched.notpip._internal.cache import WheelCache # noqa: F401
+
+
__all__ = [
"install_req_from_editable", "install_req_from_line",
"parse_editable"
@@ -39,6 +47,7 @@ operators = Specifier._operators.keys()
def _strip_extras(path):
+ # type: (str) -> Tuple[str, Optional[str]]
m = re.match(r'^(.+)(\[[^\]]+\])$', path)
extras = None
if m:
@@ -51,6 +60,7 @@ def _strip_extras(path):
def parse_editable(editable_req):
+ # type: (str) -> Tuple[Optional[str], str, Optional[Set[str]]]
"""Parses an editable requirement into:
- a requirement name
- an URL
@@ -68,10 +78,18 @@ def parse_editable(editable_req):
if os.path.isdir(url_no_extras):
if not os.path.exists(os.path.join(url_no_extras, 'setup.py')):
- raise InstallationError(
- "Directory %r is not installable. File 'setup.py' not found." %
- url_no_extras
+ msg = (
+ 'File "setup.py" not found. Directory cannot be installed '
+ 'in editable mode: {}'.format(os.path.abspath(url_no_extras))
)
+ pyproject_path = make_pyproject_path(url_no_extras)
+ if os.path.isfile(pyproject_path):
+ msg += (
+ '\n(A "pyproject.toml" file was found, but editable '
+ 'mode currently requires a setup.py based build.)'
+ )
+ raise InstallationError(msg)
+
# Treating it as code that has already been checked out
url_no_extras = path_to_url(url_no_extras)
@@ -116,6 +134,7 @@ def parse_editable(editable_req):
def deduce_helpful_msg(req):
+ # type: (str) -> str
"""Returns helpful msg in case requirements file does not exist,
or cannot be parsed.
@@ -136,7 +155,7 @@ def deduce_helpful_msg(req):
" the packages specified within it."
except RequirementParseError:
logger.debug("Cannot parse '%s' as requirements \
- file" % (req), exc_info=1)
+ file" % (req), exc_info=True)
else:
msg += " File '%s' does not exist." % (req)
return msg
@@ -146,9 +165,15 @@ def deduce_helpful_msg(req):
def install_req_from_editable(
- editable_req, comes_from=None, isolated=False, options=None,
- wheel_cache=None, constraint=False
+ editable_req, # type: str
+ comes_from=None, # type: Optional[str]
+ use_pep517=None, # type: Optional[bool]
+ isolated=False, # type: bool
+ options=None, # type: Optional[Dict[str, Any]]
+ wheel_cache=None, # type: Optional[WheelCache]
+ constraint=False # type: bool
):
+ # type: (...) -> InstallRequirement
name, url, extras_override = parse_editable(editable_req)
if url.startswith('file:'):
source_dir = url_to_path(url)
@@ -167,6 +192,7 @@ def install_req_from_editable(
editable=True,
link=Link(url),
constraint=constraint,
+ use_pep517=use_pep517,
isolated=isolated,
options=options if options else {},
wheel_cache=wheel_cache,
@@ -175,9 +201,15 @@ def install_req_from_editable(
def install_req_from_line(
- name, comes_from=None, isolated=False, options=None, wheel_cache=None,
- constraint=False
+ name, # type: str
+ comes_from=None, # type: Optional[Union[str, InstallRequirement]]
+ use_pep517=None, # type: Optional[bool]
+ isolated=False, # type: bool
+ options=None, # type: Optional[Dict[str, Any]]
+ wheel_cache=None, # type: Optional[WheelCache]
+ constraint=False # type: bool
):
+ # type: (...) -> InstallRequirement
"""Creates an InstallRequirement from a name, which might be a
requirement, directory containing 'setup.py', filename, or URL.
"""
@@ -186,24 +218,24 @@ def install_req_from_line(
else:
marker_sep = ';'
if marker_sep in name:
- name, markers = name.split(marker_sep, 1)
- markers = markers.strip()
- if not markers:
+ name, markers_as_string = name.split(marker_sep, 1)
+ markers_as_string = markers_as_string.strip()
+ if not markers_as_string:
markers = None
else:
- markers = Marker(markers)
+ markers = Marker(markers_as_string)
else:
markers = None
name = name.strip()
- req = None
+ req_as_string = None
path = os.path.normpath(os.path.abspath(name))
link = None
- extras = None
+ extras_as_string = None
if is_url(name):
link = Link(name)
else:
- p, extras = _strip_extras(path)
+ p, extras_as_string = _strip_extras(path)
looks_like_dir = os.path.isdir(p) and (
os.path.sep in name or
(os.path.altsep is not None and os.path.altsep in name) or
@@ -234,38 +266,41 @@ def install_req_from_line(
# wheel file
if link.is_wheel:
wheel = Wheel(link.filename) # can raise InvalidWheelFilename
- req = "%s==%s" % (wheel.name, wheel.version)
+ req_as_string = "%s==%s" % (wheel.name, wheel.version)
else:
# set the req to the egg fragment. when it's not there, this
# will become an 'unnamed' requirement
- req = link.egg_fragment
+ req_as_string = link.egg_fragment
# a requirement specifier
else:
- req = name
+ req_as_string = name
- if extras:
- extras = Requirement("placeholder" + extras.lower()).extras
+ if extras_as_string:
+ extras = Requirement("placeholder" + extras_as_string.lower()).extras
else:
extras = ()
- if req is not None:
+ if req_as_string is not None:
try:
- req = Requirement(req)
+ req = Requirement(req_as_string)
except InvalidRequirement:
- if os.path.sep in req:
+ if os.path.sep in req_as_string:
add_msg = "It looks like a path."
- add_msg += deduce_helpful_msg(req)
- elif '=' in req and not any(op in req for op in operators):
+ add_msg += deduce_helpful_msg(req_as_string)
+ elif ('=' in req_as_string and
+ not any(op in req_as_string for op in operators)):
add_msg = "= is not a valid operator. Did you mean == ?"
else:
- add_msg = traceback.format_exc()
+ add_msg = ""
raise InstallationError(
- "Invalid requirement: '%s'\n%s" % (req, add_msg)
+ "Invalid requirement: '%s'\n%s" % (req_as_string, add_msg)
)
+ else:
+ req = None
return InstallRequirement(
req, comes_from, link=link, markers=markers,
- isolated=isolated,
+ use_pep517=use_pep517, isolated=isolated,
options=options if options else {},
wheel_cache=wheel_cache,
constraint=constraint,
@@ -273,11 +308,16 @@ def install_req_from_line(
)
-def install_req_from_req(
- req, comes_from=None, isolated=False, wheel_cache=None
+def install_req_from_req_string(
+ req_string, # type: str
+ comes_from=None, # type: Optional[InstallRequirement]
+ isolated=False, # type: bool
+ wheel_cache=None, # type: Optional[WheelCache]
+ use_pep517=None # type: Optional[bool]
):
+ # type: (...) -> InstallRequirement
try:
- req = Requirement(req)
+ req = Requirement(req_string)
except InvalidRequirement:
raise InstallationError("Invalid requirement: '%s'" % req)
@@ -294,5 +334,6 @@ def install_req_from_req(
)
return InstallRequirement(
- req, comes_from, isolated=isolated, wheel_cache=wheel_cache
+ req, comes_from, isolated=isolated, wheel_cache=wheel_cache,
+ use_pep517=use_pep517
)
diff --git a/pipenv/patched/notpip/_internal/req/req_file.py b/pipenv/patched/notpip/_internal/req/req_file.py
index 5f23cd3a..30391cb5 100644
--- a/pipenv/patched/notpip/_internal/req/req_file.py
+++ b/pipenv/patched/notpip/_internal/req/req_file.py
@@ -19,6 +19,18 @@ from pipenv.patched.notpip._internal.exceptions import RequirementsFileParseErro
from pipenv.patched.notpip._internal.req.constructors import (
install_req_from_editable, install_req_from_line,
)
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Iterator, Tuple, Optional, List, Callable, Text
+ )
+ from pipenv.patched.notpip._internal.req import InstallRequirement # noqa: F401
+ from pipenv.patched.notpip._internal.cache import WheelCache # noqa: F401
+ from pipenv.patched.notpip._internal.index import PackageFinder # noqa: F401
+ from pipenv.patched.notpip._internal.download import PipSession # noqa: F401
+
+ ReqFileLines = Iterator[Tuple[int, Text]]
__all__ = ['parse_requirements']
@@ -43,24 +55,32 @@ SUPPORTED_OPTIONS = [
cmdoptions.no_binary,
cmdoptions.only_binary,
cmdoptions.pre,
- cmdoptions.process_dependency_links,
cmdoptions.trusted_host,
cmdoptions.require_hashes,
-]
+] # type: List[Callable[..., optparse.Option]]
# options to be passed to requirements
SUPPORTED_OPTIONS_REQ = [
cmdoptions.install_options,
cmdoptions.global_options,
cmdoptions.hash,
-]
+] # type: List[Callable[..., optparse.Option]]
# the 'dest' string values
-SUPPORTED_OPTIONS_REQ_DEST = [o().dest for o in SUPPORTED_OPTIONS_REQ]
+SUPPORTED_OPTIONS_REQ_DEST = [str(o().dest) for o in SUPPORTED_OPTIONS_REQ]
-def parse_requirements(filename, finder=None, comes_from=None, options=None,
- session=None, constraint=False, wheel_cache=None):
+def parse_requirements(
+ filename, # type: str
+ finder=None, # type: Optional[PackageFinder]
+ comes_from=None, # type: Optional[str]
+ options=None, # type: Optional[optparse.Values]
+ session=None, # type: Optional[PipSession]
+ constraint=False, # type: bool
+ wheel_cache=None, # type: Optional[WheelCache]
+ use_pep517=None # type: Optional[bool]
+):
+ # type: (...) -> Iterator[InstallRequirement]
"""Parse a requirements file and yield InstallRequirement instances.
:param filename: Path or url of requirements file.
@@ -71,6 +91,7 @@ def parse_requirements(filename, finder=None, comes_from=None, options=None,
:param constraint: If true, parsing a constraint file rather than
requirements file.
:param wheel_cache: Instance of pip.wheel.WheelCache
+ :param use_pep517: Value of the --use-pep517 option.
"""
if session is None:
raise TypeError(
@@ -87,18 +108,19 @@ def parse_requirements(filename, finder=None, comes_from=None, options=None,
for line_number, line in lines_enum:
req_iter = process_line(line, filename, line_number, finder,
comes_from, options, session, wheel_cache,
- constraint=constraint)
+ use_pep517=use_pep517, constraint=constraint)
for req in req_iter:
yield req
def preprocess(content, options):
+ # type: (Text, Optional[optparse.Values]) -> ReqFileLines
"""Split, filter, and join lines, and return a line iterator
:param content: the content of the requirements file
:param options: cli options
"""
- lines_enum = enumerate(content.splitlines(), start=1)
+ lines_enum = enumerate(content.splitlines(), start=1) # type: ReqFileLines
lines_enum = join_lines(lines_enum)
lines_enum = ignore_comments(lines_enum)
lines_enum = skip_regex(lines_enum, options)
@@ -106,9 +128,19 @@ def preprocess(content, options):
return lines_enum
-def process_line(line, filename, line_number, finder=None, comes_from=None,
- options=None, session=None, wheel_cache=None,
- constraint=False):
+def process_line(
+ line, # type: Text
+ filename, # type: str
+ line_number, # type: int
+ finder=None, # type: Optional[PackageFinder]
+ comes_from=None, # type: Optional[str]
+ options=None, # type: Optional[optparse.Values]
+ session=None, # type: Optional[PipSession]
+ wheel_cache=None, # type: Optional[WheelCache]
+ use_pep517=None, # type: Optional[bool]
+ constraint=False # type: bool
+):
+ # type: (...) -> Iterator[InstallRequirement]
"""Process a single requirements line; This can result in creating/yielding
requirements, or updating the finder.
@@ -130,13 +162,15 @@ def process_line(line, filename, line_number, finder=None, comes_from=None,
defaults = parser.get_default_values()
defaults.index_url = None
if finder:
- # `finder.format_control` will be updated during parsing
defaults.format_control = finder.format_control
args_str, options_str = break_args_options(line)
+ # Prior to 2.7.3, shlex cannot deal with unicode entries
if sys.version_info < (2, 7, 3):
- # Prior to 2.7.3, shlex cannot deal with unicode entries
- options_str = options_str.encode('utf8')
- opts, _ = parser.parse_args(shlex.split(options_str), defaults)
+ # https://github.com/python/mypy/issues/1174
+ options_str = options_str.encode('utf8') # type: ignore
+ # https://github.com/python/mypy/issues/1174
+ opts, _ = parser.parse_args(
+ shlex.split(options_str), defaults) # type: ignore
# preserve for the nested code path
line_comes_from = '%s %s (line %s)' % (
@@ -155,6 +189,7 @@ def process_line(line, filename, line_number, finder=None, comes_from=None,
req_options[dest] = opts.__dict__[dest]
yield install_req_from_line(
args_str, line_comes_from, constraint=constraint,
+ use_pep517=use_pep517,
isolated=isolated, options=req_options, wheel_cache=wheel_cache
)
@@ -163,6 +198,7 @@ def process_line(line, filename, line_number, finder=None, comes_from=None,
isolated = options.isolated_mode if options else False
yield install_req_from_editable(
opts.editables[0], comes_from=line_comes_from,
+ use_pep517=use_pep517,
constraint=constraint, isolated=isolated, wheel_cache=wheel_cache
)
@@ -183,11 +219,11 @@ def process_line(line, filename, line_number, finder=None, comes_from=None,
# do a join so relative paths work
req_path = os.path.join(os.path.dirname(filename), req_path)
# TODO: Why not use `comes_from='-r {} (line {})'` here as well?
- parser = parse_requirements(
+ parsed_reqs = parse_requirements(
req_path, finder, comes_from, options, session,
constraint=nested_constraint, wheel_cache=wheel_cache
)
- for req in parser:
+ for req in parsed_reqs:
yield req
# percolate hash-checking option upward
@@ -214,14 +250,13 @@ def process_line(line, filename, line_number, finder=None, comes_from=None,
finder.find_links.append(value)
if opts.pre:
finder.allow_all_prereleases = True
- if opts.process_dependency_links:
- finder.process_dependency_links = True
if opts.trusted_hosts:
finder.secure_origins.extend(
("*", host, "*") for host in opts.trusted_hosts)
def break_args_options(line):
+ # type: (Text) -> Tuple[str, Text]
"""Break up the line into an args and options string. We only want to shlex
(and then optparse) the options, not the args. args can contain markers
which are corrupted by shlex.
@@ -235,10 +270,11 @@ def break_args_options(line):
else:
args.append(token)
options.pop(0)
- return ' '.join(args), ' '.join(options)
+ return ' '.join(args), ' '.join(options) # type: ignore
def build_parser(line):
+ # type: (Text) -> optparse.OptionParser
"""
Return a parser for parsing requirement lines
"""
@@ -255,17 +291,20 @@ def build_parser(line):
# add offending line
msg = 'Invalid requirement: %s\n%s' % (line, msg)
raise RequirementsFileParseError(msg)
- parser.exit = parser_exit
+ # NOTE: mypy disallows assigning to a method
+ # https://github.com/python/mypy/issues/2427
+ parser.exit = parser_exit # type: ignore
return parser
def join_lines(lines_enum):
+ # type: (ReqFileLines) -> ReqFileLines
"""Joins a line ending in '\' with the previous line (except when following
comments). The joined line takes on the index of the first line.
"""
primary_line_number = None
- new_line = []
+ new_line = [] # type: List[Text]
for line_number, line in lines_enum:
if not line.endswith('\\') or COMMENT_RE.match(line):
if COMMENT_RE.match(line):
@@ -290,6 +329,7 @@ def join_lines(lines_enum):
def ignore_comments(lines_enum):
+ # type: (ReqFileLines) -> ReqFileLines
"""
Strips comments and filter empty lines.
"""
@@ -301,6 +341,7 @@ def ignore_comments(lines_enum):
def skip_regex(lines_enum, options):
+ # type: (ReqFileLines, Optional[optparse.Values]) -> ReqFileLines
"""
Skip lines that match '--skip-requirements-regex' pattern
@@ -314,6 +355,7 @@ def skip_regex(lines_enum, options):
def expand_env_variables(lines_enum):
+ # type: (ReqFileLines) -> ReqFileLines
"""Replace all environment variables that can be retrieved via `os.getenv`.
The only allowed format for environment variables defined in the
diff --git a/pipenv/patched/notpip/_internal/req/req_install.py b/pipenv/patched/notpip/_internal/req/req_install.py
index 3f32892c..fd3cead6 100644
--- a/pipenv/patched/notpip/_internal/req/req_install.py
+++ b/pipenv/patched/notpip/_internal/req/req_install.py
@@ -22,7 +22,7 @@ from pipenv.patched.notpip._internal.locations import (
PIP_DELETE_MARKER_FILENAME, running_under_virtualenv,
)
from pipenv.patched.notpip._internal.models.link import Link
-from pipenv.patched.notpip._internal.pyproject import load_pyproject_toml
+from pipenv.patched.notpip._internal.pyproject import load_pyproject_toml, make_pyproject_path
from pipenv.patched.notpip._internal.req.req_uninstall import UninstallPathSet
from pipenv.patched.notpip._internal.utils.compat import native_str
from pipenv.patched.notpip._internal.utils.hashes import Hashes
@@ -30,15 +30,28 @@ from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import (
_make_build_dir, ask_path_exists, backup_dir, call_subprocess,
display_path, dist_in_site_packages, dist_in_usersite, ensure_dir,
- get_installed_version, rmtree,
+ get_installed_version, redact_password_from_url, rmtree,
)
from pipenv.patched.notpip._internal.utils.packaging import get_metadata
from pipenv.patched.notpip._internal.utils.setuptools_build import SETUPTOOLS_SHIM
from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.utils.ui import open_spinner
from pipenv.patched.notpip._internal.vcs import vcs
from pipenv.patched.notpip._internal.wheel import move_wheel_files
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Optional, Iterable, List, Union, Any, Text, Sequence, Dict
+ )
+ from pipenv.patched.notpip._internal.build_env import BuildEnvironment # noqa: F401
+ from pipenv.patched.notpip._internal.cache import WheelCache # noqa: F401
+ from pipenv.patched.notpip._internal.index import PackageFinder # noqa: F401
+ from pipenv.patched.notpip._vendor.pkg_resources import Distribution # noqa: F401
+ from pipenv.patched.notpip._vendor.packaging.specifiers import SpecifierSet # noqa: F401
+ from pipenv.patched.notpip._vendor.packaging.markers import Marker # noqa: F401
+
+
logger = logging.getLogger(__name__)
@@ -49,10 +62,23 @@ class InstallRequirement(object):
installing the said requirement.
"""
- def __init__(self, req, comes_from, source_dir=None, editable=False,
- link=None, update=True, markers=None,
- isolated=False, options=None, wheel_cache=None,
- constraint=False, extras=()):
+ def __init__(
+ self,
+ req, # type: Optional[Requirement]
+ comes_from, # type: Optional[Union[str, InstallRequirement]]
+ source_dir=None, # type: Optional[str]
+ editable=False, # type: bool
+ link=None, # type: Optional[Link]
+ update=True, # type: bool
+ markers=None, # type: Optional[Marker]
+ use_pep517=None, # type: Optional[bool]
+ isolated=False, # type: bool
+ options=None, # type: Optional[Dict[str, Any]]
+ wheel_cache=None, # type: Optional[WheelCache]
+ constraint=False, # type: bool
+ extras=() # type: Iterable[str]
+ ):
+ # type: (...) -> None
assert req is None or isinstance(req, Requirement), req
self.req = req
self.comes_from = comes_from
@@ -64,10 +90,10 @@ class InstallRequirement(object):
self.editable = editable
self._wheel_cache = wheel_cache
- if link is not None:
- self.link = self.original_link = link
- else:
- self.link = self.original_link = req and req.url and Link(req.url)
+ if link is None and req and req.url:
+ # PEP 508 URL requirement
+ link = Link(req.url)
+ self.link = self.original_link = link
if extras:
self.extras = extras
@@ -77,11 +103,11 @@ class InstallRequirement(object):
}
else:
self.extras = set()
- if markers is not None:
- self.markers = markers
- else:
- self.markers = req and req.marker
- self._egg_info_path = None
+ if markers is None and req:
+ markers = req.marker
+ self.markers = markers
+
+ self._egg_info_path = None # type: Optional[str]
# This holds the pkg_resources.Distribution object if this requirement
# is already available:
self.satisfied_by = None
@@ -92,11 +118,11 @@ class InstallRequirement(object):
self._temp_build_dir = TempDirectory(kind="req-build")
# Used to store the global directory where the _temp_build_dir should
# have been created. Cf _correct_build_location method.
- self._ideal_build_dir = None
+ self._ideal_build_dir = None # type: Optional[str]
# True if the editable should be updated:
self.update = update
# Set to True after successful installation
- self.install_succeeded = None
+ self.install_succeeded = None # type: Optional[bool]
# UninstallPathSet of uninstalled distribution (for possible rollback)
self.uninstalled_pathset = None
self.options = options if options else {}
@@ -105,32 +131,37 @@ class InstallRequirement(object):
self.is_direct = False
self.isolated = isolated
- self.build_env = NoOpBuildEnvironment()
+ self.build_env = NoOpBuildEnvironment() # type: BuildEnvironment
+
+ # For PEP 517, the directory where we request the project metadata
+ # gets stored. We need this to pass to build_wheel, so the backend
+ # can ensure that the wheel matches the metadata (see the PEP for
+ # details).
+ self.metadata_directory = None # type: Optional[str]
# The static build requirements (from pyproject.toml)
- self.pyproject_requires = None
+ self.pyproject_requires = None # type: Optional[List[str]]
# Build requirements that we will check are available
- # TODO: We don't do this for --no-build-isolation. Should we?
- self.requirements_to_check = []
+ self.requirements_to_check = [] # type: List[str]
# The PEP 517 backend we should use to build the project
- self.pep517_backend = None
+ self.pep517_backend = None # type: Optional[Pep517HookCaller]
# Are we using PEP 517 for this requirement?
# After pyproject.toml has been loaded, the only valid values are True
# and False. Before loading, None is valid (meaning "use the default").
# Setting an explicit value before loading pyproject.toml is supported,
# but after loading this flag should be treated as read only.
- self.use_pep517 = None
+ self.use_pep517 = use_pep517
def __str__(self):
if self.req:
s = str(self.req)
if self.link:
- s += ' from %s' % self.link.url
+ s += ' from %s' % redact_password_from_url(self.link.url)
elif self.link:
- s = self.link.url
+ s = redact_password_from_url(self.link.url)
else:
s = ''
if self.satisfied_by is not None:
@@ -149,6 +180,7 @@ class InstallRequirement(object):
self.__class__.__name__, str(self), self.editable)
def populate_link(self, finder, upgrade, require_hashes):
+ # type: (PackageFinder, bool, bool) -> None
"""Ensure that if a link can be found for this, that it is found.
Note that self.link may still be None - if Upgrade is False and the
@@ -171,16 +203,19 @@ class InstallRequirement(object):
# Things that are valid for all kinds of requirements?
@property
def name(self):
+ # type: () -> Optional[str]
if self.req is None:
return None
return native_str(pkg_resources.safe_name(self.req.name))
@property
def specifier(self):
+ # type: () -> SpecifierSet
return self.req.specifier
@property
def is_pinned(self):
+ # type: () -> bool
"""Return whether I am pinned to an exact version.
For example, some-package==1.2 is pinned; some-package>1.2 is not.
@@ -194,6 +229,7 @@ class InstallRequirement(object):
return get_installed_version(self.name)
def match_markers(self, extras_requested=None):
+ # type: (Optional[Iterable[str]]) -> bool
if not extras_requested:
# Provide an extra to safely evaluate the markers
# without matching any extra
@@ -207,6 +243,7 @@ class InstallRequirement(object):
@property
def has_hash_options(self):
+ # type: () -> bool
"""Return whether any known-good hashes are specified as options.
These activate --require-hashes mode; hashes specified as part of a
@@ -216,6 +253,7 @@ class InstallRequirement(object):
return bool(self.options.get('hashes', {}))
def hashes(self, trust_internet=True):
+ # type: (bool) -> Hashes
"""Return a hash-comparer that considers my option- and URL-based
hashes to be known-good.
@@ -237,6 +275,7 @@ class InstallRequirement(object):
return Hashes(good_hashes)
def from_path(self):
+ # type: () -> Optional[str]
"""Format a nice indicator to show where this "comes from"
"""
if self.req is None:
@@ -252,6 +291,7 @@ class InstallRequirement(object):
return s
def build_location(self, build_dir):
+ # type: (str) -> Optional[str]
assert build_dir is not None
if self._temp_build_dir.path is not None:
return self._temp_build_dir.path
@@ -279,6 +319,7 @@ class InstallRequirement(object):
return os.path.join(build_dir, name)
def _correct_build_location(self):
+ # type: () -> None
"""Move self._temp_build_dir to self._ideal_build_dir/self.req.name
For some requirements (e.g. a path to a directory), the name of the
@@ -292,7 +333,8 @@ class InstallRequirement(object):
return
assert self.req is not None
assert self._temp_build_dir.path
- assert self._ideal_build_dir.path
+ assert (self._ideal_build_dir is not None and
+ self._ideal_build_dir.path) # type: ignore
old_location = self._temp_build_dir.path
self._temp_build_dir.path = None
@@ -311,7 +353,16 @@ class InstallRequirement(object):
self.source_dir = os.path.normpath(os.path.abspath(new_location))
self._egg_info_path = None
+ # Correct the metadata directory, if it exists
+ if self.metadata_directory:
+ old_meta = self.metadata_directory
+ rel = os.path.relpath(old_meta, start=old_location)
+ new_meta = os.path.join(new_location, rel)
+ new_meta = os.path.normpath(os.path.abspath(new_meta))
+ self.metadata_directory = new_meta
+
def remove_temporary_source(self):
+ # type: () -> None
"""Remove the source files from this requirement, if they are marked
for deletion"""
if self.source_dir and os.path.exists(
@@ -323,6 +374,7 @@ class InstallRequirement(object):
self.build_env.cleanup()
def check_if_exists(self, use_user_site):
+ # type: (bool) -> bool
"""Find an installed distribution that satisfies or conflicts
with this requirement, and set self.satisfied_by or
self.conflicts_with appropriately.
@@ -366,11 +418,22 @@ class InstallRequirement(object):
# Things valid for wheels
@property
def is_wheel(self):
- return self.link and self.link.is_wheel
+ # type: () -> bool
+ if not self.link:
+ return False
+ return self.link.is_wheel
- def move_wheel_files(self, wheeldir, root=None, home=None, prefix=None,
- warn_script_location=True, use_user_site=False,
- pycompile=True):
+ def move_wheel_files(
+ self,
+ wheeldir, # type: str
+ root=None, # type: Optional[str]
+ home=None, # type: Optional[str]
+ prefix=None, # type: Optional[str]
+ warn_script_location=True, # type: bool
+ use_user_site=False, # type: bool
+ pycompile=True # type: bool
+ ):
+ # type: (...) -> None
move_wheel_files(
self.name, self.req, wheeldir,
user=use_user_site,
@@ -385,12 +448,14 @@ class InstallRequirement(object):
# Things valid for sdists
@property
def setup_py_dir(self):
+ # type: () -> str
return os.path.join(
self.source_dir,
self.link and self.link.subdirectory_fragment or '')
@property
def setup_py(self):
+ # type: () -> str
assert self.source_dir, "No source dir for %s" % self
setup_py = os.path.join(self.setup_py_dir, 'setup.py')
@@ -403,17 +468,13 @@ class InstallRequirement(object):
@property
def pyproject_toml(self):
+ # type: () -> str
assert self.source_dir, "No source dir for %s" % self
- pp_toml = os.path.join(self.setup_py_dir, 'pyproject.toml')
-
- # Python2 __file__ should not be unicode
- if six.PY2 and isinstance(pp_toml, six.text_type):
- pp_toml = pp_toml.encode(sys.getfilesystemencoding())
-
- return pp_toml
+ return make_pyproject_path(self.setup_py_dir)
def load_pyproject_toml(self):
+ # type: () -> None
"""Load the pyproject.toml file.
After calling this routine, all of the attributes related to PEP 517
@@ -437,41 +498,36 @@ class InstallRequirement(object):
self.pyproject_requires = requires
self.pep517_backend = Pep517HookCaller(self.setup_py_dir, backend)
- def run_egg_info(self):
+ # Use a custom function to call subprocesses
+ self.spin_message = ""
+
+ def runner(cmd, cwd=None, extra_environ=None):
+ with open_spinner(self.spin_message) as spinner:
+ call_subprocess(
+ cmd,
+ cwd=cwd,
+ extra_environ=extra_environ,
+ show_stdout=False,
+ spinner=spinner
+ )
+ self.spin_message = ""
+
+ self.pep517_backend._subprocess_runner = runner
+
+ def prepare_metadata(self):
+ # type: () -> None
+ """Ensure that project metadata is available.
+
+ Under PEP 517, call the backend hook to prepare the metadata.
+ Under legacy processing, call setup.py egg-info.
+ """
assert self.source_dir
- if self.name:
- logger.debug(
- 'Running setup.py (path:%s) egg_info for package %s',
- self.setup_py, self.name,
- )
- else:
- logger.debug(
- 'Running setup.py (path:%s) egg_info for package from %s',
- self.setup_py, self.link,
- )
with indent_log():
- script = SETUPTOOLS_SHIM % self.setup_py
- sys_executable = os.environ.get('PIP_PYTHON_PATH', sys.executable)
- base_cmd = [sys_executable, '-c', script]
- if self.isolated:
- base_cmd += ["--no-user-cfg"]
- egg_info_cmd = base_cmd + ['egg_info']
- # We can't put the .egg-info files at the root, because then the
- # source code will be mistaken for an installed egg, causing
- # problems
- if self.editable:
- egg_base_option = []
+ if self.use_pep517:
+ self.prepare_pep517_metadata()
else:
- egg_info_dir = os.path.join(self.setup_py_dir, 'pip-egg-info')
- ensure_dir(egg_info_dir)
- egg_base_option = ['--egg-base', 'pip-egg-info']
- with self.build_env:
- call_subprocess(
- egg_info_cmd + egg_base_option,
- cwd=self.setup_py_dir,
- show_stdout=False,
- command_desc='python setup.py egg_info')
+ self.run_egg_info()
if not self.req:
if isinstance(parse_version(self.metadata["Version"]), Version):
@@ -490,15 +546,72 @@ class InstallRequirement(object):
metadata_name = canonicalize_name(self.metadata["Name"])
if canonicalize_name(self.req.name) != metadata_name:
logger.warning(
- 'Running setup.py (path:%s) egg_info for package %s '
+ 'Generating metadata for package %s '
'produced metadata for project name %s. Fix your '
'#egg=%s fragments.',
- self.setup_py, self.name, metadata_name, self.name
+ self.name, metadata_name, self.name
)
self.req = Requirement(metadata_name)
+ def prepare_pep517_metadata(self):
+ # type: () -> None
+ assert self.pep517_backend is not None
+
+ metadata_dir = os.path.join(
+ self.setup_py_dir,
+ 'pip-wheel-metadata'
+ )
+ ensure_dir(metadata_dir)
+
+ with self.build_env:
+ # Note that Pep517HookCaller implements a fallback for
+ # prepare_metadata_for_build_wheel, so we don't have to
+ # consider the possibility that this hook doesn't exist.
+ backend = self.pep517_backend
+ self.spin_message = "Preparing wheel metadata"
+ distinfo_dir = backend.prepare_metadata_for_build_wheel(
+ metadata_dir
+ )
+
+ self.metadata_directory = os.path.join(metadata_dir, distinfo_dir)
+
+ def run_egg_info(self):
+ # type: () -> None
+ if self.name:
+ logger.debug(
+ 'Running setup.py (path:%s) egg_info for package %s',
+ self.setup_py, self.name,
+ )
+ else:
+ logger.debug(
+ 'Running setup.py (path:%s) egg_info for package from %s',
+ self.setup_py, self.link,
+ )
+ script = SETUPTOOLS_SHIM % self.setup_py
+ sys_executable = os.environ.get('PIP_PYTHON_PATH', sys.executable)
+ base_cmd = [sys_executable, '-c', script]
+ if self.isolated:
+ base_cmd += ["--no-user-cfg"]
+ egg_info_cmd = base_cmd + ['egg_info']
+ # We can't put the .egg-info files at the root, because then the
+ # source code will be mistaken for an installed egg, causing
+ # problems
+ if self.editable:
+ egg_base_option = [] # type: List[str]
+ else:
+ egg_info_dir = os.path.join(self.setup_py_dir, 'pip-egg-info')
+ ensure_dir(egg_info_dir)
+ egg_base_option = ['--egg-base', 'pip-egg-info']
+ with self.build_env:
+ call_subprocess(
+ egg_info_cmd + egg_base_option,
+ cwd=self.setup_py_dir,
+ show_stdout=False,
+ command_desc='python setup.py egg_info')
+
@property
def egg_info_path(self):
+ # type: () -> str
if self._egg_info_path is None:
if self.editable:
base = self.source_dir
@@ -557,18 +670,31 @@ class InstallRequirement(object):
return self._metadata
def get_dist(self):
- """Return a pkg_resources.Distribution built from self.egg_info_path"""
- egg_info = self.egg_info_path.rstrip(os.path.sep)
- base_dir = os.path.dirname(egg_info)
- metadata = pkg_resources.PathMetadata(base_dir, egg_info)
- dist_name = os.path.splitext(os.path.basename(egg_info))[0]
- return pkg_resources.Distribution(
- os.path.dirname(egg_info),
+ # type: () -> Distribution
+ """Return a pkg_resources.Distribution for this requirement"""
+ if self.metadata_directory:
+ base_dir, distinfo = os.path.split(self.metadata_directory)
+ metadata = pkg_resources.PathMetadata(
+ base_dir, self.metadata_directory
+ )
+ dist_name = os.path.splitext(distinfo)[0]
+ typ = pkg_resources.DistInfoDistribution
+ else:
+ egg_info = self.egg_info_path.rstrip(os.path.sep)
+ base_dir = os.path.dirname(egg_info)
+ metadata = pkg_resources.PathMetadata(base_dir, egg_info)
+ dist_name = os.path.splitext(os.path.basename(egg_info))[0]
+ # https://github.com/python/mypy/issues/1174
+ typ = pkg_resources.Distribution # type: ignore
+
+ return typ(
+ base_dir,
project_name=dist_name,
metadata=metadata,
)
def assert_source_matches_version(self):
+ # type: () -> None
assert self.source_dir
version = self.metadata['version']
if self.req.specifier and version not in self.req.specifier:
@@ -587,6 +713,7 @@ class InstallRequirement(object):
# For both source distributions and editables
def ensure_has_source_dir(self, parent_dir):
+ # type: (str) -> str
"""Ensure that a source_dir is set.
This will create a temporary build dir if the name of the requirement
@@ -601,8 +728,13 @@ class InstallRequirement(object):
return self.source_dir
# For editable installations
- def install_editable(self, install_options,
- global_options=(), prefix=None):
+ def install_editable(
+ self,
+ install_options, # type: List[str]
+ global_options=(), # type: Sequence[str]
+ prefix=None # type: Optional[str]
+ ):
+ # type: (...) -> None
logger.info('Running setup.py develop for %s', self.name)
if self.isolated:
@@ -614,8 +746,8 @@ class InstallRequirement(object):
with indent_log():
# FIXME: should we do --install-headers here too?
- sys_executable = os.environ.get('PIP_PYTHON_PATH', sys.executable)
with self.build_env:
+ sys_executable = os.environ.get('PIP_PYTHON_PATH', sys.executable)
call_subprocess(
[
sys_executable,
@@ -633,6 +765,7 @@ class InstallRequirement(object):
self.install_succeeded = True
def update_editable(self, obtain=True):
+ # type: (bool) -> None
if not self.link:
logger.debug(
"Cannot update repository at %s; repository location is "
@@ -664,6 +797,7 @@ class InstallRequirement(object):
# Top-level Actions
def uninstall(self, auto_confirm=False, verbose=False,
use_user_site=False):
+ # type: (bool, bool, bool) -> Optional[UninstallPathSet]
"""
Uninstall the distribution currently satisfying this requirement.
@@ -678,7 +812,7 @@ class InstallRequirement(object):
"""
if not self.check_if_exists(use_user_site):
logger.warning("Skipping %s as it is not installed.", self.name)
- return
+ return None
dist = self.satisfied_by or self.conflicts_with
uninstalled_pathset = UninstallPathSet.from_dist(dist)
@@ -693,9 +827,16 @@ class InstallRequirement(object):
name = name.replace(os.path.sep, '/')
return name
+ def _get_archive_name(self, path, parentdir, rootdir):
+ # type: (str, str, str) -> str
+ path = os.path.join(parentdir, path)
+ name = self._clean_zip_name(path, rootdir)
+ return self.name + '/' + name
+
# TODO: Investigate if this should be kept in InstallRequirement
# Seems to be used only when VCS + downloads
def archive(self, build_dir):
+ # type: (str) -> None
assert self.source_dir
create_archive = True
archive_name = '%s-%s.zip' % (self.name, self.metadata["version"])
@@ -729,23 +870,35 @@ class InstallRequirement(object):
if 'pip-egg-info' in dirnames:
dirnames.remove('pip-egg-info')
for dirname in dirnames:
- dirname = os.path.join(dirpath, dirname)
- name = self._clean_zip_name(dirname, dir)
- zipdir = zipfile.ZipInfo(self.name + '/' + name + '/')
+ dir_arcname = self._get_archive_name(dirname,
+ parentdir=dirpath,
+ rootdir=dir)
+ zipdir = zipfile.ZipInfo(dir_arcname + '/')
zipdir.external_attr = 0x1ED << 16 # 0o755
zip.writestr(zipdir, '')
for filename in filenames:
if filename == PIP_DELETE_MARKER_FILENAME:
continue
+ file_arcname = self._get_archive_name(filename,
+ parentdir=dirpath,
+ rootdir=dir)
filename = os.path.join(dirpath, filename)
- name = self._clean_zip_name(filename, dir)
- zip.write(filename, self.name + '/' + name)
+ zip.write(filename, file_arcname)
zip.close()
logger.info('Saved %s', display_path(archive_path))
- def install(self, install_options, global_options=None, root=None,
- home=None, prefix=None, warn_script_location=True,
- use_user_site=False, pycompile=True):
+ def install(
+ self,
+ install_options, # type: List[str]
+ global_options=None, # type: Optional[Sequence[str]]
+ root=None, # type: Optional[str]
+ home=None, # type: Optional[str]
+ prefix=None, # type: Optional[str]
+ warn_script_location=True, # type: bool
+ use_user_site=False, # type: bool
+ pycompile=True # type: bool
+ ):
+ # type: (...) -> None
global_options = global_options if global_options is not None else []
if self.editable:
self.install_editable(
@@ -775,7 +928,8 @@ class InstallRequirement(object):
self.options.get('install_options', [])
if self.isolated:
- global_options = global_options + ["--no-user-cfg"]
+ # https://github.com/python/mypy/issues/1174
+ global_options = global_options + ["--no-user-cfg"] # type: ignore
with TempDirectory(kind="record") as temp_dir:
record_filename = os.path.join(temp_dir.path, 'install-record.txt')
@@ -834,8 +988,15 @@ class InstallRequirement(object):
with open(inst_files_path, 'w') as f:
f.write('\n'.join(new_lines) + '\n')
- def get_install_args(self, global_options, record_filename, root, prefix,
- pycompile):
+ def get_install_args(
+ self,
+ global_options, # type: Sequence[str]
+ record_filename, # type: str
+ root, # type: Optional[str]
+ prefix, # type: Optional[str]
+ pycompile # type: bool
+ ):
+ # type: (...) -> List[str]
sys_executable = os.environ.get('PIP_PYTHON_PATH', sys.executable)
install_args = [sys_executable, "-u"]
install_args.append('-c')
diff --git a/pipenv/patched/notpip/_internal/req/req_set.py b/pipenv/patched/notpip/_internal/req/req_set.py
index a65851ff..e7da5b71 100644
--- a/pipenv/patched/notpip/_internal/req/req_set.py
+++ b/pipenv/patched/notpip/_internal/req/req_set.py
@@ -5,29 +5,36 @@ from collections import OrderedDict
from pipenv.patched.notpip._internal.exceptions import InstallationError
from pipenv.patched.notpip._internal.utils.logging import indent_log
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.wheel import Wheel
+if MYPY_CHECK_RUNNING:
+ from typing import Optional, List, Tuple, Dict, Iterable # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_install import InstallRequirement # noqa: F401
+
+
logger = logging.getLogger(__name__)
class RequirementSet(object):
def __init__(self, require_hashes=False, check_supported_wheels=True, ignore_compatibility=True):
+ # type: (bool, bool) -> None
"""Create a RequirementSet.
"""
- self.requirements = OrderedDict()
+ self.requirements = OrderedDict() # type: Dict[str, InstallRequirement] # noqa: E501
self.require_hashes = require_hashes
self.check_supported_wheels = check_supported_wheels
if ignore_compatibility:
self.check_supported_wheels = False
- self.ignore_compatibility = True if (check_supported_wheels is False or ignore_compatibility is True) else False
+ self.ignore_compatibility = (check_supported_wheels is False or ignore_compatibility is True)
# Mapping of alias: real_name
- self.requirement_aliases = {}
- self.unnamed_requirements = []
- self.successfully_downloaded = []
- self.reqs_to_cleanup = []
+ self.requirement_aliases = {} # type: Dict[str, str]
+ self.unnamed_requirements = [] # type: List[InstallRequirement]
+ self.successfully_downloaded = [] # type: List[InstallRequirement]
+ self.reqs_to_cleanup = [] # type: List[InstallRequirement]
def __str__(self):
reqs = [req for req in self.requirements.values()
@@ -42,8 +49,13 @@ class RequirementSet(object):
return ('<%s object; %d requirement(s): %s>'
% (self.__class__.__name__, len(reqs), reqs_str))
- def add_requirement(self, install_req, parent_req_name=None,
- extras_requested=None):
+ def add_requirement(
+ self,
+ install_req, # type: InstallRequirement
+ parent_req_name=None, # type: Optional[str]
+ extras_requested=None # type: Optional[Iterable[str]]
+ ):
+ # type: (...) -> Tuple[List[InstallRequirement], Optional[InstallRequirement]] # noqa: E501
"""Add install_req as a requirement to install.
:param parent_req_name: The name of the requirement that needed this
@@ -155,6 +167,7 @@ class RequirementSet(object):
return [existing_req], existing_req
def has_requirement(self, project_name):
+ # type: (str) -> bool
name = project_name.lower()
if (name in self.requirements and
not self.requirements[name].constraint or
@@ -165,10 +178,12 @@ class RequirementSet(object):
@property
def has_requirements(self):
+ # type: () -> List[InstallRequirement]
return list(req for req in self.requirements.values() if not
req.constraint) or self.unnamed_requirements
def get_requirement(self, project_name):
+ # type: (str) -> InstallRequirement
for name in project_name, project_name.lower():
if name in self.requirements:
return self.requirements[name]
@@ -177,6 +192,7 @@ class RequirementSet(object):
pass
def cleanup_files(self):
+ # type: () -> None
"""Clean up files, remove builds."""
logger.debug('Cleaning up...')
with indent_log():
diff --git a/pipenv/patched/notpip/_internal/req/req_tracker.py b/pipenv/patched/notpip/_internal/req/req_tracker.py
index 6e0201fe..d17a4187 100644
--- a/pipenv/patched/notpip/_internal/req/req_tracker.py
+++ b/pipenv/patched/notpip/_internal/req/req_tracker.py
@@ -7,6 +7,12 @@ import logging
import os
from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Set, Iterator # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_install import InstallRequirement # noqa: F401
+ from pipenv.patched.notpip._internal.models.link import Link # noqa: F401
logger = logging.getLogger(__name__)
@@ -14,6 +20,7 @@ logger = logging.getLogger(__name__)
class RequirementTracker(object):
def __init__(self):
+ # type: () -> None
self._root = os.environ.get('PIP_REQ_TRACKER')
if self._root is None:
self._temp_dir = TempDirectory(delete=False, kind='req-tracker')
@@ -23,7 +30,7 @@ class RequirementTracker(object):
else:
self._temp_dir = None
logger.debug('Re-using requirements tracker %r', self._root)
- self._entries = set()
+ self._entries = set() # type: Set[InstallRequirement]
def __enter__(self):
return self
@@ -32,10 +39,12 @@ class RequirementTracker(object):
self.cleanup()
def _entry_path(self, link):
+ # type: (Link) -> str
hashed = hashlib.sha224(link.url_without_fragment.encode()).hexdigest()
return os.path.join(self._root, hashed)
def add(self, req):
+ # type: (InstallRequirement) -> None
link = req.link
info = str(req)
entry_path = self._entry_path(link)
@@ -54,12 +63,14 @@ class RequirementTracker(object):
logger.debug('Added %s to build tracker %r', req, self._root)
def remove(self, req):
+ # type: (InstallRequirement) -> None
link = req.link
self._entries.remove(req)
os.unlink(self._entry_path(link))
logger.debug('Removed %s from build tracker %r', req, self._root)
def cleanup(self):
+ # type: () -> None
for req in set(self._entries):
self.remove(req)
remove = self._temp_dir is not None
@@ -71,6 +82,7 @@ class RequirementTracker(object):
@contextlib.contextmanager
def track(self, req):
+ # type: (InstallRequirement) -> Iterator[None]
self.add(req)
yield
self.remove(req)
diff --git a/pipenv/patched/notpip/_internal/req/req_uninstall.py b/pipenv/patched/notpip/_internal/req/req_uninstall.py
index 4cd15d84..ce80e6dc 100644
--- a/pipenv/patched/notpip/_internal/req/req_uninstall.py
+++ b/pipenv/patched/notpip/_internal/req/req_uninstall.py
@@ -15,9 +15,9 @@ from pipenv.patched.notpip._internal.utils.compat import WINDOWS, cache_from_sou
from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import (
FakeFile, ask, dist_in_usersite, dist_is_local, egg_link_path, is_local,
- normalize_path, renames,
+ normalize_path, renames, rmtree,
)
-from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
+from pipenv.patched.notpip._internal.utils.temp_dir import AdjacentTempDirectory, TempDirectory
logger = logging.getLogger(__name__)
@@ -86,16 +86,54 @@ def compact(paths):
sep = os.path.sep
short_paths = set()
for path in sorted(paths, key=len):
- should_add = any(
+ should_skip = any(
path.startswith(shortpath.rstrip("*")) and
path[len(shortpath.rstrip("*").rstrip(sep))] == sep
for shortpath in short_paths
)
- if not should_add:
+ if not should_skip:
short_paths.add(path)
return short_paths
+def compress_for_rename(paths):
+ """Returns a set containing the paths that need to be renamed.
+
+ This set may include directories when the original sequence of paths
+ included every file on disk.
+ """
+ case_map = dict((os.path.normcase(p), p) for p in paths)
+ remaining = set(case_map)
+ unchecked = sorted(set(os.path.split(p)[0]
+ for p in case_map.values()), key=len)
+ wildcards = set()
+
+ def norm_join(*a):
+ return os.path.normcase(os.path.join(*a))
+
+ for root in unchecked:
+ if any(os.path.normcase(root).startswith(w)
+ for w in wildcards):
+ # This directory has already been handled.
+ continue
+
+ all_files = set()
+ all_subdirs = set()
+ for dirname, subdirs, files in os.walk(root):
+ all_subdirs.update(norm_join(root, dirname, d)
+ for d in subdirs)
+ all_files.update(norm_join(root, dirname, f)
+ for f in files)
+ # If all the files we found are in our remaining set of files to
+ # remove, then remove them from the latter set and add a wildcard
+ # for the directory.
+ if not (all_files - remaining):
+ remaining.difference_update(all_files)
+ wildcards.add(root + os.sep)
+
+ return set(map(case_map.__getitem__, remaining)) | wildcards
+
+
def compress_for_output_listing(paths):
"""Returns a tuple of 2 sets of which paths to display to user
@@ -145,6 +183,111 @@ def compress_for_output_listing(paths):
return will_remove, will_skip
+class StashedUninstallPathSet(object):
+ """A set of file rename operations to stash files while
+ tentatively uninstalling them."""
+ def __init__(self):
+ # Mapping from source file root to [Adjacent]TempDirectory
+ # for files under that directory.
+ self._save_dirs = {}
+ # (old path, new path) tuples for each move that may need
+ # to be undone.
+ self._moves = []
+
+ def _get_directory_stash(self, path):
+ """Stashes a directory.
+
+ Directories are stashed adjacent to their original location if
+ possible, or else moved/copied into the user's temp dir."""
+
+ try:
+ save_dir = AdjacentTempDirectory(path)
+ save_dir.create()
+ except OSError:
+ save_dir = TempDirectory(kind="uninstall")
+ save_dir.create()
+ self._save_dirs[os.path.normcase(path)] = save_dir
+
+ return save_dir.path
+
+ def _get_file_stash(self, path):
+ """Stashes a file.
+
+ If no root has been provided, one will be created for the directory
+ in the user's temp directory."""
+ path = os.path.normcase(path)
+ head, old_head = os.path.dirname(path), None
+ save_dir = None
+
+ while head != old_head:
+ try:
+ save_dir = self._save_dirs[head]
+ break
+ except KeyError:
+ pass
+ head, old_head = os.path.dirname(head), head
+ else:
+ # Did not find any suitable root
+ head = os.path.dirname(path)
+ save_dir = TempDirectory(kind='uninstall')
+ save_dir.create()
+ self._save_dirs[head] = save_dir
+
+ relpath = os.path.relpath(path, head)
+ if relpath and relpath != os.path.curdir:
+ return os.path.join(save_dir.path, relpath)
+ return save_dir.path
+
+ def stash(self, path):
+ """Stashes the directory or file and returns its new location.
+ """
+ if os.path.isdir(path):
+ new_path = self._get_directory_stash(path)
+ else:
+ new_path = self._get_file_stash(path)
+
+ self._moves.append((path, new_path))
+ if os.path.isdir(path) and os.path.isdir(new_path):
+ # If we're moving a directory, we need to
+ # remove the destination first or else it will be
+ # moved to inside the existing directory.
+ # We just created new_path ourselves, so it will
+ # be removable.
+ os.rmdir(new_path)
+ renames(path, new_path)
+ return new_path
+
+ def commit(self):
+ """Commits the uninstall by removing stashed files."""
+ for _, save_dir in self._save_dirs.items():
+ save_dir.cleanup()
+ self._moves = []
+ self._save_dirs = {}
+
+ def rollback(self):
+ """Undoes the uninstall by moving stashed files back."""
+ for p in self._moves:
+ logging.info("Moving to %s\n from %s", *p)
+
+ for new_path, path in self._moves:
+ try:
+ logger.debug('Replacing %s from %s', new_path, path)
+ if os.path.isfile(new_path):
+ os.unlink(new_path)
+ elif os.path.isdir(new_path):
+ rmtree(new_path)
+ renames(path, new_path)
+ except OSError as ex:
+ logger.error("Failed to restore %s", new_path)
+ logger.debug("Exception: %s", ex)
+
+ self.commit()
+
+ @property
+ def can_rollback(self):
+ return bool(self._moves)
+
+
class UninstallPathSet(object):
"""A set of file paths to be removed in the uninstallation of a
requirement."""
@@ -153,8 +296,7 @@ class UninstallPathSet(object):
self._refuse = set()
self.pth = {}
self.dist = dist
- self.save_dir = TempDirectory(kind="uninstall")
- self._moved_paths = []
+ self._moved_paths = StashedUninstallPathSet()
def _permitted(self, path):
"""
@@ -192,11 +334,6 @@ class UninstallPathSet(object):
else:
self._refuse.add(pth_file)
- def _stash(self, path):
- return os.path.join(
- self.save_dir.path, os.path.splitdrive(path)[1].lstrip(os.path.sep)
- )
-
def remove(self, auto_confirm=False, verbose=False):
"""Remove paths in ``self.paths`` with confirmation (unless
``auto_confirm`` is True)."""
@@ -215,13 +352,14 @@ class UninstallPathSet(object):
with indent_log():
if auto_confirm or self._allowed_to_proceed(verbose):
- self.save_dir.create()
+ moved = self._moved_paths
- for path in sorted(compact(self.paths)):
- new_path = self._stash(path)
+ for_rename = compress_for_rename(self.paths)
+
+ for path in sorted(compact(for_rename)):
+ moved.stash(path)
logger.debug('Removing file or directory %s', path)
- self._moved_paths.append(path)
- renames(path, new_path)
+
for pth in self.pth.values():
pth.remove()
@@ -251,29 +389,27 @@ class UninstallPathSet(object):
_display('Would remove:', will_remove)
_display('Would not remove (might be manually added):', will_skip)
_display('Would not remove (outside of prefix):', self._refuse)
+ if verbose:
+ _display('Will actually move:', compress_for_rename(self.paths))
return ask('Proceed (y/n)? ', ('y', 'n')) == 'y'
def rollback(self):
"""Rollback the changes previously made by remove()."""
- if self.save_dir.path is None:
+ if not self._moved_paths.can_rollback:
logger.error(
"Can't roll back %s; was not uninstalled",
self.dist.project_name,
)
return False
logger.info('Rolling back uninstall of %s', self.dist.project_name)
- for path in self._moved_paths:
- tmp_path = self._stash(path)
- logger.debug('Replacing %s', path)
- renames(tmp_path, path)
+ self._moved_paths.rollback()
for pth in self.pth.values():
pth.rollback()
def commit(self):
"""Remove temporary save dir: rollback will no longer be possible."""
- self.save_dir.cleanup()
- self._moved_paths = []
+ self._moved_paths.commit()
@classmethod
def from_dist(cls, dist):
diff --git a/pipenv/patched/notpip/_internal/resolve.py b/pipenv/patched/notpip/_internal/resolve.py
index b0d096f9..e42dd3d4 100644
--- a/pipenv/patched/notpip/_internal/resolve.py
+++ b/pipenv/patched/notpip/_internal/resolve.py
@@ -18,10 +18,23 @@ from pipenv.patched.notpip._internal.exceptions import (
BestVersionAlreadyInstalled, DistributionNotFound, HashError, HashErrors,
UnsupportedPythonVersion,
)
-from pipenv.patched.notpip._internal.req.constructors import install_req_from_req
+from pipenv.patched.notpip._internal.req.constructors import install_req_from_req_string
+from pipenv.patched.notpip._internal.req.req_install import InstallRequirement
from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import dist_in_usersite, ensure_dir
from pipenv.patched.notpip._internal.utils.packaging import check_dist_requires_python
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Optional, DefaultDict, List, Set # noqa: F401
+ from pipenv.patched.notpip._internal.download import PipSession # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_install import InstallRequirement # noqa: F401
+ from pipenv.patched.notpip._internal.index import PackageFinder # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_set import RequirementSet # noqa: F401
+ from pipenv.patched.notpip._internal.operations.prepare import ( # noqa: F401
+ DistAbstraction, RequirementPreparer
+ )
+ from pipenv.patched.notpip._internal.cache import WheelCache # noqa: F401
logger = logging.getLogger(__name__)
@@ -33,9 +46,23 @@ class Resolver(object):
_allowed_strategies = {"eager", "only-if-needed", "to-satisfy-only"}
- def __init__(self, preparer, session, finder, wheel_cache, use_user_site,
- ignore_dependencies, ignore_installed, ignore_requires_python,
- force_reinstall, isolated, upgrade_strategy, ignore_compatibility=False):
+ def __init__(
+ self,
+ preparer, # type: RequirementPreparer
+ session, # type: PipSession
+ finder, # type: PackageFinder
+ wheel_cache, # type: Optional[WheelCache]
+ use_user_site, # type: bool
+ ignore_dependencies, # type: bool
+ ignore_installed, # type: bool
+ ignore_requires_python, # type: bool
+ force_reinstall, # type: bool
+ isolated, # type: bool
+ upgrade_strategy, # type: str
+ use_pep517=None, # type: Optional[bool]
+ ignore_compatibility=False, # type: bool
+ ):
+ # type: (...) -> None
super(Resolver, self).__init__()
assert upgrade_strategy in self._allowed_strategies
@@ -47,7 +74,8 @@ class Resolver(object):
# information about both sdist and wheels transparently.
self.wheel_cache = wheel_cache
- self.require_hashes = None # This is set in resolve
+ # This is set in resolve
+ self.require_hashes = None # type: Optional[bool]
self.upgrade_strategy = upgrade_strategy
self.force_reinstall = force_reinstall
@@ -57,13 +85,16 @@ class Resolver(object):
self.ignore_requires_python = ignore_requires_python
self.ignore_compatibility = ignore_compatibility
self.use_user_site = use_user_site
+ self.use_pep517 = use_pep517
self.requires_python = None
if self.ignore_compatibility:
self.ignore_requires_python = True
- self._discovered_dependencies = defaultdict(list)
+ self._discovered_dependencies = \
+ defaultdict(list) # type: DefaultDict[str, List]
def resolve(self, requirement_set):
+ # type: (RequirementSet) -> None
"""Resolve what operations need to be done
As a side-effect of this method, the packages (and their dependencies)
@@ -98,7 +129,7 @@ class Resolver(object):
# exceptions cannot be checked ahead of time, because
# req.populate_link() needs to be called before we can make decisions
# based on link type.
- discovered_reqs = []
+ discovered_reqs = [] # type: List[InstallRequirement]
hash_errors = HashErrors()
for req in chain(root_reqs, discovered_reqs):
try:
@@ -113,6 +144,7 @@ class Resolver(object):
raise hash_errors
def _is_upgrade_allowed(self, req):
+ # type: (InstallRequirement) -> bool
if self.upgrade_strategy == "to-satisfy-only":
return False
elif self.upgrade_strategy == "eager":
@@ -122,6 +154,7 @@ class Resolver(object):
return req.is_direct
def _set_req_to_reinstall(self, req):
+ # type: (InstallRequirement) -> None
"""
Set a requirement to be installed.
"""
@@ -133,6 +166,7 @@ class Resolver(object):
# XXX: Stop passing requirement_set for options
def _check_skip_installed(self, req_to_install):
+ # type: (InstallRequirement) -> Optional[str]
"""Check if req_to_install should be skipped.
This will check if the req is installed, and whether we should upgrade
@@ -185,6 +219,7 @@ class Resolver(object):
return None
def _get_abstract_dist_for(self, req):
+ # type: (InstallRequirement) -> DistAbstraction
"""Takes a InstallRequirement and returns a single AbstractDist \
representing a prepared variant of the same.
"""
@@ -241,7 +276,13 @@ class Resolver(object):
return abstract_dist
- def _resolve_one(self, requirement_set, req_to_install, ignore_requires_python=False):
+ def _resolve_one(
+ self,
+ requirement_set, # type: RequirementSet
+ req_to_install, # type: InstallRequirement
+ ignore_requires_python=False # type: bool
+ ):
+ # type: (...) -> List[InstallRequirement]
"""Prepare a single requirements file.
:return: A list of additional InstallRequirements to also install.
@@ -260,11 +301,11 @@ class Resolver(object):
abstract_dist = self._get_abstract_dist_for(req_to_install)
# Parse and return dependencies
- dist = abstract_dist.dist(self.finder)
+ dist = abstract_dist.dist()
try:
check_dist_requires_python(dist)
except UnsupportedPythonVersion as err:
- if self.ignore_requires_python or self.ignore_compatibility:
+ if self.ignore_requires_python or ignore_requires_python or self.ignore_compatibility:
logger.warning(err.args[0])
else:
raise
@@ -275,14 +316,16 @@ class Resolver(object):
except TypeError:
self.requires_python = None
- more_reqs = []
+
+ more_reqs = [] # type: List[InstallRequirement]
def add_req(subreq, extras_requested):
- sub_install_req = install_req_from_req(
+ sub_install_req = install_req_from_req_string(
str(subreq),
req_to_install,
isolated=self.isolated,
wheel_cache=self.wheel_cache,
+ use_pep517=self.use_pep517
)
parent_req_name = req_to_install.name
to_scan_again, add_to_parent = requirement_set.add_requirement(
@@ -300,10 +343,10 @@ class Resolver(object):
# We add req_to_install before its dependencies, so that we
# can refer to it when adding dependencies.
if not requirement_set.has_requirement(req_to_install.name):
- # 'unnamed' requirements will get added here
available_requested = sorted(
set(dist.extras) & set(req_to_install.extras)
)
+ # 'unnamed' requirements will get added here
req_to_install.is_direct = True
requirement_set.add_requirement(
req_to_install, parent_req_name=None,
@@ -335,11 +378,12 @@ class Resolver(object):
for available in available_requested:
if hasattr(dist, '_DistInfoDistribution__dep_map'):
for req in dist._DistInfoDistribution__dep_map[available]:
- req = install_req_from_req(
- str(req),
+ req = InstallRequirement(
+ req,
req_to_install,
isolated=self.isolated,
wheel_cache=self.wheel_cache,
+ use_pep517=None
)
more_reqs.append(req)
@@ -353,6 +397,7 @@ class Resolver(object):
return more_reqs
def get_installation_order(self, req_set):
+ # type: (RequirementSet) -> List[InstallRequirement]
"""Create the installation order.
The installation order is topological - requirements are installed
@@ -363,7 +408,7 @@ class Resolver(object):
# installs the user specified things in the order given, except when
# dependencies must come earlier to achieve topological order.
order = []
- ordered_reqs = set()
+ ordered_reqs = set() # type: Set[InstallRequirement]
def schedule(req):
if req.satisfied_by or req in ordered_reqs:
diff --git a/pipenv/patched/notpip/_internal/utils/appdirs.py b/pipenv/patched/notpip/_internal/utils/appdirs.py
index e8e14526..9ce3a1b3 100644
--- a/pipenv/patched/notpip/_internal/utils/appdirs.py
+++ b/pipenv/patched/notpip/_internal/utils/appdirs.py
@@ -10,9 +10,16 @@ import sys
from pipenv.patched.notpip._vendor.six import PY2, text_type
from pipenv.patched.notpip._internal.utils.compat import WINDOWS, expanduser
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ List, Union
+ )
def user_cache_dir(appname):
+ # type: (str) -> str
r"""
Return full path to the user-specific cache dir for this application.
@@ -61,6 +68,7 @@ def user_cache_dir(appname):
def user_data_dir(appname, roaming=False):
+ # type: (str, bool) -> str
r"""
Return full path to the user-specific data dir for this application.
@@ -113,6 +121,7 @@ def user_data_dir(appname, roaming=False):
def user_config_dir(appname, roaming=True):
+ # type: (str, bool) -> str
"""Return full path to the user-specific config dir for this application.
"appname" is the name of application.
@@ -146,6 +155,7 @@ def user_config_dir(appname, roaming=True):
# for the discussion regarding site_config_dirs locations
# see
def site_config_dirs(appname):
+ # type: (str) -> List[str]
r"""Return a list of potential user-shared config dirs for this application.
"appname" is the name of application.
@@ -186,6 +196,7 @@ def site_config_dirs(appname):
# -- Windows support functions --
def _get_win_folder_from_registry(csidl_name):
+ # type: (str) -> str
"""
This is a fallback technique at best. I'm not sure if using the
registry for this guarantees us the correct answer for all CSIDL_*
@@ -208,6 +219,7 @@ def _get_win_folder_from_registry(csidl_name):
def _get_win_folder_with_ctypes(csidl_name):
+ # type: (str) -> str
csidl_const = {
"CSIDL_APPDATA": 26,
"CSIDL_COMMON_APPDATA": 35,
diff --git a/pipenv/patched/notpip/_internal/utils/compat.py b/pipenv/patched/notpip/_internal/utils/compat.py
index 483bfdc8..1dad56b0 100644
--- a/pipenv/patched/notpip/_internal/utils/compat.py
+++ b/pipenv/patched/notpip/_internal/utils/compat.py
@@ -11,6 +11,11 @@ import sys
from pipenv.patched.notpip._vendor.six import text_type
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Tuple, Text # noqa: F401
+
try:
import ipaddress
except ImportError:
@@ -18,8 +23,8 @@ except ImportError:
from pipenv.patched.notpip._vendor import ipaddress # type: ignore
except ImportError:
import ipaddr as ipaddress # type: ignore
- ipaddress.ip_address = ipaddress.IPAddress
- ipaddress.ip_network = ipaddress.IPNetwork
+ ipaddress.ip_address = ipaddress.IPAddress # type: ignore
+ ipaddress.ip_network = ipaddress.IPNetwork # type: ignore
__all__ = [
@@ -68,6 +73,7 @@ else:
def console_to_str(data):
+ # type: (bytes) -> Text
"""Return a string, safe for output, of subprocess output.
We assume the data is in the locale preferred encoding.
@@ -88,13 +94,13 @@ def console_to_str(data):
# Now try to decode the data - if we fail, warn the user and
# decode with replacement.
try:
- s = data.decode(encoding)
+ decoded_data = data.decode(encoding)
except UnicodeDecodeError:
logger.warning(
"Subprocess output does not appear to be encoded as %s",
encoding,
)
- s = data.decode(encoding, errors=backslashreplace_decode)
+ decoded_data = data.decode(encoding, errors=backslashreplace_decode)
# Make sure we can print the output, by encoding it to the output
# encoding with replacement of unencodable characters, and then
@@ -112,20 +118,25 @@ def console_to_str(data):
"encoding", None)
if output_encoding:
- s = s.encode(output_encoding, errors="backslashreplace")
- s = s.decode(output_encoding)
+ output_encoded = decoded_data.encode(
+ output_encoding,
+ errors="backslashreplace"
+ )
+ decoded_data = output_encoded.decode(output_encoding)
- return s
+ return decoded_data
if sys.version_info >= (3,):
def native_str(s, replace=False):
+ # type: (str, bool) -> str
if isinstance(s, bytes):
return s.decode('utf-8', 'replace' if replace else 'strict')
return s
else:
def native_str(s, replace=False):
+ # type: (str, bool) -> str
# Replace is ignored -- unicode to UTF-8 can't fail
if isinstance(s, text_type):
return s.encode('utf-8')
@@ -133,6 +144,7 @@ else:
def get_path_uid(path):
+ # type: (str) -> int
"""
Return path's uid.
@@ -174,6 +186,7 @@ else:
def expanduser(path):
+ # type: (str) -> str
"""
Expand ~ and ~user constructions.
@@ -199,6 +212,7 @@ WINDOWS = (sys.platform.startswith("win") or
def samefile(file1, file2):
+ # type: (str, str) -> bool
"""Provide an alternative for os.path.samefile on Windows/Python2"""
if hasattr(os.path, 'samefile'):
return os.path.samefile(file1, file2)
@@ -210,13 +224,15 @@ def samefile(file1, file2):
if hasattr(shutil, 'get_terminal_size'):
def get_terminal_size():
+ # type: () -> Tuple[int, int]
"""
Returns a tuple (x, y) representing the width(x) and the height(y)
in characters of the terminal window.
"""
- return tuple(shutil.get_terminal_size())
+ return tuple(shutil.get_terminal_size()) # type: ignore
else:
def get_terminal_size():
+ # type: () -> Tuple[int, int]
"""
Returns a tuple (x, y) representing the width(x) and the height(y)
in characters of the terminal window.
diff --git a/pipenv/patched/notpip/_internal/utils/deprecation.py b/pipenv/patched/notpip/_internal/utils/deprecation.py
index b140ac71..2e309ec2 100644
--- a/pipenv/patched/notpip/_internal/utils/deprecation.py
+++ b/pipenv/patched/notpip/_internal/utils/deprecation.py
@@ -41,6 +41,7 @@ def _showwarning(message, category, filename, lineno, file=None, line=None):
def install_warning_logger():
+ # type: () -> None
# Enable our Deprecation Warnings
warnings.simplefilter("default", PipDeprecationWarning, append=True)
diff --git a/pipenv/patched/notpip/_internal/utils/encoding.py b/pipenv/patched/notpip/_internal/utils/encoding.py
index 56f60361..f03fc901 100644
--- a/pipenv/patched/notpip/_internal/utils/encoding.py
+++ b/pipenv/patched/notpip/_internal/utils/encoding.py
@@ -3,6 +3,11 @@ import locale
import re
import sys
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import List, Tuple, Text # noqa: F401
+
BOMS = [
(codecs.BOM_UTF8, 'utf8'),
(codecs.BOM_UTF16, 'utf16'),
@@ -11,12 +16,13 @@ BOMS = [
(codecs.BOM_UTF32, 'utf32'),
(codecs.BOM_UTF32_BE, 'utf32-be'),
(codecs.BOM_UTF32_LE, 'utf32-le'),
-]
+] # type: List[Tuple[bytes, Text]]
ENCODING_RE = re.compile(br'coding[:=]\s*([-\w.]+)')
def auto_decode(data):
+ # type: (bytes) -> Text
"""Check a bytes string for a BOM to correctly detect the encoding
Fallback to locale.getpreferredencoding(False) like open() on Python3"""
diff --git a/pipenv/patched/notpip/_internal/utils/filesystem.py b/pipenv/patched/notpip/_internal/utils/filesystem.py
index e8d6a2bb..d4aae97d 100644
--- a/pipenv/patched/notpip/_internal/utils/filesystem.py
+++ b/pipenv/patched/notpip/_internal/utils/filesystem.py
@@ -5,6 +5,7 @@ from pipenv.patched.notpip._internal.utils.compat import get_path_uid
def check_path_owner(path):
+ # type: (str) -> bool
# If we don't have a way to check the effective uid of this process, then
# we'll just assume that we own the directory.
if not hasattr(os, "geteuid"):
@@ -26,3 +27,4 @@ def check_path_owner(path):
return os.access(path, os.W_OK)
else:
previous, path = path, os.path.dirname(path)
+ return False # assume we don't own the path
diff --git a/pipenv/patched/notpip/_internal/utils/glibc.py b/pipenv/patched/notpip/_internal/utils/glibc.py
index ebcfc5be..e2b6d505 100644
--- a/pipenv/patched/notpip/_internal/utils/glibc.py
+++ b/pipenv/patched/notpip/_internal/utils/glibc.py
@@ -4,8 +4,14 @@ import ctypes
import re
import warnings
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Optional, Tuple # noqa: F401
+
def glibc_version_string():
+ # type: () -> Optional[str]
"Returns glibc version string, or None if not using glibc."
# ctypes.CDLL(None) internally calls dlopen(NULL), and as the dlopen
@@ -32,6 +38,7 @@ def glibc_version_string():
# Separated out from have_compatible_glibc for easier unit testing
def check_glibc_version(version_str, required_major, minimum_minor):
+ # type: (str, int, int) -> bool
# Parse string and check against requested version.
#
# We use a regexp instead of str.split because we want to discard any
@@ -48,7 +55,8 @@ def check_glibc_version(version_str, required_major, minimum_minor):
def have_compatible_glibc(required_major, minimum_minor):
- version_str = glibc_version_string()
+ # type: (int, int) -> bool
+ version_str = glibc_version_string() # type: Optional[str]
if version_str is None:
return False
return check_glibc_version(version_str, required_major, minimum_minor)
@@ -72,6 +80,7 @@ def have_compatible_glibc(required_major, minimum_minor):
# misleading. Solution: instead of using platform, use our code that actually
# works.
def libc_ver():
+ # type: () -> Tuple[str, str]
"""Try to determine the glibc version
Returns a tuple of strings (lib, version) which default to empty strings
diff --git a/pipenv/patched/notpip/_internal/utils/hashes.py b/pipenv/patched/notpip/_internal/utils/hashes.py
index 818f5505..55cb8411 100644
--- a/pipenv/patched/notpip/_internal/utils/hashes.py
+++ b/pipenv/patched/notpip/_internal/utils/hashes.py
@@ -8,6 +8,18 @@ from pipenv.patched.notpip._internal.exceptions import (
HashMismatch, HashMissing, InstallationError,
)
from pipenv.patched.notpip._internal.utils.misc import read_chunks
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Dict, List, BinaryIO, NoReturn, Iterator
+ )
+ from pipenv.patched.notpip._vendor.six import PY3
+ if PY3:
+ from hashlib import _Hash # noqa: F401
+ else:
+ from hashlib import _hash as _Hash # noqa: F401
+
# The recommended hash algo of the moment. Change this whenever the state of
# the art changes; it won't hurt backward compatibility.
@@ -25,6 +37,7 @@ class Hashes(object):
"""
def __init__(self, hashes=None):
+ # type: (Dict[str, List[str]]) -> None
"""
:param hashes: A dict of algorithm names pointing to lists of allowed
hex digests
@@ -32,6 +45,7 @@ class Hashes(object):
self._allowed = {} if hashes is None else hashes
def check_against_chunks(self, chunks):
+ # type: (Iterator[bytes]) -> None
"""Check good hashes against ones built from iterable of chunks of
data.
@@ -55,9 +69,11 @@ class Hashes(object):
self._raise(gots)
def _raise(self, gots):
+ # type: (Dict[str, _Hash]) -> NoReturn
raise HashMismatch(self._allowed, gots)
def check_against_file(self, file):
+ # type: (BinaryIO) -> None
"""Check good hashes against a file-like object
Raise HashMismatch if none match.
@@ -66,14 +82,17 @@ class Hashes(object):
return self.check_against_chunks(read_chunks(file))
def check_against_path(self, path):
+ # type: (str) -> None
with open(path, 'rb') as file:
return self.check_against_file(file)
def __nonzero__(self):
+ # type: () -> bool
"""Return whether I know any known-good hashes."""
return bool(self._allowed)
def __bool__(self):
+ # type: () -> bool
return self.__nonzero__()
@@ -85,10 +104,12 @@ class MissingHashes(Hashes):
"""
def __init__(self):
+ # type: () -> None
"""Don't offer the ``hashes`` kwarg."""
# Pass our favorite hash in to generate a "gotten hash". With the
# empty list, it will never match, so an error will always raise.
super(MissingHashes, self).__init__(hashes={FAVORITE_HASH: []})
def _raise(self, gots):
+ # type: (Dict[str, _Hash]) -> NoReturn
raise HashMissing(gots[FAVORITE_HASH].hexdigest())
diff --git a/pipenv/patched/notpip/_internal/utils/logging.py b/pipenv/patched/notpip/_internal/utils/logging.py
index 576c4fa0..638c5ca9 100644
--- a/pipenv/patched/notpip/_internal/utils/logging.py
+++ b/pipenv/patched/notpip/_internal/utils/logging.py
@@ -1,9 +1,13 @@
from __future__ import absolute_import
import contextlib
+import errno
import logging
import logging.handlers
import os
+import sys
+
+from pipenv.patched.notpip._vendor.six import PY2
from pipenv.patched.notpip._internal.utils.compat import WINDOWS
from pipenv.patched.notpip._internal.utils.misc import ensure_dir
@@ -26,6 +30,48 @@ _log_state = threading.local()
_log_state.indentation = 0
+class BrokenStdoutLoggingError(Exception):
+ """
+ Raised if BrokenPipeError occurs for the stdout stream while logging.
+ """
+ pass
+
+
+# BrokenPipeError does not exist in Python 2 and, in addition, manifests
+# differently in Windows and non-Windows.
+if WINDOWS:
+ # In Windows, a broken pipe can show up as EINVAL rather than EPIPE:
+ # https://bugs.python.org/issue19612
+ # https://bugs.python.org/issue30418
+ if PY2:
+ def _is_broken_pipe_error(exc_class, exc):
+ """See the docstring for non-Windows Python 3 below."""
+ return (exc_class is IOError and
+ exc.errno in (errno.EINVAL, errno.EPIPE))
+ else:
+ # In Windows, a broken pipe IOError became OSError in Python 3.
+ def _is_broken_pipe_error(exc_class, exc):
+ """See the docstring for non-Windows Python 3 below."""
+ return ((exc_class is BrokenPipeError) or # noqa: F821
+ (exc_class is OSError and
+ exc.errno in (errno.EINVAL, errno.EPIPE)))
+elif PY2:
+ def _is_broken_pipe_error(exc_class, exc):
+ """See the docstring for non-Windows Python 3 below."""
+ return (exc_class is IOError and exc.errno == errno.EPIPE)
+else:
+ # Then we are in the non-Windows Python 3 case.
+ def _is_broken_pipe_error(exc_class, exc):
+ """
+ Return whether an exception is a broken pipe error.
+
+ Args:
+ exc_class: an exception class.
+ exc: an exception instance.
+ """
+ return (exc_class is BrokenPipeError) # noqa: F821
+
+
@contextlib.contextmanager
def indent_log(num=2):
"""
@@ -44,15 +90,28 @@ def get_indentation():
class IndentingFormatter(logging.Formatter):
+ def __init__(self, *args, **kwargs):
+ """
+ A logging.Formatter obeying containing indent_log contexts.
+
+ :param add_timestamp: A bool indicating output lines should be prefixed
+ with their record's timestamp.
+ """
+ self.add_timestamp = kwargs.pop("add_timestamp", False)
+ super(IndentingFormatter, self).__init__(*args, **kwargs)
def format(self, record):
"""
Calls the standard formatter, but will indent all of the log messages
by our current indentation level.
"""
- formatted = logging.Formatter.format(self, record)
+ formatted = super(IndentingFormatter, self).format(record)
+ prefix = ''
+ if self.add_timestamp:
+ prefix = self.formatTime(record, "%Y-%m-%dT%H:%M:%S ")
+ prefix += " " * get_indentation()
formatted = "".join([
- (" " * get_indentation()) + line
+ prefix + line
for line in formatted.splitlines(True)
])
return formatted
@@ -83,6 +142,16 @@ class ColorizedStreamHandler(logging.StreamHandler):
if WINDOWS and colorama:
self.stream = colorama.AnsiToWin32(self.stream)
+ def _using_stdout(self):
+ """
+ Return whether the handler is using sys.stdout.
+ """
+ if WINDOWS and colorama:
+ # Then self.stream is an AnsiToWin32 object.
+ return self.stream.wrapped is sys.stdout
+
+ return self.stream is sys.stdout
+
def should_color(self):
# Don't colorize things if we do not have colorama or if told not to
if not colorama or self._no_color:
@@ -115,6 +184,19 @@ class ColorizedStreamHandler(logging.StreamHandler):
return msg
+ # The logging module says handleError() can be customized.
+ def handleError(self, record):
+ exc_class, exc = sys.exc_info()[:2]
+ # If a broken pipe occurred while calling write() or flush() on the
+ # stdout stream in logging's Handler.emit(), then raise our special
+ # exception so we can handle it in main() instead of logging the
+ # broken pipe error and continuing.
+ if (exc_class and self._using_stdout() and
+ _is_broken_pipe_error(exc_class, exc)):
+ raise BrokenStdoutLoggingError()
+
+ return super(ColorizedStreamHandler, self).handleError(record)
+
class BetterRotatingFileHandler(logging.handlers.RotatingFileHandler):
@@ -134,6 +216,8 @@ class MaxLevelFilter(logging.Filter):
def setup_logging(verbosity, no_color, user_log_file):
"""Configures and sets up all of the logging
+
+ Returns the requested logging level, as its integer value.
"""
# Determine the level to be logging at.
@@ -148,6 +232,8 @@ def setup_logging(verbosity, no_color, user_log_file):
else:
level = "INFO"
+ level_number = getattr(logging, level)
+
# The "root" logger should match the "console" level *unless* we also need
# to log to a user log file.
include_user_log = user_log_file is not None
@@ -186,6 +272,11 @@ def setup_logging(verbosity, no_color, user_log_file):
"()": IndentingFormatter,
"format": "%(message)s",
},
+ "indent_with_timestamp": {
+ "()": IndentingFormatter,
+ "format": "%(message)s",
+ "add_timestamp": True,
+ },
},
"handlers": {
"console": {
@@ -208,7 +299,7 @@ def setup_logging(verbosity, no_color, user_log_file):
"class": handler_classes["file"],
"filename": additional_log_file,
"delay": True,
- "formatter": "indent",
+ "formatter": "indent_with_timestamp",
},
},
"root": {
@@ -223,3 +314,5 @@ def setup_logging(verbosity, no_color, user_log_file):
}
},
})
+
+ return level_number
diff --git a/pipenv/patched/notpip/_internal/utils/misc.py b/pipenv/patched/notpip/_internal/utils/misc.py
index 45e5204c..0a3237a2 100644
--- a/pipenv/patched/notpip/_internal/utils/misc.py
+++ b/pipenv/patched/notpip/_internal/utils/misc.py
@@ -25,6 +25,7 @@ from pipenv.patched.notpip._vendor.retrying import retry # type: ignore
from pipenv.patched.notpip._vendor.six import PY2
from pipenv.patched.notpip._vendor.six.moves import input
from pipenv.patched.notpip._vendor.six.moves.urllib import parse as urllib_parse
+from pipenv.patched.notpip._vendor.six.moves.urllib.parse import unquote as urllib_unquote
from pipenv.patched.notpip._internal.exceptions import CommandError, InstallationError
from pipenv.patched.notpip._internal.locations import (
@@ -34,12 +35,23 @@ from pipenv.patched.notpip._internal.locations import (
from pipenv.patched.notpip._internal.utils.compat import (
WINDOWS, console_to_str, expanduser, stdlib_pkgs,
)
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
if PY2:
from io import BytesIO as StringIO
else:
from io import StringIO
+if MYPY_CHECK_RUNNING:
+ from typing import ( # noqa: F401
+ Optional, Tuple, Iterable, List, Match, Union, Any, Mapping, Text,
+ AnyStr, Container
+ )
+ from pipenv.patched.notpip._vendor.pkg_resources import Distribution # noqa: F401
+ from pipenv.patched.notpip._internal.models.link import Link # noqa: F401
+ from pipenv.patched.notpip._internal.utils.ui import SpinnerInterface # noqa: F401
+
+
__all__ = ['rmtree', 'display_path', 'backup_dir',
'ask', 'splitext',
'format_size', 'is_installable_dir',
@@ -49,19 +61,21 @@ __all__ = ['rmtree', 'display_path', 'backup_dir',
'renames', 'get_prog',
'unzip_file', 'untar_file', 'unpack_file', 'call_subprocess',
'captured_stdout', 'ensure_dir',
- 'ARCHIVE_EXTENSIONS', 'SUPPORTED_EXTENSIONS',
+ 'ARCHIVE_EXTENSIONS', 'SUPPORTED_EXTENSIONS', 'WHEEL_EXTENSION',
'get_installed_version', 'remove_auth_from_url']
logger = std_logging.getLogger(__name__)
+WHEEL_EXTENSION = '.whl'
BZ2_EXTENSIONS = ('.tar.bz2', '.tbz')
XZ_EXTENSIONS = ('.tar.xz', '.txz', '.tlz', '.tar.lz', '.tar.lzma')
-ZIP_EXTENSIONS = ('.zip', '.whl')
+ZIP_EXTENSIONS = ('.zip', WHEEL_EXTENSION)
TAR_EXTENSIONS = ('.tar.gz', '.tgz', '.tar')
ARCHIVE_EXTENSIONS = (
ZIP_EXTENSIONS + BZ2_EXTENSIONS + TAR_EXTENSIONS + XZ_EXTENSIONS)
SUPPORTED_EXTENSIONS = ZIP_EXTENSIONS + TAR_EXTENSIONS
+
try:
import bz2 # noqa
SUPPORTED_EXTENSIONS += BZ2_EXTENSIONS
@@ -76,14 +90,8 @@ except ImportError:
logger.debug('lzma module is not available')
-def import_or_raise(pkg_or_module_string, ExceptionType, *args, **kwargs):
- try:
- return __import__(pkg_or_module_string)
- except ImportError:
- raise ExceptionType(*args, **kwargs)
-
-
def ensure_dir(path):
+ # type: (AnyStr) -> None
"""os.path.makedirs without EEXIST."""
try:
os.makedirs(path)
@@ -93,6 +101,7 @@ def ensure_dir(path):
def get_prog():
+ # type: () -> str
try:
prog = os.path.basename(sys.argv[0])
if prog in ('__main__.py', '-c'):
@@ -107,8 +116,9 @@ def get_prog():
# Retry every half second for up to 3 seconds
@retry(stop_max_delay=3000, wait_fixed=500)
def rmtree(dir, ignore_errors=False):
- shutil.rmtree(dir, ignore_errors=ignore_errors,
- onerror=rmtree_errorhandler)
+ # type: (str, bool) -> None
+ from pipenv.vendor.vistir.path import rmtree as vistir_rmtree, handle_remove_readonly
+ vistir_rmtree(dir, onerror=handle_remove_readonly, ignore_errors=ignore_errors)
def rmtree_errorhandler(func, path, exc_info):
@@ -127,6 +137,7 @@ def rmtree_errorhandler(func, path, exc_info):
def display_path(path):
+ # type: (Union[str, Text]) -> str
"""Gives the display value for a given path, making it relative to cwd
if possible."""
path = os.path.normcase(os.path.abspath(path))
@@ -139,6 +150,7 @@ def display_path(path):
def backup_dir(dir, ext='.bak'):
+ # type: (str, str) -> str
"""Figure out the name of a directory to back up the given dir to
(adding .bak, .bak2, etc)"""
n = 1
@@ -150,6 +162,7 @@ def backup_dir(dir, ext='.bak'):
def ask_path_exists(message, options):
+ # type: (str, Iterable[str]) -> str
for action in os.environ.get('PIP_EXISTS_ACTION', '').split():
if action in options:
return action
@@ -157,6 +170,7 @@ def ask_path_exists(message, options):
def ask(message, options):
+ # type: (str, Iterable[str]) -> str
"""Ask the message interactively, with the given possible responses"""
while 1:
if os.environ.get('PIP_NO_INPUT'):
@@ -176,6 +190,7 @@ def ask(message, options):
def format_size(bytes):
+ # type: (float) -> str
if bytes > 1000 * 1000:
return '%.1fMB' % (bytes / 1000.0 / 1000)
elif bytes > 10 * 1000:
@@ -187,6 +202,7 @@ def format_size(bytes):
def is_installable_dir(path):
+ # type: (str) -> bool
"""Is path is a directory containing setup.py or pyproject.toml?
"""
if not os.path.isdir(path):
@@ -201,6 +217,7 @@ def is_installable_dir(path):
def is_svn_page(html):
+ # type: (Union[str, Text]) -> Optional[Match[Union[str, Text]]]
"""
Returns true if the page appears to be the index page of an svn repository
"""
@@ -209,6 +226,7 @@ def is_svn_page(html):
def file_contents(filename):
+ # type: (str) -> Text
with open(filename, 'rb') as fp:
return fp.read().decode('utf-8')
@@ -223,6 +241,7 @@ def read_chunks(file, size=io.DEFAULT_BUFFER_SIZE):
def split_leading_dir(path):
+ # type: (Union[str, Text]) -> List[Union[str, Text]]
path = path.lstrip('/').lstrip('\\')
if '/' in path and (('\\' in path and path.find('/') < path.find('\\')) or
'\\' not in path):
@@ -230,10 +249,11 @@ def split_leading_dir(path):
elif '\\' in path:
return path.split('\\', 1)
else:
- return path, ''
+ return [path, '']
def has_leading_dir(paths):
+ # type: (Iterable[Union[str, Text]]) -> bool
"""Returns true if all the paths have the same leading path name
(i.e., everything is in one subdirectory in an archive)"""
common_prefix = None
@@ -249,6 +269,7 @@ def has_leading_dir(paths):
def normalize_path(path, resolve_symlinks=True):
+ # type: (str, bool) -> str
"""
Convert a path to its canonical, case-normalized, absolute version.
@@ -262,6 +283,7 @@ def normalize_path(path, resolve_symlinks=True):
def splitext(path):
+ # type: (str) -> Tuple[str, str]
"""Like os.path.splitext, but take off .tar too"""
base, ext = posixpath.splitext(path)
if base.lower().endswith('.tar'):
@@ -271,6 +293,7 @@ def splitext(path):
def renames(old, new):
+ # type: (str, str) -> None
"""Like os.renames(), but handles renaming across devices."""
# Implementation borrowed from os.renames().
head, tail = os.path.split(new)
@@ -288,6 +311,7 @@ def renames(old, new):
def is_local(path):
+ # type: (str) -> bool
"""
Return True if path is within sys.prefix, if we're running in a virtualenv.
@@ -300,6 +324,7 @@ def is_local(path):
def dist_is_local(dist):
+ # type: (Distribution) -> bool
"""
Return True if given Distribution object is installed locally
(i.e. within current virtualenv).
@@ -311,6 +336,7 @@ def dist_is_local(dist):
def dist_in_usersite(dist):
+ # type: (Distribution) -> bool
"""
Return True if given Distribution is installed in user site.
"""
@@ -319,6 +345,7 @@ def dist_in_usersite(dist):
def dist_in_site_packages(dist):
+ # type: (Distribution) -> bool
"""
Return True if given Distribution is installed in
sysconfig.get_python_lib().
@@ -329,7 +356,10 @@ def dist_in_site_packages(dist):
def dist_is_editable(dist):
- """Is distribution an editable install?"""
+ # type: (Distribution) -> bool
+ """
+ Return True if given Distribution is an editable install.
+ """
for path_item in sys.path:
egg_link = os.path.join(path_item, dist.project_name + '.egg-link')
if os.path.isfile(egg_link):
@@ -342,6 +372,7 @@ def get_installed_distributions(local_only=True,
include_editables=True,
editables_only=False,
user_only=False):
+ # type: (bool, Container[str], bool, bool, bool) -> List[Distribution]
"""
Return a list of installed Distribution objects.
@@ -385,7 +416,8 @@ def get_installed_distributions(local_only=True,
def user_test(d):
return True
- return [d for d in pkg_resources.working_set
+ # because of pkg_resources vendoring, mypy cannot find stub in typeshed
+ return [d for d in pkg_resources.working_set # type: ignore
if local_test(d) and
d.key not in skip and
editable_test(d) and
@@ -395,6 +427,7 @@ def get_installed_distributions(local_only=True,
def egg_link_path(dist):
+ # type: (Distribution) -> Optional[str]
"""
Return the path for the .egg-link file if it exists, otherwise, None.
@@ -429,9 +462,11 @@ def egg_link_path(dist):
egglink = os.path.join(site, dist.project_name) + '.egg-link'
if os.path.isfile(egglink):
return egglink
+ return None
def dist_location(dist):
+ # type: (Distribution) -> str
"""
Get the site-packages location of this distribution. Generally
this is dist.location, except in the case of develop-installed
@@ -453,6 +488,7 @@ def current_umask():
def unzip_file(filename, location, flatten=True):
+ # type: (str, str, bool) -> None
"""
Unzip the file (with path `filename`) to the destination `location`. All
files are written based on system defaults and umask (i.e. permissions are
@@ -468,7 +504,6 @@ def unzip_file(filename, location, flatten=True):
leading = has_leading_dir(zip.namelist()) and flatten
for info in zip.infolist():
name = info.filename
- data = zip.read(name)
fn = name
if leading:
fn = split_leading_dir(name)[1]
@@ -479,9 +514,12 @@ def unzip_file(filename, location, flatten=True):
ensure_dir(fn)
else:
ensure_dir(dir)
- fp = open(fn, 'wb')
+ # Don't use read() to avoid allocating an arbitrarily large
+ # chunk of memory for the file's content
+ fp = zip.open(name)
try:
- fp.write(data)
+ with open(fn, 'wb') as destfp:
+ shutil.copyfileobj(fp, destfp)
finally:
fp.close()
mode = info.external_attr >> 16
@@ -496,6 +534,7 @@ def unzip_file(filename, location, flatten=True):
def untar_file(filename, location):
+ # type: (str, str) -> None
"""
Untar the file (with path `filename`) to the destination `location`.
All files are written based on system defaults and umask (i.e. permissions
@@ -520,23 +559,21 @@ def untar_file(filename, location):
mode = 'r:*'
tar = tarfile.open(filename, mode)
try:
- # note: python<=2.5 doesn't seem to know about pax headers, filter them
leading = has_leading_dir([
member.name for member in tar.getmembers()
- if member.name != 'pax_global_header'
])
for member in tar.getmembers():
fn = member.name
- if fn == 'pax_global_header':
- continue
if leading:
- fn = split_leading_dir(fn)[1]
+ # https://github.com/python/mypy/issues/1174
+ fn = split_leading_dir(fn)[1] # type: ignore
path = os.path.join(location, fn)
if member.isdir():
ensure_dir(path)
elif member.issym():
try:
- tar._extract_member(member, path)
+ # https://github.com/python/typeshed/issues/2673
+ tar._extract_member(member, path) # type: ignore
except Exception as exc:
# Some corrupt tar files seem to produce this
# (specifically bad symlinks)
@@ -561,7 +598,8 @@ def untar_file(filename, location):
shutil.copyfileobj(fp, destfp)
fp.close()
# Update the timestamp (useful for cython compiled files)
- tar.utime(member, path)
+ # https://github.com/python/typeshed/issues/2673
+ tar.utime(member, path) # type: ignore
# member have any execute permissions for user/group/world?
if member.mode & 0o111:
# make dest file have execute for user/group/world
@@ -571,7 +609,13 @@ def untar_file(filename, location):
tar.close()
-def unpack_file(filename, location, content_type, link):
+def unpack_file(
+ filename, # type: str
+ location, # type: str
+ content_type, # type: Optional[str]
+ link # type: Optional[Link]
+):
+ # type: (...) -> None
filename = os.path.realpath(filename)
if (content_type == 'application/zip' or
filename.lower().endswith(ZIP_EXTENSIONS) or
@@ -604,15 +648,27 @@ def unpack_file(filename, location, content_type, link):
)
-def call_subprocess(cmd, show_stdout=True, cwd=None,
- on_returncode='raise',
- command_desc=None,
- extra_environ=None, unset_environ=None, spinner=None):
+def call_subprocess(
+ cmd, # type: List[str]
+ show_stdout=True, # type: bool
+ cwd=None, # type: Optional[str]
+ on_returncode='raise', # type: str
+ extra_ok_returncodes=None, # type: Optional[Iterable[int]]
+ command_desc=None, # type: Optional[str]
+ extra_environ=None, # type: Optional[Mapping[str, Any]]
+ unset_environ=None, # type: Optional[Iterable[str]]
+ spinner=None # type: Optional[SpinnerInterface]
+):
+ # type: (...) -> Optional[Text]
"""
Args:
+ extra_ok_returncodes: an iterable of integer return codes that are
+ acceptable, in addition to 0. Defaults to None, which means [].
unset_environ: an iterable of environment variable names to unset
prior to calling subprocess.Popen().
"""
+ if extra_ok_returncodes is None:
+ extra_ok_returncodes = []
if unset_environ is None:
unset_environ = []
# This function's handling of subprocess output is confusing and I
@@ -689,7 +745,7 @@ def call_subprocess(cmd, show_stdout=True, cwd=None,
spinner.finish("error")
else:
spinner.finish("done")
- if proc.returncode:
+ if proc.returncode and proc.returncode not in extra_ok_returncodes:
if on_returncode == 'raise':
if (logger.getEffectiveLevel() > std_logging.DEBUG and
not show_stdout):
@@ -715,9 +771,11 @@ def call_subprocess(cmd, show_stdout=True, cwd=None,
repr(on_returncode))
if not show_stdout:
return ''.join(all_output)
+ return None
def read_text_file(filename):
+ # type: (str) -> str
"""Return the contents of *filename*.
Try to decode the file contents with utf-8, the preferred system encoding
@@ -732,12 +790,13 @@ def read_text_file(filename):
encodings = ['utf-8', locale.getpreferredencoding(False), 'latin1']
for enc in encodings:
try:
- data = data.decode(enc)
+ # https://github.com/python/mypy/issues/1174
+ data = data.decode(enc) # type: ignore
except UnicodeDecodeError:
continue
break
- assert type(data) != bytes # Latin1 should have worked.
+ assert not isinstance(data, bytes) # Latin1 should have worked.
return data
@@ -805,6 +864,13 @@ def captured_stdout():
return captured_output('stdout')
+def captured_stderr():
+ """
+ See captured_stdout().
+ """
+ return captured_output('stderr')
+
+
class cached_property(object):
"""A property that is only computed once per instance and then replaces
itself with an ordinary attribute. Deleting the attribute resets the
@@ -856,13 +922,15 @@ def enum(*sequential, **named):
return type('Enum', (), enums)
-def make_vcs_requirement_url(repo_url, rev, egg_project_name, subdir=None):
+def make_vcs_requirement_url(repo_url, rev, project_name, subdir=None):
"""
Return the URL for a VCS requirement.
Args:
repo_url: the remote VCS url, with any needed VCS prefix (e.g. "git+").
+ project_name: the (unescaped) project name.
"""
+ egg_project_name = pkg_resources.to_filename(project_name)
req = '{}@{}#egg={}'.format(repo_url, rev, egg_project_name)
if subdir:
req += '&subdirectory={}'.format(subdir)
@@ -887,22 +955,36 @@ def split_auth_from_netloc(netloc):
# Split from the left because that's how urllib.parse.urlsplit()
# behaves if more than one : is present (which again can be checked
# using the password attribute of the return value)
- user_pass = tuple(auth.split(':', 1))
+ user_pass = auth.split(':', 1)
else:
user_pass = auth, None
+ user_pass = tuple(
+ None if x is None else urllib_unquote(x) for x in user_pass
+ )
+
return netloc, user_pass
-def remove_auth_from_url(url):
- # Return a copy of url with 'username:password@' removed.
- # username/pass params are passed to subversion through flags
- # and are not recognized in the url.
+def redact_netloc(netloc):
+ # type: (str) -> str
+ """
+ Replace the password in a netloc with "****", if it exists.
- # parsed url
+ For example, "user:pass@example.com" returns "user:****@example.com".
+ """
+ netloc, (user, password) = split_auth_from_netloc(netloc)
+ if user is None:
+ return netloc
+ password = '' if password is None else ':****'
+ return '{user}{password}@{netloc}'.format(user=urllib_parse.quote(user),
+ password=password,
+ netloc=netloc)
+
+
+def _transform_url(url, transform_netloc):
purl = urllib_parse.urlsplit(url)
- netloc, user_pass = split_auth_from_netloc(purl.netloc)
-
+ netloc = transform_netloc(purl.netloc)
# stripped url
url_pieces = (
purl.scheme, netloc, purl.path, purl.query, purl.fragment
@@ -911,6 +993,24 @@ def remove_auth_from_url(url):
return surl
+def _get_netloc(netloc):
+ return split_auth_from_netloc(netloc)[0]
+
+
+def remove_auth_from_url(url):
+ # type: (str) -> str
+ # Return a copy of url with 'username:password@' removed.
+ # username/pass params are passed to subversion through flags
+ # and are not recognized in the url.
+ return _transform_url(url, _get_netloc)
+
+
+def redact_password_from_url(url):
+ # type: (str) -> str
+ """Replace the password in a given url with ****."""
+ return _transform_url(url, redact_netloc)
+
+
def protect_pip_from_modification_on_windows(modifying_pip):
"""Protection of pip.exe from modification on Windows
diff --git a/pipenv/patched/notpip/_internal/utils/outdated.py b/pipenv/patched/notpip/_internal/utils/outdated.py
index f8b1fe04..83dc58cc 100644
--- a/pipenv/patched/notpip/_internal/utils/outdated.py
+++ b/pipenv/patched/notpip/_internal/utils/outdated.py
@@ -13,6 +13,13 @@ from pipenv.patched.notpip._internal.index import PackageFinder
from pipenv.patched.notpip._internal.utils.compat import WINDOWS
from pipenv.patched.notpip._internal.utils.filesystem import check_path_owner
from pipenv.patched.notpip._internal.utils.misc import ensure_dir, get_installed_version
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ import optparse # noqa: F401
+ from typing import Any, Dict # noqa: F401
+ from pipenv.patched.notpip._internal.download import PipSession # noqa: F401
+
SELFCHECK_DATE_FMT = "%Y-%m-%dT%H:%M:%SZ"
@@ -22,7 +29,8 @@ logger = logging.getLogger(__name__)
class SelfCheckState(object):
def __init__(self, cache_dir):
- self.state = {}
+ # type: (str) -> None
+ self.state = {} # type: Dict[str, Any]
self.statefile_path = None
# Try to load the existing state
@@ -37,6 +45,7 @@ class SelfCheckState(object):
pass
def save(self, pypi_version, current_time):
+ # type: (str, datetime.datetime) -> None
# If we do not have a path to cache in, don't bother saving.
if not self.statefile_path:
return
@@ -68,6 +77,7 @@ class SelfCheckState(object):
def was_installed_by_pip(pkg):
+ # type: (str) -> bool
"""Checks whether pkg was installed by pip
This is used not to display the upgrade message when pip is in fact
@@ -82,6 +92,7 @@ def was_installed_by_pip(pkg):
def pip_version_check(session, options):
+ # type: (PipSession, optparse.Values) -> None
"""Check for an update for pip.
Limit the frequency of checks to once per week. State is stored either in
@@ -116,7 +127,6 @@ def pip_version_check(session, options):
index_urls=[options.index_url] + options.extra_index_urls,
allow_all_prereleases=False, # Explicitly set to False
trusted_hosts=options.trusted_hosts,
- process_dependency_links=options.process_dependency_links,
session=session,
)
all_candidates = finder.find_all_candidates("pip")
diff --git a/pipenv/patched/notpip/_internal/utils/packaging.py b/pipenv/patched/notpip/_internal/utils/packaging.py
index d1e8ecaa..dc944529 100644
--- a/pipenv/patched/notpip/_internal/utils/packaging.py
+++ b/pipenv/patched/notpip/_internal/utils/packaging.py
@@ -2,18 +2,26 @@ from __future__ import absolute_import
import logging
import sys
-from email.parser import FeedParser # type: ignore
+from email.parser import FeedParser
from pipenv.patched.notpip._vendor import pkg_resources
from pipenv.patched.notpip._vendor.packaging import specifiers, version
from pipenv.patched.notpip._internal import exceptions
from pipenv.patched.notpip._internal.utils.misc import display_path
+from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+ from typing import Optional # noqa: F401
+ from email.message import Message # noqa: F401
+ from pipenv.patched.notpip._vendor.pkg_resources import Distribution # noqa: F401
+
logger = logging.getLogger(__name__)
def check_requires_python(requires_python):
+ # type: (Optional[str]) -> bool
"""
Check if the python version in use match the `requires_python` specifier.
@@ -34,6 +42,7 @@ def check_requires_python(requires_python):
def get_metadata(dist):
+ # type: (Distribution) -> Message
if (isinstance(dist, pkg_resources.DistInfoDistribution) and
dist.has_metadata('METADATA')):
metadata = dist.get_metadata('METADATA')
@@ -48,7 +57,7 @@ def get_metadata(dist):
return feed_parser.close()
-def check_dist_requires_python(dist, absorb=True):
+def check_dist_requires_python(dist, absorb=False):
pkg_info_dict = get_metadata(dist)
requires_python = pkg_info_dict.get('Requires-Python')
if absorb:
@@ -70,6 +79,7 @@ def check_dist_requires_python(dist, absorb=True):
def get_installer(dist):
+ # type: (Distribution) -> str
if dist.has_metadata('INSTALLER'):
for line in dist.get_metadata_lines('INSTALLER'):
if line.strip():
diff --git a/pipenv/patched/notpip/_internal/utils/temp_dir.py b/pipenv/patched/notpip/_internal/utils/temp_dir.py
index 893dc975..d8121a59 100644
--- a/pipenv/patched/notpip/_internal/utils/temp_dir.py
+++ b/pipenv/patched/notpip/_internal/utils/temp_dir.py
@@ -1,5 +1,7 @@
from __future__ import absolute_import
+import errno
+import itertools
import logging
import os.path
import tempfile
@@ -57,7 +59,7 @@ class TempDirectory(object):
self,
self._cleanup,
self.path,
- warn_message=None
+ warn_message = None
)
else:
self._finalizer = None
@@ -74,7 +76,7 @@ class TempDirectory(object):
self.cleanup()
def create(self):
- """Create a temporary directory and store it's path in self.path
+ """Create a temporary directory and store its path in self.path
"""
if self.path is not None:
logger.debug(
@@ -112,3 +114,75 @@ class TempDirectory(object):
pass
else:
self.path = None
+
+
+class AdjacentTempDirectory(TempDirectory):
+ """Helper class that creates a temporary directory adjacent to a real one.
+
+ Attributes:
+ original
+ The original directory to create a temp directory for.
+ path
+ After calling create() or entering, contains the full
+ path to the temporary directory.
+ delete
+ Whether the directory should be deleted when exiting
+ (when used as a contextmanager)
+
+ """
+ # The characters that may be used to name the temp directory
+ # We always prepend a ~ and then rotate through these until
+ # a usable name is found.
+ # pkg_resources raises a different error for .dist-info folder
+ # with leading '-' and invalid metadata
+ LEADING_CHARS = "-~.=%0123456789"
+
+ def __init__(self, original, delete=None):
+ super(AdjacentTempDirectory, self).__init__(delete=delete)
+ self.original = original.rstrip('/\\')
+
+ @classmethod
+ def _generate_names(cls, name):
+ """Generates a series of temporary names.
+
+ The algorithm replaces the leading characters in the name
+ with ones that are valid filesystem characters, but are not
+ valid package names (for both Python and pip definitions of
+ package).
+ """
+ for i in range(1, len(name)):
+ for candidate in itertools.combinations_with_replacement(
+ cls.LEADING_CHARS, i - 1):
+ new_name = '~' + ''.join(candidate) + name[i:]
+ if new_name != name:
+ yield new_name
+
+ # If we make it this far, we will have to make a longer name
+ for i in range(len(cls.LEADING_CHARS)):
+ for candidate in itertools.combinations_with_replacement(
+ cls.LEADING_CHARS, i):
+ new_name = '~' + ''.join(candidate) + name
+ if new_name != name:
+ yield new_name
+
+ def create(self):
+ root, name = os.path.split(self.original)
+ for candidate in self._generate_names(name):
+ path = os.path.join(root, candidate)
+ try:
+ os.mkdir(path)
+ except OSError as ex:
+ # Continue if the name exists already
+ if ex.errno != errno.EEXIST:
+ raise
+ else:
+ self.path = os.path.realpath(path)
+ break
+
+ if not self.path:
+ # Final fallback on the default behavior.
+ self.path = os.path.realpath(
+ tempfile.mkdtemp(prefix="pip-{}-".format(self.kind))
+ )
+ self._register_finalizer()
+ logger.debug("Created temporary directory: {}".format(self.path))
diff --git a/pipenv/patched/notpip/_internal/utils/ui.py b/pipenv/patched/notpip/_internal/utils/ui.py
index 6eebd17d..18119e0e 100644
--- a/pipenv/patched/notpip/_internal/utils/ui.py
+++ b/pipenv/patched/notpip/_internal/utils/ui.py
@@ -21,7 +21,7 @@ from pipenv.patched.notpip._internal.utils.misc import format_size
from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
if MYPY_CHECK_RUNNING:
- from typing import Any # noqa: F401
+ from typing import Any, Iterator, IO # noqa: F401
try:
from pipenv.patched.notpip._vendor import colorama
@@ -203,7 +203,7 @@ class BaseDownloadProgressBar(WindowsMixin, InterruptibleMixin,
class DefaultDownloadProgressBar(BaseDownloadProgressBar,
- _BaseBar): # type: ignore
+ _BaseBar):
pass
@@ -292,6 +292,7 @@ def DownloadProgressProvider(progress_bar, max=None):
@contextlib.contextmanager
def hidden_cursor(file):
+ # type: (IO) -> Iterator[None]
# The Windows terminal does not support the hide/show cursor ANSI codes,
# even via colorama. So don't even try.
if WINDOWS:
@@ -311,19 +312,32 @@ def hidden_cursor(file):
class RateLimiter(object):
def __init__(self, min_update_interval_seconds):
+ # type: (float) -> None
self._min_update_interval_seconds = min_update_interval_seconds
- self._last_update = 0
+ self._last_update = 0 # type: float
def ready(self):
+ # type: () -> bool
now = time.time()
delta = now - self._last_update
return delta >= self._min_update_interval_seconds
def reset(self):
+ # type: () -> None
self._last_update = time.time()
-class InteractiveSpinner(object):
+class SpinnerInterface(object):
+ def spin(self):
+ # type: () -> None
+ raise NotImplementedError()
+
+ def finish(self, final_status):
+ # type: (str) -> None
+ raise NotImplementedError()
+
+
+class InteractiveSpinner(SpinnerInterface):
def __init__(self, message, file=None, spin_chars="-\\|/",
# Empirically, 8 updates/second looks nice
min_update_interval_seconds=0.125):
@@ -352,6 +366,7 @@ class InteractiveSpinner(object):
self._rate_limiter.reset()
def spin(self):
+ # type: () -> None
if self._finished:
return
if not self._rate_limiter.ready():
@@ -359,6 +374,7 @@ class InteractiveSpinner(object):
self._write(next(self._spin_cycle))
def finish(self, final_status):
+ # type: (str) -> None
if self._finished:
return
self._write(final_status)
@@ -371,8 +387,9 @@ class InteractiveSpinner(object):
# We still print updates occasionally (once every 60 seconds by default) to
# act as a keep-alive for systems like Travis-CI that take lack-of-output as
# an indication that a task has frozen.
-class NonInteractiveSpinner(object):
+class NonInteractiveSpinner(SpinnerInterface):
def __init__(self, message, min_update_interval_seconds=60):
+ # type: (str, float) -> None
self._message = message
self._finished = False
self._rate_limiter = RateLimiter(min_update_interval_seconds)
@@ -384,6 +401,7 @@ class NonInteractiveSpinner(object):
logger.info("%s: %s", self._message, status)
def spin(self):
+ # type: () -> None
if self._finished:
return
if not self._rate_limiter.ready():
@@ -391,6 +409,7 @@ class NonInteractiveSpinner(object):
self._update("still running...")
def finish(self, final_status):
+ # type: (str) -> None
if self._finished:
return
self._update("finished with status '%s'" % (final_status,))
@@ -399,13 +418,14 @@ class NonInteractiveSpinner(object):
@contextlib.contextmanager
def open_spinner(message):
+ # type: (str) -> Iterator[SpinnerInterface]
# Interactive spinner goes directly to sys.stdout rather than being routed
# through the logging system, but it acts like it has level INFO,
# i.e. it's only displayed if we're at level INFO or better.
# Non-interactive spinner goes through the logging system, so it is always
# in sync with logging configuration.
if sys.stdout.isatty() and logger.getEffectiveLevel() <= logging.INFO:
- spinner = InteractiveSpinner(message)
+ spinner = InteractiveSpinner(message) # type: SpinnerInterface
else:
spinner = NonInteractiveSpinner(message)
try:
diff --git a/pipenv/patched/notpip/_internal/vcs/__init__.py b/pipenv/patched/notpip/_internal/vcs/__init__.py
index 5aeac633..06d36148 100644
--- a/pipenv/patched/notpip/_internal/vcs/__init__.py
+++ b/pipenv/patched/notpip/_internal/vcs/__init__.py
@@ -16,15 +16,23 @@ from pipenv.patched.notpip._internal.utils.misc import (
from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
if MYPY_CHECK_RUNNING:
- from typing import Dict, Optional, Tuple # noqa: F401
- from pipenv.patched.notpip._internal.cli.base_command import Command # noqa: F401
+ from typing import ( # noqa: F401
+ Any, Dict, Iterable, List, Mapping, Optional, Text, Tuple, Type
+ )
+ from pipenv.patched.notpip._internal.utils.ui import SpinnerInterface # noqa: F401
-__all__ = ['vcs', 'get_src_requirement']
+ AuthInfo = Tuple[Optional[str], Optional[str]]
+
+__all__ = ['vcs']
logger = logging.getLogger(__name__)
+class RemoteNotFoundError(Exception):
+ pass
+
+
class RevOptions(object):
"""
@@ -35,6 +43,7 @@ class RevOptions(object):
"""
def __init__(self, vcs, rev=None, extra_args=None):
+ # type: (VersionControl, Optional[str], Optional[List[str]]) -> None
"""
Args:
vcs: a VersionControl object.
@@ -53,16 +62,18 @@ class RevOptions(object):
@property
def arg_rev(self):
+ # type: () -> Optional[str]
if self.rev is None:
return self.vcs.default_arg_rev
return self.rev
def to_args(self):
+ # type: () -> List[str]
"""
Return the VCS-specific command arguments.
"""
- args = []
+ args = [] # type: List[str]
rev = self.arg_rev
if rev is not None:
args += self.vcs.get_base_rev_args(rev)
@@ -71,12 +82,14 @@ class RevOptions(object):
return args
def to_display(self):
+ # type: () -> str
if not self.rev:
return ''
return ' (to revision {})'.format(self.rev)
def make_new(self, rev):
+ # type: (str) -> RevOptions
"""
Make a copy of the current instance, but with a new rev.
@@ -87,10 +100,11 @@ class RevOptions(object):
class VcsSupport(object):
- _registry = {} # type: Dict[str, Command]
+ _registry = {} # type: Dict[str, Type[VersionControl]]
schemes = ['ssh', 'git', 'hg', 'bzr', 'sftp', 'svn']
def __init__(self):
+ # type: () -> None
# Register more schemes with urlparse for various version control
# systems
urllib_parse.uses_netloc.extend(self.schemes)
@@ -104,20 +118,24 @@ class VcsSupport(object):
@property
def backends(self):
+ # type: () -> List[Type[VersionControl]]
return list(self._registry.values())
@property
def dirnames(self):
+ # type: () -> List[str]
return [backend.dirname for backend in self.backends]
@property
def all_schemes(self):
- schemes = []
+ # type: () -> List[str]
+ schemes = [] # type: List[str]
for backend in self.backends:
schemes.extend(backend.schemes)
return schemes
def register(self, cls):
+ # type: (Type[VersionControl]) -> None
if not hasattr(cls, 'name'):
logger.warning('Cannot register VCS %s', cls.__name__)
return
@@ -126,6 +144,7 @@ class VcsSupport(object):
logger.debug('Registered VCS backend: %s', cls.name)
def unregister(self, cls=None, name=None):
+ # type: (Optional[Type[VersionControl]], Optional[str]) -> None
if name in self._registry:
del self._registry[name]
elif cls in self._registry.values():
@@ -133,27 +152,24 @@ class VcsSupport(object):
else:
logger.warning('Cannot unregister because no class or name given')
- def get_backend_name(self, location):
+ def get_backend_type(self, location):
+ # type: (str) -> Optional[Type[VersionControl]]
"""
- Return the name of the version control backend if found at given
- location, e.g. vcs.get_backend_name('/path/to/vcs/checkout')
+ Return the type of the version control backend if found at given
+ location, e.g. vcs.get_backend_type('/path/to/vcs/checkout')
"""
for vc_type in self._registry.values():
if vc_type.controls_location(location):
logger.debug('Determine that %s uses VCS: %s',
location, vc_type.name)
- return vc_type.name
+ return vc_type
return None
def get_backend(self, name):
+ # type: (str) -> Optional[Type[VersionControl]]
name = name.lower()
if name in self._registry:
return self._registry[name]
-
- def get_backend_from_location(self, location):
- vc_type = self.get_backend_name(location)
- if vc_type:
- return self.get_backend(vc_type)
return None
@@ -163,6 +179,7 @@ vcs = VcsSupport()
class VersionControl(object):
name = ''
dirname = ''
+ repo_name = ''
# List of supported schemes for this Version Control
schemes = () # type: Tuple[str, ...]
# Iterable of environment variable names to pass to call_subprocess().
@@ -183,6 +200,7 @@ class VersionControl(object):
raise NotImplementedError
def make_rev_options(self, rev=None, extra_args=None):
+ # type: (Optional[str], Optional[List[str]]) -> RevOptions
"""
Return a RevOptions object.
@@ -192,13 +210,15 @@ class VersionControl(object):
"""
return RevOptions(self, rev, extra_args=extra_args)
- def _is_local_repository(self, repo):
+ @classmethod
+ def _is_local_repository(cls, repo):
+ # type: (str) -> bool
"""
posix absolute paths start with os.path.sep,
win32 ones start with drive (like c:\\folder)
"""
drive, tail = os.path.splitdrive(repo)
- return repo.startswith(os.path.sep) or drive
+ return repo.startswith(os.path.sep) or bool(drive)
def export(self, location):
"""
@@ -226,6 +246,7 @@ class VersionControl(object):
return netloc, (None, None)
def get_url_rev_and_auth(self, url):
+ # type: (str) -> Tuple[str, Optional[str], AuthInfo]
"""
Parse the repository URL to use, and return the URL, revision,
and auth info to use.
@@ -255,6 +276,7 @@ class VersionControl(object):
return []
def get_url_rev_options(self, url):
+ # type: (str) -> Tuple[str, RevOptions]
"""
Return the URL and RevOptions object to use in obtain() and in
some cases export(), as a tuple (url, rev_options).
@@ -267,6 +289,7 @@ class VersionControl(object):
return url, rev_options
def normalize_url(self, url):
+ # type: (str) -> str
"""
Normalize a URL for comparison by unquoting it and removing any
trailing slash.
@@ -274,6 +297,7 @@ class VersionControl(object):
return urllib_parse.unquote(url).rstrip('/')
def compare_urls(self, url1, url2):
+ # type: (str, str) -> bool
"""
Compare two repo URLs for identity, ignoring incidental differences.
"""
@@ -319,6 +343,7 @@ class VersionControl(object):
raise NotImplementedError
def obtain(self, dest):
+ # type: (str) -> None
"""
Install or update in editable mode the package represented by this
VersionControl object.
@@ -334,7 +359,7 @@ class VersionControl(object):
rev_display = rev_options.to_display()
if self.is_repository_directory(dest):
- existing_url = self.get_url(dest)
+ existing_url = self.get_remote_url(dest)
if self.compare_urls(existing_url, url):
logger.debug(
'%s in %s exists, and has correct URL (%s)',
@@ -370,7 +395,9 @@ class VersionControl(object):
self.name,
self.repo_name,
)
- prompt = ('(i)gnore, (w)ipe, (b)ackup ', ('i', 'w', 'b'))
+ # https://github.com/python/mypy/issues/1174
+ prompt = ('(i)gnore, (w)ipe, (b)ackup ', # type: ignore
+ ('i', 'w', 'b'))
logger.warning(
'The plan is to install the %s repository %s',
@@ -409,6 +436,7 @@ class VersionControl(object):
self.switch(dest, url, rev_options)
def unpack(self, location):
+ # type: (str) -> None
"""
Clean up current location and download the url repository
(and vcs infos) into location
@@ -417,7 +445,8 @@ class VersionControl(object):
rmtree(location)
self.obtain(location)
- def get_src_requirement(self, dist, location):
+ @classmethod
+ def get_src_requirement(cls, location, project_name):
"""
Return a string representing the requirement needed to
redownload the files currently present in location, something
@@ -426,33 +455,49 @@ class VersionControl(object):
"""
raise NotImplementedError
- def get_url(self, location):
+ @classmethod
+ def get_remote_url(cls, location):
"""
Return the url used at location
+
+ Raises RemoteNotFoundError if the repository does not have a remote
+ url configured.
"""
raise NotImplementedError
- def get_revision(self, location):
+ @classmethod
+ def get_revision(cls, location):
"""
Return the current commit id of the files at the given location.
"""
raise NotImplementedError
- def run_command(self, cmd, show_stdout=True, cwd=None,
- on_returncode='raise',
- command_desc=None,
- extra_environ=None, spinner=None):
+ @classmethod
+ def run_command(
+ cls,
+ cmd, # type: List[str]
+ show_stdout=True, # type: bool
+ cwd=None, # type: Optional[str]
+ on_returncode='raise', # type: str
+ extra_ok_returncodes=None, # type: Optional[Iterable[int]]
+ command_desc=None, # type: Optional[str]
+ extra_environ=None, # type: Optional[Mapping[str, Any]]
+ spinner=None # type: Optional[SpinnerInterface]
+ ):
+ # type: (...) -> Optional[Text]
"""
Run a VCS subcommand
This is simply a wrapper around call_subprocess that adds the VCS
command name, and checks that the VCS is available
"""
- cmd = [self.name] + cmd
+ cmd = [cls.name] + cmd
try:
return call_subprocess(cmd, show_stdout, cwd,
- on_returncode,
- command_desc, extra_environ,
- unset_environ=self.unset_environ,
+ on_returncode=on_returncode,
+ extra_ok_returncodes=extra_ok_returncodes,
+ command_desc=command_desc,
+ extra_environ=extra_environ,
+ unset_environ=cls.unset_environ,
spinner=spinner)
except OSError as e:
# errno.ENOENT = no such file or directory
@@ -461,12 +506,13 @@ class VersionControl(object):
raise BadCommand(
'Cannot find command %r - do you have '
'%r installed and in your '
- 'PATH?' % (self.name, self.name))
+ 'PATH?' % (cls.name, cls.name))
else:
raise # re-raise exception if a different error occurred
@classmethod
def is_repository_directory(cls, path):
+ # type: (str) -> bool
"""
Return whether a directory path is a repository directory.
"""
@@ -476,6 +522,7 @@ class VersionControl(object):
@classmethod
def controls_location(cls, location):
+ # type: (str) -> bool
"""
Check if a location is controlled by the vcs.
It is meant to be overridden to implement smarter detection
@@ -485,25 +532,3 @@ class VersionControl(object):
the Git override checks that Git is actually available.
"""
return cls.is_repository_directory(location)
-
-
-def get_src_requirement(dist, location):
- version_control = vcs.get_backend_from_location(location)
- if version_control:
- try:
- return version_control().get_src_requirement(dist,
- location)
- except BadCommand:
- logger.warning(
- 'cannot determine version of editable source in %s '
- '(%s command not found in path)',
- location,
- version_control.name,
- )
- return dist.as_requirement()
- logger.warning(
- 'cannot determine version of editable source in %s (is not SVN '
- 'checkout, Git clone, Mercurial clone or Bazaar branch)',
- location,
- )
- return dist.as_requirement()
diff --git a/pipenv/patched/notpip/_internal/vcs/bazaar.py b/pipenv/patched/notpip/_internal/vcs/bazaar.py
index 890448ed..256eb9e2 100644
--- a/pipenv/patched/notpip/_internal/vcs/bazaar.py
+++ b/pipenv/patched/notpip/_internal/vcs/bazaar.py
@@ -75,34 +75,36 @@ class Bazaar(VersionControl):
url = 'bzr+' + url
return url, rev, user_pass
- def get_url(self, location):
- urls = self.run_command(['info'], show_stdout=False, cwd=location)
+ @classmethod
+ def get_remote_url(cls, location):
+ urls = cls.run_command(['info'], show_stdout=False, cwd=location)
for line in urls.splitlines():
line = line.strip()
for x in ('checkout of branch: ',
'parent branch: '):
if line.startswith(x):
repo = line.split(x)[1]
- if self._is_local_repository(repo):
+ if cls._is_local_repository(repo):
return path_to_url(repo)
return repo
return None
- def get_revision(self, location):
- revision = self.run_command(
+ @classmethod
+ def get_revision(cls, location):
+ revision = cls.run_command(
['revno'], show_stdout=False, cwd=location,
)
return revision.splitlines()[-1]
- def get_src_requirement(self, dist, location):
- repo = self.get_url(location)
+ @classmethod
+ def get_src_requirement(cls, location, project_name):
+ repo = cls.get_remote_url(location)
if not repo:
return None
if not repo.lower().startswith('bzr:'):
repo = 'bzr+' + repo
- current_rev = self.get_revision(location)
- egg_project_name = dist.egg_name().split('-', 1)[0]
- return make_vcs_requirement_url(repo, current_rev, egg_project_name)
+ current_rev = cls.get_revision(location)
+ return make_vcs_requirement_url(repo, current_rev, project_name)
def is_commit_id_equal(self, dest, name):
"""Always assume the versions don't match"""
diff --git a/pipenv/patched/notpip/_internal/vcs/git.py b/pipenv/patched/notpip/_internal/vcs/git.py
index 3db56144..310eb9fb 100644
--- a/pipenv/patched/notpip/_internal/vcs/git.py
+++ b/pipenv/patched/notpip/_internal/vcs/git.py
@@ -10,9 +10,11 @@ from pipenv.patched.notpip._vendor.six.moves.urllib import request as urllib_req
from pipenv.patched.notpip._internal.exceptions import BadCommand
from pipenv.patched.notpip._internal.utils.compat import samefile
-from pipenv.patched.notpip._internal.utils.misc import display_path, make_vcs_requirement_url
+from pipenv.patched.notpip._internal.utils.misc import (
+ display_path, make_vcs_requirement_url, redact_password_from_url,
+)
from pipenv.patched.notpip._internal.utils.temp_dir import TempDirectory
-from pipenv.patched.notpip._internal.vcs import VersionControl, vcs
+from pipenv.patched.notpip._internal.vcs import RemoteNotFoundError, VersionControl, vcs
urlsplit = urllib_parse.urlsplit
urlunsplit = urllib_parse.urlunsplit
@@ -77,19 +79,25 @@ class Git(VersionControl):
version = '.'.join(version.split('.')[:3])
return parse_version(version)
- def get_branch(self, location):
+ def get_current_branch(self, location):
"""
Return the current branch, or None if HEAD isn't at a branch
(e.g. detached HEAD).
"""
- args = ['rev-parse', '--abbrev-ref', 'HEAD']
- output = self.run_command(args, show_stdout=False, cwd=location)
- branch = output.strip()
+ # git-symbolic-ref exits with empty stdout if "HEAD" is a detached
+ # HEAD rather than a symbolic ref. In addition, the -q causes the
+ # command to exit with status code 1 instead of 128 in this case
+ # and to suppress the message to stderr.
+ args = ['symbolic-ref', '-q', 'HEAD']
+ output = self.run_command(
+ args, extra_ok_returncodes=(1, ), show_stdout=False, cwd=location,
+ )
+ ref = output.strip()
- if branch == 'HEAD':
- return None
+ if ref.startswith('refs/heads/'):
+ return ref[len('refs/heads/'):]
- return branch
+ return None
def export(self, location):
"""Export the Git repository at the url to the destination location"""
@@ -193,7 +201,8 @@ class Git(VersionControl):
def fetch_new(self, dest, url, rev_options):
rev_display = rev_options.to_display()
logger.info(
- 'Cloning %s%s to %s', url, rev_display, display_path(dest),
+ 'Cloning %s%s to %s', redact_password_from_url(url),
+ rev_display, display_path(dest),
)
self.run_command(['clone', '-q', url, dest])
@@ -207,7 +216,7 @@ class Git(VersionControl):
if not self.is_commit_id_equal(dest, rev_options.rev):
cmd_args = ['checkout', '-q'] + rev_options.to_args()
self.run_command(cmd_args, cwd=dest)
- elif self.get_branch(dest) != branch_name:
+ elif self.get_current_branch(dest) != branch_name:
# Then a specific branch was requested, and that branch
# is not yet checked out.
track_branch = 'origin/{}'.format(branch_name)
@@ -240,14 +249,26 @@ class Git(VersionControl):
#: update submodules
self.update_submodules(dest)
- def get_url(self, location):
- """Return URL of the first remote encountered."""
- remotes = self.run_command(
+ @classmethod
+ def get_remote_url(cls, location):
+ """
+ Return URL of the first remote encountered.
+
+ Raises RemoteNotFoundError if the repository does not have a remote
+ url configured.
+ """
+ # We need to pass 1 for extra_ok_returncodes since the command
+ # exits with return code 1 if there are no matching lines.
+ stdout = cls.run_command(
['config', '--get-regexp', r'remote\..*\.url'],
- show_stdout=False, cwd=location,
+ extra_ok_returncodes=(1, ), show_stdout=False, cwd=location,
)
- remotes = remotes.splitlines()
- found_remote = remotes[0]
+ remotes = stdout.splitlines()
+ try:
+ found_remote = remotes[0]
+ except IndexError:
+ raise RemoteNotFoundError
+
for remote in remotes:
if remote.startswith('remote.origin.url '):
found_remote = remote
@@ -255,19 +276,21 @@ class Git(VersionControl):
url = found_remote.split(' ')[1]
return url.strip()
- def get_revision(self, location, rev=None):
+ @classmethod
+ def get_revision(cls, location, rev=None):
if rev is None:
rev = 'HEAD'
- current_rev = self.run_command(
+ current_rev = cls.run_command(
['rev-parse', rev], show_stdout=False, cwd=location,
)
return current_rev.strip()
- def _get_subdirectory(self, location):
+ @classmethod
+ def _get_subdirectory(cls, location):
"""Return the relative path of setup.py to the git repo root."""
# find the repo root
- git_dir = self.run_command(['rev-parse', '--git-dir'],
- show_stdout=False, cwd=location).strip()
+ git_dir = cls.run_command(['rev-parse', '--git-dir'],
+ show_stdout=False, cwd=location).strip()
if not os.path.isabs(git_dir):
git_dir = os.path.join(location, git_dir)
root_dir = os.path.join(git_dir, '..')
@@ -290,14 +313,14 @@ class Git(VersionControl):
return None
return os.path.relpath(location, root_dir)
- def get_src_requirement(self, dist, location):
- repo = self.get_url(location)
+ @classmethod
+ def get_src_requirement(cls, location, project_name):
+ repo = cls.get_remote_url(location)
if not repo.lower().startswith('git:'):
repo = 'git+' + repo
- current_rev = self.get_revision(location)
- egg_project_name = dist.egg_name().split('-', 1)[0]
- subdir = self._get_subdirectory(location)
- req = make_vcs_requirement_url(repo, current_rev, egg_project_name,
+ current_rev = cls.get_revision(location)
+ subdir = cls._get_subdirectory(location)
+ req = make_vcs_requirement_url(repo, current_rev, project_name,
subdir=subdir)
return req
@@ -332,10 +355,10 @@ class Git(VersionControl):
if super(Git, cls).controls_location(location):
return True
try:
- r = cls().run_command(['rev-parse'],
- cwd=location,
- show_stdout=False,
- on_returncode='ignore')
+ r = cls.run_command(['rev-parse'],
+ cwd=location,
+ show_stdout=False,
+ on_returncode='ignore')
return not r
except BadCommand:
logger.debug("could not determine if %s is under git control "
diff --git a/pipenv/patched/notpip/_internal/vcs/mercurial.py b/pipenv/patched/notpip/_internal/vcs/mercurial.py
index d76d47f3..37a3f7c2 100644
--- a/pipenv/patched/notpip/_internal/vcs/mercurial.py
+++ b/pipenv/patched/notpip/_internal/vcs/mercurial.py
@@ -64,34 +64,36 @@ class Mercurial(VersionControl):
cmd_args = ['update', '-q'] + rev_options.to_args()
self.run_command(cmd_args, cwd=dest)
- def get_url(self, location):
- url = self.run_command(
+ @classmethod
+ def get_remote_url(cls, location):
+ url = cls.run_command(
['showconfig', 'paths.default'],
show_stdout=False, cwd=location).strip()
- if self._is_local_repository(url):
+ if cls._is_local_repository(url):
url = path_to_url(url)
return url.strip()
- def get_revision(self, location):
- current_revision = self.run_command(
+ @classmethod
+ def get_revision(cls, location):
+ current_revision = cls.run_command(
['parents', '--template={rev}'],
show_stdout=False, cwd=location).strip()
return current_revision
- def get_revision_hash(self, location):
- current_rev_hash = self.run_command(
+ @classmethod
+ def get_revision_hash(cls, location):
+ current_rev_hash = cls.run_command(
['parents', '--template={node}'],
show_stdout=False, cwd=location).strip()
return current_rev_hash
- def get_src_requirement(self, dist, location):
- repo = self.get_url(location)
+ @classmethod
+ def get_src_requirement(cls, location, project_name):
+ repo = cls.get_remote_url(location)
if not repo.lower().startswith('hg:'):
repo = 'hg+' + repo
- current_rev_hash = self.get_revision_hash(location)
- egg_project_name = dist.egg_name().split('-', 1)[0]
- return make_vcs_requirement_url(repo, current_rev_hash,
- egg_project_name)
+ current_rev_hash = cls.get_revision_hash(location)
+ return make_vcs_requirement_url(repo, current_rev_hash, project_name)
def is_commit_id_equal(self, dest, name):
"""Always assume the versions don't match"""
diff --git a/pipenv/patched/notpip/_internal/vcs/subversion.py b/pipenv/patched/notpip/_internal/vcs/subversion.py
index f3c3db4d..e62d3def 100644
--- a/pipenv/patched/notpip/_internal/vcs/subversion.py
+++ b/pipenv/patched/notpip/_internal/vcs/subversion.py
@@ -4,7 +4,6 @@ import logging
import os
import re
-from pipenv.patched.notpip._internal.models.link import Link
from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import (
display_path, make_vcs_requirement_url, rmtree, split_auth_from_netloc,
@@ -61,21 +60,8 @@ class Subversion(VersionControl):
cmd_args = ['update'] + rev_options.to_args() + [dest]
self.run_command(cmd_args)
- def get_location(self, dist, dependency_links):
- for url in dependency_links:
- egg_fragment = Link(url).egg_fragment
- if not egg_fragment:
- continue
- if '-' in egg_fragment:
- # FIXME: will this work when a package has - in the name?
- key = '-'.join(egg_fragment.split('-')[:-1]).lower()
- else:
- key = egg_fragment
- if key == dist.key:
- return url.split('#', 1)[0]
- return None
-
- def get_revision(self, location):
+ @classmethod
+ def get_revision(cls, location):
"""
Return the maximum revision for all files under a given location
"""
@@ -83,16 +69,16 @@ class Subversion(VersionControl):
revision = 0
for base, dirs, files in os.walk(location):
- if self.dirname not in dirs:
+ if cls.dirname not in dirs:
dirs[:] = []
continue # no sense walking uncontrolled subdirs
- dirs.remove(self.dirname)
- entries_fn = os.path.join(base, self.dirname, 'entries')
+ dirs.remove(cls.dirname)
+ entries_fn = os.path.join(base, cls.dirname, 'entries')
if not os.path.exists(entries_fn):
# FIXME: should we warn?
continue
- dirurl, localrev = self._get_svn_url_rev(base)
+ dirurl, localrev = cls._get_svn_url_rev(base)
if base == location:
base = dirurl + '/' # save the root url
@@ -131,7 +117,8 @@ class Subversion(VersionControl):
return extra_args
- def get_url(self, location):
+ @classmethod
+ def get_remote_url(cls, location):
# In cases where the source is in a subdirectory, not alongside
# setup.py we have to look up in the location until we find a real
# setup.py
@@ -149,12 +136,13 @@ class Subversion(VersionControl):
)
return None
- return self._get_svn_url_rev(location)[0]
+ return cls._get_svn_url_rev(location)[0]
- def _get_svn_url_rev(self, location):
+ @classmethod
+ def _get_svn_url_rev(cls, location):
from pipenv.patched.notpip._internal.exceptions import InstallationError
- entries_path = os.path.join(location, self.dirname, 'entries')
+ entries_path = os.path.join(location, cls.dirname, 'entries')
if os.path.exists(entries_path):
with open(entries_path) as f:
data = f.read()
@@ -177,7 +165,7 @@ class Subversion(VersionControl):
else:
try:
# subversion >= 1.7
- xml = self.run_command(
+ xml = cls.run_command(
['info', '--xml', location],
show_stdout=False,
)
@@ -195,15 +183,14 @@ class Subversion(VersionControl):
return url, rev
- def get_src_requirement(self, dist, location):
- repo = self.get_url(location)
+ @classmethod
+ def get_src_requirement(cls, location, project_name):
+ repo = cls.get_remote_url(location)
if repo is None:
return None
repo = 'svn+' + repo
- rev = self.get_revision(location)
- # FIXME: why not project name?
- egg_project_name = dist.egg_name().split('-', 1)[0]
- return make_vcs_requirement_url(repo, rev, egg_project_name)
+ rev = cls.get_revision(location)
+ return make_vcs_requirement_url(repo, rev, project_name)
def is_commit_id_equal(self, dest, name):
"""Always assume the versions don't match"""
diff --git a/pipenv/patched/notpip/_internal/wheel.py b/pipenv/patched/notpip/_internal/wheel.py
index 6df5a3a3..06c880e8 100644
--- a/pipenv/patched/notpip/_internal/wheel.py
+++ b/pipenv/patched/notpip/_internal/wheel.py
@@ -30,6 +30,7 @@ from pipenv.patched.notpip._internal.exceptions import (
from pipenv.patched.notpip._internal.locations import (
PIP_DELETE_MARKER_FILENAME, distutils_scheme,
)
+from pipenv.patched.notpip._internal.models.link import Link
from pipenv.patched.notpip._internal.utils.logging import indent_log
from pipenv.patched.notpip._internal.utils.misc import (
call_subprocess, captured_stdout, ensure_dir, read_chunks,
@@ -40,9 +41,22 @@ from pipenv.patched.notpip._internal.utils.typing import MYPY_CHECK_RUNNING
from pipenv.patched.notpip._internal.utils.ui import open_spinner
if MYPY_CHECK_RUNNING:
- from typing import Dict, List, Optional # noqa: F401
+ from typing import ( # noqa: F401
+ Dict, List, Optional, Sequence, Mapping, Tuple, IO, Text, Any,
+ Union, Iterable
+ )
+ from pipenv.patched.notpip._vendor.packaging.requirements import Requirement # noqa: F401
+ from pipenv.patched.notpip._internal.req.req_install import InstallRequirement # noqa: F401
+ from pipenv.patched.notpip._internal.download import PipSession # noqa: F401
+ from pipenv.patched.notpip._internal.index import FormatControl, PackageFinder # noqa: F401
+ from pipenv.patched.notpip._internal.operations.prepare import ( # noqa: F401
+ RequirementPreparer
+ )
+ from pipenv.patched.notpip._internal.cache import WheelCache # noqa: F401
+ from pipenv.patched.notpip._internal.pep425tags import Pep425Tag # noqa: F401
+
+ InstalledCSVRow = Tuple[str, ...]
-wheel_ext = '.whl'
VERSION_COMPATIBLE = (1, 0)
@@ -50,7 +64,12 @@ VERSION_COMPATIBLE = (1, 0)
logger = logging.getLogger(__name__)
+def normpath(src, p):
+ return os.path.relpath(src, p).replace(os.path.sep, '/')
+
+
def rehash(path, blocksize=1 << 20):
+ # type: (str, int) -> Tuple[str, str]
"""Return (hash, length) for path using hashlib.sha256()"""
h = hashlib.sha256()
length = 0
@@ -61,20 +80,32 @@ def rehash(path, blocksize=1 << 20):
digest = 'sha256=' + urlsafe_b64encode(
h.digest()
).decode('latin1').rstrip('=')
- return (digest, length)
+ # unicode/str python2 issues
+ return (digest, str(length)) # type: ignore
def open_for_csv(name, mode):
+ # type: (str, Text) -> IO
if sys.version_info[0] < 3:
- nl = {}
+ nl = {} # type: Dict[str, Any]
bin = 'b'
else:
- nl = {'newline': ''}
+ nl = {'newline': ''} # type: Dict[str, Any]
bin = ''
return open(name, mode + bin, **nl)
+def replace_python_tag(wheelname, new_tag):
+ # type: (str, str) -> str
+ """Replace the Python tag in a wheel file name with a new value.
+ """
+ parts = wheelname.split('-')
+ parts[-3] = new_tag
+ return '-'.join(parts)
+
+
def fix_script(path):
+ # type: (str) -> Optional[bool]
"""Replace #!python with #!/path/to/python
Return True if file was changed."""
# XXX RECORD hashes will need to be updated
@@ -90,6 +121,7 @@ def fix_script(path):
script.write(firstline)
script.write(rest)
return True
+ return None
dist_info_re = re.compile(r"""^(?P(?P.+?)(-(?P.+?))?)
@@ -97,6 +129,7 @@ dist_info_re = re.compile(r"""^(?P(?P.+?)(-(?P.+?))?)
def root_is_purelib(name, wheeldir):
+ # type: (str, str) -> bool
"""
Return True if the extracted wheel in wheeldir should go into purelib.
"""
@@ -113,6 +146,7 @@ def root_is_purelib(name, wheeldir):
def get_entrypoints(filename):
+ # type: (str) -> Tuple[Dict[str, str], Dict[str, str]]
if not os.path.exists(filename):
return {}, {}
@@ -144,7 +178,7 @@ def get_entrypoints(filename):
def message_about_scripts_not_on_PATH(scripts):
- # type: (List[str]) -> Optional[str]
+ # type: (Sequence[str]) -> Optional[str]
"""Determine if any scripts are not on PATH and format a warning.
Returns a warning message if one or more scripts are not on PATH,
@@ -205,10 +239,81 @@ def message_about_scripts_not_on_PATH(scripts):
return "\n".join(msg_lines)
-def move_wheel_files(name, req, wheeldir, user=False, home=None, root=None,
- pycompile=True, scheme=None, isolated=False, prefix=None,
- warn_script_location=True):
+def sorted_outrows(outrows):
+ # type: (Iterable[InstalledCSVRow]) -> List[InstalledCSVRow]
+ """
+ Return the given rows of a RECORD file in sorted order.
+
+ Each row is a 3-tuple (path, hash, size) and corresponds to a record of
+ a RECORD file (see PEP 376 and PEP 427 for details). For the rows
+ passed to this function, the size can be an integer as an int or string,
+ or the empty string.
+ """
+ # Normally, there should only be one row per path, in which case the
+ # second and third elements don't come into play when sorting.
+ # However, in cases in the wild where a path might happen to occur twice,
+ # we don't want the sort operation to trigger an error (but still want
+ # determinism). Since the third element can be an int or string, we
+ # coerce each element to a string to avoid a TypeError in this case.
+ # For additional background, see--
+ # https://github.com/pypa/pip/issues/5868
+ return sorted(outrows, key=lambda row: tuple(str(x) for x in row))
+
+
+def get_csv_rows_for_installed(
+ old_csv_rows, # type: Iterable[List[str]]
+ installed, # type: Dict[str, str]
+ changed, # type: set
+ generated, # type: List[str]
+ lib_dir, # type: str
+):
+ # type: (...) -> List[InstalledCSVRow]
+ """
+ :param installed: A map from archive RECORD path to installation RECORD
+ path.
+ """
+ installed_rows = [] # type: List[InstalledCSVRow]
+ for row in old_csv_rows:
+ if len(row) > 3:
+ logger.warning(
+ 'RECORD line has more than three elements: {}'.format(row)
+ )
+ # Make a copy because we are mutating the row.
+ row = list(row)
+ old_path = row[0]
+ new_path = installed.pop(old_path, old_path)
+ row[0] = new_path
+ if new_path in changed:
+ digest, length = rehash(new_path)
+ row[1] = digest
+ row[2] = length
+ installed_rows.append(tuple(row))
+ for f in generated:
+ digest, length = rehash(f)
+ installed_rows.append((normpath(f, lib_dir), digest, str(length)))
+ for f in installed:
+ installed_rows.append((installed[f], '', ''))
+ return installed_rows
+
+
+def move_wheel_files(
+ name, # type: str
+ req, # type: Requirement
+ wheeldir, # type: str
+ user=False, # type: bool
+ home=None, # type: Optional[str]
+ root=None, # type: Optional[str]
+ pycompile=True, # type: bool
+ scheme=None, # type: Optional[Mapping[str, str]]
+ isolated=False, # type: bool
+ prefix=None, # type: Optional[str]
+ warn_script_location=True # type: bool
+):
+ # type: (...) -> None
"""Install a wheel"""
+ # TODO: Investigate and break this up.
+ # TODO: Look into moving this into a dedicated class for representing an
+ # installation.
if not scheme:
scheme = distutils_scheme(
@@ -221,7 +326,7 @@ def move_wheel_files(name, req, wheeldir, user=False, home=None, root=None,
else:
lib_dir = scheme['platlib']
- info_dir = []
+ info_dir = [] # type: List[str]
data_dirs = []
source = wheeldir.rstrip(os.path.sep) + os.path.sep
@@ -229,9 +334,9 @@ def move_wheel_files(name, req, wheeldir, user=False, home=None, root=None,
# installed = files copied from the wheel to the destination
# changed = files changed while installing (scripts #! line typically)
# generated = files newly generated during the install (script wrappers)
- installed = {}
+ installed = {} # type: Dict[str, str]
changed = set()
- generated = []
+ generated = [] # type: List[str]
# Compile all of the pyc files that we're going to be installing
if pycompile:
@@ -241,9 +346,6 @@ def move_wheel_files(name, req, wheeldir, user=False, home=None, root=None,
compileall.compile_dir(source, force=True, quiet=True)
logger.debug(stdout.getvalue())
- def normpath(src, p):
- return os.path.relpath(src, p).replace(os.path.sep, '/')
-
def record_installed(srcfile, destfile, modified=False):
"""Map archive RECORD paths to installation RECORD paths."""
oldpath = normpath(srcfile, wheeldir)
@@ -389,8 +491,9 @@ def move_wheel_files(name, req, wheeldir, user=False, home=None, root=None,
"import_name": entry.suffix.split(".")[0],
"func": entry.suffix,
}
-
- maker._get_script_text = _get_script_text
+ # ignore type, because mypy disallows assigning to a method,
+ # see https://github.com/python/mypy/issues/2427
+ maker._get_script_text = _get_script_text # type: ignore
maker.script_template = r"""# -*- coding: utf-8 -*-
import re
import sys
@@ -500,28 +603,23 @@ if __name__ == '__main__':
with open_for_csv(record, 'r') as record_in:
with open_for_csv(temp_record, 'w+') as record_out:
reader = csv.reader(record_in)
+ outrows = get_csv_rows_for_installed(
+ reader, installed=installed, changed=changed,
+ generated=generated, lib_dir=lib_dir,
+ )
writer = csv.writer(record_out)
- outrows = []
- for row in reader:
- row[0] = installed.pop(row[0], row[0])
- if row[0] in changed:
- row[1], row[2] = rehash(row[0])
- outrows.append(tuple(row))
- for f in generated:
- digest, length = rehash(f)
- outrows.append((normpath(f, lib_dir), digest, length))
- for f in installed:
- outrows.append((installed[f], '', ''))
- for row in sorted(outrows):
+ # Sort to simplify testing.
+ for row in sorted_outrows(outrows):
writer.writerow(row)
shutil.move(temp_record, record)
def wheel_version(source_dir):
+ # type: (Optional[str]) -> Optional[Tuple[int, ...]]
"""
Return the Wheel-Version of an extracted wheel, if possible.
- Otherwise, return False if we couldn't parse / extract it.
+ Otherwise, return None if we couldn't parse / extract it.
"""
try:
dist = [d for d in pkg_resources.find_on_path(None, source_dir)][0]
@@ -533,10 +631,11 @@ def wheel_version(source_dir):
version = tuple(map(int, version.split('.')))
return version
except Exception:
- return False
+ return None
def check_compatibility(version, name):
+ # type: (Optional[Tuple[int, ...]], str) -> None
"""
Raises errors or warns if called with an incompatible Wheel-Version.
@@ -568,7 +667,8 @@ def check_compatibility(version, name):
class Wheel(object):
"""A wheel file"""
- # TODO: maybe move the install code into this class
+ # TODO: Maybe move the class into the models sub-package
+ # TODO: Maybe move the install code into this class
wheel_file_re = re.compile(
r"""^(?P(?P.+?)-(?P.*?))
@@ -578,6 +678,7 @@ class Wheel(object):
)
def __init__(self, filename):
+ # type: (str) -> None
"""
:raises InvalidWheelFilename: when the filename is invalid for a wheel
"""
@@ -603,6 +704,7 @@ class Wheel(object):
}
def support_index_min(self, tags=None):
+ # type: (Optional[List[Pep425Tag]]) -> Optional[int]
"""
Return the lowest index that one of the wheel's file_tag combinations
achieves in the supported_tags list e.g. if there are 8 supported tags,
@@ -615,17 +717,146 @@ class Wheel(object):
return min(indexes) if indexes else None
def supported(self, tags=None):
+ # type: (Optional[List[Pep425Tag]]) -> bool
"""Is this wheel supported on this system?"""
if tags is None: # for mock
tags = pep425tags.get_supported()
return bool(set(tags).intersection(self.file_tags))
+def _contains_egg_info(
+ s, _egg_info_re=re.compile(r'([a-z0-9_.]+)-([a-z0-9_.!+-]+)', re.I)):
+ """Determine whether the string looks like an egg_info.
+
+ :param s: The string to parse. E.g. foo-2.1
+ """
+ return bool(_egg_info_re.search(s))
+
+
+def should_use_ephemeral_cache(
+ req, # type: InstallRequirement
+ format_control, # type: FormatControl
+ autobuilding, # type: bool
+ cache_available # type: bool
+):
+ # type: (...) -> Optional[bool]
+ """
+ Return whether to build an InstallRequirement object using the
+ ephemeral cache.
+
+ :param cache_available: whether a cache directory is available for the
+ autobuilding=True case.
+
+ :return: True or False to build the requirement with ephem_cache=True
+ or False, respectively; or None not to build the requirement.
+ """
+ if req.constraint:
+ return None
+ if req.is_wheel:
+ if not autobuilding:
+ logger.info(
+ 'Skipping %s, due to already being wheel.', req.name,
+ )
+ return None
+ if not autobuilding:
+ return False
+
+ if req.editable or not req.source_dir:
+ return None
+
+ if req.link and not req.link.is_artifact:
+ # VCS checkout. Build wheel just for this run.
+ return True
+
+ if "binary" not in format_control.get_allowed_formats(
+ canonicalize_name(req.name)):
+ logger.info(
+ "Skipping bdist_wheel for %s, due to binaries "
+ "being disabled for it.", req.name,
+ )
+ return None
+
+ link = req.link
+ base, ext = link.splitext()
+ if cache_available and _contains_egg_info(base):
+ return False
+
+ # Otherwise, build the wheel just for this run using the ephemeral
+ # cache since we are either in the case of e.g. a local directory, or
+ # no cache directory is available to use.
+ return True
+
+
+def format_command(
+ command_args, # type: List[str]
+ command_output, # type: str
+):
+ # type: (...) -> str
+ """
+ Format command information for logging.
+ """
+ text = 'Command arguments: {}\n'.format(command_args)
+
+ if not command_output:
+ text += 'Command output: None'
+ elif logger.getEffectiveLevel() > logging.DEBUG:
+ text += 'Command output: [use --verbose to show]'
+ else:
+ if not command_output.endswith('\n'):
+ command_output += '\n'
+ text += (
+ 'Command output:\n{}'
+ '-----------------------------------------'
+ ).format(command_output)
+
+ return text
+
+
+def get_legacy_build_wheel_path(
+ names, # type: List[str]
+ temp_dir, # type: str
+ req, # type: InstallRequirement
+ command_args, # type: List[str]
+ command_output, # type: str
+):
+ # type: (...) -> Optional[str]
+ """
+ Return the path to the wheel in the temporary build directory.
+ """
+ # Sort for determinism.
+ names = sorted(names)
+ if not names:
+ msg = (
+ 'Legacy build of wheel for {!r} created no files.\n'
+ ).format(req.name)
+ msg += format_command(command_args, command_output)
+ logger.warning(msg)
+ return None
+
+ if len(names) > 1:
+ msg = (
+ 'Legacy build of wheel for {!r} created more than one file.\n'
+ 'Filenames (choosing first): {}\n'
+ ).format(req.name, names)
+ msg += format_command(command_args, command_output)
+ logger.warning(msg)
+
+ return os.path.join(temp_dir, names[0])
+
+
class WheelBuilder(object):
"""Build wheels from a RequirementSet."""
- def __init__(self, finder, preparer, wheel_cache,
- build_options=None, global_options=None, no_clean=False):
+ def __init__(
+ self,
+ finder, # type: PackageFinder
+ preparer, # type: RequirementPreparer
+ wheel_cache, # type: WheelCache
+ build_options=None, # type: Optional[List[str]]
+ global_options=None, # type: Optional[List[str]]
+ no_clean=False # type: bool
+ ):
+ # type: (...) -> None
self.finder = finder
self.preparer = preparer
self.wheel_cache = wheel_cache
@@ -648,15 +879,18 @@ class WheelBuilder(object):
def _build_one_inside_env(self, req, output_dir, python_tag=None):
with TempDirectory(kind="wheel") as temp_dir:
- if self.__build_one(req, temp_dir.path, python_tag=python_tag):
+ if req.use_pep517:
+ builder = self._build_one_pep517
+ else:
+ builder = self._build_one_legacy
+ wheel_path = builder(req, temp_dir.path, python_tag=python_tag)
+ if wheel_path is not None:
+ wheel_name = os.path.basename(wheel_path)
+ dest_path = os.path.join(output_dir, wheel_name)
try:
- wheel_name = os.listdir(temp_dir.path)[0]
- wheel_path = os.path.join(output_dir, wheel_name)
- shutil.move(
- os.path.join(temp_dir.path, wheel_name), wheel_path
- )
+ shutil.move(wheel_path, dest_path)
logger.info('Stored in directory: %s', output_dir)
- return wheel_path
+ return dest_path
except Exception:
pass
# Ignore return, we can't do anything else useful.
@@ -674,10 +908,43 @@ class WheelBuilder(object):
SETUPTOOLS_SHIM % req.setup_py
] + list(self.global_options)
- def __build_one(self, req, tempd, python_tag=None):
+ def _build_one_pep517(self, req, tempd, python_tag=None):
+ """Build one InstallRequirement using the PEP 517 build process.
+
+ Returns path to wheel if successfully built. Otherwise, returns None.
+ """
+ assert req.metadata_directory is not None
+ try:
+ req.spin_message = 'Building wheel for %s (PEP 517)' % (req.name,)
+ logger.debug('Destination directory: %s', tempd)
+ wheel_name = req.pep517_backend.build_wheel(
+ tempd,
+ metadata_directory=req.metadata_directory
+ )
+ if python_tag:
+ # General PEP 517 backends don't necessarily support
+ # a "--python-tag" option, so we rename the wheel
+ # file directly.
+ new_name = replace_python_tag(wheel_name, python_tag)
+ os.rename(
+ os.path.join(tempd, wheel_name),
+ os.path.join(tempd, new_name)
+ )
+ # Reassign to simplify the return at the end of function
+ wheel_name = new_name
+ except Exception:
+ logger.error('Failed building wheel for %s', req.name)
+ return None
+ return os.path.join(tempd, wheel_name)
+
+ def _build_one_legacy(self, req, tempd, python_tag=None):
+ """Build one InstallRequirement using the "legacy" build process.
+
+ Returns path to wheel if successfully built. Otherwise, returns None.
+ """
base_args = self._base_setup_args(req)
- spin_message = 'Running setup.py bdist_wheel for %s' % (req.name,)
+ spin_message = 'Building wheel for %s (setup.py)' % (req.name,)
with open_spinner(spin_message) as spinner:
logger.debug('Destination directory: %s', tempd)
wheel_args = base_args + ['bdist_wheel', '-d', tempd] \
@@ -687,13 +954,21 @@ class WheelBuilder(object):
wheel_args += ["--python-tag", python_tag]
try:
- call_subprocess(wheel_args, cwd=req.setup_py_dir,
- show_stdout=False, spinner=spinner)
- return True
+ output = call_subprocess(wheel_args, cwd=req.setup_py_dir,
+ show_stdout=False, spinner=spinner)
except Exception:
spinner.finish("error")
logger.error('Failed building wheel for %s', req.name)
- return False
+ return None
+ names = os.listdir(tempd)
+ wheel_path = get_legacy_build_wheel_path(
+ names=names,
+ temp_dir=tempd,
+ req=req,
+ command_args=wheel_args,
+ command_output=output,
+ )
+ return wheel_path
def _clean_one(self, req):
base_args = self._base_setup_args(req)
@@ -707,57 +982,46 @@ class WheelBuilder(object):
logger.error('Failed cleaning build dir for %s', req.name)
return False
- def build(self, requirements, session, autobuilding=False):
+ def build(
+ self,
+ requirements, # type: Iterable[InstallRequirement]
+ session, # type: PipSession
+ autobuilding=False # type: bool
+ ):
+ # type: (...) -> List[InstallRequirement]
"""Build wheels.
:param unpack: If True, replace the sdist we built from with the
newly built wheel, in preparation for installation.
:return: True if all the wheels built correctly.
"""
- from pipenv.patched.notpip._internal import index
- from pipenv.patched.notpip._internal.models.link import Link
-
- building_is_possible = self._wheel_dir or (
- autobuilding and self.wheel_cache.cache_dir
- )
- assert building_is_possible
-
buildset = []
format_control = self.finder.format_control
+ # Whether a cache directory is available for autobuilding=True.
+ cache_available = bool(self._wheel_dir or self.wheel_cache.cache_dir)
+
for req in requirements:
- if req.constraint:
+ ephem_cache = should_use_ephemeral_cache(
+ req, format_control=format_control, autobuilding=autobuilding,
+ cache_available=cache_available,
+ )
+ if ephem_cache is None:
continue
- if req.is_wheel:
- if not autobuilding:
- logger.info(
- 'Skipping %s, due to already being wheel.', req.name,
- )
- elif autobuilding and req.editable:
- pass
- elif autobuilding and not req.source_dir:
- pass
- elif autobuilding and req.link and not req.link.is_artifact:
- # VCS checkout. Build wheel just for this run.
- buildset.append((req, True))
- else:
- ephem_cache = False
- if autobuilding:
- link = req.link
- base, ext = link.splitext()
- if index.egg_info_matches(base, None, link) is None:
- # E.g. local directory. Build wheel just for this run.
- ephem_cache = True
- if "binary" not in format_control.get_allowed_formats(
- canonicalize_name(req.name)):
- logger.info(
- "Skipping bdist_wheel for %s, due to binaries "
- "being disabled for it.", req.name,
- )
- continue
- buildset.append((req, ephem_cache))
+
+ buildset.append((req, ephem_cache))
if not buildset:
- return True
+ return []
+
+ # Is any wheel build not using the ephemeral cache?
+ if any(not ephem_cache for _, ephem_cache in buildset):
+ have_directory_for_build = self._wheel_dir or (
+ autobuilding and self.wheel_cache.cache_dir
+ )
+ assert have_directory_for_build
+
+ # TODO by @pradyunsg
+ # Should break up this method into 2 separate methods.
# Build the wheels.
logger.info(
@@ -829,5 +1093,5 @@ class WheelBuilder(object):
'Failed to build %s',
' '.join([req.name for req in build_failure]),
)
- # Return True if all builds were successful
- return len(build_failure) == 0
+ # Return a list of requirements that failed to build
+ return build_failure
diff --git a/pipenv/patched/notpip/_vendor/__init__.py b/pipenv/patched/notpip/_vendor/__init__.py
index b6294b21..1256c039 100644
--- a/pipenv/patched/notpip/_vendor/__init__.py
+++ b/pipenv/patched/notpip/_vendor/__init__.py
@@ -74,6 +74,7 @@ if DEBUNDLED:
vendored("packaging")
vendored("packaging.version")
vendored("packaging.specifiers")
+ vendored("pep517")
vendored("pkg_resources")
vendored("progress")
vendored("pytoml")
diff --git a/pipenv/patched/notpip/_vendor/certifi/__init__.py b/pipenv/patched/notpip/_vendor/certifi/__init__.py
index aa329fbb..ef71f3af 100644
--- a/pipenv/patched/notpip/_vendor/certifi/__init__.py
+++ b/pipenv/patched/notpip/_vendor/certifi/__init__.py
@@ -1,3 +1,3 @@
-from .core import where, old_where
+from .core import where
-__version__ = "2018.08.24"
+__version__ = "2018.11.29"
diff --git a/pipenv/patched/notpip/_vendor/certifi/cacert.pem b/pipenv/patched/notpip/_vendor/certifi/cacert.pem
index 85de024e..db68797e 100644
--- a/pipenv/patched/notpip/_vendor/certifi/cacert.pem
+++ b/pipenv/patched/notpip/_vendor/certifi/cacert.pem
@@ -326,36 +326,6 @@ OCiNUW7dFGdTbHFcJoRNdVq2fmBWqU2t+5sel/MN2dKXVHfaPRK34B7vCAas+YWH
QMAJKOSLakhT2+zNVVXxxvjpoixMptEmX36vWkzaH6byHCx+rgIW0lbQL1dTR+iS
-----END CERTIFICATE-----
-# Issuer: CN=Visa eCommerce Root O=VISA OU=Visa International Service Association
-# Subject: CN=Visa eCommerce Root O=VISA OU=Visa International Service Association
-# Label: "Visa eCommerce Root"
-# Serial: 25952180776285836048024890241505565794
-# MD5 Fingerprint: fc:11:b8:d8:08:93:30:00:6d:23:f9:7e:eb:52:1e:02
-# SHA1 Fingerprint: 70:17:9b:86:8c:00:a4:fa:60:91:52:22:3f:9f:3e:32:bd:e0:05:62
-# SHA256 Fingerprint: 69:fa:c9:bd:55:fb:0a:c7:8d:53:bb:ee:5c:f1:d5:97:98:9f:d0:aa:ab:20:a2:51:51:bd:f1:73:3e:e7:d1:22
------BEGIN CERTIFICATE-----
-MIIDojCCAoqgAwIBAgIQE4Y1TR0/BvLB+WUF1ZAcYjANBgkqhkiG9w0BAQUFADBr
-MQswCQYDVQQGEwJVUzENMAsGA1UEChMEVklTQTEvMC0GA1UECxMmVmlzYSBJbnRl
-cm5hdGlvbmFsIFNlcnZpY2UgQXNzb2NpYXRpb24xHDAaBgNVBAMTE1Zpc2EgZUNv
-bW1lcmNlIFJvb3QwHhcNMDIwNjI2MDIxODM2WhcNMjIwNjI0MDAxNjEyWjBrMQsw
-CQYDVQQGEwJVUzENMAsGA1UEChMEVklTQTEvMC0GA1UECxMmVmlzYSBJbnRlcm5h
-dGlvbmFsIFNlcnZpY2UgQXNzb2NpYXRpb24xHDAaBgNVBAMTE1Zpc2EgZUNvbW1l
-cmNlIFJvb3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvV95WHm6h
-2mCxlCfLF9sHP4CFT8icttD0b0/Pmdjh28JIXDqsOTPHH2qLJj0rNfVIsZHBAk4E
-lpF7sDPwsRROEW+1QK8bRaVK7362rPKgH1g/EkZgPI2h4H3PVz4zHvtH8aoVlwdV
-ZqW1LS7YgFmypw23RuwhY/81q6UCzyr0TP579ZRdhE2o8mCP2w4lPJ9zcc+U30rq
-299yOIzzlr3xF7zSujtFWsan9sYXiwGd/BmoKoMWuDpI/k4+oKsGGelT84ATB+0t
-vz8KPFUgOSwsAGl0lUq8ILKpeeUYiZGo3BxN77t+Nwtd/jmliFKMAGzsGHxBvfaL
-dXe6YJ2E5/4tAgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQD
-AgEGMB0GA1UdDgQWBBQVOIMPPyw/cDMezUb+B4wg4NfDtzANBgkqhkiG9w0BAQUF
-AAOCAQEAX/FBfXxcCLkr4NWSR/pnXKUTwwMhmytMiUbPWU3J/qVAtmPN3XEolWcR
-zCSs00Rsca4BIGsDoo8Ytyk6feUWYFN4PMCvFYP3j1IzJL1kk5fui/fbGKhtcbP3
-LBfQdCVp9/5rPJS+TUtBjE7ic9DjkCJzQ83z7+pzzkWKsKZJ/0x9nXGIxHYdkFsd
-7v3M9+79YKWxehZx0RbQfBI8bGmX265fOZpwLwU8GUYEmSA20GBuYQa7FkKMcPcw
-++DbZqMAAb3mLNqRX6BGi01qnD093QVG/na/oAo85ADmJ7f/hC3euiInlhBx6yLt
-398znM/jra6O1I7mT1GvFpLgXPYHDw==
------END CERTIFICATE-----
-
# Issuer: CN=AAA Certificate Services O=Comodo CA Limited
# Subject: CN=AAA Certificate Services O=Comodo CA Limited
# Label: "Comodo AAA Services root"
@@ -4298,3 +4268,245 @@ rYy0UGYwEAYJKwYBBAGCNxUBBAMCAQAwCgYIKoZIzj0EAwMDaAAwZQIwJsdpW9zV
57LnyAyMjMPdeYwbY9XJUpROTYJKcx6ygISpJcBMWm1JKWB4E+J+SOtkAjEA2zQg
Mgj/mkkCtojeFK9dbJlxjRo/i9fgojaGHAeCOnZT/cKi7e97sIBPWA9LUzm9
-----END CERTIFICATE-----
+
+# Issuer: CN=GTS Root R1 O=Google Trust Services LLC
+# Subject: CN=GTS Root R1 O=Google Trust Services LLC
+# Label: "GTS Root R1"
+# Serial: 146587175971765017618439757810265552097
+# MD5 Fingerprint: 82:1a:ef:d4:d2:4a:f2:9f:e2:3d:97:06:14:70:72:85
+# SHA1 Fingerprint: e1:c9:50:e6:ef:22:f8:4c:56:45:72:8b:92:20:60:d7:d5:a7:a3:e8
+# SHA256 Fingerprint: 2a:57:54:71:e3:13:40:bc:21:58:1c:bd:2c:f1:3e:15:84:63:20:3e:ce:94:bc:f9:d3:cc:19:6b:f0:9a:54:72
+-----BEGIN CERTIFICATE-----
+MIIFWjCCA0KgAwIBAgIQbkepxUtHDA3sM9CJuRz04TANBgkqhkiG9w0BAQwFADBH
+MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
+QzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIy
+MDAwMDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNl
+cnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwggIiMA0GCSqGSIb3DQEB
+AQUAA4ICDwAwggIKAoICAQC2EQKLHuOhd5s73L+UPreVp0A8of2C+X0yBoJx9vaM
+f/vo27xqLpeXo4xL+Sv2sfnOhB2x+cWX3u+58qPpvBKJXqeqUqv4IyfLpLGcY9vX
+mX7wCl7raKb0xlpHDU0QM+NOsROjyBhsS+z8CZDfnWQpJSMHobTSPS5g4M/SCYe7
+zUjwTcLCeoiKu7rPWRnWr4+wB7CeMfGCwcDfLqZtbBkOtdh+JhpFAz2weaSUKK0P
+fyblqAj+lug8aJRT7oM6iCsVlgmy4HqMLnXWnOunVmSPlk9orj2XwoSPwLxAwAtc
+vfaHszVsrBhQf4TgTM2S0yDpM7xSma8ytSmzJSq0SPly4cpk9+aCEI3oncKKiPo4
+Zor8Y/kB+Xj9e1x3+naH+uzfsQ55lVe0vSbv1gHR6xYKu44LtcXFilWr06zqkUsp
+zBmkMiVOKvFlRNACzqrOSbTqn3yDsEB750Orp2yjj32JgfpMpf/VjsPOS+C12LOO
+Rc92wO1AK/1TD7Cn1TsNsYqiA94xrcx36m97PtbfkSIS5r762DL8EGMUUXLeXdYW
+k70paDPvOmbsB4om3xPXV2V4J95eSRQAogB/mqghtqmxlbCluQ0WEdrHbEg8QOB+
+DVrNVjzRlwW5y0vtOUucxD/SVRNuJLDWcfr0wbrM7Rv1/oFB2ACYPTrIrnqYNxgF
+lQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV
+HQ4EFgQU5K8rJnEaK0gnhS9SZizv8IkTcT4wDQYJKoZIhvcNAQEMBQADggIBADiW
+Cu49tJYeX++dnAsznyvgyv3SjgofQXSlfKqE1OXyHuY3UjKcC9FhHb8owbZEKTV1
+d5iyfNm9dKyKaOOpMQkpAWBz40d8U6iQSifvS9efk+eCNs6aaAyC58/UEBZvXw6Z
+XPYfcX3v73svfuo21pdwCxXu11xWajOl40k4DLh9+42FpLFZXvRq4d2h9mREruZR
+gyFmxhE+885H7pwoHyXa/6xmld01D1zvICxi/ZG6qcz8WpyTgYMpl0p8WnK0OdC3
+d8t5/Wk6kjftbjhlRn7pYL15iJdfOBL07q9bgsiG1eGZbYwE8na6SfZu6W0eX6Dv
+J4J2QPim01hcDyxC2kLGe4g0x8HYRZvBPsVhHdljUEn2NIVq4BjFbkerQUIpm/Zg
+DdIx02OYI5NaAIFItO/Nis3Jz5nu2Z6qNuFoS3FJFDYoOj0dzpqPJeaAcWErtXvM
++SUWgeExX6GjfhaknBZqlxi9dnKlC54dNuYvoS++cJEPqOba+MSSQGwlfnuzCdyy
+F62ARPBopY+Udf90WuioAnwMCeKpSwughQtiue+hMZL77/ZRBIls6Kl0obsXs7X9
+SQ98POyDGCBDTtWTurQ0sR8WNh8M5mQ5Fkzc4P4dyKliPUDqysU0ArSuiYgzNdws
+E3PYJ/HQcu51OyLemGhmW/HGY0dVHLqlCFF1pkgl
+-----END CERTIFICATE-----
+
+# Issuer: CN=GTS Root R2 O=Google Trust Services LLC
+# Subject: CN=GTS Root R2 O=Google Trust Services LLC
+# Label: "GTS Root R2"
+# Serial: 146587176055767053814479386953112547951
+# MD5 Fingerprint: 44:ed:9a:0e:a4:09:3b:00:f2:ae:4c:a3:c6:61:b0:8b
+# SHA1 Fingerprint: d2:73:96:2a:2a:5e:39:9f:73:3f:e1:c7:1e:64:3f:03:38:34:fc:4d
+# SHA256 Fingerprint: c4:5d:7b:b0:8e:6d:67:e6:2e:42:35:11:0b:56:4e:5f:78:fd:92:ef:05:8c:84:0a:ea:4e:64:55:d7:58:5c:60
+-----BEGIN CERTIFICATE-----
+MIIFWjCCA0KgAwIBAgIQbkepxlqz5yDFMJo/aFLybzANBgkqhkiG9w0BAQwFADBH
+MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
+QzEUMBIGA1UEAxMLR1RTIFJvb3QgUjIwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIy
+MDAwMDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNl
+cnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjIwggIiMA0GCSqGSIb3DQEB
+AQUAA4ICDwAwggIKAoICAQDO3v2m++zsFDQ8BwZabFn3GTXd98GdVarTzTukk3Lv
+CvptnfbwhYBboUhSnznFt+4orO/LdmgUud+tAWyZH8QiHZ/+cnfgLFuv5AS/T3Kg
+GjSY6Dlo7JUle3ah5mm5hRm9iYz+re026nO8/4Piy33B0s5Ks40FnotJk9/BW9Bu
+XvAuMC6C/Pq8tBcKSOWIm8Wba96wyrQD8Nr0kLhlZPdcTK3ofmZemde4wj7I0BOd
+re7kRXuJVfeKH2JShBKzwkCX44ofR5GmdFrS+LFjKBC4swm4VndAoiaYecb+3yXu
+PuWgf9RhD1FLPD+M2uFwdNjCaKH5wQzpoeJ/u1U8dgbuak7MkogwTZq9TwtImoS1
+mKPV+3PBV2HdKFZ1E66HjucMUQkQdYhMvI35ezzUIkgfKtzra7tEscszcTJGr61K
+8YzodDqs5xoic4DSMPclQsciOzsSrZYuxsN2B6ogtzVJV+mSSeh2FnIxZyuWfoqj
+x5RWIr9qS34BIbIjMt/kmkRtWVtd9QCgHJvGeJeNkP+byKq0rxFROV7Z+2et1VsR
+nTKaG73VululycslaVNVJ1zgyjbLiGH7HrfQy+4W+9OmTN6SpdTi3/UGVN4unUu0
+kzCqgc7dGtxRcw1PcOnlthYhGXmy5okLdWTK1au8CcEYof/UVKGFPP0UJAOyh9Ok
+twIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV
+HQ4EFgQUu//KjiOfT5nK2+JopqUVJxce2Q4wDQYJKoZIhvcNAQEMBQADggIBALZp
+8KZ3/p7uC4Gt4cCpx/k1HUCCq+YEtN/L9x0Pg/B+E02NjO7jMyLDOfxA325BS0JT
+vhaI8dI4XsRomRyYUpOM52jtG2pzegVATX9lO9ZY8c6DR2Dj/5epnGB3GFW1fgiT
+z9D2PGcDFWEJ+YF59exTpJ/JjwGLc8R3dtyDovUMSRqodt6Sm2T4syzFJ9MHwAiA
+pJiS4wGWAqoC7o87xdFtCjMwc3i5T1QWvwsHoaRc5svJXISPD+AVdyx+Jn7axEvb
+pxZ3B7DNdehyQtaVhJ2Gg/LkkM0JR9SLA3DaWsYDQvTtN6LwG1BUSw7YhN4ZKJmB
+R64JGz9I0cNv4rBgF/XuIwKl2gBbbZCr7qLpGzvpx0QnRY5rn/WkhLx3+WuXrD5R
+RaIRpsyF7gpo8j5QOHokYh4XIDdtak23CZvJ/KRY9bb7nE4Yu5UC56GtmwfuNmsk
+0jmGwZODUNKBRqhfYlcsu2xkiAhu7xNUX90txGdj08+JN7+dIPT7eoOboB6BAFDC
+5AwiWVIQ7UNWhwD4FFKnHYuTjKJNRn8nxnGbJN7k2oaLDX5rIMHAnuFl2GqjpuiF
+izoHCBy69Y9Vmhh1fuXsgWbRIXOhNUQLgD1bnF5vKheW0YMjiGZt5obicDIvUiLn
+yOd/xCxgXS/Dr55FBcOEArf9LAhST4Ldo/DUhgkC
+-----END CERTIFICATE-----
+
+# Issuer: CN=GTS Root R3 O=Google Trust Services LLC
+# Subject: CN=GTS Root R3 O=Google Trust Services LLC
+# Label: "GTS Root R3"
+# Serial: 146587176140553309517047991083707763997
+# MD5 Fingerprint: 1a:79:5b:6b:04:52:9c:5d:c7:74:33:1b:25:9a:f9:25
+# SHA1 Fingerprint: 30:d4:24:6f:07:ff:db:91:89:8a:0b:e9:49:66:11:eb:8c:5e:46:e5
+# SHA256 Fingerprint: 15:d5:b8:77:46:19:ea:7d:54:ce:1c:a6:d0:b0:c4:03:e0:37:a9:17:f1:31:e8:a0:4e:1e:6b:7a:71:ba:bc:e5
+-----BEGIN CERTIFICATE-----
+MIICDDCCAZGgAwIBAgIQbkepx2ypcyRAiQ8DVd2NHTAKBggqhkjOPQQDAzBHMQsw
+CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU
+MBIGA1UEAxMLR1RTIFJvb3QgUjMwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAw
+MDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp
+Y2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjMwdjAQBgcqhkjOPQIBBgUrgQQA
+IgNiAAQfTzOHMymKoYTey8chWEGJ6ladK0uFxh1MJ7x/JlFyb+Kf1qPKzEUURout
+736GjOyxfi//qXGdGIRFBEFVbivqJn+7kAHjSxm65FSWRQmx1WyRRK2EE46ajA2A
+DDL24CejQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
+DgQWBBTB8Sa6oC2uhYHP0/EqEr24Cmf9vDAKBggqhkjOPQQDAwNpADBmAjEAgFuk
+fCPAlaUs3L6JbyO5o91lAFJekazInXJ0glMLfalAvWhgxeG4VDvBNhcl2MG9AjEA
+njWSdIUlUfUk7GRSJFClH9voy8l27OyCbvWFGFPouOOaKaqW04MjyaR7YbPMAuhd
+-----END CERTIFICATE-----
+
+# Issuer: CN=GTS Root R4 O=Google Trust Services LLC
+# Subject: CN=GTS Root R4 O=Google Trust Services LLC
+# Label: "GTS Root R4"
+# Serial: 146587176229350439916519468929765261721
+# MD5 Fingerprint: 5d:b6:6a:c4:60:17:24:6a:1a:99:a8:4b:ee:5e:b4:26
+# SHA1 Fingerprint: 2a:1d:60:27:d9:4a:b1:0a:1c:4d:91:5c:cd:33:a0:cb:3e:2d:54:cb
+# SHA256 Fingerprint: 71:cc:a5:39:1f:9e:79:4b:04:80:25:30:b3:63:e1:21:da:8a:30:43:bb:26:66:2f:ea:4d:ca:7f:c9:51:a4:bd
+-----BEGIN CERTIFICATE-----
+MIICCjCCAZGgAwIBAgIQbkepyIuUtui7OyrYorLBmTAKBggqhkjOPQQDAzBHMQsw
+CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU
+MBIGA1UEAxMLR1RTIFJvb3QgUjQwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAw
+MDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp
+Y2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjQwdjAQBgcqhkjOPQIBBgUrgQQA
+IgNiAATzdHOnaItgrkO4NcWBMHtLSZ37wWHO5t5GvWvVYRg1rkDdc/eJkTBa6zzu
+hXyiQHY7qca4R9gq55KRanPpsXI5nymfopjTX15YhmUPoYRlBtHci8nHc8iMai/l
+xKvRHYqjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
+DgQWBBSATNbrdP9JNqPV2Py1PsVq8JQdjDAKBggqhkjOPQQDAwNnADBkAjBqUFJ0
+CMRw3J5QdCHojXohw0+WbhXRIjVhLfoIN+4Zba3bssx9BzT1YBkstTTZbyACMANx
+sbqjYAuG7ZoIapVon+Kz4ZNkfF6Tpt95LY2F45TPI11xzPKwTdb+mciUqXWi4w==
+-----END CERTIFICATE-----
+
+# Issuer: CN=UCA Global G2 Root O=UniTrust
+# Subject: CN=UCA Global G2 Root O=UniTrust
+# Label: "UCA Global G2 Root"
+# Serial: 124779693093741543919145257850076631279
+# MD5 Fingerprint: 80:fe:f0:c4:4a:f0:5c:62:32:9f:1c:ba:78:a9:50:f8
+# SHA1 Fingerprint: 28:f9:78:16:19:7a:ff:18:25:18:aa:44:fe:c1:a0:ce:5c:b6:4c:8a
+# SHA256 Fingerprint: 9b:ea:11:c9:76:fe:01:47:64:c1:be:56:a6:f9:14:b5:a5:60:31:7a:bd:99:88:39:33:82:e5:16:1a:a0:49:3c
+-----BEGIN CERTIFICATE-----
+MIIFRjCCAy6gAwIBAgIQXd+x2lqj7V2+WmUgZQOQ7zANBgkqhkiG9w0BAQsFADA9
+MQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxGzAZBgNVBAMMElVDQSBH
+bG9iYWwgRzIgUm9vdDAeFw0xNjAzMTEwMDAwMDBaFw00MDEyMzEwMDAwMDBaMD0x
+CzAJBgNVBAYTAkNOMREwDwYDVQQKDAhVbmlUcnVzdDEbMBkGA1UEAwwSVUNBIEds
+b2JhbCBHMiBSb290MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxeYr
+b3zvJgUno4Ek2m/LAfmZmqkywiKHYUGRO8vDaBsGxUypK8FnFyIdK+35KYmToni9
+kmugow2ifsqTs6bRjDXVdfkX9s9FxeV67HeToI8jrg4aA3++1NDtLnurRiNb/yzm
+VHqUwCoV8MmNsHo7JOHXaOIxPAYzRrZUEaalLyJUKlgNAQLx+hVRZ2zA+te2G3/R
+VogvGjqNO7uCEeBHANBSh6v7hn4PJGtAnTRnvI3HLYZveT6OqTwXS3+wmeOwcWDc
+C/Vkw85DvG1xudLeJ1uK6NjGruFZfc8oLTW4lVYa8bJYS7cSN8h8s+1LgOGN+jIj
+tm+3SJUIsUROhYw6AlQgL9+/V087OpAh18EmNVQg7Mc/R+zvWr9LesGtOxdQXGLY
+D0tK3Cv6brxzks3sx1DoQZbXqX5t2Okdj4q1uViSukqSKwxW/YDrCPBeKW4bHAyv
+j5OJrdu9o54hyokZ7N+1wxrrFv54NkzWbtA+FxyQF2smuvt6L78RHBgOLXMDj6Dl
+NaBa4kx1HXHhOThTeEDMg5PXCp6dW4+K5OXgSORIskfNTip1KnvyIvbJvgmRlld6
+iIis7nCs+dwp4wwcOxJORNanTrAmyPPZGpeRaOrvjUYG0lZFWJo8DA+DuAUlwznP
+O6Q0ibd5Ei9Hxeepl2n8pndntd978XplFeRhVmUCAwEAAaNCMEAwDgYDVR0PAQH/
+BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFIHEjMz15DD/pQwIX4wV
+ZyF0Ad/fMA0GCSqGSIb3DQEBCwUAA4ICAQATZSL1jiutROTL/7lo5sOASD0Ee/oj
+L3rtNtqyzm325p7lX1iPyzcyochltq44PTUbPrw7tgTQvPlJ9Zv3hcU2tsu8+Mg5
+1eRfB70VVJd0ysrtT7q6ZHafgbiERUlMjW+i67HM0cOU2kTC5uLqGOiiHycFutfl
+1qnN3e92mI0ADs0b+gO3joBYDic/UvuUospeZcnWhNq5NXHzJsBPd+aBJ9J3O5oU
+b3n09tDh05S60FdRvScFDcH9yBIw7m+NESsIndTUv4BFFJqIRNow6rSn4+7vW4LV
+PtateJLbXDzz2K36uGt/xDYotgIVilQsnLAXc47QN6MUPJiVAAwpBVueSUmxX8fj
+y88nZY41F7dXyDDZQVu5FLbowg+UMaeUmMxq67XhJ/UQqAHojhJi6IjMtX9Gl8Cb
+EGY4GjZGXyJoPd/JxhMnq1MGrKI8hgZlb7F+sSlEmqO6SWkoaY/X5V+tBIZkbxqg
+DMUIYs6Ao9Dz7GjevjPHF1t/gMRMTLGmhIrDO7gJzRSBuhjjVFc2/tsvfEehOjPI
++Vg7RE+xygKJBJYoaMVLuCaJu9YzL1DV/pqJuhgyklTGW+Cd+V7lDSKb9triyCGy
+YiGqhkCyLmTTX8jjfhFnRR8F/uOi77Oos/N9j/gMHyIfLXC0uAE0djAA5SN4p1bX
+UB+K+wb1whnw0A==
+-----END CERTIFICATE-----
+
+# Issuer: CN=UCA Extended Validation Root O=UniTrust
+# Subject: CN=UCA Extended Validation Root O=UniTrust
+# Label: "UCA Extended Validation Root"
+# Serial: 106100277556486529736699587978573607008
+# MD5 Fingerprint: a1:f3:5f:43:c6:34:9b:da:bf:8c:7e:05:53:ad:96:e2
+# SHA1 Fingerprint: a3:a1:b0:6f:24:61:23:4a:e3:36:a5:c2:37:fc:a6:ff:dd:f0:d7:3a
+# SHA256 Fingerprint: d4:3a:f9:b3:54:73:75:5c:96:84:fc:06:d7:d8:cb:70:ee:5c:28:e7:73:fb:29:4e:b4:1e:e7:17:22:92:4d:24
+-----BEGIN CERTIFICATE-----
+MIIFWjCCA0KgAwIBAgIQT9Irj/VkyDOeTzRYZiNwYDANBgkqhkiG9w0BAQsFADBH
+MQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNVBAMMHFVDQSBF
+eHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwHhcNMTUwMzEzMDAwMDAwWhcNMzgxMjMx
+MDAwMDAwWjBHMQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNV
+BAMMHFVDQSBFeHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwggIiMA0GCSqGSIb3DQEB
+AQUAA4ICDwAwggIKAoICAQCpCQcoEwKwmeBkqh5DFnpzsZGgdT6o+uM4AHrsiWog
+D4vFsJszA1qGxliG1cGFu0/GnEBNyr7uaZa4rYEwmnySBesFK5pI0Lh2PpbIILvS
+sPGP2KxFRv+qZ2C0d35qHzwaUnoEPQc8hQ2E0B92CvdqFN9y4zR8V05WAT558aop
+O2z6+I9tTcg1367r3CTueUWnhbYFiN6IXSV8l2RnCdm/WhUFhvMJHuxYMjMR83dk
+sHYf5BA1FxvyDrFspCqjc/wJHx4yGVMR59mzLC52LqGj3n5qiAno8geK+LLNEOfi
+c0CTuwjRP+H8C5SzJe98ptfRr5//lpr1kXuYC3fUfugH0mK1lTnj8/FtDw5lhIpj
+VMWAtuCeS31HJqcBCF3RiJ7XwzJE+oJKCmhUfzhTA8ykADNkUVkLo4KRel7sFsLz
+KuZi2irbWWIQJUoqgQtHB0MGcIfS+pMRKXpITeuUx3BNr2fVUbGAIAEBtHoIppB/
+TuDvB0GHr2qlXov7z1CymlSvw4m6WC31MJixNnI5fkkE/SmnTHnkBVfblLkWU41G
+sx2VYVdWf6/wFlthWG82UBEL2KwrlRYaDh8IzTY0ZRBiZtWAXxQgXy0MoHgKaNYs
+1+lvK9JKBZP8nm9rZ/+I8U6laUpSNwXqxhaN0sSZ0YIrO7o1dfdRUVjzyAfd5LQD
+fwIDAQABo0IwQDAdBgNVHQ4EFgQU2XQ65DA9DfcS3H5aBZ8eNJr34RQwDwYDVR0T
+AQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBADaN
+l8xCFWQpN5smLNb7rhVpLGsaGvdftvkHTFnq88nIua7Mui563MD1sC3AO6+fcAUR
+ap8lTwEpcOPlDOHqWnzcSbvBHiqB9RZLcpHIojG5qtr8nR/zXUACE/xOHAbKsxSQ
+VBcZEhrxH9cMaVr2cXj0lH2RC47skFSOvG+hTKv8dGT9cZr4QQehzZHkPJrgmzI5
+c6sq1WnIeJEmMX3ixzDx/BR4dxIOE/TdFpS/S2d7cFOFyrC78zhNLJA5wA3CXWvp
+4uXViI3WLL+rG761KIcSF3Ru/H38j9CHJrAb+7lsq+KePRXBOy5nAliRn+/4Qh8s
+t2j1da3Ptfb/EX3C8CSlrdP6oDyp+l3cpaDvRKS+1ujl5BOWF3sGPjLtx7dCvHaj
+2GU4Kzg1USEODm8uNBNA4StnDG1KQTAYI1oyVZnJF+A83vbsea0rWBmirSwiGpWO
+vpaQXUJXxPkUAzUrHC1RVwinOt4/5Mi0A3PCwSaAuwtCH60NryZy2sy+s6ODWA2C
+xR9GUeOcGMyNm43sSet1UNWMKFnKdDTajAshqx7qG+XH/RU+wBeq+yNuJkbL+vmx
+cmtpzyKEC2IPrNkZAJSidjzULZrtBJ4tBmIQN1IchXIbJ+XMxjHsN+xjWZsLHXbM
+fjKaiJUINlK73nZfdklJrX+9ZSCyycErdhh2n1ax
+-----END CERTIFICATE-----
+
+# Issuer: CN=Certigna Root CA O=Dhimyotis OU=0002 48146308100036
+# Subject: CN=Certigna Root CA O=Dhimyotis OU=0002 48146308100036
+# Label: "Certigna Root CA"
+# Serial: 269714418870597844693661054334862075617
+# MD5 Fingerprint: 0e:5c:30:62:27:eb:5b:bc:d7:ae:62:ba:e9:d5:df:77
+# SHA1 Fingerprint: 2d:0d:52:14:ff:9e:ad:99:24:01:74:20:47:6e:6c:85:27:27:f5:43
+# SHA256 Fingerprint: d4:8d:3d:23:ee:db:50:a4:59:e5:51:97:60:1c:27:77:4b:9d:7b:18:c9:4d:5a:05:95:11:a1:02:50:b9:31:68
+-----BEGIN CERTIFICATE-----
+MIIGWzCCBEOgAwIBAgIRAMrpG4nxVQMNo+ZBbcTjpuEwDQYJKoZIhvcNAQELBQAw
+WjELMAkGA1UEBhMCRlIxEjAQBgNVBAoMCURoaW15b3RpczEcMBoGA1UECwwTMDAw
+MiA0ODE0NjMwODEwMDAzNjEZMBcGA1UEAwwQQ2VydGlnbmEgUm9vdCBDQTAeFw0x
+MzEwMDEwODMyMjdaFw0zMzEwMDEwODMyMjdaMFoxCzAJBgNVBAYTAkZSMRIwEAYD
+VQQKDAlEaGlteW90aXMxHDAaBgNVBAsMEzAwMDIgNDgxNDYzMDgxMDAwMzYxGTAX
+BgNVBAMMEENlcnRpZ25hIFJvb3QgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAw
+ggIKAoICAQDNGDllGlmx6mQWDoyUJJV8g9PFOSbcDO8WV43X2KyjQn+Cyu3NW9sO
+ty3tRQgXstmzy9YXUnIo245Onoq2C/mehJpNdt4iKVzSs9IGPjA5qXSjklYcoW9M
+CiBtnyN6tMbaLOQdLNyzKNAT8kxOAkmhVECe5uUFoC2EyP+YbNDrihqECB63aCPu
+I9Vwzm1RaRDuoXrC0SIxwoKF0vJVdlB8JXrJhFwLrN1CTivngqIkicuQstDuI7pm
+TLtipPlTWmR7fJj6o0ieD5Wupxj0auwuA0Wv8HT4Ks16XdG+RCYyKfHx9WzMfgIh
+C59vpD++nVPiz32pLHxYGpfhPTc3GGYo0kDFUYqMwy3OU4gkWGQwFsWq4NYKpkDf
+ePb1BHxpE4S80dGnBs8B92jAqFe7OmGtBIyT46388NtEbVncSVmurJqZNjBBe3Yz
+IoejwpKGbvlw7q6Hh5UbxHq9MfPU0uWZ/75I7HX1eBYdpnDBfzwboZL7z8g81sWT
+Co/1VTp2lc5ZmIoJlXcymoO6LAQ6l73UL77XbJuiyn1tJslV1c/DeVIICZkHJC1k
+JWumIWmbat10TWuXekG9qxf5kBdIjzb5LdXF2+6qhUVB+s06RbFo5jZMm5BX7CO5
+hwjCxAnxl4YqKE3idMDaxIzb3+KhF1nOJFl0Mdp//TBt2dzhauH8XwIDAQABo4IB
+GjCCARYwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYE
+FBiHVuBud+4kNTxOc5of1uHieX4rMB8GA1UdIwQYMBaAFBiHVuBud+4kNTxOc5of
+1uHieX4rMEQGA1UdIAQ9MDswOQYEVR0gADAxMC8GCCsGAQUFBwIBFiNodHRwczov
+L3d3d3cuY2VydGlnbmEuZnIvYXV0b3JpdGVzLzBtBgNVHR8EZjBkMC+gLaArhilo
+dHRwOi8vY3JsLmNlcnRpZ25hLmZyL2NlcnRpZ25hcm9vdGNhLmNybDAxoC+gLYYr
+aHR0cDovL2NybC5kaGlteW90aXMuY29tL2NlcnRpZ25hcm9vdGNhLmNybDANBgkq
+hkiG9w0BAQsFAAOCAgEAlLieT/DjlQgi581oQfccVdV8AOItOoldaDgvUSILSo3L
+6btdPrtcPbEo/uRTVRPPoZAbAh1fZkYJMyjhDSSXcNMQH+pkV5a7XdrnxIxPTGRG
+HVyH41neQtGbqH6mid2PHMkwgu07nM3A6RngatgCdTer9zQoKJHyBApPNeNgJgH6
+0BGM+RFq7q89w1DTj18zeTyGqHNFkIwgtnJzFyO+B2XleJINugHA64wcZr+shncB
+lA2c5uk5jR+mUYyZDDl34bSb+hxnV29qao6pK0xXeXpXIs/NX2NGjVxZOob4Mkdi
+o2cNGJHc+6Zr9UhhcyNZjgKnvETq9Emd8VRY+WCv2hikLyhF3HqgiIZd8zvn/yk1
+gPxkQ5Tm4xxvvq0OKmOZK8l+hfZx6AYDlf7ej0gcWtSS6Cvu5zHbugRqh5jnxV/v
+faci9wHYTfmJ0A6aBVmknpjZbyvKcL5kwlWj9Omvw5Ip3IgWJJk8jSaYtlu3zM63
+Nwf9JtmYhST/WSMDmu2dnajkXjjO11INb9I/bbEFa0nOipFGc/T2L/Coc3cOZayh
+jWZSaX5LaAzHHjcng6WMxwLkFM1JAbBzs/3GkDpv0mztO+7skb6iQ12LAEpmJURw
+3kAP+HwV96LOPNdeE4yBFxgX0b3xdxA61GU5wSesVywlVP+i2k+KYTlerj1KjL0=
+-----END CERTIFICATE-----
diff --git a/pipenv/patched/notpip/_vendor/certifi/core.py b/pipenv/patched/notpip/_vendor/certifi/core.py
index eab9d1d1..2d02ea44 100644
--- a/pipenv/patched/notpip/_vendor/certifi/core.py
+++ b/pipenv/patched/notpip/_vendor/certifi/core.py
@@ -8,14 +8,6 @@ certifi.py
This module returns the installation location of cacert.pem.
"""
import os
-import warnings
-
-
-class DeprecatedBundleWarning(DeprecationWarning):
- """
- The weak security bundle is being deprecated. Please bother your service
- provider to get them to stop using cross-signed roots.
- """
def where():
@@ -24,14 +16,5 @@ def where():
return os.path.join(f, 'cacert.pem')
-def old_where():
- warnings.warn(
- "The weak security bundle has been removed. certifi.old_where() is now an alias "
- "of certifi.where(). Please update your code to use certifi.where() instead. "
- "certifi.old_where() will be removed in 2018.",
- DeprecatedBundleWarning
- )
- return where()
-
if __name__ == '__main__':
print(where())
diff --git a/pipenv/patched/notpip/_vendor/colorama/LICENSE.txt b/pipenv/patched/notpip/_vendor/colorama/LICENSE.txt
index 5f567799..3105888e 100644
--- a/pipenv/patched/notpip/_vendor/colorama/LICENSE.txt
+++ b/pipenv/patched/notpip/_vendor/colorama/LICENSE.txt
@@ -25,4 +25,3 @@ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
diff --git a/pipenv/patched/notpip/_vendor/colorama/__init__.py b/pipenv/patched/notpip/_vendor/colorama/__init__.py
index f4d9ce21..2a3bf471 100644
--- a/pipenv/patched/notpip/_vendor/colorama/__init__.py
+++ b/pipenv/patched/notpip/_vendor/colorama/__init__.py
@@ -3,5 +3,4 @@ from .initialise import init, deinit, reinit, colorama_text
from .ansi import Fore, Back, Style, Cursor
from .ansitowin32 import AnsiToWin32
-__version__ = '0.3.9'
-
+__version__ = '0.4.1'
diff --git a/pipenv/patched/notpip/_vendor/colorama/ansitowin32.py b/pipenv/patched/notpip/_vendor/colorama/ansitowin32.py
index 1d6e6059..359c92be 100644
--- a/pipenv/patched/notpip/_vendor/colorama/ansitowin32.py
+++ b/pipenv/patched/notpip/_vendor/colorama/ansitowin32.py
@@ -13,14 +13,6 @@ if windll is not None:
winterm = WinTerm()
-def is_stream_closed(stream):
- return not hasattr(stream, 'closed') or stream.closed
-
-
-def is_a_tty(stream):
- return hasattr(stream, 'isatty') and stream.isatty()
-
-
class StreamWrapper(object):
'''
Wraps a stream (such as stdout), acting as a transparent proxy for all
@@ -36,9 +28,38 @@ class StreamWrapper(object):
def __getattr__(self, name):
return getattr(self.__wrapped, name)
+ def __enter__(self, *args, **kwargs):
+ # special method lookup bypasses __getattr__/__getattribute__, see
+ # https://stackoverflow.com/questions/12632894/why-doesnt-getattr-work-with-exit
+ # thus, contextlib magic methods are not proxied via __getattr__
+ return self.__wrapped.__enter__(*args, **kwargs)
+
+ def __exit__(self, *args, **kwargs):
+ return self.__wrapped.__exit__(*args, **kwargs)
+
def write(self, text):
self.__convertor.write(text)
+ def isatty(self):
+ stream = self.__wrapped
+ if 'PYCHARM_HOSTED' in os.environ:
+ if stream is not None and (stream is sys.__stdout__ or stream is sys.__stderr__):
+ return True
+ try:
+ stream_isatty = stream.isatty
+ except AttributeError:
+ return False
+ else:
+ return stream_isatty()
+
+ @property
+ def closed(self):
+ stream = self.__wrapped
+ try:
+ return stream.closed
+ except AttributeError:
+ return True
+
class AnsiToWin32(object):
'''
@@ -68,12 +89,12 @@ class AnsiToWin32(object):
# should we strip ANSI sequences from our output?
if strip is None:
- strip = conversion_supported or (not is_stream_closed(wrapped) and not is_a_tty(wrapped))
+ strip = conversion_supported or (not self.stream.closed and not self.stream.isatty())
self.strip = strip
# should we should convert ANSI sequences into win32 calls?
if convert is None:
- convert = conversion_supported and not is_stream_closed(wrapped) and is_a_tty(wrapped)
+ convert = conversion_supported and not self.stream.closed and self.stream.isatty()
self.convert = convert
# dict of ansi codes to win32 functions and parameters
@@ -149,7 +170,7 @@ class AnsiToWin32(object):
def reset_all(self):
if self.convert:
self.call_win32('m', (0,))
- elif not self.strip and not is_stream_closed(self.wrapped):
+ elif not self.strip and not self.stream.closed:
self.wrapped.write(Style.RESET_ALL)
diff --git a/pipenv/patched/notpip/_vendor/colorama/initialise.py b/pipenv/patched/notpip/_vendor/colorama/initialise.py
index 834962a3..430d0668 100644
--- a/pipenv/patched/notpip/_vendor/colorama/initialise.py
+++ b/pipenv/patched/notpip/_vendor/colorama/initialise.py
@@ -78,5 +78,3 @@ def wrap_stream(stream, convert, strip, autoreset, wrap):
if wrapper.should_wrap():
stream = wrapper.stream
return stream
-
-
diff --git a/pipenv/patched/notpip/_vendor/colorama/win32.py b/pipenv/patched/notpip/_vendor/colorama/win32.py
index 8262e350..c2d83603 100644
--- a/pipenv/patched/notpip/_vendor/colorama/win32.py
+++ b/pipenv/patched/notpip/_vendor/colorama/win32.py
@@ -89,11 +89,6 @@ else:
]
_SetConsoleTitleW.restype = wintypes.BOOL
- handles = {
- STDOUT: _GetStdHandle(STDOUT),
- STDERR: _GetStdHandle(STDERR),
- }
-
def _winapi_test(handle):
csbi = CONSOLE_SCREEN_BUFFER_INFO()
success = _GetConsoleScreenBufferInfo(
@@ -101,17 +96,18 @@ else:
return bool(success)
def winapi_test():
- return any(_winapi_test(h) for h in handles.values())
+ return any(_winapi_test(h) for h in
+ (_GetStdHandle(STDOUT), _GetStdHandle(STDERR)))
def GetConsoleScreenBufferInfo(stream_id=STDOUT):
- handle = handles[stream_id]
+ handle = _GetStdHandle(stream_id)
csbi = CONSOLE_SCREEN_BUFFER_INFO()
success = _GetConsoleScreenBufferInfo(
handle, byref(csbi))
return csbi
def SetConsoleTextAttribute(stream_id, attrs):
- handle = handles[stream_id]
+ handle = _GetStdHandle(stream_id)
return _SetConsoleTextAttribute(handle, attrs)
def SetConsoleCursorPosition(stream_id, position, adjust=True):
@@ -129,11 +125,11 @@ else:
adjusted_position.Y += sr.Top
adjusted_position.X += sr.Left
# Resume normal processing
- handle = handles[stream_id]
+ handle = _GetStdHandle(stream_id)
return _SetConsoleCursorPosition(handle, adjusted_position)
def FillConsoleOutputCharacter(stream_id, char, length, start):
- handle = handles[stream_id]
+ handle = _GetStdHandle(stream_id)
char = c_char(char.encode())
length = wintypes.DWORD(length)
num_written = wintypes.DWORD(0)
@@ -144,7 +140,7 @@ else:
def FillConsoleOutputAttribute(stream_id, attr, length, start):
''' FillConsoleOutputAttribute( hConsole, csbi.wAttributes, dwConSize, coordScreen, &cCharsWritten )'''
- handle = handles[stream_id]
+ handle = _GetStdHandle(stream_id)
attribute = wintypes.WORD(attr)
length = wintypes.DWORD(length)
num_written = wintypes.DWORD(0)
diff --git a/pipenv/patched/notpip/_vendor/colorama/winterm.py b/pipenv/patched/notpip/_vendor/colorama/winterm.py
index 60309d3c..0fdb4ec4 100644
--- a/pipenv/patched/notpip/_vendor/colorama/winterm.py
+++ b/pipenv/patched/notpip/_vendor/colorama/winterm.py
@@ -44,6 +44,7 @@ class WinTerm(object):
def reset_all(self, on_stderr=None):
self.set_attrs(self._default)
self.set_console(attrs=self._default)
+ self._light = 0
def fore(self, fore=None, light=False, on_stderr=False):
if fore is None:
@@ -122,12 +123,15 @@ class WinTerm(object):
if mode == 0:
from_coord = csbi.dwCursorPosition
cells_to_erase = cells_in_screen - cells_before_cursor
- if mode == 1:
+ elif mode == 1:
from_coord = win32.COORD(0, 0)
cells_to_erase = cells_before_cursor
elif mode == 2:
from_coord = win32.COORD(0, 0)
cells_to_erase = cells_in_screen
+ else:
+ # invalid mode
+ return
# fill the entire screen with blanks
win32.FillConsoleOutputCharacter(handle, ' ', cells_to_erase, from_coord)
# now set the buffer's attributes accordingly
@@ -147,12 +151,15 @@ class WinTerm(object):
if mode == 0:
from_coord = csbi.dwCursorPosition
cells_to_erase = csbi.dwSize.X - csbi.dwCursorPosition.X
- if mode == 1:
+ elif mode == 1:
from_coord = win32.COORD(0, csbi.dwCursorPosition.Y)
cells_to_erase = csbi.dwCursorPosition.X
elif mode == 2:
from_coord = win32.COORD(0, csbi.dwCursorPosition.Y)
cells_to_erase = csbi.dwSize.X
+ else:
+ # invalid mode
+ return
# fill the entire screen with blanks
win32.FillConsoleOutputCharacter(handle, ' ', cells_to_erase, from_coord)
# now set the buffer's attributes accordingly
diff --git a/pipenv/patched/notpip/_vendor/distlib/__init__.py b/pipenv/patched/notpip/_vendor/distlib/__init__.py
index d4aab453..a786b4d3 100644
--- a/pipenv/patched/notpip/_vendor/distlib/__init__.py
+++ b/pipenv/patched/notpip/_vendor/distlib/__init__.py
@@ -6,7 +6,7 @@
#
import logging
-__version__ = '0.2.7'
+__version__ = '0.2.8'
class DistlibException(Exception):
pass
diff --git a/pipenv/patched/notpip/_vendor/distlib/database.py b/pipenv/patched/notpip/_vendor/distlib/database.py
index a19905e2..b13cdac9 100644
--- a/pipenv/patched/notpip/_vendor/distlib/database.py
+++ b/pipenv/patched/notpip/_vendor/distlib/database.py
@@ -20,7 +20,8 @@ import zipimport
from . import DistlibException, resources
from .compat import StringIO
from .version import get_scheme, UnsupportedVersionError
-from .metadata import Metadata, METADATA_FILENAME, WHEEL_METADATA_FILENAME
+from .metadata import (Metadata, METADATA_FILENAME, WHEEL_METADATA_FILENAME,
+ LEGACY_METADATA_FILENAME)
from .util import (parse_requirement, cached_property, parse_name_and_version,
read_exports, write_exports, CSVReader, CSVWriter)
@@ -132,7 +133,9 @@ class DistributionPath(object):
if not r or r.path in seen:
continue
if self._include_dist and entry.endswith(DISTINFO_EXT):
- possible_filenames = [METADATA_FILENAME, WHEEL_METADATA_FILENAME]
+ possible_filenames = [METADATA_FILENAME,
+ WHEEL_METADATA_FILENAME,
+ LEGACY_METADATA_FILENAME]
for metadata_filename in possible_filenames:
metadata_path = posixpath.join(entry, metadata_filename)
pydist = finder.find(metadata_path)
diff --git a/pipenv/patched/notpip/_vendor/distlib/locators.py b/pipenv/patched/notpip/_vendor/distlib/locators.py
index cb05b184..a7ed9469 100644
--- a/pipenv/patched/notpip/_vendor/distlib/locators.py
+++ b/pipenv/patched/notpip/_vendor/distlib/locators.py
@@ -255,7 +255,9 @@ class Locator(object):
if path.endswith('.whl'):
try:
wheel = Wheel(path)
- if is_compatible(wheel, self.wheel_tags):
+ if not is_compatible(wheel, self.wheel_tags):
+ logger.debug('Wheel not compatible: %s', path)
+ else:
if project_name is None:
include = True
else:
@@ -613,6 +615,7 @@ class SimpleScrapingLocator(Locator):
# as it is for coordinating our internal threads - the ones created
# in _prepare_threads.
self._gplock = threading.RLock()
+ self.platform_check = False # See issue #112
def _prepare_threads(self):
"""
@@ -658,8 +661,8 @@ class SimpleScrapingLocator(Locator):
del self.result
return result
- platform_dependent = re.compile(r'\b(linux-(i\d86|x86_64|arm\w+)|'
- r'win(32|-amd64)|macosx-?\d+)\b', re.I)
+ platform_dependent = re.compile(r'\b(linux_(i\d86|x86_64|arm\w+)|'
+ r'win(32|_amd64)|macosx_?\d+)\b', re.I)
def _is_platform_dependent(self, url):
"""
@@ -677,7 +680,7 @@ class SimpleScrapingLocator(Locator):
Note that the return value isn't actually used other than as a boolean
value.
"""
- if self._is_platform_dependent(url):
+ if self.platform_check and self._is_platform_dependent(url):
info = None
else:
info = self.convert_url_to_download_info(url, self.project_name)
diff --git a/pipenv/patched/notpip/_vendor/distlib/metadata.py b/pipenv/patched/notpip/_vendor/distlib/metadata.py
index 6d6470ff..77eed7f9 100644
--- a/pipenv/patched/notpip/_vendor/distlib/metadata.py
+++ b/pipenv/patched/notpip/_vendor/distlib/metadata.py
@@ -91,7 +91,9 @@ _426_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform',
_426_MARKERS = ('Private-Version', 'Provides-Extra', 'Obsoleted-By',
'Setup-Requires-Dist', 'Extension')
-_566_FIELDS = _426_FIELDS + ('Description-Content-Type',)
+# See issue #106: Sometimes 'Requires' occurs wrongly in the metadata. Include
+# it in the tuple literal below to allow it (for now)
+_566_FIELDS = _426_FIELDS + ('Description-Content-Type', 'Requires')
_566_MARKERS = ('Description-Content-Type',)
@@ -377,8 +379,8 @@ class LegacyMetadata(object):
value = msg[field]
if value is not None and value != 'UNKNOWN':
self.set(field, value)
- logger.debug('Attempting to set metadata for %s', self)
- self.set_metadata_version()
+ # logger.debug('Attempting to set metadata for %s', self)
+ # self.set_metadata_version()
def write(self, filepath, skip_unknown=False):
"""Write the metadata fields to filepath."""
@@ -648,6 +650,7 @@ class LegacyMetadata(object):
METADATA_FILENAME = 'pydist.json'
WHEEL_METADATA_FILENAME = 'metadata.json'
+LEGACY_METADATA_FILENAME = 'METADATA'
class Metadata(object):
diff --git a/pipenv/patched/notpip/_vendor/distlib/scripts.py b/pipenv/patched/notpip/_vendor/distlib/scripts.py
index 0b7c3d0b..8e22cb91 100644
--- a/pipenv/patched/notpip/_vendor/distlib/scripts.py
+++ b/pipenv/patched/notpip/_vendor/distlib/scripts.py
@@ -236,8 +236,10 @@ class ScriptMaker(object):
def _write_script(self, names, shebang, script_bytes, filenames, ext):
use_launcher = self.add_launchers and self._is_nt
linesep = os.linesep.encode('utf-8')
+ if not shebang.endswith(linesep):
+ shebang += linesep
if not use_launcher:
- script_bytes = shebang + linesep + script_bytes
+ script_bytes = shebang + script_bytes
else: # pragma: no cover
if ext == 'py':
launcher = self._get_launcher('t')
@@ -247,7 +249,7 @@ class ScriptMaker(object):
with ZipFile(stream, 'w') as zf:
zf.writestr('__main__.py', script_bytes)
zip_data = stream.getvalue()
- script_bytes = launcher + shebang + linesep + zip_data
+ script_bytes = launcher + shebang + zip_data
for name in names:
outname = os.path.join(self.target_dir, name)
if use_launcher: # pragma: no cover
diff --git a/pipenv/patched/notpip/_vendor/distlib/util.py b/pipenv/patched/notpip/_vendor/distlib/util.py
index 0b14a93b..9d4bfd3b 100644
--- a/pipenv/patched/notpip/_vendor/distlib/util.py
+++ b/pipenv/patched/notpip/_vendor/distlib/util.py
@@ -545,16 +545,14 @@ class FileOperator(object):
def write_binary_file(self, path, data):
self.ensure_dir(os.path.dirname(path))
if not self.dry_run:
+ if os.path.exists(path):
+ os.remove(path)
with open(path, 'wb') as f:
f.write(data)
self.record_as_written(path)
def write_text_file(self, path, data, encoding):
- self.ensure_dir(os.path.dirname(path))
- if not self.dry_run:
- with open(path, 'wb') as f:
- f.write(data.encode(encoding))
- self.record_as_written(path)
+ self.write_binary_file(path, data.encode(encoding))
def set_mode(self, bits, mask, files):
if os.name == 'posix' or (os.name == 'java' and os._name == 'posix'):
@@ -582,7 +580,7 @@ class FileOperator(object):
if self.record:
self.dirs_created.add(path)
- def byte_compile(self, path, optimize=False, force=False, prefix=None):
+ def byte_compile(self, path, optimize=False, force=False, prefix=None, hashed_invalidation=False):
dpath = cache_from_source(path, not optimize)
logger.info('Byte-compiling %s to %s', path, dpath)
if not self.dry_run:
@@ -592,7 +590,10 @@ class FileOperator(object):
else:
assert path.startswith(prefix)
diagpath = path[len(prefix):]
- py_compile.compile(path, dpath, diagpath, True) # raise error
+ compile_kwargs = {}
+ if hashed_invalidation and hasattr(py_compile, 'PycInvalidationMode'):
+ compile_kwargs['invalidation_mode'] = py_compile.PycInvalidationMode.CHECKED_HASH
+ py_compile.compile(path, dpath, diagpath, True, **compile_kwargs) # raise error
self.record_as_written(dpath)
return dpath
diff --git a/pipenv/patched/notpip/_vendor/distlib/wheel.py b/pipenv/patched/notpip/_vendor/distlib/wheel.py
index 77372235..b04bfaef 100644
--- a/pipenv/patched/notpip/_vendor/distlib/wheel.py
+++ b/pipenv/patched/notpip/_vendor/distlib/wheel.py
@@ -442,7 +442,9 @@ class Wheel(object):
This can be used to issue any warnings to raise any exceptions.
If kwarg ``lib_only`` is True, only the purelib/platlib files are
installed, and the headers, scripts, data and dist-info metadata are
- not written.
+ not written. If kwarg ``bytecode_hashed_invalidation`` is True, written
+ bytecode will try to use file-hash based invalidation (PEP-552) on
+ supported interpreter versions (CPython 2.7+).
The return value is a :class:`InstalledDistribution` instance unless
``options.lib_only`` is True, in which case the return value is ``None``.
@@ -451,6 +453,7 @@ class Wheel(object):
dry_run = maker.dry_run
warner = kwargs.get('warner')
lib_only = kwargs.get('lib_only', False)
+ bc_hashed_invalidation = kwargs.get('bytecode_hashed_invalidation', False)
pathname = os.path.join(self.dirname, self.filename)
name_ver = '%s-%s' % (self.name, self.version)
@@ -557,7 +560,8 @@ class Wheel(object):
'%s' % outfile)
if bc and outfile.endswith('.py'):
try:
- pyc = fileop.byte_compile(outfile)
+ pyc = fileop.byte_compile(outfile,
+ hashed_invalidation=bc_hashed_invalidation)
outfiles.append(pyc)
except Exception:
# Don't give up if byte-compilation fails,
diff --git a/pipenv/patched/notpip/_vendor/idna/core.py b/pipenv/patched/notpip/_vendor/idna/core.py
index 090c2c18..104624ad 100644
--- a/pipenv/patched/notpip/_vendor/idna/core.py
+++ b/pipenv/patched/notpip/_vendor/idna/core.py
@@ -267,10 +267,7 @@ def alabel(label):
try:
label = label.encode('ascii')
- try:
- ulabel(label)
- except IDNAError:
- raise IDNAError('The label {0} is not a valid A-label'.format(label))
+ ulabel(label)
if not valid_label_length(label):
raise IDNAError('Label too long')
return label
diff --git a/pipenv/patched/notpip/_vendor/idna/idnadata.py b/pipenv/patched/notpip/_vendor/idna/idnadata.py
index 17974e23..a80c959d 100644
--- a/pipenv/patched/notpip/_vendor/idna/idnadata.py
+++ b/pipenv/patched/notpip/_vendor/idna/idnadata.py
@@ -1,6 +1,6 @@
# This file is automatically generated by tools/idna-data
-__version__ = "10.0.0"
+__version__ = "11.0.0"
scripts = {
'Greek': (
0x37000000374,
@@ -49,7 +49,7 @@ scripts = {
0x30210000302a,
0x30380000303c,
0x340000004db6,
- 0x4e0000009feb,
+ 0x4e0000009ff0,
0xf9000000fa6e,
0xfa700000fada,
0x200000002a6d7,
@@ -62,7 +62,7 @@ scripts = {
'Hebrew': (
0x591000005c8,
0x5d0000005eb,
- 0x5f0000005f5,
+ 0x5ef000005f5,
0xfb1d0000fb37,
0xfb380000fb3d,
0xfb3e0000fb3f,
@@ -248,6 +248,7 @@ joining_types = {
0x6fb: 68,
0x6fc: 68,
0x6ff: 68,
+ 0x70f: 84,
0x710: 82,
0x712: 68,
0x713: 68,
@@ -522,6 +523,7 @@ joining_types = {
0x1875: 68,
0x1876: 68,
0x1877: 68,
+ 0x1878: 68,
0x1880: 85,
0x1881: 85,
0x1882: 85,
@@ -690,6 +692,70 @@ joining_types = {
0x10bad: 68,
0x10bae: 68,
0x10baf: 85,
+ 0x10d00: 76,
+ 0x10d01: 68,
+ 0x10d02: 68,
+ 0x10d03: 68,
+ 0x10d04: 68,
+ 0x10d05: 68,
+ 0x10d06: 68,
+ 0x10d07: 68,
+ 0x10d08: 68,
+ 0x10d09: 68,
+ 0x10d0a: 68,
+ 0x10d0b: 68,
+ 0x10d0c: 68,
+ 0x10d0d: 68,
+ 0x10d0e: 68,
+ 0x10d0f: 68,
+ 0x10d10: 68,
+ 0x10d11: 68,
+ 0x10d12: 68,
+ 0x10d13: 68,
+ 0x10d14: 68,
+ 0x10d15: 68,
+ 0x10d16: 68,
+ 0x10d17: 68,
+ 0x10d18: 68,
+ 0x10d19: 68,
+ 0x10d1a: 68,
+ 0x10d1b: 68,
+ 0x10d1c: 68,
+ 0x10d1d: 68,
+ 0x10d1e: 68,
+ 0x10d1f: 68,
+ 0x10d20: 68,
+ 0x10d21: 68,
+ 0x10d22: 82,
+ 0x10d23: 68,
+ 0x10f30: 68,
+ 0x10f31: 68,
+ 0x10f32: 68,
+ 0x10f33: 82,
+ 0x10f34: 68,
+ 0x10f35: 68,
+ 0x10f36: 68,
+ 0x10f37: 68,
+ 0x10f38: 68,
+ 0x10f39: 68,
+ 0x10f3a: 68,
+ 0x10f3b: 68,
+ 0x10f3c: 68,
+ 0x10f3d: 68,
+ 0x10f3e: 68,
+ 0x10f3f: 68,
+ 0x10f40: 68,
+ 0x10f41: 68,
+ 0x10f42: 68,
+ 0x10f43: 68,
+ 0x10f44: 68,
+ 0x10f45: 85,
+ 0x10f51: 68,
+ 0x10f52: 68,
+ 0x10f53: 68,
+ 0x10f54: 82,
+ 0x110bd: 85,
+ 0x110cd: 85,
0x1e900: 68,
0x1e901: 68,
0x1e902: 68,
@@ -1034,14 +1100,15 @@ codepoint_classes = {
0x52d0000052e,
0x52f00000530,
0x5590000055a,
- 0x56100000587,
+ 0x56000000587,
+ 0x58800000589,
0x591000005be,
0x5bf000005c0,
0x5c1000005c3,
0x5c4000005c6,
0x5c7000005c8,
0x5d0000005eb,
- 0x5f0000005f3,
+ 0x5ef000005f3,
0x6100000061b,
0x62000000640,
0x64100000660,
@@ -1054,12 +1121,13 @@ codepoint_classes = {
0x7100000074b,
0x74d000007b2,
0x7c0000007f6,
+ 0x7fd000007fe,
0x8000000082e,
0x8400000085c,
0x8600000086b,
0x8a0000008b5,
0x8b6000008be,
- 0x8d4000008e2,
+ 0x8d3000008e2,
0x8e300000958,
0x96000000964,
0x96600000970,
@@ -1077,6 +1145,7 @@ codepoint_classes = {
0x9e0000009e4,
0x9e6000009f2,
0x9fc000009fd,
+ 0x9fe000009ff,
0xa0100000a04,
0xa0500000a0b,
0xa0f00000a11,
@@ -1136,8 +1205,7 @@ codepoint_classes = {
0xbd000000bd1,
0xbd700000bd8,
0xbe600000bf0,
- 0xc0000000c04,
- 0xc0500000c0d,
+ 0xc0000000c0d,
0xc0e00000c11,
0xc1200000c29,
0xc2a00000c3a,
@@ -1276,7 +1344,7 @@ codepoint_classes = {
0x17dc000017de,
0x17e0000017ea,
0x18100000181a,
- 0x182000001878,
+ 0x182000001879,
0x1880000018ab,
0x18b0000018f6,
0x19000000191f,
@@ -1544,11 +1612,11 @@ codepoint_classes = {
0x309d0000309f,
0x30a1000030fb,
0x30fc000030ff,
- 0x31050000312f,
+ 0x310500003130,
0x31a0000031bb,
0x31f000003200,
0x340000004db6,
- 0x4e0000009feb,
+ 0x4e0000009ff0,
0xa0000000a48d,
0xa4d00000a4fe,
0xa5000000a60d,
@@ -1655,8 +1723,10 @@ codepoint_classes = {
0xa7a50000a7a6,
0xa7a70000a7a8,
0xa7a90000a7aa,
+ 0xa7af0000a7b0,
0xa7b50000a7b6,
0xa7b70000a7b8,
+ 0xa7b90000a7ba,
0xa7f70000a7f8,
0xa7fa0000a828,
0xa8400000a874,
@@ -1664,8 +1734,7 @@ codepoint_classes = {
0xa8d00000a8da,
0xa8e00000a8f8,
0xa8fb0000a8fc,
- 0xa8fd0000a8fe,
- 0xa9000000a92e,
+ 0xa8fd0000a92e,
0xa9300000a954,
0xa9800000a9c1,
0xa9cf0000a9da,
@@ -1743,7 +1812,7 @@ codepoint_classes = {
0x10a0500010a07,
0x10a0c00010a14,
0x10a1500010a18,
- 0x10a1900010a34,
+ 0x10a1900010a36,
0x10a3800010a3b,
0x10a3f00010a40,
0x10a6000010a7d,
@@ -1756,6 +1825,11 @@ codepoint_classes = {
0x10b8000010b92,
0x10c0000010c49,
0x10cc000010cf3,
+ 0x10d0000010d28,
+ 0x10d3000010d3a,
+ 0x10f0000010f1d,
+ 0x10f2700010f28,
+ 0x10f3000010f51,
0x1100000011047,
0x1106600011070,
0x1107f000110bb,
@@ -1763,10 +1837,11 @@ codepoint_classes = {
0x110f0000110fa,
0x1110000011135,
0x1113600011140,
+ 0x1114400011147,
0x1115000011174,
0x1117600011177,
0x11180000111c5,
- 0x111ca000111cd,
+ 0x111c9000111cd,
0x111d0000111db,
0x111dc000111dd,
0x1120000011212,
@@ -1786,7 +1861,7 @@ codepoint_classes = {
0x1132a00011331,
0x1133200011334,
0x113350001133a,
- 0x1133c00011345,
+ 0x1133b00011345,
0x1134700011349,
0x1134b0001134e,
0x1135000011351,
@@ -1796,6 +1871,7 @@ codepoint_classes = {
0x1137000011375,
0x114000001144b,
0x114500001145a,
+ 0x1145e0001145f,
0x11480000114c6,
0x114c7000114c8,
0x114d0000114da,
@@ -1807,15 +1883,17 @@ codepoint_classes = {
0x116500001165a,
0x11680000116b8,
0x116c0000116ca,
- 0x117000001171a,
+ 0x117000001171b,
0x1171d0001172c,
0x117300001173a,
+ 0x118000001183b,
0x118c0000118ea,
0x118ff00011900,
0x11a0000011a3f,
0x11a4700011a48,
0x11a5000011a84,
0x11a8600011a9a,
+ 0x11a9d00011a9e,
0x11ac000011af9,
0x11c0000011c09,
0x11c0a00011c37,
@@ -1831,6 +1909,13 @@ codepoint_classes = {
0x11d3c00011d3e,
0x11d3f00011d48,
0x11d5000011d5a,
+ 0x11d6000011d66,
+ 0x11d6700011d69,
+ 0x11d6a00011d8f,
+ 0x11d9000011d92,
+ 0x11d9300011d99,
+ 0x11da000011daa,
+ 0x11ee000011ef7,
0x120000001239a,
0x1248000012544,
0x130000001342f,
@@ -1845,11 +1930,12 @@ codepoint_classes = {
0x16b5000016b5a,
0x16b6300016b78,
0x16b7d00016b90,
+ 0x16e6000016e80,
0x16f0000016f45,
0x16f5000016f7f,
0x16f8f00016fa0,
0x16fe000016fe2,
- 0x17000000187ed,
+ 0x17000000187f2,
0x1880000018af3,
0x1b0000001b11f,
0x1b1700001b2fc,
diff --git a/pipenv/patched/notpip/_vendor/idna/package_data.py b/pipenv/patched/notpip/_vendor/idna/package_data.py
index 39c192ba..257e8989 100644
--- a/pipenv/patched/notpip/_vendor/idna/package_data.py
+++ b/pipenv/patched/notpip/_vendor/idna/package_data.py
@@ -1,2 +1,2 @@
-__version__ = '2.7'
+__version__ = '2.8'
diff --git a/pipenv/patched/notpip/_vendor/idna/uts46data.py b/pipenv/patched/notpip/_vendor/idna/uts46data.py
index 79731cb9..a68ed4c0 100644
--- a/pipenv/patched/notpip/_vendor/idna/uts46data.py
+++ b/pipenv/patched/notpip/_vendor/idna/uts46data.py
@@ -4,7 +4,7 @@
"""IDNA Mapping Table from UTS46."""
-__version__ = "10.0.0"
+__version__ = "11.0.0"
def _seg_0():
return [
(0x0, '3'),
@@ -1029,11 +1029,8 @@ def _seg_9():
(0x556, 'M', u'ֆ'),
(0x557, 'X'),
(0x559, 'V'),
- (0x560, 'X'),
- (0x561, 'V'),
(0x587, 'M', u'եւ'),
- (0x588, 'X'),
- (0x589, 'V'),
+ (0x588, 'V'),
(0x58B, 'X'),
(0x58D, 'V'),
(0x590, 'X'),
@@ -1041,15 +1038,15 @@ def _seg_9():
(0x5C8, 'X'),
(0x5D0, 'V'),
(0x5EB, 'X'),
- (0x5F0, 'V'),
+ (0x5EF, 'V'),
(0x5F5, 'X'),
+ (0x606, 'V'),
+ (0x61C, 'X'),
+ (0x61E, 'V'),
]
def _seg_10():
return [
- (0x606, 'V'),
- (0x61C, 'X'),
- (0x61E, 'V'),
(0x675, 'M', u'اٴ'),
(0x676, 'M', u'وٴ'),
(0x677, 'M', u'ۇٴ'),
@@ -1064,7 +1061,7 @@ def _seg_10():
(0x7B2, 'X'),
(0x7C0, 'V'),
(0x7FB, 'X'),
- (0x800, 'V'),
+ (0x7FD, 'V'),
(0x82E, 'X'),
(0x830, 'V'),
(0x83F, 'X'),
@@ -1078,7 +1075,7 @@ def _seg_10():
(0x8B5, 'X'),
(0x8B6, 'V'),
(0x8BE, 'X'),
- (0x8D4, 'V'),
+ (0x8D3, 'V'),
(0x8E2, 'X'),
(0x8E3, 'V'),
(0x958, 'M', u'क़'),
@@ -1118,7 +1115,7 @@ def _seg_10():
(0x9E0, 'V'),
(0x9E4, 'X'),
(0x9E6, 'V'),
- (0x9FE, 'X'),
+ (0x9FF, 'X'),
(0xA01, 'V'),
(0xA04, 'X'),
(0xA05, 'V'),
@@ -1147,19 +1144,19 @@ def _seg_10():
(0xA4E, 'X'),
(0xA51, 'V'),
(0xA52, 'X'),
+ (0xA59, 'M', u'ਖ਼'),
+ (0xA5A, 'M', u'ਗ਼'),
+ (0xA5B, 'M', u'ਜ਼'),
]
def _seg_11():
return [
- (0xA59, 'M', u'ਖ਼'),
- (0xA5A, 'M', u'ਗ਼'),
- (0xA5B, 'M', u'ਜ਼'),
(0xA5C, 'V'),
(0xA5D, 'X'),
(0xA5E, 'M', u'ਫ਼'),
(0xA5F, 'X'),
(0xA66, 'V'),
- (0xA76, 'X'),
+ (0xA77, 'X'),
(0xA81, 'V'),
(0xA84, 'X'),
(0xA85, 'V'),
@@ -1250,16 +1247,14 @@ def _seg_11():
(0xBE6, 'V'),
(0xBFB, 'X'),
(0xC00, 'V'),
- (0xC04, 'X'),
- ]
-
-def _seg_12():
- return [
- (0xC05, 'V'),
(0xC0D, 'X'),
(0xC0E, 'V'),
(0xC11, 'X'),
(0xC12, 'V'),
+ ]
+
+def _seg_12():
+ return [
(0xC29, 'X'),
(0xC2A, 'V'),
(0xC3A, 'X'),
@@ -1278,8 +1273,6 @@ def _seg_12():
(0xC66, 'V'),
(0xC70, 'X'),
(0xC78, 'V'),
- (0xC84, 'X'),
- (0xC85, 'V'),
(0xC8D, 'X'),
(0xC8E, 'V'),
(0xC91, 'X'),
@@ -1355,10 +1348,6 @@ def _seg_12():
(0xE83, 'X'),
(0xE84, 'V'),
(0xE85, 'X'),
- ]
-
-def _seg_13():
- return [
(0xE87, 'V'),
(0xE89, 'X'),
(0xE8A, 'V'),
@@ -1366,6 +1355,10 @@ def _seg_13():
(0xE8D, 'V'),
(0xE8E, 'X'),
(0xE94, 'V'),
+ ]
+
+def _seg_13():
+ return [
(0xE98, 'X'),
(0xE99, 'V'),
(0xEA0, 'X'),
@@ -1459,10 +1452,6 @@ def _seg_13():
(0x124E, 'X'),
(0x1250, 'V'),
(0x1257, 'X'),
- ]
-
-def _seg_14():
- return [
(0x1258, 'V'),
(0x1259, 'X'),
(0x125A, 'V'),
@@ -1470,6 +1459,10 @@ def _seg_14():
(0x1260, 'V'),
(0x1289, 'X'),
(0x128A, 'V'),
+ ]
+
+def _seg_14():
+ return [
(0x128E, 'X'),
(0x1290, 'V'),
(0x12B1, 'X'),
@@ -1538,7 +1531,7 @@ def _seg_14():
(0x1810, 'V'),
(0x181A, 'X'),
(0x1820, 'V'),
- (0x1878, 'X'),
+ (0x1879, 'X'),
(0x1880, 'V'),
(0x18AB, 'X'),
(0x18B0, 'V'),
@@ -1563,10 +1556,6 @@ def _seg_14():
(0x19DB, 'X'),
(0x19DE, 'V'),
(0x1A1C, 'X'),
- ]
-
-def _seg_15():
- return [
(0x1A1E, 'V'),
(0x1A5F, 'X'),
(0x1A60, 'V'),
@@ -1574,6 +1563,10 @@ def _seg_15():
(0x1A7F, 'V'),
(0x1A8A, 'X'),
(0x1A90, 'V'),
+ ]
+
+def _seg_15():
+ return [
(0x1A9A, 'X'),
(0x1AA0, 'V'),
(0x1AAE, 'X'),
@@ -1667,10 +1660,6 @@ def _seg_15():
(0x1D68, 'M', u'ρ'),
(0x1D69, 'M', u'φ'),
(0x1D6A, 'M', u'χ'),
- ]
-
-def _seg_16():
- return [
(0x1D6B, 'V'),
(0x1D78, 'M', u'н'),
(0x1D79, 'V'),
@@ -1678,6 +1667,10 @@ def _seg_16():
(0x1D9C, 'M', u'c'),
(0x1D9D, 'M', u'ɕ'),
(0x1D9E, 'M', u'ð'),
+ ]
+
+def _seg_16():
+ return [
(0x1D9F, 'M', u'ɜ'),
(0x1DA0, 'M', u'f'),
(0x1DA1, 'M', u'ɟ'),
@@ -1771,10 +1764,6 @@ def _seg_16():
(0x1E36, 'M', u'ḷ'),
(0x1E37, 'V'),
(0x1E38, 'M', u'ḹ'),
- ]
-
-def _seg_17():
- return [
(0x1E39, 'V'),
(0x1E3A, 'M', u'ḻ'),
(0x1E3B, 'V'),
@@ -1782,6 +1771,10 @@ def _seg_17():
(0x1E3D, 'V'),
(0x1E3E, 'M', u'ḿ'),
(0x1E3F, 'V'),
+ ]
+
+def _seg_17():
+ return [
(0x1E40, 'M', u'ṁ'),
(0x1E41, 'V'),
(0x1E42, 'M', u'ṃ'),
@@ -1875,10 +1868,6 @@ def _seg_17():
(0x1E9F, 'V'),
(0x1EA0, 'M', u'ạ'),
(0x1EA1, 'V'),
- ]
-
-def _seg_18():
- return [
(0x1EA2, 'M', u'ả'),
(0x1EA3, 'V'),
(0x1EA4, 'M', u'ấ'),
@@ -1886,6 +1875,10 @@ def _seg_18():
(0x1EA6, 'M', u'ầ'),
(0x1EA7, 'V'),
(0x1EA8, 'M', u'ẩ'),
+ ]
+
+def _seg_18():
+ return [
(0x1EA9, 'V'),
(0x1EAA, 'M', u'ẫ'),
(0x1EAB, 'V'),
@@ -1979,10 +1972,6 @@ def _seg_18():
(0x1F0B, 'M', u'ἃ'),
(0x1F0C, 'M', u'ἄ'),
(0x1F0D, 'M', u'ἅ'),
- ]
-
-def _seg_19():
- return [
(0x1F0E, 'M', u'ἆ'),
(0x1F0F, 'M', u'ἇ'),
(0x1F10, 'V'),
@@ -1990,6 +1979,10 @@ def _seg_19():
(0x1F18, 'M', u'ἐ'),
(0x1F19, 'M', u'ἑ'),
(0x1F1A, 'M', u'ἒ'),
+ ]
+
+def _seg_19():
+ return [
(0x1F1B, 'M', u'ἓ'),
(0x1F1C, 'M', u'ἔ'),
(0x1F1D, 'M', u'ἕ'),
@@ -2083,10 +2076,6 @@ def _seg_19():
(0x1F9A, 'M', u'ἢι'),
(0x1F9B, 'M', u'ἣι'),
(0x1F9C, 'M', u'ἤι'),
- ]
-
-def _seg_20():
- return [
(0x1F9D, 'M', u'ἥι'),
(0x1F9E, 'M', u'ἦι'),
(0x1F9F, 'M', u'ἧι'),
@@ -2094,6 +2083,10 @@ def _seg_20():
(0x1FA1, 'M', u'ὡι'),
(0x1FA2, 'M', u'ὢι'),
(0x1FA3, 'M', u'ὣι'),
+ ]
+
+def _seg_20():
+ return [
(0x1FA4, 'M', u'ὤι'),
(0x1FA5, 'M', u'ὥι'),
(0x1FA6, 'M', u'ὦι'),
@@ -2187,10 +2180,6 @@ def _seg_20():
(0x2024, 'X'),
(0x2027, 'V'),
(0x2028, 'X'),
- ]
-
-def _seg_21():
- return [
(0x202F, '3', u' '),
(0x2030, 'V'),
(0x2033, 'M', u'′′'),
@@ -2198,6 +2187,10 @@ def _seg_21():
(0x2035, 'V'),
(0x2036, 'M', u'‵‵'),
(0x2037, 'M', u'‵‵‵'),
+ ]
+
+def _seg_21():
+ return [
(0x2038, 'V'),
(0x203C, '3', u'!!'),
(0x203D, 'V'),
@@ -2291,10 +2284,6 @@ def _seg_21():
(0x2120, 'M', u'sm'),
(0x2121, 'M', u'tel'),
(0x2122, 'M', u'tm'),
- ]
-
-def _seg_22():
- return [
(0x2123, 'V'),
(0x2124, 'M', u'z'),
(0x2125, 'V'),
@@ -2302,6 +2291,10 @@ def _seg_22():
(0x2127, 'V'),
(0x2128, 'M', u'z'),
(0x2129, 'V'),
+ ]
+
+def _seg_22():
+ return [
(0x212A, 'M', u'k'),
(0x212B, 'M', u'å'),
(0x212C, 'M', u'b'),
@@ -2395,10 +2388,6 @@ def _seg_22():
(0x226E, '3'),
(0x2270, 'V'),
(0x2329, 'M', u'〈'),
- ]
-
-def _seg_23():
- return [
(0x232A, 'M', u'〉'),
(0x232B, 'V'),
(0x2427, 'X'),
@@ -2406,6 +2395,10 @@ def _seg_23():
(0x244B, 'X'),
(0x2460, 'M', u'1'),
(0x2461, 'M', u'2'),
+ ]
+
+def _seg_23():
+ return [
(0x2462, 'M', u'3'),
(0x2463, 'M', u'4'),
(0x2464, 'M', u'5'),
@@ -2499,10 +2492,6 @@ def _seg_23():
(0x24CF, 'M', u'z'),
(0x24D0, 'M', u'a'),
(0x24D1, 'M', u'b'),
- ]
-
-def _seg_24():
- return [
(0x24D2, 'M', u'c'),
(0x24D3, 'M', u'd'),
(0x24D4, 'M', u'e'),
@@ -2510,6 +2499,10 @@ def _seg_24():
(0x24D6, 'M', u'g'),
(0x24D7, 'M', u'h'),
(0x24D8, 'M', u'i'),
+ ]
+
+def _seg_24():
+ return [
(0x24D9, 'M', u'j'),
(0x24DA, 'M', u'k'),
(0x24DB, 'M', u'l'),
@@ -2541,13 +2534,9 @@ def _seg_24():
(0x2B76, 'V'),
(0x2B96, 'X'),
(0x2B98, 'V'),
- (0x2BBA, 'X'),
- (0x2BBD, 'V'),
(0x2BC9, 'X'),
(0x2BCA, 'V'),
- (0x2BD3, 'X'),
- (0x2BEC, 'V'),
- (0x2BF0, 'X'),
+ (0x2BFF, 'X'),
(0x2C00, 'M', u'ⰰ'),
(0x2C01, 'M', u'ⰱ'),
(0x2C02, 'M', u'ⰲ'),
@@ -2603,10 +2592,6 @@ def _seg_24():
(0x2C62, 'M', u'ɫ'),
(0x2C63, 'M', u'ᵽ'),
(0x2C64, 'M', u'ɽ'),
- ]
-
-def _seg_25():
- return [
(0x2C65, 'V'),
(0x2C67, 'M', u'ⱨ'),
(0x2C68, 'V'),
@@ -2618,6 +2603,10 @@ def _seg_25():
(0x2C6E, 'M', u'ɱ'),
(0x2C6F, 'M', u'ɐ'),
(0x2C70, 'M', u'ɒ'),
+ ]
+
+def _seg_25():
+ return [
(0x2C71, 'V'),
(0x2C72, 'M', u'ⱳ'),
(0x2C73, 'V'),
@@ -2707,10 +2696,6 @@ def _seg_25():
(0x2CCD, 'V'),
(0x2CCE, 'M', u'ⳏ'),
(0x2CCF, 'V'),
- ]
-
-def _seg_26():
- return [
(0x2CD0, 'M', u'ⳑ'),
(0x2CD1, 'V'),
(0x2CD2, 'M', u'ⳓ'),
@@ -2722,6 +2707,10 @@ def _seg_26():
(0x2CD8, 'M', u'ⳙ'),
(0x2CD9, 'V'),
(0x2CDA, 'M', u'ⳛ'),
+ ]
+
+def _seg_26():
+ return [
(0x2CDB, 'V'),
(0x2CDC, 'M', u'ⳝ'),
(0x2CDD, 'V'),
@@ -2768,7 +2757,7 @@ def _seg_26():
(0x2DD8, 'V'),
(0x2DDF, 'X'),
(0x2DE0, 'V'),
- (0x2E4A, 'X'),
+ (0x2E4F, 'X'),
(0x2E80, 'V'),
(0x2E9A, 'X'),
(0x2E9B, 'V'),
@@ -2811,10 +2800,6 @@ def _seg_26():
(0x2F20, 'M', u'士'),
(0x2F21, 'M', u'夂'),
(0x2F22, 'M', u'夊'),
- ]
-
-def _seg_27():
- return [
(0x2F23, 'M', u'夕'),
(0x2F24, 'M', u'大'),
(0x2F25, 'M', u'女'),
@@ -2826,6 +2811,10 @@ def _seg_27():
(0x2F2B, 'M', u'尸'),
(0x2F2C, 'M', u'屮'),
(0x2F2D, 'M', u'山'),
+ ]
+
+def _seg_27():
+ return [
(0x2F2E, 'M', u'巛'),
(0x2F2F, 'M', u'工'),
(0x2F30, 'M', u'己'),
@@ -2915,10 +2904,6 @@ def _seg_27():
(0x2F84, 'M', u'至'),
(0x2F85, 'M', u'臼'),
(0x2F86, 'M', u'舌'),
- ]
-
-def _seg_28():
- return [
(0x2F87, 'M', u'舛'),
(0x2F88, 'M', u'舟'),
(0x2F89, 'M', u'艮'),
@@ -2930,6 +2915,10 @@ def _seg_28():
(0x2F8F, 'M', u'行'),
(0x2F90, 'M', u'衣'),
(0x2F91, 'M', u'襾'),
+ ]
+
+def _seg_28():
+ return [
(0x2F92, 'M', u'見'),
(0x2F93, 'M', u'角'),
(0x2F94, 'M', u'言'),
@@ -3019,13 +3008,9 @@ def _seg_28():
(0x309F, 'M', u'より'),
(0x30A0, 'V'),
(0x30FF, 'M', u'コト'),
- ]
-
-def _seg_29():
- return [
(0x3100, 'X'),
(0x3105, 'V'),
- (0x312F, 'X'),
+ (0x3130, 'X'),
(0x3131, 'M', u'ᄀ'),
(0x3132, 'M', u'ᄁ'),
(0x3133, 'M', u'ᆪ'),
@@ -3034,6 +3019,10 @@ def _seg_29():
(0x3136, 'M', u'ᆭ'),
(0x3137, 'M', u'ᄃ'),
(0x3138, 'M', u'ᄄ'),
+ ]
+
+def _seg_29():
+ return [
(0x3139, 'M', u'ᄅ'),
(0x313A, 'M', u'ᆰ'),
(0x313B, 'M', u'ᆱ'),
@@ -3123,10 +3112,6 @@ def _seg_29():
(0x318F, 'X'),
(0x3190, 'V'),
(0x3192, 'M', u'一'),
- ]
-
-def _seg_30():
- return [
(0x3193, 'M', u'二'),
(0x3194, 'M', u'三'),
(0x3195, 'M', u'四'),
@@ -3138,6 +3123,10 @@ def _seg_30():
(0x319B, 'M', u'丙'),
(0x319C, 'M', u'丁'),
(0x319D, 'M', u'天'),
+ ]
+
+def _seg_30():
+ return [
(0x319E, 'M', u'地'),
(0x319F, 'M', u'人'),
(0x31A0, 'V'),
@@ -3227,10 +3216,6 @@ def _seg_30():
(0x3256, 'M', u'26'),
(0x3257, 'M', u'27'),
(0x3258, 'M', u'28'),
- ]
-
-def _seg_31():
- return [
(0x3259, 'M', u'29'),
(0x325A, 'M', u'30'),
(0x325B, 'M', u'31'),
@@ -3242,6 +3227,10 @@ def _seg_31():
(0x3261, 'M', u'ᄂ'),
(0x3262, 'M', u'ᄃ'),
(0x3263, 'M', u'ᄅ'),
+ ]
+
+def _seg_31():
+ return [
(0x3264, 'M', u'ᄆ'),
(0x3265, 'M', u'ᄇ'),
(0x3266, 'M', u'ᄉ'),
@@ -3331,10 +3320,6 @@ def _seg_31():
(0x32BA, 'M', u'45'),
(0x32BB, 'M', u'46'),
(0x32BC, 'M', u'47'),
- ]
-
-def _seg_32():
- return [
(0x32BD, 'M', u'48'),
(0x32BE, 'M', u'49'),
(0x32BF, 'M', u'50'),
@@ -3346,6 +3331,10 @@ def _seg_32():
(0x32C5, 'M', u'6月'),
(0x32C6, 'M', u'7月'),
(0x32C7, 'M', u'8月'),
+ ]
+
+def _seg_32():
+ return [
(0x32C8, 'M', u'9月'),
(0x32C9, 'M', u'10月'),
(0x32CA, 'M', u'11月'),
@@ -3435,10 +3424,6 @@ def _seg_32():
(0x331E, 'M', u'コーポ'),
(0x331F, 'M', u'サイクル'),
(0x3320, 'M', u'サンチーム'),
- ]
-
-def _seg_33():
- return [
(0x3321, 'M', u'シリング'),
(0x3322, 'M', u'センチ'),
(0x3323, 'M', u'セント'),
@@ -3450,6 +3435,10 @@ def _seg_33():
(0x3329, 'M', u'ノット'),
(0x332A, 'M', u'ハイツ'),
(0x332B, 'M', u'パーセント'),
+ ]
+
+def _seg_33():
+ return [
(0x332C, 'M', u'パーツ'),
(0x332D, 'M', u'バーレル'),
(0x332E, 'M', u'ピアストル'),
@@ -3539,10 +3528,6 @@ def _seg_33():
(0x3382, 'M', u'μa'),
(0x3383, 'M', u'ma'),
(0x3384, 'M', u'ka'),
- ]
-
-def _seg_34():
- return [
(0x3385, 'M', u'kb'),
(0x3386, 'M', u'mb'),
(0x3387, 'M', u'gb'),
@@ -3554,6 +3539,10 @@ def _seg_34():
(0x338D, 'M', u'μg'),
(0x338E, 'M', u'mg'),
(0x338F, 'M', u'kg'),
+ ]
+
+def _seg_34():
+ return [
(0x3390, 'M', u'hz'),
(0x3391, 'M', u'khz'),
(0x3392, 'M', u'mhz'),
@@ -3643,10 +3632,6 @@ def _seg_34():
(0x33E6, 'M', u'7日'),
(0x33E7, 'M', u'8日'),
(0x33E8, 'M', u'9日'),
- ]
-
-def _seg_35():
- return [
(0x33E9, 'M', u'10日'),
(0x33EA, 'M', u'11日'),
(0x33EB, 'M', u'12日'),
@@ -3658,6 +3643,10 @@ def _seg_35():
(0x33F1, 'M', u'18日'),
(0x33F2, 'M', u'19日'),
(0x33F3, 'M', u'20日'),
+ ]
+
+def _seg_35():
+ return [
(0x33F4, 'M', u'21日'),
(0x33F5, 'M', u'22日'),
(0x33F6, 'M', u'23日'),
@@ -3673,7 +3662,7 @@ def _seg_35():
(0x3400, 'V'),
(0x4DB6, 'X'),
(0x4DC0, 'V'),
- (0x9FEB, 'X'),
+ (0x9FF0, 'X'),
(0xA000, 'V'),
(0xA48D, 'X'),
(0xA490, 'V'),
@@ -3747,10 +3736,6 @@ def _seg_35():
(0xA692, 'M', u'ꚓ'),
(0xA693, 'V'),
(0xA694, 'M', u'ꚕ'),
- ]
-
-def _seg_36():
- return [
(0xA695, 'V'),
(0xA696, 'M', u'ꚗ'),
(0xA697, 'V'),
@@ -3762,6 +3747,10 @@ def _seg_36():
(0xA69D, 'M', u'ь'),
(0xA69E, 'V'),
(0xA6F8, 'X'),
+ ]
+
+def _seg_36():
+ return [
(0xA700, 'V'),
(0xA722, 'M', u'ꜣ'),
(0xA723, 'V'),
@@ -3851,10 +3840,6 @@ def _seg_36():
(0xA780, 'M', u'ꞁ'),
(0xA781, 'V'),
(0xA782, 'M', u'ꞃ'),
- ]
-
-def _seg_37():
- return [
(0xA783, 'V'),
(0xA784, 'M', u'ꞅ'),
(0xA785, 'V'),
@@ -3866,6 +3851,10 @@ def _seg_37():
(0xA78E, 'V'),
(0xA790, 'M', u'ꞑ'),
(0xA791, 'V'),
+ ]
+
+def _seg_37():
+ return [
(0xA792, 'M', u'ꞓ'),
(0xA793, 'V'),
(0xA796, 'M', u'ꞗ'),
@@ -3893,7 +3882,7 @@ def _seg_37():
(0xA7AC, 'M', u'ɡ'),
(0xA7AD, 'M', u'ɬ'),
(0xA7AE, 'M', u'ɪ'),
- (0xA7AF, 'X'),
+ (0xA7AF, 'V'),
(0xA7B0, 'M', u'ʞ'),
(0xA7B1, 'M', u'ʇ'),
(0xA7B2, 'M', u'ʝ'),
@@ -3903,6 +3892,8 @@ def _seg_37():
(0xA7B6, 'M', u'ꞷ'),
(0xA7B7, 'V'),
(0xA7B8, 'X'),
+ (0xA7B9, 'V'),
+ (0xA7BA, 'X'),
(0xA7F7, 'V'),
(0xA7F8, 'M', u'ħ'),
(0xA7F9, 'M', u'œ'),
@@ -3917,8 +3908,6 @@ def _seg_37():
(0xA8CE, 'V'),
(0xA8DA, 'X'),
(0xA8E0, 'V'),
- (0xA8FE, 'X'),
- (0xA900, 'V'),
(0xA954, 'X'),
(0xA95F, 'V'),
(0xA97D, 'X'),
@@ -3955,10 +3944,6 @@ def _seg_37():
(0xAB5F, 'M', u'ꭒ'),
(0xAB60, 'V'),
(0xAB66, 'X'),
- ]
-
-def _seg_38():
- return [
(0xAB70, 'M', u'Ꭰ'),
(0xAB71, 'M', u'Ꭱ'),
(0xAB72, 'M', u'Ꭲ'),
@@ -3970,6 +3955,10 @@ def _seg_38():
(0xAB78, 'M', u'Ꭸ'),
(0xAB79, 'M', u'Ꭹ'),
(0xAB7A, 'M', u'Ꭺ'),
+ ]
+
+def _seg_38():
+ return [
(0xAB7B, 'M', u'Ꭻ'),
(0xAB7C, 'M', u'Ꭼ'),
(0xAB7D, 'M', u'Ꭽ'),
@@ -4059,10 +4048,6 @@ def _seg_38():
(0xF907, 'M', u'龜'),
(0xF909, 'M', u'契'),
(0xF90A, 'M', u'金'),
- ]
-
-def _seg_39():
- return [
(0xF90B, 'M', u'喇'),
(0xF90C, 'M', u'奈'),
(0xF90D, 'M', u'懶'),
@@ -4074,6 +4059,10 @@ def _seg_39():
(0xF913, 'M', u'邏'),
(0xF914, 'M', u'樂'),
(0xF915, 'M', u'洛'),
+ ]
+
+def _seg_39():
+ return [
(0xF916, 'M', u'烙'),
(0xF917, 'M', u'珞'),
(0xF918, 'M', u'落'),
@@ -4163,10 +4152,6 @@ def _seg_39():
(0xF96C, 'M', u'塞'),
(0xF96D, 'M', u'省'),
(0xF96E, 'M', u'葉'),
- ]
-
-def _seg_40():
- return [
(0xF96F, 'M', u'說'),
(0xF970, 'M', u'殺'),
(0xF971, 'M', u'辰'),
@@ -4178,6 +4163,10 @@ def _seg_40():
(0xF977, 'M', u'亮'),
(0xF978, 'M', u'兩'),
(0xF979, 'M', u'凉'),
+ ]
+
+def _seg_40():
+ return [
(0xF97A, 'M', u'梁'),
(0xF97B, 'M', u'糧'),
(0xF97C, 'M', u'良'),
@@ -4267,10 +4256,6 @@ def _seg_40():
(0xF9D0, 'M', u'類'),
(0xF9D1, 'M', u'六'),
(0xF9D2, 'M', u'戮'),
- ]
-
-def _seg_41():
- return [
(0xF9D3, 'M', u'陸'),
(0xF9D4, 'M', u'倫'),
(0xF9D5, 'M', u'崙'),
@@ -4282,6 +4267,10 @@ def _seg_41():
(0xF9DB, 'M', u'率'),
(0xF9DC, 'M', u'隆'),
(0xF9DD, 'M', u'利'),
+ ]
+
+def _seg_41():
+ return [
(0xF9DE, 'M', u'吏'),
(0xF9DF, 'M', u'履'),
(0xF9E0, 'M', u'易'),
@@ -4371,10 +4360,6 @@ def _seg_41():
(0xFA39, 'M', u'塀'),
(0xFA3A, 'M', u'墨'),
(0xFA3B, 'M', u'層'),
- ]
-
-def _seg_42():
- return [
(0xFA3C, 'M', u'屮'),
(0xFA3D, 'M', u'悔'),
(0xFA3E, 'M', u'慨'),
@@ -4386,6 +4371,10 @@ def _seg_42():
(0xFA44, 'M', u'梅'),
(0xFA45, 'M', u'海'),
(0xFA46, 'M', u'渚'),
+ ]
+
+def _seg_42():
+ return [
(0xFA47, 'M', u'漢'),
(0xFA48, 'M', u'煮'),
(0xFA49, 'M', u'爫'),
@@ -4475,10 +4464,6 @@ def _seg_42():
(0xFA9F, 'M', u'犯'),
(0xFAA0, 'M', u'猪'),
(0xFAA1, 'M', u'瑱'),
- ]
-
-def _seg_43():
- return [
(0xFAA2, 'M', u'甆'),
(0xFAA3, 'M', u'画'),
(0xFAA4, 'M', u'瘝'),
@@ -4490,6 +4475,10 @@ def _seg_43():
(0xFAAA, 'M', u'着'),
(0xFAAB, 'M', u'磌'),
(0xFAAC, 'M', u'窱'),
+ ]
+
+def _seg_43():
+ return [
(0xFAAD, 'M', u'節'),
(0xFAAE, 'M', u'类'),
(0xFAAF, 'M', u'絛'),
@@ -4579,10 +4568,6 @@ def _seg_43():
(0xFB38, 'M', u'טּ'),
(0xFB39, 'M', u'יּ'),
(0xFB3A, 'M', u'ךּ'),
- ]
-
-def _seg_44():
- return [
(0xFB3B, 'M', u'כּ'),
(0xFB3C, 'M', u'לּ'),
(0xFB3D, 'X'),
@@ -4594,6 +4579,10 @@ def _seg_44():
(0xFB43, 'M', u'ףּ'),
(0xFB44, 'M', u'פּ'),
(0xFB45, 'X'),
+ ]
+
+def _seg_44():
+ return [
(0xFB46, 'M', u'צּ'),
(0xFB47, 'M', u'קּ'),
(0xFB48, 'M', u'רּ'),
@@ -4683,10 +4672,6 @@ def _seg_44():
(0xFC19, 'M', u'خج'),
(0xFC1A, 'M', u'خح'),
(0xFC1B, 'M', u'خم'),
- ]
-
-def _seg_45():
- return [
(0xFC1C, 'M', u'سج'),
(0xFC1D, 'M', u'سح'),
(0xFC1E, 'M', u'سخ'),
@@ -4698,6 +4683,10 @@ def _seg_45():
(0xFC24, 'M', u'ضخ'),
(0xFC25, 'M', u'ضم'),
(0xFC26, 'M', u'طح'),
+ ]
+
+def _seg_45():
+ return [
(0xFC27, 'M', u'طم'),
(0xFC28, 'M', u'ظم'),
(0xFC29, 'M', u'عج'),
@@ -4787,10 +4776,6 @@ def _seg_45():
(0xFC7D, 'M', u'في'),
(0xFC7E, 'M', u'قى'),
(0xFC7F, 'M', u'قي'),
- ]
-
-def _seg_46():
- return [
(0xFC80, 'M', u'كا'),
(0xFC81, 'M', u'كل'),
(0xFC82, 'M', u'كم'),
@@ -4802,6 +4787,10 @@ def _seg_46():
(0xFC88, 'M', u'ما'),
(0xFC89, 'M', u'مم'),
(0xFC8A, 'M', u'نر'),
+ ]
+
+def _seg_46():
+ return [
(0xFC8B, 'M', u'نز'),
(0xFC8C, 'M', u'نم'),
(0xFC8D, 'M', u'نن'),
@@ -4891,10 +4880,6 @@ def _seg_46():
(0xFCE1, 'M', u'بم'),
(0xFCE2, 'M', u'به'),
(0xFCE3, 'M', u'تم'),
- ]
-
-def _seg_47():
- return [
(0xFCE4, 'M', u'ته'),
(0xFCE5, 'M', u'ثم'),
(0xFCE6, 'M', u'ثه'),
@@ -4906,6 +4891,10 @@ def _seg_47():
(0xFCEC, 'M', u'كم'),
(0xFCED, 'M', u'لم'),
(0xFCEE, 'M', u'نم'),
+ ]
+
+def _seg_47():
+ return [
(0xFCEF, 'M', u'نه'),
(0xFCF0, 'M', u'يم'),
(0xFCF1, 'M', u'يه'),
@@ -4995,10 +4984,6 @@ def _seg_47():
(0xFD57, 'M', u'تمخ'),
(0xFD58, 'M', u'جمح'),
(0xFD5A, 'M', u'حمي'),
- ]
-
-def _seg_48():
- return [
(0xFD5B, 'M', u'حمى'),
(0xFD5C, 'M', u'سحج'),
(0xFD5D, 'M', u'سجح'),
@@ -5010,6 +4995,10 @@ def _seg_48():
(0xFD66, 'M', u'صمم'),
(0xFD67, 'M', u'شحم'),
(0xFD69, 'M', u'شجي'),
+ ]
+
+def _seg_48():
+ return [
(0xFD6A, 'M', u'شمخ'),
(0xFD6C, 'M', u'شمم'),
(0xFD6E, 'M', u'ضحى'),
@@ -5099,10 +5088,6 @@ def _seg_48():
(0xFDF3, 'M', u'اكبر'),
(0xFDF4, 'M', u'محمد'),
(0xFDF5, 'M', u'صلعم'),
- ]
-
-def _seg_49():
- return [
(0xFDF6, 'M', u'رسول'),
(0xFDF7, 'M', u'عليه'),
(0xFDF8, 'M', u'وسلم'),
@@ -5114,6 +5099,10 @@ def _seg_49():
(0xFDFE, 'X'),
(0xFE00, 'I'),
(0xFE10, '3', u','),
+ ]
+
+def _seg_49():
+ return [
(0xFE11, 'M', u'、'),
(0xFE12, 'X'),
(0xFE13, '3', u':'),
@@ -5203,10 +5192,6 @@ def _seg_49():
(0xFE8F, 'M', u'ب'),
(0xFE93, 'M', u'ة'),
(0xFE95, 'M', u'ت'),
- ]
-
-def _seg_50():
- return [
(0xFE99, 'M', u'ث'),
(0xFE9D, 'M', u'ج'),
(0xFEA1, 'M', u'ح'),
@@ -5218,6 +5203,10 @@ def _seg_50():
(0xFEB1, 'M', u'س'),
(0xFEB5, 'M', u'ش'),
(0xFEB9, 'M', u'ص'),
+ ]
+
+def _seg_50():
+ return [
(0xFEBD, 'M', u'ض'),
(0xFEC1, 'M', u'ط'),
(0xFEC5, 'M', u'ظ'),
@@ -5307,10 +5296,6 @@ def _seg_50():
(0xFF41, 'M', u'a'),
(0xFF42, 'M', u'b'),
(0xFF43, 'M', u'c'),
- ]
-
-def _seg_51():
- return [
(0xFF44, 'M', u'd'),
(0xFF45, 'M', u'e'),
(0xFF46, 'M', u'f'),
@@ -5322,6 +5307,10 @@ def _seg_51():
(0xFF4C, 'M', u'l'),
(0xFF4D, 'M', u'm'),
(0xFF4E, 'M', u'n'),
+ ]
+
+def _seg_51():
+ return [
(0xFF4F, 'M', u'o'),
(0xFF50, 'M', u'p'),
(0xFF51, 'M', u'q'),
@@ -5411,10 +5400,6 @@ def _seg_51():
(0xFFA5, 'M', u'ᆬ'),
(0xFFA6, 'M', u'ᆭ'),
(0xFFA7, 'M', u'ᄃ'),
- ]
-
-def _seg_52():
- return [
(0xFFA8, 'M', u'ᄄ'),
(0xFFA9, 'M', u'ᄅ'),
(0xFFAA, 'M', u'ᆰ'),
@@ -5426,6 +5411,10 @@ def _seg_52():
(0xFFB0, 'M', u'ᄚ'),
(0xFFB1, 'M', u'ᄆ'),
(0xFFB2, 'M', u'ᄇ'),
+ ]
+
+def _seg_52():
+ return [
(0xFFB3, 'M', u'ᄈ'),
(0xFFB4, 'M', u'ᄡ'),
(0xFFB5, 'M', u'ᄉ'),
@@ -5515,10 +5504,6 @@ def _seg_52():
(0x10300, 'V'),
(0x10324, 'X'),
(0x1032D, 'V'),
- ]
-
-def _seg_53():
- return [
(0x1034B, 'X'),
(0x10350, 'V'),
(0x1037B, 'X'),
@@ -5530,6 +5515,10 @@ def _seg_53():
(0x103D6, 'X'),
(0x10400, 'M', u'𐐨'),
(0x10401, 'M', u'𐐩'),
+ ]
+
+def _seg_53():
+ return [
(0x10402, 'M', u'𐐪'),
(0x10403, 'M', u'𐐫'),
(0x10404, 'M', u'𐐬'),
@@ -5619,10 +5608,6 @@ def _seg_53():
(0x10570, 'X'),
(0x10600, 'V'),
(0x10737, 'X'),
- ]
-
-def _seg_54():
- return [
(0x10740, 'V'),
(0x10756, 'X'),
(0x10760, 'V'),
@@ -5634,6 +5619,10 @@ def _seg_54():
(0x1080A, 'V'),
(0x10836, 'X'),
(0x10837, 'V'),
+ ]
+
+def _seg_54():
+ return [
(0x10839, 'X'),
(0x1083C, 'V'),
(0x1083D, 'X'),
@@ -5666,11 +5655,11 @@ def _seg_54():
(0x10A15, 'V'),
(0x10A18, 'X'),
(0x10A19, 'V'),
- (0x10A34, 'X'),
+ (0x10A36, 'X'),
(0x10A38, 'V'),
(0x10A3B, 'X'),
(0x10A3F, 'V'),
- (0x10A48, 'X'),
+ (0x10A49, 'X'),
(0x10A50, 'V'),
(0x10A59, 'X'),
(0x10A60, 'V'),
@@ -5723,10 +5712,6 @@ def _seg_54():
(0x10C9B, 'M', u'𐳛'),
(0x10C9C, 'M', u'𐳜'),
(0x10C9D, 'M', u'𐳝'),
- ]
-
-def _seg_55():
- return [
(0x10C9E, 'M', u'𐳞'),
(0x10C9F, 'M', u'𐳟'),
(0x10CA0, 'M', u'𐳠'),
@@ -5738,6 +5723,10 @@ def _seg_55():
(0x10CA6, 'M', u'𐳦'),
(0x10CA7, 'M', u'𐳧'),
(0x10CA8, 'M', u'𐳨'),
+ ]
+
+def _seg_55():
+ return [
(0x10CA9, 'M', u'𐳩'),
(0x10CAA, 'M', u'𐳪'),
(0x10CAB, 'M', u'𐳫'),
@@ -5752,9 +5741,15 @@ def _seg_55():
(0x10CC0, 'V'),
(0x10CF3, 'X'),
(0x10CFA, 'V'),
- (0x10D00, 'X'),
+ (0x10D28, 'X'),
+ (0x10D30, 'V'),
+ (0x10D3A, 'X'),
(0x10E60, 'V'),
(0x10E7F, 'X'),
+ (0x10F00, 'V'),
+ (0x10F28, 'X'),
+ (0x10F30, 'V'),
+ (0x10F5A, 'X'),
(0x11000, 'V'),
(0x1104E, 'X'),
(0x11052, 'V'),
@@ -5770,7 +5765,7 @@ def _seg_55():
(0x11100, 'V'),
(0x11135, 'X'),
(0x11136, 'V'),
- (0x11144, 'X'),
+ (0x11147, 'X'),
(0x11150, 'V'),
(0x11177, 'X'),
(0x11180, 'V'),
@@ -5811,7 +5806,7 @@ def _seg_55():
(0x11334, 'X'),
(0x11335, 'V'),
(0x1133A, 'X'),
- (0x1133C, 'V'),
+ (0x1133B, 'V'),
(0x11345, 'X'),
(0x11347, 'V'),
(0x11349, 'X'),
@@ -5827,16 +5822,16 @@ def _seg_55():
(0x1136D, 'X'),
(0x11370, 'V'),
(0x11375, 'X'),
- ]
-
-def _seg_56():
- return [
(0x11400, 'V'),
(0x1145A, 'X'),
(0x1145B, 'V'),
(0x1145C, 'X'),
(0x1145D, 'V'),
- (0x1145E, 'X'),
+ ]
+
+def _seg_56():
+ return [
+ (0x1145F, 'X'),
(0x11480, 'V'),
(0x114C8, 'X'),
(0x114D0, 'V'),
@@ -5856,11 +5851,13 @@ def _seg_56():
(0x116C0, 'V'),
(0x116CA, 'X'),
(0x11700, 'V'),
- (0x1171A, 'X'),
+ (0x1171B, 'X'),
(0x1171D, 'V'),
(0x1172C, 'X'),
(0x11730, 'V'),
(0x11740, 'X'),
+ (0x11800, 'V'),
+ (0x1183C, 'X'),
(0x118A0, 'M', u'𑣀'),
(0x118A1, 'M', u'𑣁'),
(0x118A2, 'M', u'𑣂'),
@@ -5902,8 +5899,6 @@ def _seg_56():
(0x11A50, 'V'),
(0x11A84, 'X'),
(0x11A86, 'V'),
- (0x11A9D, 'X'),
- (0x11A9E, 'V'),
(0x11AA3, 'X'),
(0x11AC0, 'V'),
(0x11AF9, 'X'),
@@ -5931,14 +5926,28 @@ def _seg_56():
(0x11D3B, 'X'),
(0x11D3C, 'V'),
(0x11D3E, 'X'),
- ]
-
-def _seg_57():
- return [
(0x11D3F, 'V'),
(0x11D48, 'X'),
(0x11D50, 'V'),
(0x11D5A, 'X'),
+ (0x11D60, 'V'),
+ ]
+
+def _seg_57():
+ return [
+ (0x11D66, 'X'),
+ (0x11D67, 'V'),
+ (0x11D69, 'X'),
+ (0x11D6A, 'V'),
+ (0x11D8F, 'X'),
+ (0x11D90, 'V'),
+ (0x11D92, 'X'),
+ (0x11D93, 'V'),
+ (0x11D99, 'X'),
+ (0x11DA0, 'V'),
+ (0x11DAA, 'X'),
+ (0x11EE0, 'V'),
+ (0x11EF9, 'X'),
(0x12000, 'V'),
(0x1239A, 'X'),
(0x12400, 'V'),
@@ -5973,6 +5982,8 @@ def _seg_57():
(0x16B78, 'X'),
(0x16B7D, 'V'),
(0x16B90, 'X'),
+ (0x16E60, 'V'),
+ (0x16E9B, 'X'),
(0x16F00, 'V'),
(0x16F45, 'X'),
(0x16F50, 'V'),
@@ -5982,7 +5993,7 @@ def _seg_57():
(0x16FE0, 'V'),
(0x16FE2, 'X'),
(0x17000, 'V'),
- (0x187ED, 'X'),
+ (0x187F2, 'X'),
(0x18800, 'V'),
(0x18AF3, 'X'),
(0x1B000, 'V'),
@@ -6024,21 +6035,23 @@ def _seg_57():
(0x1D1C1, 'V'),
(0x1D1E9, 'X'),
(0x1D200, 'V'),
+ ]
+
+def _seg_58():
+ return [
(0x1D246, 'X'),
+ (0x1D2E0, 'V'),
+ (0x1D2F4, 'X'),
(0x1D300, 'V'),
(0x1D357, 'X'),
(0x1D360, 'V'),
- (0x1D372, 'X'),
+ (0x1D379, 'X'),
(0x1D400, 'M', u'a'),
(0x1D401, 'M', u'b'),
(0x1D402, 'M', u'c'),
(0x1D403, 'M', u'd'),
(0x1D404, 'M', u'e'),
(0x1D405, 'M', u'f'),
- ]
-
-def _seg_58():
- return [
(0x1D406, 'M', u'g'),
(0x1D407, 'M', u'h'),
(0x1D408, 'M', u'i'),
@@ -6126,6 +6139,10 @@ def _seg_58():
(0x1D45A, 'M', u'm'),
(0x1D45B, 'M', u'n'),
(0x1D45C, 'M', u'o'),
+ ]
+
+def _seg_59():
+ return [
(0x1D45D, 'M', u'p'),
(0x1D45E, 'M', u'q'),
(0x1D45F, 'M', u'r'),
@@ -6139,10 +6156,6 @@ def _seg_58():
(0x1D467, 'M', u'z'),
(0x1D468, 'M', u'a'),
(0x1D469, 'M', u'b'),
- ]
-
-def _seg_59():
- return [
(0x1D46A, 'M', u'c'),
(0x1D46B, 'M', u'd'),
(0x1D46C, 'M', u'e'),
@@ -6230,6 +6243,10 @@ def _seg_59():
(0x1D4C1, 'M', u'l'),
(0x1D4C2, 'M', u'm'),
(0x1D4C3, 'M', u'n'),
+ ]
+
+def _seg_60():
+ return [
(0x1D4C4, 'X'),
(0x1D4C5, 'M', u'p'),
(0x1D4C6, 'M', u'q'),
@@ -6243,10 +6260,6 @@ def _seg_59():
(0x1D4CE, 'M', u'y'),
(0x1D4CF, 'M', u'z'),
(0x1D4D0, 'M', u'a'),
- ]
-
-def _seg_60():
- return [
(0x1D4D1, 'M', u'b'),
(0x1D4D2, 'M', u'c'),
(0x1D4D3, 'M', u'd'),
@@ -6334,6 +6347,10 @@ def _seg_60():
(0x1D526, 'M', u'i'),
(0x1D527, 'M', u'j'),
(0x1D528, 'M', u'k'),
+ ]
+
+def _seg_61():
+ return [
(0x1D529, 'M', u'l'),
(0x1D52A, 'M', u'm'),
(0x1D52B, 'M', u'n'),
@@ -6347,10 +6364,6 @@ def _seg_60():
(0x1D533, 'M', u'v'),
(0x1D534, 'M', u'w'),
(0x1D535, 'M', u'x'),
- ]
-
-def _seg_61():
- return [
(0x1D536, 'M', u'y'),
(0x1D537, 'M', u'z'),
(0x1D538, 'M', u'a'),
@@ -6438,6 +6451,10 @@ def _seg_61():
(0x1D58C, 'M', u'g'),
(0x1D58D, 'M', u'h'),
(0x1D58E, 'M', u'i'),
+ ]
+
+def _seg_62():
+ return [
(0x1D58F, 'M', u'j'),
(0x1D590, 'M', u'k'),
(0x1D591, 'M', u'l'),
@@ -6451,10 +6468,6 @@ def _seg_61():
(0x1D599, 'M', u't'),
(0x1D59A, 'M', u'u'),
(0x1D59B, 'M', u'v'),
- ]
-
-def _seg_62():
- return [
(0x1D59C, 'M', u'w'),
(0x1D59D, 'M', u'x'),
(0x1D59E, 'M', u'y'),
@@ -6542,6 +6555,10 @@ def _seg_62():
(0x1D5F0, 'M', u'c'),
(0x1D5F1, 'M', u'd'),
(0x1D5F2, 'M', u'e'),
+ ]
+
+def _seg_63():
+ return [
(0x1D5F3, 'M', u'f'),
(0x1D5F4, 'M', u'g'),
(0x1D5F5, 'M', u'h'),
@@ -6555,10 +6572,6 @@ def _seg_62():
(0x1D5FD, 'M', u'p'),
(0x1D5FE, 'M', u'q'),
(0x1D5FF, 'M', u'r'),
- ]
-
-def _seg_63():
- return [
(0x1D600, 'M', u's'),
(0x1D601, 'M', u't'),
(0x1D602, 'M', u'u'),
@@ -6646,6 +6659,10 @@ def _seg_63():
(0x1D654, 'M', u'y'),
(0x1D655, 'M', u'z'),
(0x1D656, 'M', u'a'),
+ ]
+
+def _seg_64():
+ return [
(0x1D657, 'M', u'b'),
(0x1D658, 'M', u'c'),
(0x1D659, 'M', u'd'),
@@ -6659,10 +6676,6 @@ def _seg_63():
(0x1D661, 'M', u'l'),
(0x1D662, 'M', u'm'),
(0x1D663, 'M', u'n'),
- ]
-
-def _seg_64():
- return [
(0x1D664, 'M', u'o'),
(0x1D665, 'M', u'p'),
(0x1D666, 'M', u'q'),
@@ -6750,6 +6763,10 @@ def _seg_64():
(0x1D6B9, 'M', u'θ'),
(0x1D6BA, 'M', u'σ'),
(0x1D6BB, 'M', u'τ'),
+ ]
+
+def _seg_65():
+ return [
(0x1D6BC, 'M', u'υ'),
(0x1D6BD, 'M', u'φ'),
(0x1D6BE, 'M', u'χ'),
@@ -6763,10 +6780,6 @@ def _seg_64():
(0x1D6C6, 'M', u'ε'),
(0x1D6C7, 'M', u'ζ'),
(0x1D6C8, 'M', u'η'),
- ]
-
-def _seg_65():
- return [
(0x1D6C9, 'M', u'θ'),
(0x1D6CA, 'M', u'ι'),
(0x1D6CB, 'M', u'κ'),
@@ -6854,6 +6867,10 @@ def _seg_65():
(0x1D71F, 'M', u'δ'),
(0x1D720, 'M', u'ε'),
(0x1D721, 'M', u'ζ'),
+ ]
+
+def _seg_66():
+ return [
(0x1D722, 'M', u'η'),
(0x1D723, 'M', u'θ'),
(0x1D724, 'M', u'ι'),
@@ -6867,10 +6884,6 @@ def _seg_65():
(0x1D72C, 'M', u'ρ'),
(0x1D72D, 'M', u'θ'),
(0x1D72E, 'M', u'σ'),
- ]
-
-def _seg_66():
- return [
(0x1D72F, 'M', u'τ'),
(0x1D730, 'M', u'υ'),
(0x1D731, 'M', u'φ'),
@@ -6958,6 +6971,10 @@ def _seg_66():
(0x1D785, 'M', u'φ'),
(0x1D786, 'M', u'χ'),
(0x1D787, 'M', u'ψ'),
+ ]
+
+def _seg_67():
+ return [
(0x1D788, 'M', u'ω'),
(0x1D789, 'M', u'∂'),
(0x1D78A, 'M', u'ε'),
@@ -6971,10 +6988,6 @@ def _seg_66():
(0x1D792, 'M', u'γ'),
(0x1D793, 'M', u'δ'),
(0x1D794, 'M', u'ε'),
- ]
-
-def _seg_67():
- return [
(0x1D795, 'M', u'ζ'),
(0x1D796, 'M', u'η'),
(0x1D797, 'M', u'θ'),
@@ -7062,6 +7075,10 @@ def _seg_67():
(0x1D7EC, 'M', u'0'),
(0x1D7ED, 'M', u'1'),
(0x1D7EE, 'M', u'2'),
+ ]
+
+def _seg_68():
+ return [
(0x1D7EF, 'M', u'3'),
(0x1D7F0, 'M', u'4'),
(0x1D7F1, 'M', u'5'),
@@ -7075,10 +7092,6 @@ def _seg_67():
(0x1D7F9, 'M', u'3'),
(0x1D7FA, 'M', u'4'),
(0x1D7FB, 'M', u'5'),
- ]
-
-def _seg_68():
- return [
(0x1D7FC, 'M', u'6'),
(0x1D7FD, 'M', u'7'),
(0x1D7FE, 'M', u'8'),
@@ -7143,6 +7156,8 @@ def _seg_68():
(0x1E95A, 'X'),
(0x1E95E, 'V'),
(0x1E960, 'X'),
+ (0x1EC71, 'V'),
+ (0x1ECB5, 'X'),
(0x1EE00, 'M', u'ا'),
(0x1EE01, 'M', u'ب'),
(0x1EE02, 'M', u'ج'),
@@ -7164,6 +7179,10 @@ def _seg_68():
(0x1EE12, 'M', u'ق'),
(0x1EE13, 'M', u'ر'),
(0x1EE14, 'M', u'ش'),
+ ]
+
+def _seg_69():
+ return [
(0x1EE15, 'M', u'ت'),
(0x1EE16, 'M', u'ث'),
(0x1EE17, 'M', u'خ'),
@@ -7179,10 +7198,6 @@ def _seg_68():
(0x1EE21, 'M', u'ب'),
(0x1EE22, 'M', u'ج'),
(0x1EE23, 'X'),
- ]
-
-def _seg_69():
- return [
(0x1EE24, 'M', u'ه'),
(0x1EE25, 'X'),
(0x1EE27, 'M', u'ح'),
@@ -7268,6 +7283,10 @@ def _seg_69():
(0x1EE81, 'M', u'ب'),
(0x1EE82, 'M', u'ج'),
(0x1EE83, 'M', u'د'),
+ ]
+
+def _seg_70():
+ return [
(0x1EE84, 'M', u'ه'),
(0x1EE85, 'M', u'و'),
(0x1EE86, 'M', u'ز'),
@@ -7283,10 +7302,6 @@ def _seg_69():
(0x1EE90, 'M', u'ف'),
(0x1EE91, 'M', u'ص'),
(0x1EE92, 'M', u'ق'),
- ]
-
-def _seg_70():
- return [
(0x1EE93, 'M', u'ر'),
(0x1EE94, 'M', u'ش'),
(0x1EE95, 'M', u'ت'),
@@ -7372,6 +7387,10 @@ def _seg_70():
(0x1F122, '3', u'(s)'),
(0x1F123, '3', u'(t)'),
(0x1F124, '3', u'(u)'),
+ ]
+
+def _seg_71():
+ return [
(0x1F125, '3', u'(v)'),
(0x1F126, '3', u'(w)'),
(0x1F127, '3', u'(x)'),
@@ -7382,15 +7401,11 @@ def _seg_70():
(0x1F12C, 'M', u'r'),
(0x1F12D, 'M', u'cd'),
(0x1F12E, 'M', u'wz'),
- (0x1F12F, 'X'),
+ (0x1F12F, 'V'),
(0x1F130, 'M', u'a'),
(0x1F131, 'M', u'b'),
(0x1F132, 'M', u'c'),
(0x1F133, 'M', u'd'),
- ]
-
-def _seg_71():
- return [
(0x1F134, 'M', u'e'),
(0x1F135, 'M', u'f'),
(0x1F136, 'M', u'g'),
@@ -7476,6 +7491,10 @@ def _seg_71():
(0x1F239, 'M', u'割'),
(0x1F23A, 'M', u'営'),
(0x1F23B, 'M', u'配'),
+ ]
+
+def _seg_72():
+ return [
(0x1F23C, 'X'),
(0x1F240, 'M', u'〔本〕'),
(0x1F241, 'M', u'〔三〕'),
@@ -7491,21 +7510,17 @@ def _seg_71():
(0x1F251, 'M', u'可'),
(0x1F252, 'X'),
(0x1F260, 'V'),
- ]
-
-def _seg_72():
- return [
(0x1F266, 'X'),
(0x1F300, 'V'),
(0x1F6D5, 'X'),
(0x1F6E0, 'V'),
(0x1F6ED, 'X'),
(0x1F6F0, 'V'),
- (0x1F6F9, 'X'),
+ (0x1F6FA, 'X'),
(0x1F700, 'V'),
(0x1F774, 'X'),
(0x1F780, 'V'),
- (0x1F7D5, 'X'),
+ (0x1F7D9, 'X'),
(0x1F800, 'V'),
(0x1F80C, 'X'),
(0x1F810, 'V'),
@@ -7521,15 +7536,21 @@ def _seg_72():
(0x1F910, 'V'),
(0x1F93F, 'X'),
(0x1F940, 'V'),
- (0x1F94D, 'X'),
- (0x1F950, 'V'),
- (0x1F96C, 'X'),
- (0x1F980, 'V'),
- (0x1F998, 'X'),
+ (0x1F971, 'X'),
+ (0x1F973, 'V'),
+ (0x1F977, 'X'),
+ (0x1F97A, 'V'),
+ (0x1F97B, 'X'),
+ (0x1F97C, 'V'),
+ (0x1F9A3, 'X'),
+ (0x1F9B0, 'V'),
+ (0x1F9BA, 'X'),
(0x1F9C0, 'V'),
- (0x1F9C1, 'X'),
+ (0x1F9C3, 'X'),
(0x1F9D0, 'V'),
- (0x1F9E7, 'X'),
+ (0x1FA00, 'X'),
+ (0x1FA60, 'V'),
+ (0x1FA6E, 'X'),
(0x20000, 'V'),
(0x2A6D7, 'X'),
(0x2A700, 'V'),
@@ -7574,6 +7595,10 @@ def _seg_72():
(0x2F81F, 'M', u'㓟'),
(0x2F820, 'M', u'刻'),
(0x2F821, 'M', u'剆'),
+ ]
+
+def _seg_73():
+ return [
(0x2F822, 'M', u'割'),
(0x2F823, 'M', u'剷'),
(0x2F824, 'M', u'㔕'),
@@ -7595,10 +7620,6 @@ def _seg_72():
(0x2F836, 'M', u'及'),
(0x2F837, 'M', u'叟'),
(0x2F838, 'M', u'𠭣'),
- ]
-
-def _seg_73():
- return [
(0x2F839, 'M', u'叫'),
(0x2F83A, 'M', u'叱'),
(0x2F83B, 'M', u'吆'),
@@ -7678,6 +7699,10 @@ def _seg_73():
(0x2F887, 'M', u'幩'),
(0x2F888, 'M', u'㡢'),
(0x2F889, 'M', u'𢆃'),
+ ]
+
+def _seg_74():
+ return [
(0x2F88A, 'M', u'㡼'),
(0x2F88B, 'M', u'庰'),
(0x2F88C, 'M', u'庳'),
@@ -7699,10 +7724,6 @@ def _seg_73():
(0x2F89E, 'M', u'志'),
(0x2F89F, 'M', u'忹'),
(0x2F8A0, 'M', u'悁'),
- ]
-
-def _seg_74():
- return [
(0x2F8A1, 'M', u'㤺'),
(0x2F8A2, 'M', u'㤜'),
(0x2F8A3, 'M', u'悔'),
@@ -7782,6 +7803,10 @@ def _seg_74():
(0x2F8ED, 'M', u'櫛'),
(0x2F8EE, 'M', u'㰘'),
(0x2F8EF, 'M', u'次'),
+ ]
+
+def _seg_75():
+ return [
(0x2F8F0, 'M', u'𣢧'),
(0x2F8F1, 'M', u'歔'),
(0x2F8F2, 'M', u'㱎'),
@@ -7803,10 +7828,6 @@ def _seg_74():
(0x2F902, 'M', u'流'),
(0x2F903, 'M', u'浩'),
(0x2F904, 'M', u'浸'),
- ]
-
-def _seg_75():
- return [
(0x2F905, 'M', u'涅'),
(0x2F906, 'M', u'𣴞'),
(0x2F907, 'M', u'洴'),
@@ -7886,6 +7907,10 @@ def _seg_75():
(0x2F953, 'M', u'祖'),
(0x2F954, 'M', u'𥚚'),
(0x2F955, 'M', u'𥛅'),
+ ]
+
+def _seg_76():
+ return [
(0x2F956, 'M', u'福'),
(0x2F957, 'M', u'秫'),
(0x2F958, 'M', u'䄯'),
@@ -7907,10 +7932,6 @@ def _seg_75():
(0x2F969, 'M', u'糣'),
(0x2F96A, 'M', u'紀'),
(0x2F96B, 'M', u'𥾆'),
- ]
-
-def _seg_76():
- return [
(0x2F96C, 'M', u'絣'),
(0x2F96D, 'M', u'䌁'),
(0x2F96E, 'M', u'緇'),
@@ -7990,6 +8011,10 @@ def _seg_76():
(0x2F9B8, 'M', u'蚈'),
(0x2F9B9, 'M', u'蜎'),
(0x2F9BA, 'M', u'蛢'),
+ ]
+
+def _seg_77():
+ return [
(0x2F9BB, 'M', u'蝹'),
(0x2F9BC, 'M', u'蜨'),
(0x2F9BD, 'M', u'蝫'),
@@ -8011,10 +8036,6 @@ def _seg_76():
(0x2F9CD, 'M', u'䚾'),
(0x2F9CE, 'M', u'䛇'),
(0x2F9CF, 'M', u'誠'),
- ]
-
-def _seg_77():
- return [
(0x2F9D0, 'M', u'諭'),
(0x2F9D1, 'M', u'變'),
(0x2F9D2, 'M', u'豕'),
@@ -8094,6 +8115,10 @@ def _seg_77():
(0x2FA1D, 'M', u'𪘀'),
(0x2FA1E, 'X'),
(0xE0100, 'I'),
+ ]
+
+def _seg_78():
+ return [
(0xE01F0, 'X'),
]
@@ -8176,4 +8201,5 @@ uts46data = tuple(
+ _seg_75()
+ _seg_76()
+ _seg_77()
+ + _seg_78()
)
diff --git a/pipenv/patched/notpip/_vendor/packaging/__about__.py b/pipenv/patched/notpip/_vendor/packaging/__about__.py
index 21fc6ce3..7481c9e2 100644
--- a/pipenv/patched/notpip/_vendor/packaging/__about__.py
+++ b/pipenv/patched/notpip/_vendor/packaging/__about__.py
@@ -4,18 +4,24 @@
from __future__ import absolute_import, division, print_function
__all__ = [
- "__title__", "__summary__", "__uri__", "__version__", "__author__",
- "__email__", "__license__", "__copyright__",
+ "__title__",
+ "__summary__",
+ "__uri__",
+ "__version__",
+ "__author__",
+ "__email__",
+ "__license__",
+ "__copyright__",
]
__title__ = "packaging"
__summary__ = "Core utilities for Python packages"
__uri__ = "https://github.com/pypa/packaging"
-__version__ = "18.0"
+__version__ = "19.0"
__author__ = "Donald Stufft and individual contributors"
__email__ = "donald@stufft.io"
__license__ = "BSD or Apache License, Version 2.0"
-__copyright__ = "Copyright 2014-2018 %s" % __author__
+__copyright__ = "Copyright 2014-2019 %s" % __author__
diff --git a/pipenv/patched/notpip/_vendor/packaging/__init__.py b/pipenv/patched/notpip/_vendor/packaging/__init__.py
index 5ee62202..a0cf67df 100644
--- a/pipenv/patched/notpip/_vendor/packaging/__init__.py
+++ b/pipenv/patched/notpip/_vendor/packaging/__init__.py
@@ -4,11 +4,23 @@
from __future__ import absolute_import, division, print_function
from .__about__ import (
- __author__, __copyright__, __email__, __license__, __summary__, __title__,
- __uri__, __version__
+ __author__,
+ __copyright__,
+ __email__,
+ __license__,
+ __summary__,
+ __title__,
+ __uri__,
+ __version__,
)
__all__ = [
- "__title__", "__summary__", "__uri__", "__version__", "__author__",
- "__email__", "__license__", "__copyright__",
+ "__title__",
+ "__summary__",
+ "__uri__",
+ "__version__",
+ "__author__",
+ "__email__",
+ "__license__",
+ "__copyright__",
]
diff --git a/pipenv/patched/notpip/_vendor/packaging/_compat.py b/pipenv/patched/notpip/_vendor/packaging/_compat.py
index 210bb80b..25da473c 100644
--- a/pipenv/patched/notpip/_vendor/packaging/_compat.py
+++ b/pipenv/patched/notpip/_vendor/packaging/_compat.py
@@ -12,9 +12,9 @@ PY3 = sys.version_info[0] == 3
# flake8: noqa
if PY3:
- string_types = str,
+ string_types = (str,)
else:
- string_types = basestring,
+ string_types = (basestring,)
def with_metaclass(meta, *bases):
@@ -27,4 +27,5 @@ def with_metaclass(meta, *bases):
class metaclass(meta):
def __new__(cls, name, this_bases, d):
return meta(name, bases, d)
- return type.__new__(metaclass, 'temporary_class', (), {})
+
+ return type.__new__(metaclass, "temporary_class", (), {})
diff --git a/pipenv/patched/notpip/_vendor/packaging/_structures.py b/pipenv/patched/notpip/_vendor/packaging/_structures.py
index e9fc4a04..68dcca63 100644
--- a/pipenv/patched/notpip/_vendor/packaging/_structures.py
+++ b/pipenv/patched/notpip/_vendor/packaging/_structures.py
@@ -5,7 +5,6 @@ from __future__ import absolute_import, division, print_function
class Infinity(object):
-
def __repr__(self):
return "Infinity"
@@ -38,7 +37,6 @@ Infinity = Infinity()
class NegativeInfinity(object):
-
def __repr__(self):
return "-Infinity"
diff --git a/pipenv/patched/notpip/_vendor/packaging/markers.py b/pipenv/patched/notpip/_vendor/packaging/markers.py
index dc3cef81..50a08091 100644
--- a/pipenv/patched/notpip/_vendor/packaging/markers.py
+++ b/pipenv/patched/notpip/_vendor/packaging/markers.py
@@ -17,8 +17,11 @@ from .specifiers import Specifier, InvalidSpecifier
__all__ = [
- "InvalidMarker", "UndefinedComparison", "UndefinedEnvironmentName",
- "Marker", "default_environment",
+ "InvalidMarker",
+ "UndefinedComparison",
+ "UndefinedEnvironmentName",
+ "Marker",
+ "default_environment",
]
@@ -42,7 +45,6 @@ class UndefinedEnvironmentName(ValueError):
class Node(object):
-
def __init__(self, value):
self.value = value
@@ -57,62 +59,52 @@ class Node(object):
class Variable(Node):
-
def serialize(self):
return str(self)
class Value(Node):
-
def serialize(self):
return '"{0}"'.format(self)
class Op(Node):
-
def serialize(self):
return str(self)
VARIABLE = (
- L("implementation_version") |
- L("platform_python_implementation") |
- L("implementation_name") |
- L("python_full_version") |
- L("platform_release") |
- L("platform_version") |
- L("platform_machine") |
- L("platform_system") |
- L("python_version") |
- L("sys_platform") |
- L("os_name") |
- L("os.name") | # PEP-345
- L("sys.platform") | # PEP-345
- L("platform.version") | # PEP-345
- L("platform.machine") | # PEP-345
- L("platform.python_implementation") | # PEP-345
- L("python_implementation") | # undocumented setuptools legacy
- L("extra")
+ L("implementation_version")
+ | L("platform_python_implementation")
+ | L("implementation_name")
+ | L("python_full_version")
+ | L("platform_release")
+ | L("platform_version")
+ | L("platform_machine")
+ | L("platform_system")
+ | L("python_version")
+ | L("sys_platform")
+ | L("os_name")
+ | L("os.name")
+ | L("sys.platform") # PEP-345
+ | L("platform.version") # PEP-345
+ | L("platform.machine") # PEP-345
+ | L("platform.python_implementation") # PEP-345
+ | L("python_implementation") # PEP-345
+ | L("extra") # undocumented setuptools legacy
)
ALIASES = {
- 'os.name': 'os_name',
- 'sys.platform': 'sys_platform',
- 'platform.version': 'platform_version',
- 'platform.machine': 'platform_machine',
- 'platform.python_implementation': 'platform_python_implementation',
- 'python_implementation': 'platform_python_implementation'
+ "os.name": "os_name",
+ "sys.platform": "sys_platform",
+ "platform.version": "platform_version",
+ "platform.machine": "platform_machine",
+ "platform.python_implementation": "platform_python_implementation",
+ "python_implementation": "platform_python_implementation",
}
VARIABLE.setParseAction(lambda s, l, t: Variable(ALIASES.get(t[0], t[0])))
VERSION_CMP = (
- L("===") |
- L("==") |
- L(">=") |
- L("<=") |
- L("!=") |
- L("~=") |
- L(">") |
- L("<")
+ L("===") | L("==") | L(">=") | L("<=") | L("!=") | L("~=") | L(">") | L("<")
)
MARKER_OP = VERSION_CMP | L("not in") | L("in")
@@ -152,8 +144,11 @@ def _format_marker(marker, first=True):
# where the single item is itself it's own list. In that case we want skip
# the rest of this function so that we don't get extraneous () on the
# outside.
- if (isinstance(marker, list) and len(marker) == 1 and
- isinstance(marker[0], (list, tuple))):
+ if (
+ isinstance(marker, list)
+ and len(marker) == 1
+ and isinstance(marker[0], (list, tuple))
+ ):
return _format_marker(marker[0])
if isinstance(marker, list):
@@ -239,20 +234,20 @@ def _evaluate_markers(markers, environment):
def format_full_version(info):
- version = '{0.major}.{0.minor}.{0.micro}'.format(info)
+ version = "{0.major}.{0.minor}.{0.micro}".format(info)
kind = info.releaselevel
- if kind != 'final':
+ if kind != "final":
version += kind[0] + str(info.serial)
return version
def default_environment():
- if hasattr(sys, 'implementation'):
+ if hasattr(sys, "implementation"):
iver = format_full_version(sys.implementation.version)
implementation_name = sys.implementation.name
else:
- iver = '0'
- implementation_name = ''
+ iver = "0"
+ implementation_name = ""
return {
"implementation_name": implementation_name,
@@ -270,13 +265,13 @@ def default_environment():
class Marker(object):
-
def __init__(self, marker):
try:
self._markers = _coerce_parse_result(MARKER.parseString(marker))
except ParseException as e:
err_str = "Invalid marker: {0!r}, parse error at {1!r}".format(
- marker, marker[e.loc:e.loc + 8])
+ marker, marker[e.loc : e.loc + 8]
+ )
raise InvalidMarker(err_str)
def __str__(self):
diff --git a/pipenv/patched/notpip/_vendor/packaging/requirements.py b/pipenv/patched/notpip/_vendor/packaging/requirements.py
index 5ec5d74a..a3de7673 100644
--- a/pipenv/patched/notpip/_vendor/packaging/requirements.py
+++ b/pipenv/patched/notpip/_vendor/packaging/requirements.py
@@ -38,8 +38,8 @@ IDENTIFIER = Combine(ALPHANUM + ZeroOrMore(IDENTIFIER_END))
NAME = IDENTIFIER("name")
EXTRA = IDENTIFIER
-URI = Regex(r'[^ ]+')("url")
-URL = (AT + URI)
+URI = Regex(r"[^ ]+")("url")
+URL = AT + URI
EXTRAS_LIST = EXTRA + ZeroOrMore(COMMA + EXTRA)
EXTRAS = (LBRACKET + Optional(EXTRAS_LIST) + RBRACKET)("extras")
@@ -48,17 +48,18 @@ VERSION_PEP440 = Regex(Specifier._regex_str, re.VERBOSE | re.IGNORECASE)
VERSION_LEGACY = Regex(LegacySpecifier._regex_str, re.VERBOSE | re.IGNORECASE)
VERSION_ONE = VERSION_PEP440 ^ VERSION_LEGACY
-VERSION_MANY = Combine(VERSION_ONE + ZeroOrMore(COMMA + VERSION_ONE),
- joinString=",", adjacent=False)("_raw_spec")
+VERSION_MANY = Combine(
+ VERSION_ONE + ZeroOrMore(COMMA + VERSION_ONE), joinString=",", adjacent=False
+)("_raw_spec")
_VERSION_SPEC = Optional(((LPAREN + VERSION_MANY + RPAREN) | VERSION_MANY))
-_VERSION_SPEC.setParseAction(lambda s, l, t: t._raw_spec or '')
+_VERSION_SPEC.setParseAction(lambda s, l, t: t._raw_spec or "")
VERSION_SPEC = originalTextFor(_VERSION_SPEC)("specifier")
VERSION_SPEC.setParseAction(lambda s, l, t: t[1])
MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker")
MARKER_EXPR.setParseAction(
- lambda s, l, t: Marker(s[t._original_start:t._original_end])
+ lambda s, l, t: Marker(s[t._original_start : t._original_end])
)
MARKER_SEPARATOR = SEMICOLON
MARKER = MARKER_SEPARATOR + MARKER_EXPR
@@ -66,8 +67,7 @@ MARKER = MARKER_SEPARATOR + MARKER_EXPR
VERSION_AND_MARKER = VERSION_SPEC + Optional(MARKER)
URL_AND_MARKER = URL + Optional(MARKER)
-NAMED_REQUIREMENT = \
- NAME + Optional(EXTRAS) + (URL_AND_MARKER | VERSION_AND_MARKER)
+NAMED_REQUIREMENT = NAME + Optional(EXTRAS) + (URL_AND_MARKER | VERSION_AND_MARKER)
REQUIREMENT = stringStart + NAMED_REQUIREMENT + stringEnd
# pyparsing isn't thread safe during initialization, so we do it eagerly, see
@@ -92,15 +92,21 @@ class Requirement(object):
try:
req = REQUIREMENT.parseString(requirement_string)
except ParseException as e:
- raise InvalidRequirement("Parse error at \"{0!r}\": {1}".format(
- requirement_string[e.loc:e.loc + 8], e.msg
- ))
+ raise InvalidRequirement(
+ 'Parse error at "{0!r}": {1}'.format(
+ requirement_string[e.loc : e.loc + 8], e.msg
+ )
+ )
self.name = req.name
if req.url:
parsed_url = urlparse.urlparse(req.url)
- if not (parsed_url.scheme and parsed_url.netloc) or (
- not parsed_url.scheme and not parsed_url.netloc):
+ if parsed_url.scheme == "file":
+ if urlparse.urlunparse(parsed_url) != req.url:
+ raise InvalidRequirement("Invalid URL given")
+ elif not (parsed_url.scheme and parsed_url.netloc) or (
+ not parsed_url.scheme and not parsed_url.netloc
+ ):
raise InvalidRequirement("Invalid URL: {0}".format(req.url))
self.url = req.url
else:
@@ -120,6 +126,8 @@ class Requirement(object):
if self.url:
parts.append("@ {0}".format(self.url))
+ if self.marker:
+ parts.append(" ")
if self.marker:
parts.append("; {0}".format(self.marker))
diff --git a/pipenv/patched/notpip/_vendor/packaging/specifiers.py b/pipenv/patched/notpip/_vendor/packaging/specifiers.py
index 4c798999..743576a0 100644
--- a/pipenv/patched/notpip/_vendor/packaging/specifiers.py
+++ b/pipenv/patched/notpip/_vendor/packaging/specifiers.py
@@ -19,7 +19,6 @@ class InvalidSpecifier(ValueError):
class BaseSpecifier(with_metaclass(abc.ABCMeta, object)):
-
@abc.abstractmethod
def __str__(self):
"""
@@ -84,10 +83,7 @@ class _IndividualSpecifier(BaseSpecifier):
if not match:
raise InvalidSpecifier("Invalid specifier: '{0}'".format(spec))
- self._spec = (
- match.group("operator").strip(),
- match.group("version").strip(),
- )
+ self._spec = (match.group("operator").strip(), match.group("version").strip())
# Store whether or not this Specifier should accept prereleases
self._prereleases = prereleases
@@ -99,11 +95,7 @@ class _IndividualSpecifier(BaseSpecifier):
else ""
)
- return "<{0}({1!r}{2})>".format(
- self.__class__.__name__,
- str(self),
- pre,
- )
+ return "<{0}({1!r}{2})>".format(self.__class__.__name__, str(self), pre)
def __str__(self):
return "{0}{1}".format(*self._spec)
@@ -194,8 +186,9 @@ class _IndividualSpecifier(BaseSpecifier):
# If our version is a prerelease, and we were not set to allow
# prereleases, then we'll store it for later incase nothing
# else matches this specifier.
- if (parsed_version.is_prerelease and not
- (prereleases or self.prereleases)):
+ if parsed_version.is_prerelease and not (
+ prereleases or self.prereleases
+ ):
found_prereleases.append(version)
# Either this is not a prerelease, or we should have been
# accepting prereleases from the beginning.
@@ -213,8 +206,7 @@ class _IndividualSpecifier(BaseSpecifier):
class LegacySpecifier(_IndividualSpecifier):
- _regex_str = (
- r"""
+ _regex_str = r"""
(?P(==|!=|<=|>=|<|>))
\s*
(?P
@@ -225,10 +217,8 @@ class LegacySpecifier(_IndividualSpecifier):
# them, and a comma since it's a version separator.
)
"""
- )
- _regex = re.compile(
- r"^\s*" + _regex_str + r"\s*$", re.VERBOSE | re.IGNORECASE)
+ _regex = re.compile(r"^\s*" + _regex_str + r"\s*$", re.VERBOSE | re.IGNORECASE)
_operators = {
"==": "equal",
@@ -269,13 +259,13 @@ def _require_version_compare(fn):
if not isinstance(prospective, Version):
return False
return fn(self, prospective, spec)
+
return wrapped
class Specifier(_IndividualSpecifier):
- _regex_str = (
- r"""
+ _regex_str = r"""
(?P(~=|==|!=|<=|>=|<|>|===))
(?P
(?:
@@ -367,10 +357,8 @@ class Specifier(_IndividualSpecifier):
)
)
"""
- )
- _regex = re.compile(
- r"^\s*" + _regex_str + r"\s*$", re.VERBOSE | re.IGNORECASE)
+ _regex = re.compile(r"^\s*" + _regex_str + r"\s*$", re.VERBOSE | re.IGNORECASE)
_operators = {
"~=": "compatible",
@@ -397,8 +385,7 @@ class Specifier(_IndividualSpecifier):
prefix = ".".join(
list(
itertools.takewhile(
- lambda x: (not x.startswith("post") and not
- x.startswith("dev")),
+ lambda x: (not x.startswith("post") and not x.startswith("dev")),
_version_split(spec),
)
)[:-1]
@@ -407,8 +394,9 @@ class Specifier(_IndividualSpecifier):
# Add the prefix notation to the end of our string
prefix += ".*"
- return (self._get_operator(">=")(prospective, spec) and
- self._get_operator("==")(prospective, prefix))
+ return self._get_operator(">=")(prospective, spec) and self._get_operator("==")(
+ prospective, prefix
+ )
@_require_version_compare
def _compare_equal(self, prospective, spec):
@@ -428,7 +416,7 @@ class Specifier(_IndividualSpecifier):
# Shorten the prospective version to be the same length as the spec
# so that we can determine if the specifier is a prefix of the
# prospective version or not.
- prospective = prospective[:len(spec)]
+ prospective = prospective[: len(spec)]
# Pad out our two sides with zeros so that they both equal the same
# length.
@@ -567,27 +555,17 @@ def _pad_version(left, right):
right_split.append(list(itertools.takewhile(lambda x: x.isdigit(), right)))
# Get the rest of our versions
- left_split.append(left[len(left_split[0]):])
- right_split.append(right[len(right_split[0]):])
+ left_split.append(left[len(left_split[0]) :])
+ right_split.append(right[len(right_split[0]) :])
# Insert our padding
- left_split.insert(
- 1,
- ["0"] * max(0, len(right_split[0]) - len(left_split[0])),
- )
- right_split.insert(
- 1,
- ["0"] * max(0, len(left_split[0]) - len(right_split[0])),
- )
+ left_split.insert(1, ["0"] * max(0, len(right_split[0]) - len(left_split[0])))
+ right_split.insert(1, ["0"] * max(0, len(left_split[0]) - len(right_split[0])))
- return (
- list(itertools.chain(*left_split)),
- list(itertools.chain(*right_split)),
- )
+ return (list(itertools.chain(*left_split)), list(itertools.chain(*right_split)))
class SpecifierSet(BaseSpecifier):
-
def __init__(self, specifiers="", prereleases=None):
# Split on , to break each indidivual specifier into it's own item, and
# strip each item to remove leading/trailing whitespace.
@@ -721,10 +699,7 @@ class SpecifierSet(BaseSpecifier):
# given version is contained within all of them.
# Note: This use of all() here means that an empty set of specifiers
# will always return True, this is an explicit design decision.
- return all(
- s.contains(item, prereleases=prereleases)
- for s in self._specs
- )
+ return all(s.contains(item, prereleases=prereleases) for s in self._specs)
def filter(self, iterable, prereleases=None):
# Determine if we're forcing a prerelease or not, if we're not forcing
diff --git a/pipenv/patched/notpip/_vendor/packaging/utils.py b/pipenv/patched/notpip/_vendor/packaging/utils.py
index 4b94a82f..88418786 100644
--- a/pipenv/patched/notpip/_vendor/packaging/utils.py
+++ b/pipenv/patched/notpip/_vendor/packaging/utils.py
@@ -36,13 +36,7 @@ def canonicalize_version(version):
# Release segment
# NB: This strips trailing '.0's to normalize
- parts.append(
- re.sub(
- r'(\.0)+$',
- '',
- ".".join(str(x) for x in version.release)
- )
- )
+ parts.append(re.sub(r"(\.0)+$", "", ".".join(str(x) for x in version.release)))
# Pre-release
if version.pre is not None:
diff --git a/pipenv/patched/notpip/_vendor/packaging/version.py b/pipenv/patched/notpip/_vendor/packaging/version.py
index 6ed5cbbd..95157a1f 100644
--- a/pipenv/patched/notpip/_vendor/packaging/version.py
+++ b/pipenv/patched/notpip/_vendor/packaging/version.py
@@ -10,14 +10,11 @@ import re
from ._structures import Infinity
-__all__ = [
- "parse", "Version", "LegacyVersion", "InvalidVersion", "VERSION_PATTERN"
-]
+__all__ = ["parse", "Version", "LegacyVersion", "InvalidVersion", "VERSION_PATTERN"]
_Version = collections.namedtuple(
- "_Version",
- ["epoch", "release", "dev", "pre", "post", "local"],
+ "_Version", ["epoch", "release", "dev", "pre", "post", "local"]
)
@@ -40,7 +37,6 @@ class InvalidVersion(ValueError):
class _BaseVersion(object):
-
def __hash__(self):
return hash(self._key)
@@ -70,7 +66,6 @@ class _BaseVersion(object):
class LegacyVersion(_BaseVersion):
-
def __init__(self, version):
self._version = str(version)
self._key = _legacy_cmpkey(self._version)
@@ -126,12 +121,14 @@ class LegacyVersion(_BaseVersion):
return False
-_legacy_version_component_re = re.compile(
- r"(\d+ | [a-z]+ | \.| -)", re.VERBOSE,
-)
+_legacy_version_component_re = re.compile(r"(\d+ | [a-z]+ | \.| -)", re.VERBOSE)
_legacy_version_replacement_map = {
- "pre": "c", "preview": "c", "-": "final-", "rc": "c", "dev": "@",
+ "pre": "c",
+ "preview": "c",
+ "-": "final-",
+ "rc": "c",
+ "dev": "@",
}
@@ -215,10 +212,7 @@ VERSION_PATTERN = r"""
class Version(_BaseVersion):
- _regex = re.compile(
- r"^\s*" + VERSION_PATTERN + r"\s*$",
- re.VERBOSE | re.IGNORECASE,
- )
+ _regex = re.compile(r"^\s*" + VERSION_PATTERN + r"\s*$", re.VERBOSE | re.IGNORECASE)
def __init__(self, version):
# Validate the version and parse it into pieces
@@ -230,18 +224,11 @@ class Version(_BaseVersion):
self._version = _Version(
epoch=int(match.group("epoch")) if match.group("epoch") else 0,
release=tuple(int(i) for i in match.group("release").split(".")),
- pre=_parse_letter_version(
- match.group("pre_l"),
- match.group("pre_n"),
- ),
+ pre=_parse_letter_version(match.group("pre_l"), match.group("pre_n")),
post=_parse_letter_version(
- match.group("post_l"),
- match.group("post_n1") or match.group("post_n2"),
- ),
- dev=_parse_letter_version(
- match.group("dev_l"),
- match.group("dev_n"),
+ match.group("post_l"), match.group("post_n1") or match.group("post_n2")
),
+ dev=_parse_letter_version(match.group("dev_l"), match.group("dev_n")),
local=_parse_local_version(match.group("local")),
)
@@ -395,12 +382,7 @@ def _cmpkey(epoch, release, pre, post, dev, local):
# re-reverse it back into the correct order and make it a tuple and use
# that for our sorting key.
release = tuple(
- reversed(list(
- itertools.dropwhile(
- lambda x: x == 0,
- reversed(release),
- )
- ))
+ reversed(list(itertools.dropwhile(lambda x: x == 0, reversed(release))))
)
# We need to "trick" the sorting algorithm to put 1.0.dev0 before 1.0a0.
@@ -433,9 +415,6 @@ def _cmpkey(epoch, release, pre, post, dev, local):
# - Numeric segments sort numerically
# - Shorter versions sort before longer versions when the prefixes
# match exactly
- local = tuple(
- (i, "") if isinstance(i, int) else (-Infinity, i)
- for i in local
- )
+ local = tuple((i, "") if isinstance(i, int) else (-Infinity, i) for i in local)
return epoch, release, pre, post, dev, local
diff --git a/pipenv/patched/notpip/_vendor/pep517/__init__.py b/pipenv/patched/notpip/_vendor/pep517/__init__.py
index 8beedea4..9c1a098f 100644
--- a/pipenv/patched/notpip/_vendor/pep517/__init__.py
+++ b/pipenv/patched/notpip/_vendor/pep517/__init__.py
@@ -1,4 +1,4 @@
"""Wrappers to build Python packages using PEP 517 hooks
"""
-__version__ = '0.2'
+__version__ = '0.5.0'
diff --git a/pipenv/patched/notpip/_vendor/pep517/_in_process.py b/pipenv/patched/notpip/_vendor/pep517/_in_process.py
index baa14d38..d6524b66 100644
--- a/pipenv/patched/notpip/_vendor/pep517/_in_process.py
+++ b/pipenv/patched/notpip/_vendor/pep517/_in_process.py
@@ -21,19 +21,28 @@ import sys
# This is run as a script, not a module, so it can't do a relative import
import compat
+
+class BackendUnavailable(Exception):
+ """Raised if we cannot import the backend"""
+
+
def _build_backend():
"""Find and load the build backend"""
ep = os.environ['PEP517_BUILD_BACKEND']
mod_path, _, obj_path = ep.partition(':')
- obj = import_module(mod_path)
+ try:
+ obj = import_module(mod_path)
+ except ImportError:
+ raise BackendUnavailable
if obj_path:
for path_part in obj_path.split('.'):
obj = getattr(obj, path_part)
return obj
+
def get_requires_for_build_wheel(config_settings):
"""Invoke the optional get_requires_for_build_wheel hook
-
+
Returns [] if the hook is not defined.
"""
backend = _build_backend()
@@ -44,9 +53,10 @@ def get_requires_for_build_wheel(config_settings):
else:
return hook(config_settings)
+
def prepare_metadata_for_build_wheel(metadata_directory, config_settings):
"""Invoke optional prepare_metadata_for_build_wheel
-
+
Implements a fallback by building a wheel if the hook isn't defined.
"""
backend = _build_backend()
@@ -58,8 +68,10 @@ def prepare_metadata_for_build_wheel(metadata_directory, config_settings):
else:
return hook(metadata_directory, config_settings)
+
WHEEL_BUILT_MARKER = 'PEP517_ALREADY_BUILT_WHEEL'
+
def _dist_info_files(whl_zip):
"""Identify the .dist-info folder inside a wheel ZipFile."""
res = []
@@ -71,11 +83,13 @@ def _dist_info_files(whl_zip):
return res
raise Exception("No .dist-info folder found in wheel")
-def _get_wheel_metadata_from_wheel(backend, metadata_directory, config_settings):
+
+def _get_wheel_metadata_from_wheel(
+ backend, metadata_directory, config_settings):
"""Build a wheel and extract the metadata from it.
-
- Fallback for when the build backend does not define the 'get_wheel_metadata'
- hook.
+
+ Fallback for when the build backend does not
+ define the 'get_wheel_metadata' hook.
"""
from zipfile import ZipFile
whl_basename = backend.build_wheel(metadata_directory, config_settings)
@@ -88,6 +102,7 @@ def _get_wheel_metadata_from_wheel(backend, metadata_directory, config_settings)
zipf.extractall(path=metadata_directory, members=dist_info)
return dist_info[0].split('/')[0]
+
def _find_already_built_wheel(metadata_directory):
"""Check for a wheel already built during the get_wheel_metadata hook.
"""
@@ -105,14 +120,16 @@ def _find_already_built_wheel(metadata_directory):
print('Found multiple .whl files; unspecified behaviour. '
'Will call build_wheel.')
return None
-
+
# Exactly one .whl file
return whl_files[0]
+
def build_wheel(wheel_directory, config_settings, metadata_directory=None):
"""Invoke the mandatory build_wheel hook.
-
- If a wheel was already built in the prepare_metadata_for_build_wheel fallback, this
+
+ If a wheel was already built in the
+ prepare_metadata_for_build_wheel fallback, this
will copy it rather than rebuilding the wheel.
"""
prebuilt_whl = _find_already_built_wheel(metadata_directory)
@@ -137,12 +154,15 @@ def get_requires_for_build_sdist(config_settings):
else:
return hook(config_settings)
+
class _DummyException(Exception):
"""Nothing should ever raise this exception"""
+
class GotUnsupportedOperation(Exception):
"""For internal use when backend raises UnsupportedOperation"""
+
def build_sdist(sdist_directory, config_settings):
"""Invoke the mandatory build_sdist hook."""
backend = _build_backend()
@@ -151,6 +171,7 @@ def build_sdist(sdist_directory, config_settings):
except getattr(backend, 'UnsupportedOperation', _DummyException):
raise GotUnsupportedOperation
+
HOOK_NAMES = {
'get_requires_for_build_wheel',
'prepare_metadata_for_build_wheel',
@@ -159,6 +180,7 @@ HOOK_NAMES = {
'build_sdist',
}
+
def main():
if len(sys.argv) < 3:
sys.exit("Needs args: hook_name, control_dir")
@@ -173,10 +195,13 @@ def main():
json_out = {'unsupported': False, 'return_val': None}
try:
json_out['return_val'] = hook(**hook_input['kwargs'])
+ except BackendUnavailable:
+ json_out['no_backend'] = True
except GotUnsupportedOperation:
json_out['unsupported'] = True
-
+
compat.write_json(json_out, pjoin(control_dir, 'output.json'), indent=2)
+
if __name__ == '__main__':
main()
diff --git a/pipenv/patched/notpip/_vendor/pep517/build.py b/pipenv/patched/notpip/_vendor/pep517/build.py
new file mode 100644
index 00000000..db9a0799
--- /dev/null
+++ b/pipenv/patched/notpip/_vendor/pep517/build.py
@@ -0,0 +1,108 @@
+"""Build a project using PEP 517 hooks.
+"""
+import argparse
+import logging
+import os
+import contextlib
+from pipenv.patched.notpip._vendor import pytoml
+import shutil
+import errno
+import tempfile
+
+from .envbuild import BuildEnvironment
+from .wrappers import Pep517HookCaller
+
+log = logging.getLogger(__name__)
+
+
+@contextlib.contextmanager
+def tempdir():
+ td = tempfile.mkdtemp()
+ try:
+ yield td
+ finally:
+ shutil.rmtree(td)
+
+
+def _do_build(hooks, env, dist, dest):
+ get_requires_name = 'get_requires_for_build_{dist}'.format(**locals())
+ get_requires = getattr(hooks, get_requires_name)
+ reqs = get_requires({})
+ log.info('Got build requires: %s', reqs)
+
+ env.pip_install(reqs)
+ log.info('Installed dynamic build dependencies')
+
+ with tempdir() as td:
+ log.info('Trying to build %s in %s', dist, td)
+ build_name = 'build_{dist}'.format(**locals())
+ build = getattr(hooks, build_name)
+ filename = build(td, {})
+ source = os.path.join(td, filename)
+ shutil.move(source, os.path.join(dest, os.path.basename(filename)))
+
+
+def mkdir_p(*args, **kwargs):
+ """Like `mkdir`, but does not raise an exception if the
+ directory already exists.
+ """
+ try:
+ return os.mkdir(*args, **kwargs)
+ except OSError as exc:
+ if exc.errno != errno.EEXIST:
+ raise
+
+
+def build(source_dir, dist, dest=None):
+ pyproject = os.path.join(source_dir, 'pyproject.toml')
+ dest = os.path.join(source_dir, dest or 'dist')
+ mkdir_p(dest)
+
+ with open(pyproject) as f:
+ pyproject_data = pytoml.load(f)
+ # Ensure the mandatory data can be loaded
+ buildsys = pyproject_data['build-system']
+ requires = buildsys['requires']
+ backend = buildsys['build-backend']
+
+ hooks = Pep517HookCaller(source_dir, backend)
+
+ with BuildEnvironment() as env:
+ env.pip_install(requires)
+ _do_build(hooks, env, dist, dest)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+ 'source_dir',
+ help="A directory containing pyproject.toml",
+)
+parser.add_argument(
+ '--binary', '-b',
+ action='store_true',
+ default=False,
+)
+parser.add_argument(
+ '--source', '-s',
+ action='store_true',
+ default=False,
+)
+parser.add_argument(
+ '--out-dir', '-o',
+ help="Destination in which to save the builds relative to source dir",
+)
+
+
+def main(args):
+ # determine which dists to build
+ dists = list(filter(None, (
+ 'sdist' if args.source or not args.binary else None,
+ 'wheel' if args.binary or not args.source else None,
+ )))
+
+ for dist in dists:
+ build(args.source_dir, dist, args.out_dir)
+
+
+if __name__ == '__main__':
+ main(parser.parse_args())
diff --git a/pipenv/patched/notpip/_vendor/pep517/check.py b/pipenv/patched/notpip/_vendor/pep517/check.py
index 3dffd2e0..9d28ba44 100644
--- a/pipenv/patched/notpip/_vendor/pep517/check.py
+++ b/pipenv/patched/notpip/_vendor/pep517/check.py
@@ -18,10 +18,11 @@ from .wrappers import Pep517HookCaller
log = logging.getLogger(__name__)
-def check_build_sdist(hooks):
+
+def check_build_sdist(hooks, build_sys_requires):
with BuildEnvironment() as env:
try:
- env.pip_install(hooks.build_sys_requires)
+ env.pip_install(build_sys_requires)
log.info('Installed static build dependencies')
except CalledProcessError:
log.error('Failed to install static build dependencies')
@@ -30,7 +31,7 @@ def check_build_sdist(hooks):
try:
reqs = hooks.get_requires_for_build_sdist({})
log.info('Got build requires: %s', reqs)
- except:
+ except Exception:
log.error('Failure in get_requires_for_build_sdist', exc_info=True)
return False
@@ -47,12 +48,13 @@ def check_build_sdist(hooks):
try:
filename = hooks.build_sdist(td, {})
log.info('build_sdist returned %r', filename)
- except:
+ except Exception:
log.info('Failure in build_sdist', exc_info=True)
return False
if not filename.endswith('.tar.gz'):
- log.error("Filename %s doesn't have .tar.gz extension", filename)
+ log.error(
+ "Filename %s doesn't have .tar.gz extension", filename)
return False
path = pjoin(td, filename)
@@ -73,10 +75,11 @@ def check_build_sdist(hooks):
return True
-def check_build_wheel(hooks):
+
+def check_build_wheel(hooks, build_sys_requires):
with BuildEnvironment() as env:
try:
- env.pip_install(hooks.build_sys_requires)
+ env.pip_install(build_sys_requires)
log.info('Installed static build dependencies')
except CalledProcessError:
log.error('Failed to install static build dependencies')
@@ -85,7 +88,7 @@ def check_build_wheel(hooks):
try:
reqs = hooks.get_requires_for_build_wheel({})
log.info('Got build requires: %s', reqs)
- except:
+ except Exception:
log.error('Failure in get_requires_for_build_sdist', exc_info=True)
return False
@@ -102,7 +105,7 @@ def check_build_wheel(hooks):
try:
filename = hooks.build_wheel(td, {})
log.info('build_wheel returned %r', filename)
- except:
+ except Exception:
log.info('Failure in build_wheel', exc_info=True)
return False
@@ -151,8 +154,8 @@ def check(source_dir):
hooks = Pep517HookCaller(source_dir, backend)
- sdist_ok = check_build_sdist(hooks)
- wheel_ok = check_build_wheel(hooks)
+ sdist_ok = check_build_sdist(hooks, requires)
+ wheel_ok = check_build_wheel(hooks, requires)
if not sdist_ok:
log.warning('Sdist checks failed; scroll up to see')
@@ -164,7 +167,8 @@ def check(source_dir):
def main(argv=None):
ap = argparse.ArgumentParser()
- ap.add_argument('source_dir',
+ ap.add_argument(
+ 'source_dir',
help="A directory containing pyproject.toml")
args = ap.parse_args(argv)
@@ -178,17 +182,21 @@ def main(argv=None):
print(ansi('Checks failed', 'red'))
sys.exit(1)
+
ansi_codes = {
'reset': '\x1b[0m',
'bold': '\x1b[1m',
'red': '\x1b[31m',
'green': '\x1b[32m',
}
+
+
def ansi(s, attr):
if os.name != 'nt' and sys.stdout.isatty():
return ansi_codes[attr] + str(s) + ansi_codes['reset']
else:
return str(s)
+
if __name__ == '__main__':
main()
diff --git a/pipenv/patched/notpip/_vendor/pep517/colorlog.py b/pipenv/patched/notpip/_vendor/pep517/colorlog.py
index 26cf7480..69c8a59d 100644
--- a/pipenv/patched/notpip/_vendor/pep517/colorlog.py
+++ b/pipenv/patched/notpip/_vendor/pep517/colorlog.py
@@ -24,6 +24,7 @@ try:
except ImportError:
curses = None
+
def _stderr_supports_color():
color = False
if curses and hasattr(sys.stderr, 'isatty') and sys.stderr.isatty():
@@ -35,13 +36,14 @@ def _stderr_supports_color():
pass
return color
+
class LogFormatter(logging.Formatter):
"""Log formatter with colour support
"""
DEFAULT_COLORS = {
- logging.INFO: 2, # Green
- logging.WARNING: 3, # Yellow
- logging.ERROR: 1, # Red
+ logging.INFO: 2, # Green
+ logging.WARNING: 3, # Yellow
+ logging.ERROR: 1, # Red
logging.CRITICAL: 1,
}
@@ -75,7 +77,8 @@ class LogFormatter(logging.Formatter):
fg_color = str(fg_color, "ascii")
for levelno, code in self.DEFAULT_COLORS.items():
- self._colors[levelno] = str(curses.tparm(fg_color, code), "ascii")
+ self._colors[levelno] = str(
+ curses.tparm(fg_color, code), "ascii")
self._normal = str(curses.tigetstr("sgr0"), "ascii")
scr = curses.initscr()
@@ -83,15 +86,16 @@ class LogFormatter(logging.Formatter):
curses.endwin()
else:
self._normal = ''
- # Default width is usually 80, but too wide is worse than too narrow
+ # Default width is usually 80, but too wide is
+ # worse than too narrow
self.termwidth = 70
def formatMessage(self, record):
- l = len(record.message)
+ mlen = len(record.message)
right_text = '{initial}-{name}'.format(initial=record.levelname[0],
name=record.name)
- if l + len(right_text) < self.termwidth:
- space = ' ' * (self.termwidth - (l + len(right_text)))
+ if mlen + len(right_text) < self.termwidth:
+ space = ' ' * (self.termwidth - (mlen + len(right_text)))
else:
space = ' '
@@ -103,6 +107,7 @@ class LogFormatter(logging.Formatter):
return record.message + space + start_color + right_text + end_color
+
def enable_colourful_output(level=logging.INFO):
handler = logging.StreamHandler()
handler.setFormatter(LogFormatter())
diff --git a/pipenv/patched/notpip/_vendor/pep517/envbuild.py b/pipenv/patched/notpip/_vendor/pep517/envbuild.py
index c54d3585..8a5ad4d7 100644
--- a/pipenv/patched/notpip/_vendor/pep517/envbuild.py
+++ b/pipenv/patched/notpip/_vendor/pep517/envbuild.py
@@ -14,6 +14,7 @@ from .wrappers import Pep517HookCaller
log = logging.getLogger(__name__)
+
def _load_pyproject(source_dir):
with open(os.path.join(source_dir, 'pyproject.toml')) as f:
pyproject_data = pytoml.load(f)
@@ -89,11 +90,17 @@ class BuildEnvironment(object):
if not reqs:
return
log.info('Calling pip to install %s', reqs)
- check_call([sys.executable, '-m', 'pip', 'install', '--ignore-installed',
- '--prefix', self.path] + list(reqs))
+ check_call([
+ sys.executable, '-m', 'pip', 'install', '--ignore-installed',
+ '--prefix', self.path] + list(reqs))
def __exit__(self, exc_type, exc_val, exc_tb):
- if self._cleanup and (self.path is not None) and os.path.isdir(self.path):
+ needs_cleanup = (
+ self._cleanup and
+ self.path is not None and
+ os.path.isdir(self.path)
+ )
+ if needs_cleanup:
shutil.rmtree(self.path)
if self.save_path is None:
@@ -106,6 +113,7 @@ class BuildEnvironment(object):
else:
os.environ['PYTHONPATH'] = self.save_pythonpath
+
def build_wheel(source_dir, wheel_dir, config_settings=None):
"""Build a wheel from a source directory using PEP 517 hooks.
diff --git a/pipenv/patched/notpip/_vendor/pep517/wrappers.py b/pipenv/patched/notpip/_vendor/pep517/wrappers.py
index 28260f32..b14b8991 100644
--- a/pipenv/patched/notpip/_vendor/pep517/wrappers.py
+++ b/pipenv/patched/notpip/_vendor/pep517/wrappers.py
@@ -10,6 +10,7 @@ from . import compat
_in_proc_script = pjoin(dirname(abspath(__file__)), '_in_process.py')
+
@contextmanager
def tempdir():
td = mkdtemp()
@@ -18,9 +19,24 @@ def tempdir():
finally:
shutil.rmtree(td)
+
+class BackendUnavailable(Exception):
+ """Will be raised if the backend cannot be imported in the hook process."""
+
+
class UnsupportedOperation(Exception):
"""May be raised by build_sdist if the backend indicates that it can't."""
+
+def default_subprocess_runner(cmd, cwd=None, extra_environ=None):
+ """The default method of calling the wrapper subprocess."""
+ env = os.environ.copy()
+ if extra_environ:
+ env.update(extra_environ)
+
+ check_call(cmd, cwd=cwd, env=env)
+
+
class Pep517HookCaller(object):
"""A wrapper around a source directory to be built with a PEP 517 backend.
@@ -30,6 +46,16 @@ class Pep517HookCaller(object):
def __init__(self, source_dir, build_backend):
self.source_dir = abspath(source_dir)
self.build_backend = build_backend
+ self._subprocess_runner = default_subprocess_runner
+
+ # TODO: Is this over-engineered? Maybe frontends only need to
+ # set this when creating the wrapper, not on every call.
+ @contextmanager
+ def subprocess_runner(self, runner):
+ prev = self._subprocess_runner
+ self._subprocess_runner = runner
+ yield
+ self._subprocess_runner = prev
def get_requires_for_build_wheel(self, config_settings=None):
"""Identify packages required for building a wheel
@@ -45,7 +71,8 @@ class Pep517HookCaller(object):
'config_settings': config_settings
})
- def prepare_metadata_for_build_wheel(self, metadata_directory, config_settings=None):
+ def prepare_metadata_for_build_wheel(
+ self, metadata_directory, config_settings=None):
"""Prepare a *.dist-info folder with metadata for this project.
Returns the name of the newly created folder.
@@ -59,7 +86,9 @@ class Pep517HookCaller(object):
'config_settings': config_settings,
})
- def build_wheel(self, wheel_directory, config_settings=None, metadata_directory=None):
+ def build_wheel(
+ self, wheel_directory, config_settings=None,
+ metadata_directory=None):
"""Build a wheel from this project.
Returns the name of the newly created file.
@@ -103,10 +132,7 @@ class Pep517HookCaller(object):
'config_settings': config_settings,
})
-
def _call_hook(self, hook_name, kwargs):
- env = os.environ.copy()
-
# On Python 2, pytoml returns Unicode values (which is correct) but the
# environment passed to check_call needs to contain string values. We
# convert here by encoding using ASCII (the backend can only contain
@@ -118,17 +144,20 @@ class Pep517HookCaller(object):
else:
build_backend = self.build_backend
- env['PEP517_BUILD_BACKEND'] = build_backend
with tempdir() as td:
compat.write_json({'kwargs': kwargs}, pjoin(td, 'input.json'),
indent=2)
# Run the hook in a subprocess
- check_call([sys.executable, _in_proc_script, hook_name, td],
- cwd=self.source_dir, env=env)
+ self._subprocess_runner(
+ [sys.executable, _in_proc_script, hook_name, td],
+ cwd=self.source_dir,
+ extra_environ={'PEP517_BUILD_BACKEND': build_backend}
+ )
data = compat.read_json(pjoin(td, 'output.json'))
if data.get('unsupported'):
raise UnsupportedOperation
+ if data.get('no_backend'):
+ raise BackendUnavailable
return data['return_val']
-
diff --git a/pipenv/patched/notpip/_vendor/pkg_resources/__init__.py b/pipenv/patched/notpip/_vendor/pkg_resources/__init__.py
index ac893b66..0459a7da 100644
--- a/pipenv/patched/notpip/_vendor/pkg_resources/__init__.py
+++ b/pipenv/patched/notpip/_vendor/pkg_resources/__init__.py
@@ -238,6 +238,9 @@ __all__ = [
'register_finder', 'register_namespace_handler', 'register_loader_type',
'fixup_namespace_packages', 'get_importer',
+ # Warnings
+ 'PkgResourcesDeprecationWarning',
+
# Deprecated/backward compatibility only
'run_main', 'AvailableDistributions',
]
@@ -2228,7 +2231,18 @@ register_namespace_handler(object, null_ns_handler)
def normalize_path(filename):
"""Normalize a file/dir name for comparison purposes"""
- return os.path.normcase(os.path.realpath(filename))
+ return os.path.normcase(os.path.realpath(os.path.normpath(_cygwin_patch(filename))))
+
+
+def _cygwin_patch(filename): # pragma: nocover
+ """
+ Contrary to POSIX 2008, on Cygwin, getcwd (3) contains
+ symlink components. Using
+ os.path.abspath() works around this limitation. A fix in os.getcwd()
+ would probably better, in Cygwin even more so, except
+ that this seems to be by design...
+ """
+ return os.path.abspath(filename) if sys.platform == 'cygwin' else filename
def _normalize_cached(filename, _cache={}):
@@ -2324,7 +2338,7 @@ class EntryPoint:
warnings.warn(
"Parameters to load are deprecated. Call .resolve and "
".require separately.",
- DeprecationWarning,
+ PkgResourcesDeprecationWarning,
stacklevel=2,
)
if require:
@@ -3147,3 +3161,11 @@ def _initialize_master_working_set():
# match order
list(map(working_set.add_entry, sys.path))
globals().update(locals())
+
+class PkgResourcesDeprecationWarning(Warning):
+ """
+ Base class for warning about deprecations in ``pkg_resources``
+
+ This class is not derived from ``DeprecationWarning``, and as such is
+ visible by default.
+ """
diff --git a/pipenv/patched/notpip/_vendor/pyparsing.py b/pipenv/patched/notpip/_vendor/pyparsing.py
index 455d1151..3972b370 100644
--- a/pipenv/patched/notpip/_vendor/pyparsing.py
+++ b/pipenv/patched/notpip/_vendor/pyparsing.py
@@ -1,6 +1,7 @@
+#-*- coding: utf-8 -*-
# module pyparsing.py
#
-# Copyright (c) 2003-2018 Paul T. McGuire
+# Copyright (c) 2003-2019 Paul T. McGuire
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
@@ -27,15 +28,18 @@ __doc__ = \
pyparsing module - Classes and methods to define and execute parsing grammars
=============================================================================
-The pyparsing module is an alternative approach to creating and executing simple grammars,
-vs. the traditional lex/yacc approach, or the use of regular expressions. With pyparsing, you
-don't need to learn a new syntax for defining grammars or matching expressions - the parsing module
-provides a library of classes that you use to construct the grammar directly in Python.
+The pyparsing module is an alternative approach to creating and
+executing simple grammars, vs. the traditional lex/yacc approach, or the
+use of regular expressions. With pyparsing, you don't need to learn
+a new syntax for defining grammars or matching expressions - the parsing
+module provides a library of classes that you use to construct the
+grammar directly in Python.
-Here is a program to parse "Hello, World!" (or any greeting of the form
-C{", !"}), built up using L{Word}, L{Literal}, and L{And} elements
-(L{'+'} operator gives L{And} expressions, strings are auto-converted to
-L{Literal} expressions)::
+Here is a program to parse "Hello, World!" (or any greeting of the form
+``", !"``), built up using :class:`Word`,
+:class:`Literal`, and :class:`And` elements
+(the :class:`'+'` operators create :class:`And` expressions,
+and the strings are auto-converted to :class:`Literal` expressions)::
from pipenv.patched.notpip._vendor.pyparsing import Word, alphas
@@ -49,33 +53,48 @@ The program outputs the following::
Hello, World! -> ['Hello', ',', 'World', '!']
-The Python representation of the grammar is quite readable, owing to the self-explanatory
-class names, and the use of '+', '|' and '^' operators.
+The Python representation of the grammar is quite readable, owing to the
+self-explanatory class names, and the use of '+', '|' and '^' operators.
-The L{ParseResults} object returned from L{ParserElement.parseString} can be accessed as a nested list, a dictionary, or an
-object with named attributes.
+The :class:`ParseResults` object returned from
+:class:`ParserElement.parseString` can be
+accessed as a nested list, a dictionary, or an object with named
+attributes.
-The pyparsing module handles some of the problems that are typically vexing when writing text parsers:
- - extra or missing whitespace (the above program will also handle "Hello,World!", "Hello , World !", etc.)
- - quoted strings
- - embedded comments
+The pyparsing module handles some of the problems that are typically
+vexing when writing text parsers:
+
+ - extra or missing whitespace (the above program will also handle
+ "Hello,World!", "Hello , World !", etc.)
+ - quoted strings
+ - embedded comments
Getting Started -
-----------------
-Visit the classes L{ParserElement} and L{ParseResults} to see the base classes that most other pyparsing
+Visit the classes :class:`ParserElement` and :class:`ParseResults` to
+see the base classes that most other pyparsing
classes inherit from. Use the docstrings for examples of how to:
- - construct literal match expressions from L{Literal} and L{CaselessLiteral} classes
- - construct character word-group expressions using the L{Word} class
- - see how to create repetitive expressions using L{ZeroOrMore} and L{OneOrMore} classes
- - use L{'+'}, L{'|'}, L{'^'}, and L{'&'} operators to combine simple expressions into more complex ones
- - associate names with your parsed results using L{ParserElement.setResultsName}
- - find some helpful expression short-cuts like L{delimitedList} and L{oneOf}
- - find more useful common expressions in the L{pyparsing_common} namespace class
+
+ - construct literal match expressions from :class:`Literal` and
+ :class:`CaselessLiteral` classes
+ - construct character word-group expressions using the :class:`Word`
+ class
+ - see how to create repetitive expressions using :class:`ZeroOrMore`
+ and :class:`OneOrMore` classes
+ - use :class:`'+'`, :class:`'|'`, :class:`'^'`,
+ and :class:`'&'` operators to combine simple expressions into
+ more complex ones
+ - associate names with your parsed results using
+ :class:`ParserElement.setResultsName`
+ - find some helpful expression short-cuts like :class:`delimitedList`
+ and :class:`oneOf`
+ - find more useful common expressions in the :class:`pyparsing_common`
+ namespace class
"""
-__version__ = "2.2.1"
-__versionTime__ = "18 Sep 2018 00:49 UTC"
+__version__ = "2.3.1"
+__versionTime__ = "09 Jan 2019 23:26 UTC"
__author__ = "Paul McGuire "
import string
@@ -91,6 +110,12 @@ import traceback
import types
from datetime import datetime
+try:
+ # Python 3
+ from itertools import filterfalse
+except ImportError:
+ from itertools import ifilterfalse as filterfalse
+
try:
from _thread import RLock
except ImportError:
@@ -113,27 +138,33 @@ except ImportError:
except ImportError:
_OrderedDict = None
+try:
+ from types import SimpleNamespace
+except ImportError:
+ class SimpleNamespace: pass
+
+
#~ sys.stderr.write( "testing pyparsing module, version %s, %s\n" % (__version__,__versionTime__ ) )
__all__ = [
'And', 'CaselessKeyword', 'CaselessLiteral', 'CharsNotIn', 'Combine', 'Dict', 'Each', 'Empty',
'FollowedBy', 'Forward', 'GoToColumn', 'Group', 'Keyword', 'LineEnd', 'LineStart', 'Literal',
-'MatchFirst', 'NoMatch', 'NotAny', 'OneOrMore', 'OnlyOnce', 'Optional', 'Or',
+'PrecededBy', 'MatchFirst', 'NoMatch', 'NotAny', 'OneOrMore', 'OnlyOnce', 'Optional', 'Or',
'ParseBaseException', 'ParseElementEnhance', 'ParseException', 'ParseExpression', 'ParseFatalException',
'ParseResults', 'ParseSyntaxException', 'ParserElement', 'QuotedString', 'RecursiveGrammarException',
-'Regex', 'SkipTo', 'StringEnd', 'StringStart', 'Suppress', 'Token', 'TokenConverter',
-'White', 'Word', 'WordEnd', 'WordStart', 'ZeroOrMore',
+'Regex', 'SkipTo', 'StringEnd', 'StringStart', 'Suppress', 'Token', 'TokenConverter',
+'White', 'Word', 'WordEnd', 'WordStart', 'ZeroOrMore', 'Char',
'alphanums', 'alphas', 'alphas8bit', 'anyCloseTag', 'anyOpenTag', 'cStyleComment', 'col',
'commaSeparatedList', 'commonHTMLEntity', 'countedArray', 'cppStyleComment', 'dblQuotedString',
'dblSlashComment', 'delimitedList', 'dictOf', 'downcaseTokens', 'empty', 'hexnums',
'htmlComment', 'javaStyleComment', 'line', 'lineEnd', 'lineStart', 'lineno',
'makeHTMLTags', 'makeXMLTags', 'matchOnlyAtCol', 'matchPreviousExpr', 'matchPreviousLiteral',
'nestedExpr', 'nullDebugAction', 'nums', 'oneOf', 'opAssoc', 'operatorPrecedence', 'printables',
-'punc8bit', 'pythonStyleComment', 'quotedString', 'removeQuotes', 'replaceHTMLEntity',
+'punc8bit', 'pythonStyleComment', 'quotedString', 'removeQuotes', 'replaceHTMLEntity',
'replaceWith', 'restOfLine', 'sglQuotedString', 'srange', 'stringEnd',
'stringStart', 'traceParseAction', 'unicodeString', 'upcaseTokens', 'withAttribute',
'indentedBlock', 'originalTextFor', 'ungroup', 'infixNotation','locatedExpr', 'withClass',
-'CloseMatch', 'tokenMap', 'pyparsing_common',
+'CloseMatch', 'tokenMap', 'pyparsing_common', 'pyparsing_unicode', 'unicode_set',
]
system_version = tuple(sys.version_info)[:3]
@@ -142,6 +173,7 @@ if PY_3:
_MAX_INT = sys.maxsize
basestring = str
unichr = chr
+ unicode = str
_ustr = str
# build list of single arg builtins, that can be used as parse actions
@@ -152,9 +184,11 @@ else:
range = xrange
def _ustr(obj):
- """Drop-in replacement for str(obj) that tries to be Unicode friendly. It first tries
- str(obj). If that fails with a UnicodeEncodeError, then it tries unicode(obj). It
- then < returns the unicode object | encodes it with the default encoding | ... >.
+ """Drop-in replacement for str(obj) that tries to be Unicode
+ friendly. It first tries str(obj). If that fails with
+ a UnicodeEncodeError, then it tries unicode(obj). It then
+ < returns the unicode object | encodes it with the default
+ encoding | ... >.
"""
if isinstance(obj,unicode):
return obj
@@ -179,9 +213,9 @@ else:
singleArgBuiltins.append(getattr(__builtin__,fname))
except AttributeError:
continue
-
+
_generatorType = type((y for y in range(1)))
-
+
def _xml_escape(data):
"""Escape &, <, >, ", ', etc. in a string of data."""
@@ -192,9 +226,6 @@ def _xml_escape(data):
data = data.replace(from_, to_)
return data
-class _Constants(object):
- pass
-
alphas = string.ascii_uppercase + string.ascii_lowercase
nums = "0123456789"
hexnums = nums + "ABCDEFabcdef"
@@ -220,16 +251,16 @@ class ParseBaseException(Exception):
@classmethod
def _from_exception(cls, pe):
"""
- internal factory method to simplify creating one type of ParseException
+ internal factory method to simplify creating one type of ParseException
from another - avoids having __init__ signature conflicts among subclasses
"""
return cls(pe.pstr, pe.loc, pe.msg, pe.parserElement)
def __getattr__( self, aname ):
"""supported attributes by name are:
- - lineno - returns the line number of the exception text
- - col - returns the column number of the exception text
- - line - returns the line containing the exception text
+ - lineno - returns the line number of the exception text
+ - col - returns the column number of the exception text
+ - line - returns the line containing the exception text
"""
if( aname == "lineno" ):
return lineno( self.loc, self.pstr )
@@ -262,22 +293,94 @@ class ParseException(ParseBaseException):
"""
Exception thrown when parse expressions don't match class;
supported attributes by name are:
- - lineno - returns the line number of the exception text
- - col - returns the column number of the exception text
- - line - returns the line containing the exception text
-
+ - lineno - returns the line number of the exception text
+ - col - returns the column number of the exception text
+ - line - returns the line containing the exception text
+
Example::
+
try:
Word(nums).setName("integer").parseString("ABC")
except ParseException as pe:
print(pe)
print("column: {}".format(pe.col))
-
+
prints::
+
Expected integer (at char 0), (line:1, col:1)
column: 1
+
"""
- pass
+
+ @staticmethod
+ def explain(exc, depth=16):
+ """
+ Method to take an exception and translate the Python internal traceback into a list
+ of the pyparsing expressions that caused the exception to be raised.
+
+ Parameters:
+
+ - exc - exception raised during parsing (need not be a ParseException, in support
+ of Python exceptions that might be raised in a parse action)
+ - depth (default=16) - number of levels back in the stack trace to list expression
+ and function names; if None, the full stack trace names will be listed; if 0, only
+ the failing input line, marker, and exception string will be shown
+
+ Returns a multi-line string listing the ParserElements and/or function names in the
+ exception's stack trace.
+
+ Note: the diagnostic output will include string representations of the expressions
+ that failed to parse. These representations will be more helpful if you use `setName` to
+ give identifiable names to your expressions. Otherwise they will use the default string
+ forms, which may be cryptic to read.
+
+ explain() is only supported under Python 3.
+ """
+ import inspect
+
+ if depth is None:
+ depth = sys.getrecursionlimit()
+ ret = []
+ if isinstance(exc, ParseBaseException):
+ ret.append(exc.line)
+ ret.append(' ' * (exc.col - 1) + '^')
+ ret.append("{0}: {1}".format(type(exc).__name__, exc))
+
+ if depth > 0:
+ callers = inspect.getinnerframes(exc.__traceback__, context=depth)
+ seen = set()
+ for i, ff in enumerate(callers[-depth:]):
+ frm = ff.frame
+
+ f_self = frm.f_locals.get('self', None)
+ if isinstance(f_self, ParserElement):
+ if frm.f_code.co_name not in ('parseImpl', '_parseNoCache'):
+ continue
+ if f_self in seen:
+ continue
+ seen.add(f_self)
+
+ self_type = type(f_self)
+ ret.append("{0}.{1} - {2}".format(self_type.__module__,
+ self_type.__name__,
+ f_self))
+ elif f_self is not None:
+ self_type = type(f_self)
+ ret.append("{0}.{1}".format(self_type.__module__,
+ self_type.__name__))
+ else:
+ code = frm.f_code
+ if code.co_name in ('wrapper', ''):
+ continue
+
+ ret.append("{0}".format(code.co_name))
+
+ depth -= 1
+ if not depth:
+ break
+
+ return '\n'.join(ret)
+
class ParseFatalException(ParseBaseException):
"""user-throwable exception thrown when inconsistent parse content
@@ -285,9 +388,11 @@ class ParseFatalException(ParseBaseException):
pass
class ParseSyntaxException(ParseFatalException):
- """just like L{ParseFatalException}, but thrown internally when an
- L{ErrorStop} ('-' operator) indicates that parsing is to stop
- immediately because an unbacktrackable syntax error has been found"""
+ """just like :class:`ParseFatalException`, but thrown internally
+ when an :class:`ErrorStop` ('-' operator) indicates
+ that parsing is to stop immediately because an unbacktrackable
+ syntax error has been found.
+ """
pass
#~ class ReparseException(ParseBaseException):
@@ -304,7 +409,9 @@ class ParseSyntaxException(ParseFatalException):
#~ self.reparseLoc = restartLoc
class RecursiveGrammarException(Exception):
- """exception thrown by L{ParserElement.validate} if the grammar could be improperly recursive"""
+ """exception thrown by :class:`ParserElement.validate` if the
+ grammar could be improperly recursive
+ """
def __init__( self, parseElementList ):
self.parseElementTrace = parseElementList
@@ -322,16 +429,18 @@ class _ParseResultsWithOffset(object):
self.tup = (self.tup[0],i)
class ParseResults(object):
- """
- Structured parse results, to provide multiple means of access to the parsed data:
- - as a list (C{len(results)})
- - by list index (C{results[0], results[1]}, etc.)
- - by attribute (C{results.} - see L{ParserElement.setResultsName})
+ """Structured parse results, to provide multiple means of access to
+ the parsed data:
+
+ - as a list (``len(results)``)
+ - by list index (``results[0], results[1]``, etc.)
+ - by attribute (``results.`` - see :class:`ParserElement.setResultsName`)
Example::
+
integer = Word(nums)
- date_str = (integer.setResultsName("year") + '/'
- + integer.setResultsName("month") + '/'
+ date_str = (integer.setResultsName("year") + '/'
+ + integer.setResultsName("month") + '/'
+ integer.setResultsName("day"))
# equivalent form:
# date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
@@ -348,7 +457,9 @@ class ParseResults(object):
test("'month' in result")
test("'minutes' in result")
test("result.dump()", str)
+
prints::
+
list(result) -> ['1999', '/', '12', '/', '31']
result[0] -> '1999'
result['month'] -> '12'
@@ -398,7 +509,7 @@ class ParseResults(object):
toklist = [ toklist ]
if asList:
if isinstance(toklist,ParseResults):
- self[name] = _ParseResultsWithOffset(toklist.copy(),0)
+ self[name] = _ParseResultsWithOffset(ParseResults(toklist.__toklist), 0)
else:
self[name] = _ParseResultsWithOffset(ParseResults(toklist[0]),0)
self[name].__name = name
@@ -467,19 +578,19 @@ class ParseResults(object):
def _itervalues( self ):
return (self[k] for k in self._iterkeys())
-
+
def _iteritems( self ):
return ((k, self[k]) for k in self._iterkeys())
if PY_3:
- keys = _iterkeys
- """Returns an iterator of all named result keys (Python 3.x only)."""
+ keys = _iterkeys
+ """Returns an iterator of all named result keys."""
values = _itervalues
- """Returns an iterator of all named result values (Python 3.x only)."""
+ """Returns an iterator of all named result values."""
items = _iteritems
- """Returns an iterator of all named result key-value tuples (Python 3.x only)."""
+ """Returns an iterator of all named result key-value tuples."""
else:
iterkeys = _iterkeys
@@ -498,7 +609,7 @@ class ParseResults(object):
def values( self ):
"""Returns all named result values (as a list in Python 2.x, as an iterator in Python 3.x)."""
return list(self.itervalues())
-
+
def items( self ):
"""Returns all named result key-values (as a list of tuples in Python 2.x, as an iterator in Python 3.x)."""
return list(self.iteritems())
@@ -507,19 +618,20 @@ class ParseResults(object):
"""Since keys() returns an iterator, this method is helpful in bypassing
code that looks for the existence of any defined results names."""
return bool(self.__tokdict)
-
+
def pop( self, *args, **kwargs):
"""
- Removes and returns item at specified index (default=C{last}).
- Supports both C{list} and C{dict} semantics for C{pop()}. If passed no
- argument or an integer argument, it will use C{list} semantics
- and pop tokens from the list of parsed tokens. If passed a
- non-integer argument (most likely a string), it will use C{dict}
- semantics and pop the corresponding value from any defined
- results names. A second default return value argument is
- supported, just as in C{dict.pop()}.
+ Removes and returns item at specified index (default= ``last``).
+ Supports both ``list`` and ``dict`` semantics for ``pop()``. If
+ passed no argument or an integer argument, it will use ``list``
+ semantics and pop tokens from the list of parsed tokens. If passed
+ a non-integer argument (most likely a string), it will use ``dict``
+ semantics and pop the corresponding value from any defined results
+ names. A second default return value argument is supported, just as in
+ ``dict.pop()``.
Example::
+
def remove_first(tokens):
tokens.pop(0)
print(OneOrMore(Word(nums)).parseString("0 123 321")) # -> ['0', '123', '321']
@@ -536,7 +648,9 @@ class ParseResults(object):
return tokens
patt.addParseAction(remove_LABEL)
print(patt.parseString("AAB 123 321").dump())
+
prints::
+
['AAB', '123', '321']
- LABEL: AAB
@@ -549,8 +663,8 @@ class ParseResults(object):
args = (args[0], v)
else:
raise TypeError("pop() got an unexpected keyword argument '%s'" % k)
- if (isinstance(args[0], int) or
- len(args) == 1 or
+ if (isinstance(args[0], int) or
+ len(args) == 1 or
args[0] in self):
index = args[0]
ret = self[index]
@@ -563,14 +677,15 @@ class ParseResults(object):
def get(self, key, defaultValue=None):
"""
Returns named result matching the given key, or if there is no
- such name, then returns the given C{defaultValue} or C{None} if no
- C{defaultValue} is specified.
+ such name, then returns the given ``defaultValue`` or ``None`` if no
+ ``defaultValue`` is specified.
+
+ Similar to ``dict.get()``.
- Similar to C{dict.get()}.
-
Example::
+
integer = Word(nums)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
result = date_str.parseString("1999/12/31")
print(result.get("year")) # -> '1999'
@@ -585,10 +700,11 @@ class ParseResults(object):
def insert( self, index, insStr ):
"""
Inserts new element at location index in the list of parsed tokens.
-
- Similar to C{list.insert()}.
+
+ Similar to ``list.insert()``.
Example::
+
print(OneOrMore(Word(nums)).parseString("0 123 321")) # -> ['0', '123', '321']
# use a parse action to insert the parse location in the front of the parsed results
@@ -607,8 +723,9 @@ class ParseResults(object):
Add single element to end of ParseResults list of elements.
Example::
+
print(OneOrMore(Word(nums)).parseString("0 123 321")) # -> ['0', '123', '321']
-
+
# use a parse action to compute the sum of the parsed integers, and add it to the end
def append_sum(tokens):
tokens.append(sum(map(int, tokens)))
@@ -621,8 +738,9 @@ class ParseResults(object):
Add sequence of elements to end of ParseResults list of elements.
Example::
+
patt = OneOrMore(Word(alphas))
-
+
# use a parse action to append the reverse of the matched strings, to make a palindrome
def make_palindrome(tokens):
tokens.extend(reversed([t[::-1] for t in tokens]))
@@ -646,7 +764,7 @@ class ParseResults(object):
return self[name]
except KeyError:
return ""
-
+
if name in self.__tokdict:
if name not in self.__accumNames:
return self.__tokdict[name][-1][0]
@@ -671,7 +789,7 @@ class ParseResults(object):
self[k] = v
if isinstance(v[0],ParseResults):
v[0].__parent = wkref(self)
-
+
self.__toklist += other.__toklist
self.__accumNames.update( other.__accumNames )
return self
@@ -683,7 +801,7 @@ class ParseResults(object):
else:
# this may raise a TypeError - so be it
return other + self
-
+
def __repr__( self ):
return "(%s, %s)" % ( repr( self.__toklist ), repr( self.__tokdict ) )
@@ -706,11 +824,12 @@ class ParseResults(object):
Returns the parse results as a nested list of matching tokens, all converted to strings.
Example::
+
patt = OneOrMore(Word(alphas))
result = patt.parseString("sldkj lsdkj sldkj")
# even though the result prints in string-like form, it is actually a pyparsing ParseResults
print(type(result), result) # -> ['sldkj', 'lsdkj', 'sldkj']
-
+
# Use asList() to create an actual list
result_list = result.asList()
print(type(result_list), result_list) # -> ['sldkj', 'lsdkj', 'sldkj']
@@ -722,12 +841,13 @@ class ParseResults(object):
Returns the named parse results as a nested dictionary.
Example::
+
integer = Word(nums)
date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
+
result = date_str.parseString('12/31/1999')
print(type(result), repr(result)) # -> (['12', '/', '31', '/', '1999'], {'day': [('1999', 4)], 'year': [('12', 0)], 'month': [('31', 2)]})
-
+
result_dict = result.asDict()
print(type(result_dict), repr(result_dict)) # -> {'day': '1999', 'year': '12', 'month': '31'}
@@ -740,7 +860,7 @@ class ParseResults(object):
item_fn = self.items
else:
item_fn = self.iteritems
-
+
def toItem(obj):
if isinstance(obj, ParseResults):
if obj.haskeys():
@@ -749,15 +869,15 @@ class ParseResults(object):
return [toItem(v) for v in obj]
else:
return obj
-
+
return dict((k,toItem(v)) for k,v in item_fn())
def copy( self ):
"""
- Returns a new copy of a C{ParseResults} object.
+ Returns a new copy of a :class:`ParseResults` object.
"""
ret = ParseResults( self.__toklist )
- ret.__tokdict = self.__tokdict.copy()
+ ret.__tokdict = dict(self.__tokdict.items())
ret.__parent = self.__parent
ret.__accumNames.update( self.__accumNames )
ret.__name = self.__name
@@ -833,22 +953,25 @@ class ParseResults(object):
def getName(self):
r"""
- Returns the results name for this token expression. Useful when several
+ Returns the results name for this token expression. Useful when several
different expressions might match at a particular location.
Example::
+
integer = Word(nums)
ssn_expr = Regex(r"\d\d\d-\d\d-\d\d\d\d")
house_number_expr = Suppress('#') + Word(nums, alphanums)
- user_data = (Group(house_number_expr)("house_number")
+ user_data = (Group(house_number_expr)("house_number")
| Group(ssn_expr)("ssn")
| Group(integer)("age"))
user_info = OneOrMore(user_data)
-
+
result = user_info.parseString("22 111-22-3333 #221B")
for item in result:
print(item.getName(), ':', item[0])
+
prints::
+
age : 22
ssn : 111-22-3333
house_number : 221B
@@ -870,17 +993,20 @@ class ParseResults(object):
def dump(self, indent='', depth=0, full=True):
"""
- Diagnostic method for listing out the contents of a C{ParseResults}.
- Accepts an optional C{indent} argument so that this string can be embedded
- in a nested display of other data.
+ Diagnostic method for listing out the contents of
+ a :class:`ParseResults`. Accepts an optional ``indent`` argument so
+ that this string can be embedded in a nested display of other data.
Example::
+
integer = Word(nums)
date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
-
+
result = date_str.parseString('12/31/1999')
print(result.dump())
+
prints::
+
['12', '/', '31', '/', '1999']
- day: 1999
- month: 31
@@ -910,16 +1036,18 @@ class ParseResults(object):
out.append("\n%s%s[%d]:\n%s%s%s" % (indent,(' '*(depth)),i,indent,(' '*(depth+1)),vv.dump(indent,depth+1) ))
else:
out.append("\n%s%s[%d]:\n%s%s%s" % (indent,(' '*(depth)),i,indent,(' '*(depth+1)),_ustr(vv)))
-
+
return "".join(out)
def pprint(self, *args, **kwargs):
"""
- Pretty-printer for parsed results as a list, using the C{pprint} module.
- Accepts additional positional or keyword args as defined for the
- C{pprint.pprint} method. (U{http://docs.python.org/3/library/pprint.html#pprint.pprint})
+ Pretty-printer for parsed results as a list, using the
+ `pprint `_ module.
+ Accepts additional positional or keyword args as defined for
+ `pprint.pprint `_ .
Example::
+
ident = Word(alphas, alphanums)
num = Word(nums)
func = Forward()
@@ -927,7 +1055,9 @@ class ParseResults(object):
func <<= ident + Group(Optional(delimitedList(term)))
result = func.parseString("fna a,b,(fnb c,d,200),100")
result.pprint(width=40)
+
prints::
+
['fna',
['a',
'b',
@@ -970,24 +1100,25 @@ def col (loc,strg):
The first column is number 1.
Note: the default parsing behavior is to expand tabs in the input string
- before starting the parsing process. See L{I{ParserElement.parseString}} for more information
- on parsing strings containing C{}s, and suggested methods to maintain a
- consistent view of the parsed string, the parse location, and line and column
- positions within the parsed string.
+ before starting the parsing process. See
+ :class:`ParserElement.parseString` for more
+ information on parsing strings containing ```` s, and suggested
+ methods to maintain a consistent view of the parsed string, the parse
+ location, and line and column positions within the parsed string.
"""
s = strg
return 1 if 0} for more information
- on parsing strings containing C{}s, and suggested methods to maintain a
- consistent view of the parsed string, the parse location, and line and column
- positions within the parsed string.
- """
+ Note - the default parsing behavior is to expand tabs in the input string
+ before starting the parsing process. See :class:`ParserElement.parseString`
+ for more information on parsing strings containing ```` s, and
+ suggested methods to maintain a consistent view of the parsed string, the
+ parse location, and line and column positions within the parsed string.
+ """
return strg.count("\n",0,loc) + 1
def line( loc, strg ):
@@ -1041,7 +1172,7 @@ def _trim_arity(func, maxargs=2):
return lambda s,l,t: func(t)
limit = [0]
foundArity = [False]
-
+
# traceback return data structure changed in Py3.5 - normalize back to plain tuples
if system_version[:2] >= (3,5):
def extract_stack(limit=0):
@@ -1056,12 +1187,12 @@ def _trim_arity(func, maxargs=2):
else:
extract_stack = traceback.extract_stack
extract_tb = traceback.extract_tb
-
- # synthesize what would be returned by traceback.extract_stack at the call to
+
+ # synthesize what would be returned by traceback.extract_stack at the call to
# user's parse action 'func', so that we don't incur call penalty at parse time
-
+
LINE_DIFF = 6
- # IF ANY CODE CHANGES, EVEN JUST COMMENTS OR BLANK LINES, BETWEEN THE NEXT LINE AND
+ # IF ANY CODE CHANGES, EVEN JUST COMMENTS OR BLANK LINES, BETWEEN THE NEXT LINE AND
# THE CALL TO FUNC INSIDE WRAPPER, LINE_DIFF MUST BE MODIFIED!!!!
this_line = extract_stack(limit=2)[-1]
pa_call_line_synth = (this_line[0], this_line[1]+LINE_DIFF)
@@ -1092,7 +1223,7 @@ def _trim_arity(func, maxargs=2):
# copy func name to wrapper for sensible debug output
func_name = ""
try:
- func_name = getattr(func, '__name__',
+ func_name = getattr(func, '__name__',
getattr(func, '__class__').__name__)
except Exception:
func_name = str(func)
@@ -1111,9 +1242,10 @@ class ParserElement(object):
Overrides the default whitespace chars
Example::
+
# default whitespace chars are space, and newline
OneOrMore(Word(alphas)).parseString("abc def\nghi jkl") # -> ['abc', 'def', 'ghi', 'jkl']
-
+
# change to just treat newline as significant
ParserElement.setDefaultWhitespaceChars(" \t")
OneOrMore(Word(alphas)).parseString("abc def\nghi jkl") # -> ['abc', 'def']
@@ -1124,18 +1256,19 @@ class ParserElement(object):
def inlineLiteralsUsing(cls):
"""
Set class to be used for inclusion of string literals into a parser.
-
+
Example::
+
# default literal class used is Literal
integer = Word(nums)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
date_str.parseString("1999/12/31") # -> ['1999', '/', '12', '/', '31']
# change to Suppress
ParserElement.inlineLiteralsUsing(Suppress)
- date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
+ date_str = integer("year") + '/' + integer("month") + '/' + integer("day")
date_str.parseString("1999/12/31") # -> ['1999', '12', '31']
"""
@@ -1149,7 +1282,7 @@ class ParserElement(object):
self.resultsName = None
self.saveAsList = savelist
self.skipWhitespace = True
- self.whiteChars = ParserElement.DEFAULT_WHITE_CHARS
+ self.whiteChars = set(ParserElement.DEFAULT_WHITE_CHARS)
self.copyDefaultWhiteChars = True
self.mayReturnEmpty = False # used when checking for left-recursion
self.keepTabs = False
@@ -1166,18 +1299,24 @@ class ParserElement(object):
def copy( self ):
"""
- Make a copy of this C{ParserElement}. Useful for defining different parse actions
- for the same parsing pattern, using copies of the original parse element.
-
+ Make a copy of this :class:`ParserElement`. Useful for defining
+ different parse actions for the same parsing pattern, using copies of
+ the original parse element.
+
Example::
+
integer = Word(nums).setParseAction(lambda toks: int(toks[0]))
integerK = integer.copy().addParseAction(lambda toks: toks[0]*1024) + Suppress("K")
integerM = integer.copy().addParseAction(lambda toks: toks[0]*1024*1024) + Suppress("M")
-
+
print(OneOrMore(integerK | integerM | integer).parseString("5K 100 640K 256M"))
+
prints::
+
[5120, 100, 655360, 268435456]
- Equivalent form of C{expr.copy()} is just C{expr()}::
+
+ Equivalent form of ``expr.copy()`` is just ``expr()``::
+
integerM = integer().addParseAction(lambda toks: toks[0]*1024*1024) + Suppress("M")
"""
cpy = copy.copy( self )
@@ -1190,8 +1329,9 @@ class ParserElement(object):
def setName( self, name ):
"""
Define name for this expression, makes debugging and exception messages clearer.
-
+
Example::
+
Word(nums).parseString("ABC") # -> Exception: Expected W:(0123...) (at char 0), (line:1, col:1)
Word(nums).setName("integer").parseString("ABC") # -> Exception: Expected integer (at char 0), (line:1, col:1)
"""
@@ -1205,17 +1345,18 @@ class ParserElement(object):
"""
Define name for referencing matching tokens as a nested attribute
of the returned parse results.
- NOTE: this returns a *copy* of the original C{ParserElement} object;
+ NOTE: this returns a *copy* of the original :class:`ParserElement` object;
this is so that the client can define a basic element, such as an
integer, and reference it in multiple places with different names.
You can also set results names using the abbreviated syntax,
- C{expr("name")} in place of C{expr.setResultsName("name")} -
- see L{I{__call__}<__call__>}.
+ ``expr("name")`` in place of ``expr.setResultsName("name")``
+ - see :class:`__call__`.
Example::
- date_str = (integer.setResultsName("year") + '/'
- + integer.setResultsName("month") + '/'
+
+ date_str = (integer.setResultsName("year") + '/'
+ + integer.setResultsName("month") + '/'
+ integer.setResultsName("day"))
# equivalent form:
@@ -1231,7 +1372,7 @@ class ParserElement(object):
def setBreak(self,breakFlag = True):
"""Method to invoke the Python pdb debugger when this element is
- about to be parsed. Set C{breakFlag} to True to enable, False to
+ about to be parsed. Set ``breakFlag`` to True to enable, False to
disable.
"""
if breakFlag:
@@ -1250,25 +1391,28 @@ class ParserElement(object):
def setParseAction( self, *fns, **kwargs ):
"""
Define one or more actions to perform when successfully matching parse element definition.
- Parse action fn is a callable method with 0-3 arguments, called as C{fn(s,loc,toks)},
- C{fn(loc,toks)}, C{fn(toks)}, or just C{fn()}, where:
- - s = the original string being parsed (see note below)
- - loc = the location of the matching substring
- - toks = a list of the matched tokens, packaged as a C{L{ParseResults}} object
+ Parse action fn is a callable method with 0-3 arguments, called as ``fn(s,loc,toks)`` ,
+ ``fn(loc,toks)`` , ``fn(toks)`` , or just ``fn()`` , where:
+
+ - s = the original string being parsed (see note below)
+ - loc = the location of the matching substring
+ - toks = a list of the matched tokens, packaged as a :class:`ParseResults` object
+
If the functions in fns modify the tokens, they can return them as the return
value from fn, and the modified list of tokens will replace the original.
Otherwise, fn does not need to return any value.
Optional keyword arguments:
- - callDuringTry = (default=C{False}) indicate if parse action should be run during lookaheads and alternate testing
+ - callDuringTry = (default= ``False`` ) indicate if parse action should be run during lookaheads and alternate testing
Note: the default parsing behavior is to expand tabs in the input string
- before starting the parsing process. See L{I{parseString}} for more information
- on parsing strings containing C{}s, and suggested methods to maintain a
- consistent view of the parsed string, the parse location, and line and column
- positions within the parsed string.
-
+ before starting the parsing process. See :class:`parseString for more
+ information on parsing strings containing ```` s, and suggested
+ methods to maintain a consistent view of the parsed string, the parse
+ location, and line and column positions within the parsed string.
+
Example::
+
integer = Word(nums)
date_str = integer + '/' + integer + '/' + integer
@@ -1287,24 +1431,25 @@ class ParserElement(object):
def addParseAction( self, *fns, **kwargs ):
"""
- Add one or more parse actions to expression's list of parse actions. See L{I{setParseAction}}.
-
- See examples in L{I{copy}}.
+ Add one or more parse actions to expression's list of parse actions. See :class:`setParseAction`.
+
+ See examples in :class:`copy`.
"""
self.parseAction += list(map(_trim_arity, list(fns)))
self.callDuringTry = self.callDuringTry or kwargs.get("callDuringTry", False)
return self
def addCondition(self, *fns, **kwargs):
- """Add a boolean predicate function to expression's list of parse actions. See
- L{I{setParseAction}} for function call signatures. Unlike C{setParseAction},
- functions passed to C{addCondition} need to return boolean success/fail of the condition.
+ """Add a boolean predicate function to expression's list of parse actions. See
+ :class:`setParseAction` for function call signatures. Unlike ``setParseAction``,
+ functions passed to ``addCondition`` need to return boolean success/fail of the condition.
Optional keyword arguments:
- - message = define a custom message to be used in the raised exception
- - fatal = if True, will raise ParseFatalException to stop parsing immediately; otherwise will raise ParseException
-
+ - message = define a custom message to be used in the raised exception
+ - fatal = if True, will raise ParseFatalException to stop parsing immediately; otherwise will raise ParseException
+
Example::
+
integer = Word(nums).setParseAction(lambda toks: int(toks[0]))
year_int = integer.copy()
year_int.addCondition(lambda toks: toks[0] >= 2000, message="Only support years 2000 and later")
@@ -1315,8 +1460,9 @@ class ParserElement(object):
msg = kwargs.get("message", "failed user-defined condition")
exc_type = ParseFatalException if kwargs.get("fatal", False) else ParseException
for fn in fns:
+ fn = _trim_arity(fn)
def pa(s,l,t):
- if not bool(_trim_arity(fn)(s,l,t)):
+ if not bool(fn(s,l,t)):
raise exc_type(s,l,msg)
self.parseAction.append(pa)
self.callDuringTry = self.callDuringTry or kwargs.get("callDuringTry", False)
@@ -1325,12 +1471,12 @@ class ParserElement(object):
def setFailAction( self, fn ):
"""Define action to perform if parsing fails at this expression.
Fail acton fn is a callable function that takes the arguments
- C{fn(s,loc,expr,err)} where:
- - s = string being parsed
- - loc = location where expression match was attempted and failed
- - expr = the parse expression that failed
- - err = the exception thrown
- The function returns no value. It may throw C{L{ParseFatalException}}
+ ``fn(s,loc,expr,err)`` where:
+ - s = string being parsed
+ - loc = location where expression match was attempted and failed
+ - expr = the parse expression that failed
+ - err = the exception thrown
+ The function returns no value. It may throw :class:`ParseFatalException`
if it is desired to stop parsing immediately."""
self.failAction = fn
return self
@@ -1412,8 +1558,14 @@ class ParserElement(object):
if debugging:
try:
for fn in self.parseAction:
- tokens = fn( instring, tokensStart, retTokens )
- if tokens is not None:
+ try:
+ tokens = fn( instring, tokensStart, retTokens )
+ except IndexError as parse_action_exc:
+ exc = ParseException("exception raised in parse action")
+ exc.__cause__ = parse_action_exc
+ raise exc
+
+ if tokens is not None and tokens is not retTokens:
retTokens = ParseResults( tokens,
self.resultsName,
asList=self.saveAsList and isinstance(tokens,(ParseResults,list)),
@@ -1425,8 +1577,14 @@ class ParserElement(object):
raise
else:
for fn in self.parseAction:
- tokens = fn( instring, tokensStart, retTokens )
- if tokens is not None:
+ try:
+ tokens = fn( instring, tokensStart, retTokens )
+ except IndexError as parse_action_exc:
+ exc = ParseException("exception raised in parse action")
+ exc.__cause__ = parse_action_exc
+ raise exc
+
+ if tokens is not None and tokens is not retTokens:
retTokens = ParseResults( tokens,
self.resultsName,
asList=self.saveAsList and isinstance(tokens,(ParseResults,list)),
@@ -1443,7 +1601,7 @@ class ParserElement(object):
return self._parse( instring, loc, doActions=False )[0]
except ParseFatalException:
raise ParseException( instring, loc, self.errmsg, self)
-
+
def canParseNext(self, instring, loc):
try:
self.tryParse(instring, loc)
@@ -1465,7 +1623,7 @@ class ParserElement(object):
def clear(self):
cache.clear()
-
+
def cache_len(self):
return len(cache)
@@ -1577,23 +1735,23 @@ class ParserElement(object):
often in many complex grammars) can immediately return a cached value,
instead of re-executing parsing/validating code. Memoizing is done of
both valid results and parsing exceptions.
-
+
Parameters:
- - cache_size_limit - (default=C{128}) - if an integer value is provided
- will limit the size of the packrat cache; if None is passed, then
- the cache size will be unbounded; if 0 is passed, the cache will
- be effectively disabled.
-
+
+ - cache_size_limit - (default= ``128``) - if an integer value is provided
+ will limit the size of the packrat cache; if None is passed, then
+ the cache size will be unbounded; if 0 is passed, the cache will
+ be effectively disabled.
+
This speedup may break existing programs that use parse actions that
have side-effects. For this reason, packrat parsing is disabled when
you first import pyparsing. To activate the packrat feature, your
- program must call the class method C{ParserElement.enablePackrat()}. If
- your program uses C{psyco} to "compile as you go", you must call
- C{enablePackrat} before calling C{psyco.full()}. If you do not do this,
- Python will crash. For best results, call C{enablePackrat()} immediately
- after importing pyparsing.
-
+ program must call the class method :class:`ParserElement.enablePackrat`.
+ For best results, call ``enablePackrat()`` immediately after
+ importing pyparsing.
+
Example::
+
from pipenv.patched.notpip._vendor import pyparsing
pyparsing.ParserElement.enablePackrat()
"""
@@ -1612,23 +1770,25 @@ class ParserElement(object):
expression has been built.
If you want the grammar to require that the entire input string be
- successfully parsed, then set C{parseAll} to True (equivalent to ending
- the grammar with C{L{StringEnd()}}).
+ successfully parsed, then set ``parseAll`` to True (equivalent to ending
+ the grammar with ``StringEnd()``).
- Note: C{parseString} implicitly calls C{expandtabs()} on the input string,
+ Note: ``parseString`` implicitly calls ``expandtabs()`` on the input string,
in order to report proper column numbers in parse actions.
If the input string contains tabs and
- the grammar uses parse actions that use the C{loc} argument to index into the
+ the grammar uses parse actions that use the ``loc`` argument to index into the
string being parsed, you can ensure you have a consistent view of the input
string by:
- - calling C{parseWithTabs} on your grammar before calling C{parseString}
- (see L{I{parseWithTabs}})
- - define your parse action using the full C{(s,loc,toks)} signature, and
- reference the input string using the parse action's C{s} argument
- - explictly expand the tabs in your input string before calling
- C{parseString}
-
+
+ - calling ``parseWithTabs`` on your grammar before calling ``parseString``
+ (see :class:`parseWithTabs`)
+ - define your parse action using the full ``(s,loc,toks)`` signature, and
+ reference the input string using the parse action's ``s`` argument
+ - explictly expand the tabs in your input string before calling
+ ``parseString``
+
Example::
+
Word('a').parseString('aaaaabaaa') # -> ['aaaaa']
Word('a').parseString('aaaaabaaa', parseAll=True) # -> Exception: Expected end of text
"""
@@ -1659,22 +1819,23 @@ class ParserElement(object):
"""
Scan the input string for expression matches. Each match will return the
matching tokens, start location, and end location. May be called with optional
- C{maxMatches} argument, to clip scanning after 'n' matches are found. If
- C{overlap} is specified, then overlapping matches will be reported.
+ ``maxMatches`` argument, to clip scanning after 'n' matches are found. If
+ ``overlap`` is specified, then overlapping matches will be reported.
Note that the start and end locations are reported relative to the string
- being parsed. See L{I{parseString}} for more information on parsing
+ being parsed. See :class:`parseString` for more information on parsing
strings with embedded tabs.
Example::
+
source = "sldjf123lsdjjkf345sldkjf879lkjsfd987"
print(source)
for tokens,start,end in Word(alphas).scanString(source):
print(' '*start + '^'*(end-start))
print(' '*start + tokens[0])
-
+
prints::
-
+
sldjf123lsdjjkf345sldkjf879lkjsfd987
^^^^^
sldjf
@@ -1728,19 +1889,22 @@ class ParserElement(object):
def transformString( self, instring ):
"""
- Extension to C{L{scanString}}, to modify matching text with modified tokens that may
- be returned from a parse action. To use C{transformString}, define a grammar and
+ Extension to :class:`scanString`, to modify matching text with modified tokens that may
+ be returned from a parse action. To use ``transformString``, define a grammar and
attach a parse action to it that modifies the returned token list.
- Invoking C{transformString()} on a target string will then scan for matches,
+ Invoking ``transformString()`` on a target string will then scan for matches,
and replace the matched text patterns according to the logic in the parse
- action. C{transformString()} returns the resulting transformed string.
-
+ action. ``transformString()`` returns the resulting transformed string.
+
Example::
+
wd = Word(alphas)
wd.setParseAction(lambda toks: toks[0].title())
-
+
print(wd.transformString("now is the winter of our discontent made glorious summer by this sun of york."))
- Prints::
+
+ prints::
+
Now Is The Winter Of Our Discontent Made Glorious Summer By This Sun Of York.
"""
out = []
@@ -1771,19 +1935,22 @@ class ParserElement(object):
def searchString( self, instring, maxMatches=_MAX_INT ):
"""
- Another extension to C{L{scanString}}, simplifying the access to the tokens found
+ Another extension to :class:`scanString`, simplifying the access to the tokens found
to match the given parse expression. May be called with optional
- C{maxMatches} argument, to clip searching after 'n' matches are found.
-
+ ``maxMatches`` argument, to clip searching after 'n' matches are found.
+
Example::
+
# a capitalized word starts with an uppercase letter, followed by zero or more lowercase letters
cap_word = Word(alphas.upper(), alphas.lower())
-
+
print(cap_word.searchString("More than Iron, more than Lead, more than Gold I need Electricity"))
# the sum() builtin can be used to merge results into a single ParseResults object
print(sum(cap_word.searchString("More than Iron, more than Lead, more than Gold I need Electricity")))
+
prints::
+
[['More'], ['Iron'], ['Lead'], ['Gold'], ['I'], ['Electricity']]
['More', 'Iron', 'Lead', 'Gold', 'I', 'Electricity']
"""
@@ -1799,14 +1966,17 @@ class ParserElement(object):
def split(self, instring, maxsplit=_MAX_INT, includeSeparators=False):
"""
Generator method to split a string using the given expression as a separator.
- May be called with optional C{maxsplit} argument, to limit the number of splits;
- and the optional C{includeSeparators} argument (default=C{False}), if the separating
+ May be called with optional ``maxsplit`` argument, to limit the number of splits;
+ and the optional ``includeSeparators`` argument (default= ``False``), if the separating
matching text should be included in the split results.
-
- Example::
+
+ Example::
+
punc = oneOf(list(".,;:/-!?"))
print(list(punc.split("This, this?, this sentence, is badly punctuated!")))
+
prints::
+
['This', ' this', '', ' this sentence', ' is badly punctuated', '']
"""
splits = 0
@@ -1820,14 +1990,17 @@ class ParserElement(object):
def __add__(self, other ):
"""
- Implementation of + operator - returns C{L{And}}. Adding strings to a ParserElement
- converts them to L{Literal}s by default.
-
+ Implementation of + operator - returns :class:`And`. Adding strings to a ParserElement
+ converts them to :class:`Literal`s by default.
+
Example::
+
greet = Word(alphas) + "," + Word(alphas) + "!"
hello = "Hello, World!"
print (hello, "->", greet.parseString(hello))
- Prints::
+
+ prints::
+
Hello, World! -> ['Hello', ',', 'World', '!']
"""
if isinstance( other, basestring ):
@@ -1840,7 +2013,7 @@ class ParserElement(object):
def __radd__(self, other ):
"""
- Implementation of + operator when left operand is not a C{L{ParserElement}}
+ Implementation of + operator when left operand is not a :class:`ParserElement`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -1852,7 +2025,7 @@ class ParserElement(object):
def __sub__(self, other):
"""
- Implementation of - operator, returns C{L{And}} with error stop
+ Implementation of - operator, returns :class:`And` with error stop
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -1864,7 +2037,7 @@ class ParserElement(object):
def __rsub__(self, other ):
"""
- Implementation of - operator when left operand is not a C{L{ParserElement}}
+ Implementation of - operator when left operand is not a :class:`ParserElement`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -1876,23 +2049,23 @@ class ParserElement(object):
def __mul__(self,other):
"""
- Implementation of * operator, allows use of C{expr * 3} in place of
- C{expr + expr + expr}. Expressions may also me multiplied by a 2-integer
- tuple, similar to C{{min,max}} multipliers in regular expressions. Tuples
- may also include C{None} as in:
- - C{expr*(n,None)} or C{expr*(n,)} is equivalent
- to C{expr*n + L{ZeroOrMore}(expr)}
- (read as "at least n instances of C{expr}")
- - C{expr*(None,n)} is equivalent to C{expr*(0,n)}
- (read as "0 to n instances of C{expr}")
- - C{expr*(None,None)} is equivalent to C{L{ZeroOrMore}(expr)}
- - C{expr*(1,None)} is equivalent to C{L{OneOrMore}(expr)}
+ Implementation of * operator, allows use of ``expr * 3`` in place of
+ ``expr + expr + expr``. Expressions may also me multiplied by a 2-integer
+ tuple, similar to ``{min,max}`` multipliers in regular expressions. Tuples
+ may also include ``None`` as in:
+ - ``expr*(n,None)`` or ``expr*(n,)`` is equivalent
+ to ``expr*n + ZeroOrMore(expr)``
+ (read as "at least n instances of ``expr``")
+ - ``expr*(None,n)`` is equivalent to ``expr*(0,n)``
+ (read as "0 to n instances of ``expr``")
+ - ``expr*(None,None)`` is equivalent to ``ZeroOrMore(expr)``
+ - ``expr*(1,None)`` is equivalent to ``OneOrMore(expr)``
- Note that C{expr*(None,n)} does not raise an exception if
+ Note that ``expr*(None,n)`` does not raise an exception if
more than n exprs exist in the input stream; that is,
- C{expr*(None,n)} does not enforce a maximum number of expr
+ ``expr*(None,n)`` does not enforce a maximum number of expr
occurrences. If this behavior is desired, then write
- C{expr*(None,n) + ~expr}
+ ``expr*(None,n) + ~expr``
"""
if isinstance(other,int):
minElements, optElements = other,0
@@ -1947,7 +2120,7 @@ class ParserElement(object):
def __or__(self, other ):
"""
- Implementation of | operator - returns C{L{MatchFirst}}
+ Implementation of | operator - returns :class:`MatchFirst`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -1959,7 +2132,7 @@ class ParserElement(object):
def __ror__(self, other ):
"""
- Implementation of | operator when left operand is not a C{L{ParserElement}}
+ Implementation of | operator when left operand is not a :class:`ParserElement`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -1971,7 +2144,7 @@ class ParserElement(object):
def __xor__(self, other ):
"""
- Implementation of ^ operator - returns C{L{Or}}
+ Implementation of ^ operator - returns :class:`Or`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -1983,7 +2156,7 @@ class ParserElement(object):
def __rxor__(self, other ):
"""
- Implementation of ^ operator when left operand is not a C{L{ParserElement}}
+ Implementation of ^ operator when left operand is not a :class:`ParserElement`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -1995,7 +2168,7 @@ class ParserElement(object):
def __and__(self, other ):
"""
- Implementation of & operator - returns C{L{Each}}
+ Implementation of & operator - returns :class:`Each`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -2007,7 +2180,7 @@ class ParserElement(object):
def __rand__(self, other ):
"""
- Implementation of & operator when left operand is not a C{L{ParserElement}}
+ Implementation of & operator when left operand is not a :class:`ParserElement`
"""
if isinstance( other, basestring ):
other = ParserElement._literalStringClass( other )
@@ -2019,23 +2192,24 @@ class ParserElement(object):
def __invert__( self ):
"""
- Implementation of ~ operator - returns C{L{NotAny}}
+ Implementation of ~ operator - returns :class:`NotAny`
"""
return NotAny( self )
def __call__(self, name=None):
"""
- Shortcut for C{L{setResultsName}}, with C{listAllMatches=False}.
-
- If C{name} is given with a trailing C{'*'} character, then C{listAllMatches} will be
- passed as C{True}.
-
- If C{name} is omitted, same as calling C{L{copy}}.
+ Shortcut for :class:`setResultsName`, with ``listAllMatches=False``.
+
+ If ``name`` is given with a trailing ``'*'`` character, then ``listAllMatches`` will be
+ passed as ``True``.
+
+ If ``name` is omitted, same as calling :class:`copy`.
Example::
+
# these are equivalent
userdata = Word(alphas).setResultsName("name") + Word(nums+"-").setResultsName("socsecno")
- userdata = Word(alphas)("name") + Word(nums+"-")("socsecno")
+ userdata = Word(alphas)("name") + Word(nums+"-")("socsecno")
"""
if name is not None:
return self.setResultsName(name)
@@ -2044,7 +2218,7 @@ class ParserElement(object):
def suppress( self ):
"""
- Suppresses the output of this C{ParserElement}; useful to keep punctuation from
+ Suppresses the output of this :class:`ParserElement`; useful to keep punctuation from
cluttering up returned output.
"""
return Suppress( self )
@@ -2052,7 +2226,7 @@ class ParserElement(object):
def leaveWhitespace( self ):
"""
Disables the skipping of whitespace before matching the characters in the
- C{ParserElement}'s defined pattern. This is normally only used internally by
+ :class:`ParserElement`'s defined pattern. This is normally only used internally by
the pyparsing module, but may be needed in some whitespace-sensitive grammars.
"""
self.skipWhitespace = False
@@ -2069,9 +2243,9 @@ class ParserElement(object):
def parseWithTabs( self ):
"""
- Overrides default behavior to expand C{}s to spaces before parsing the input string.
- Must be called before C{parseString} when the input grammar contains elements that
- match C{} characters.
+ Overrides default behavior to expand ````s to spaces before parsing the input string.
+ Must be called before ``parseString`` when the input grammar contains elements that
+ match ```` characters.
"""
self.keepTabs = True
return self
@@ -2081,11 +2255,12 @@ class ParserElement(object):
Define expression to be ignored (e.g., comments) while doing pattern
matching; may be called repeatedly, to define multiple comment or other
ignorable patterns.
-
+
Example::
+
patt = OneOrMore(Word(alphas))
patt.parseString('ablaj /* comment */ lskjd') # -> ['ablaj']
-
+
patt.ignore(cStyleComment)
patt.parseString('ablaj /* comment */ lskjd') # -> ['ablaj', 'lskjd']
"""
@@ -2112,19 +2287,21 @@ class ParserElement(object):
def setDebug( self, flag=True ):
"""
Enable display of debugging messages while doing pattern matching.
- Set C{flag} to True to enable, False to disable.
+ Set ``flag`` to True to enable, False to disable.
Example::
+
wd = Word(alphas).setName("alphaword")
integer = Word(nums).setName("numword")
term = wd | integer
-
+
# turn on debugging for wd
wd.setDebug()
OneOrMore(term).parseString("abc 123 xyz 890")
-
+
prints::
+
Match alphaword at loc 0(1,1)
Matched alphaword -> ['abc']
Match alphaword at loc 3(1,4)
@@ -2137,12 +2314,12 @@ class ParserElement(object):
Exception raised:Expected alphaword (at char 15), (line:1, col:16)
The output shown is that produced by the default debug actions - custom debug actions can be
- specified using L{setDebugActions}. Prior to attempting
- to match the C{wd} expression, the debugging message C{"Match at loc (,
)"}
- is shown. Then if the parse succeeds, a C{"Matched"} message is shown, or an C{"Exception raised"}
- message is shown. Also note the use of L{setName} to assign a human-readable name to the expression,
+ specified using :class:`setDebugActions`. Prior to attempting
+ to match the ``wd`` expression, the debugging message ``"Match at loc (,
)"``
+ is shown. Then if the parse succeeds, a ``"Matched"`` message is shown, or an ``"Exception raised"``
+ message is shown. Also note the use of :class:`setName` to assign a human-readable name to the expression,
which makes debugging and exception messages easier to understand - for instance, the default
- name created for the C{Word} expression without calling C{setName} is C{"W:(ABCD...)"}.
+ name created for the :class:`Word` expression without calling ``setName`` is ``"W:(ABCD...)"``.
"""
if flag:
self.setDebugActions( _defaultStartDebugAction, _defaultSuccessDebugAction, _defaultExceptionDebugAction )
@@ -2212,14 +2389,15 @@ class ParserElement(object):
def matches(self, testString, parseAll=True):
"""
- Method for quick testing of a parser against a test string. Good for simple
+ Method for quick testing of a parser against a test string. Good for simple
inline microtests of sub expressions while building up larger parser.
-
+
Parameters:
- testString - to test against this expression for a match
- - parseAll - (default=C{True}) - flag to pass to C{L{parseString}} when running tests
-
+ - parseAll - (default= ``True``) - flag to pass to :class:`parseString` when running tests
+
Example::
+
expr = Word(nums)
assert expr.matches("100")
"""
@@ -2228,28 +2406,32 @@ class ParserElement(object):
return True
except ParseBaseException:
return False
-
- def runTests(self, tests, parseAll=True, comment='#', fullDump=True, printResults=True, failureTests=False):
+
+ def runTests(self, tests, parseAll=True, comment='#',
+ fullDump=True, printResults=True, failureTests=False, postParse=None):
"""
Execute the parse expression on a series of test strings, showing each
test, the parsed results or where the parse failed. Quick and easy way to
run a parse expression against a list of sample strings.
-
+
Parameters:
- tests - a list of separate test strings, or a multiline string of test strings
- - parseAll - (default=C{True}) - flag to pass to C{L{parseString}} when running tests
- - comment - (default=C{'#'}) - expression for indicating embedded comments in the test
+ - parseAll - (default= ``True``) - flag to pass to :class:`parseString` when running tests
+ - comment - (default= ``'#'``) - expression for indicating embedded comments in the test
string; pass None to disable comment filtering
- - fullDump - (default=C{True}) - dump results as list followed by results names in nested outline;
+ - fullDump - (default= ``True``) - dump results as list followed by results names in nested outline;
if False, only dump nested list
- - printResults - (default=C{True}) prints test output to stdout
- - failureTests - (default=C{False}) indicates if these tests are expected to fail parsing
+ - printResults - (default= ``True``) prints test output to stdout
+ - failureTests - (default= ``False``) indicates if these tests are expected to fail parsing
+ - postParse - (default= ``None``) optional callback for successful parse results; called as
+ `fn(test_string, parse_results)` and returns a string to be added to the test output
Returns: a (success, results) tuple, where success indicates that all tests succeeded
- (or failed if C{failureTests} is True), and the results contain a list of lines of each
+ (or failed if ``failureTests`` is True), and the results contain a list of lines of each
test's output
-
+
Example::
+
number_expr = pyparsing_common.number.copy()
result = number_expr.runTests('''
@@ -2273,7 +2455,9 @@ class ParserElement(object):
3.14.159
''', failureTests=True)
print("Success" if result[0] else "Failed!")
+
prints::
+
# unsigned integer
100
[100]
@@ -2291,7 +2475,7 @@ class ParserElement(object):
[1e-12]
Success
-
+
# stray character
100Z
^
@@ -2313,7 +2497,7 @@ class ParserElement(object):
lines, create a test like this::
expr.runTest(r"this is a test\\n of strings that spans \\n 3 lines")
-
+
(Note that this is a raw string literal, you must include the leading 'r'.)
"""
if isinstance(tests, basestring):
@@ -2332,10 +2516,18 @@ class ParserElement(object):
out = ['\n'.join(comments), t]
comments = []
try:
- t = t.replace(r'\n','\n')
+ # convert newline marks to actual newlines, and strip leading BOM if present
+ t = t.replace(r'\n','\n').lstrip('\ufeff')
result = self.parseString(t, parseAll=parseAll)
out.append(result.dump(full=fullDump))
success = success and not failureTests
+ if postParse is not None:
+ try:
+ pp_value = postParse(t, result)
+ if pp_value is not None:
+ out.append(str(pp_value))
+ except Exception as e:
+ out.append("{0} failed: {1}: {2}".format(postParse.__name__, type(e).__name__, e))
except ParseBaseException as pe:
fatal = "(FATAL)" if isinstance(pe, ParseFatalException) else ""
if '\n' in t:
@@ -2357,21 +2549,20 @@ class ParserElement(object):
print('\n'.join(out))
allResults.append((t, result))
-
+
return success, allResults
-
+
class Token(ParserElement):
- """
- Abstract C{ParserElement} subclass, for defining atomic matching patterns.
+ """Abstract :class:`ParserElement` subclass, for defining atomic
+ matching patterns.
"""
def __init__( self ):
super(Token,self).__init__( savelist=False )
class Empty(Token):
- """
- An empty token, will always match.
+ """An empty token, will always match.
"""
def __init__( self ):
super(Empty,self).__init__()
@@ -2381,8 +2572,7 @@ class Empty(Token):
class NoMatch(Token):
- """
- A token that will never match.
+ """A token that will never match.
"""
def __init__( self ):
super(NoMatch,self).__init__()
@@ -2396,18 +2586,18 @@ class NoMatch(Token):
class Literal(Token):
- """
- Token to exactly match a specified string.
-
+ """Token to exactly match a specified string.
+
Example::
+
Literal('blah').parseString('blah') # -> ['blah']
Literal('blah').parseString('blahfooblah') # -> ['blah']
Literal('blah').parseString('bla') # -> Exception: Expected "blah"
-
- For case-insensitive matching, use L{CaselessLiteral}.
-
+
+ For case-insensitive matching, use :class:`CaselessLiteral`.
+
For keyword matching (force word break before and after the matched string),
- use L{Keyword} or L{CaselessKeyword}.
+ use :class:`Keyword` or :class:`CaselessKeyword`.
"""
def __init__( self, matchString ):
super(Literal,self).__init__()
@@ -2437,21 +2627,29 @@ _L = Literal
ParserElement._literalStringClass = Literal
class Keyword(Token):
- """
- Token to exactly match a specified string as a keyword, that is, it must be
- immediately followed by a non-keyword character. Compare with C{L{Literal}}:
- - C{Literal("if")} will match the leading C{'if'} in C{'ifAndOnlyIf'}.
- - C{Keyword("if")} will not; it will only match the leading C{'if'} in C{'if x=1'}, or C{'if(y==2)'}
- Accepts two optional constructor arguments in addition to the keyword string:
- - C{identChars} is a string of characters that would be valid identifier characters,
- defaulting to all alphanumerics + "_" and "$"
- - C{caseless} allows case-insensitive matching, default is C{False}.
-
+ """Token to exactly match a specified string as a keyword, that is,
+ it must be immediately followed by a non-keyword character. Compare
+ with :class:`Literal`:
+
+ - ``Literal("if")`` will match the leading ``'if'`` in
+ ``'ifAndOnlyIf'``.
+ - ``Keyword("if")`` will not; it will only match the leading
+ ``'if'`` in ``'if x=1'``, or ``'if(y==2)'``
+
+ Accepts two optional constructor arguments in addition to the
+ keyword string:
+
+ - ``identChars`` is a string of characters that would be valid
+ identifier characters, defaulting to all alphanumerics + "_" and
+ "$"
+ - ``caseless`` allows case-insensitive matching, default is ``False``.
+
Example::
+
Keyword("start").parseString("start") # -> ['start']
Keyword("start").parseString("starting") # -> Exception
- For case-insensitive matching, use L{CaselessKeyword}.
+ For case-insensitive matching, use :class:`CaselessKeyword`.
"""
DEFAULT_KEYWORD_CHARS = alphanums+"_$"
@@ -2502,15 +2700,15 @@ class Keyword(Token):
Keyword.DEFAULT_KEYWORD_CHARS = chars
class CaselessLiteral(Literal):
- """
- Token to match a specified string, ignoring case of letters.
+ """Token to match a specified string, ignoring case of letters.
Note: the matched results will always be in the case of the given
match string, NOT the case of the input text.
Example::
+
OneOrMore(CaselessLiteral("CMD")).parseString("cmd CMD Cmd10") # -> ['CMD', 'CMD', 'CMD']
-
- (Contrast with example for L{CaselessKeyword}.)
+
+ (Contrast with example for :class:`CaselessKeyword`.)
"""
def __init__( self, matchString ):
super(CaselessLiteral,self).__init__( matchString.upper() )
@@ -2526,36 +2724,39 @@ class CaselessLiteral(Literal):
class CaselessKeyword(Keyword):
"""
- Caseless version of L{Keyword}.
+ Caseless version of :class:`Keyword`.
Example::
+
OneOrMore(CaselessKeyword("CMD")).parseString("cmd CMD Cmd10") # -> ['CMD', 'CMD']
-
- (Contrast with example for L{CaselessLiteral}.)
+
+ (Contrast with example for :class:`CaselessLiteral`.)
"""
def __init__( self, matchString, identChars=None ):
super(CaselessKeyword,self).__init__( matchString, identChars, caseless=True )
- def parseImpl( self, instring, loc, doActions=True ):
- if ( (instring[ loc:loc+self.matchLen ].upper() == self.caselessmatch) and
- (loc >= len(instring)-self.matchLen or instring[loc+self.matchLen].upper() not in self.identChars) ):
- return loc+self.matchLen, self.match
- raise ParseException(instring, loc, self.errmsg, self)
-
class CloseMatch(Token):
- """
- A variation on L{Literal} which matches "close" matches, that is,
- strings with at most 'n' mismatching characters. C{CloseMatch} takes parameters:
- - C{match_string} - string to be matched
- - C{maxMismatches} - (C{default=1}) maximum number of mismatches allowed to count as a match
-
- The results from a successful parse will contain the matched text from the input string and the following named results:
- - C{mismatches} - a list of the positions within the match_string where mismatches were found
- - C{original} - the original match_string used to compare against the input string
-
- If C{mismatches} is an empty list, then the match was an exact match.
-
+ """A variation on :class:`Literal` which matches "close" matches,
+ that is, strings with at most 'n' mismatching characters.
+ :class:`CloseMatch` takes parameters:
+
+ - ``match_string`` - string to be matched
+ - ``maxMismatches`` - (``default=1``) maximum number of
+ mismatches allowed to count as a match
+
+ The results from a successful parse will contain the matched text
+ from the input string and the following named results:
+
+ - ``mismatches`` - a list of the positions within the
+ match_string where mismatches were found
+ - ``original`` - the original match_string used to compare
+ against the input string
+
+ If ``mismatches`` is an empty list, then the match was an exact
+ match.
+
Example::
+
patt = CloseMatch("ATCATCGAATGGA")
patt.parseString("ATCATCGAAXGGA") # -> (['ATCATCGAAXGGA'], {'mismatches': [[9]], 'original': ['ATCATCGAATGGA']})
patt.parseString("ATCAXCGAAXGGA") # -> Exception: Expected 'ATCATCGAATGGA' (with up to 1 mismatches) (at char 0), (line:1, col:1)
@@ -2604,49 +2805,55 @@ class CloseMatch(Token):
class Word(Token):
- """
- Token for matching words composed of allowed character sets.
- Defined with string containing all allowed initial characters,
- an optional string containing allowed body characters (if omitted,
+ """Token for matching words composed of allowed character sets.
+ Defined with string containing all allowed initial characters, an
+ optional string containing allowed body characters (if omitted,
defaults to the initial character set), and an optional minimum,
- maximum, and/or exact length. The default value for C{min} is 1 (a
- minimum value < 1 is not valid); the default values for C{max} and C{exact}
- are 0, meaning no maximum or exact length restriction. An optional
- C{excludeChars} parameter can list characters that might be found in
- the input C{bodyChars} string; useful to define a word of all printables
- except for one or two characters, for instance.
-
- L{srange} is useful for defining custom character set strings for defining
- C{Word} expressions, using range notation from regular expression character sets.
-
- A common mistake is to use C{Word} to match a specific literal string, as in
- C{Word("Address")}. Remember that C{Word} uses the string argument to define
- I{sets} of matchable characters. This expression would match "Add", "AAA",
- "dAred", or any other word made up of the characters 'A', 'd', 'r', 'e', and 's'.
- To match an exact literal string, use L{Literal} or L{Keyword}.
+ maximum, and/or exact length. The default value for ``min`` is
+ 1 (a minimum value < 1 is not valid); the default values for
+ ``max`` and ``exact`` are 0, meaning no maximum or exact
+ length restriction. An optional ``excludeChars`` parameter can
+ list characters that might be found in the input ``bodyChars``
+ string; useful to define a word of all printables except for one or
+ two characters, for instance.
+
+ :class:`srange` is useful for defining custom character set strings
+ for defining ``Word`` expressions, using range notation from
+ regular expression character sets.
+
+ A common mistake is to use :class:`Word` to match a specific literal
+ string, as in ``Word("Address")``. Remember that :class:`Word`
+ uses the string argument to define *sets* of matchable characters.
+ This expression would match "Add", "AAA", "dAred", or any other word
+ made up of the characters 'A', 'd', 'r', 'e', and 's'. To match an
+ exact literal string, use :class:`Literal` or :class:`Keyword`.
pyparsing includes helper strings for building Words:
- - L{alphas}
- - L{nums}
- - L{alphanums}
- - L{hexnums}
- - L{alphas8bit} (alphabetic characters in ASCII range 128-255 - accented, tilded, umlauted, etc.)
- - L{punc8bit} (non-alphabetic characters in ASCII range 128-255 - currency, symbols, superscripts, diacriticals, etc.)
- - L{printables} (any non-whitespace character)
+
+ - :class:`alphas`
+ - :class:`nums`
+ - :class:`alphanums`
+ - :class:`hexnums`
+ - :class:`alphas8bit` (alphabetic characters in ASCII range 128-255
+ - accented, tilded, umlauted, etc.)
+ - :class:`punc8bit` (non-alphabetic characters in ASCII range
+ 128-255 - currency, symbols, superscripts, diacriticals, etc.)
+ - :class:`printables` (any non-whitespace character)
Example::
+
# a word composed of digits
integer = Word(nums) # equivalent to Word("0123456789") or Word(srange("0-9"))
-
+
# a word with a leading capital, and zero or more lowercase
capital_word = Word(alphas.upper(), alphas.lower())
# hostnames are alphanumeric, with leading alpha, and '-'
hostname = Word(alphas, alphanums+'-')
-
+
# roman numeral (not a strict parser, accepts invalid mix of characters)
roman = Word("IVXLCDM")
-
+
# any string of non-whitespace characters, except for ','
csv_value = Word(printables, excludeChars=",")
"""
@@ -2762,22 +2969,38 @@ class Word(Token):
return self.strRepr
+class Char(Word):
+ """A short-cut class for defining ``Word(characters, exact=1)``,
+ when defining a match of any single character in a string of
+ characters.
+ """
+ def __init__(self, charset):
+ super(Char, self).__init__(charset, exact=1)
+ self.reString = "[%s]" % _escapeRegexRangeChars(self.initCharsOrig)
+ self.re = re.compile( self.reString )
+
+
class Regex(Token):
- r"""
- Token for matching strings that match a given regular expression.
- Defined with string specifying the regular expression in a form recognized by the inbuilt Python re module.
- If the given regex contains named groups (defined using C{(?P...)}), these will be preserved as
- named parse results.
+ r"""Token for matching strings that match a given regular
+ expression. Defined with string specifying the regular expression in
+ a form recognized by the stdlib Python `re module `_.
+ If the given regex contains named groups (defined using ``(?P...)``),
+ these will be preserved as named parse results.
Example::
+
realnum = Regex(r"[+-]?\d+\.\d*")
date = Regex(r'(?P\d{4})-(?P\d\d?)-(?P\d\d?)')
- # ref: http://stackoverflow.com/questions/267399/how-do-you-match-only-valid-roman-numerals-with-a-regular-expression
- roman = Regex(r"M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})")
+ # ref: https://stackoverflow.com/questions/267399/how-do-you-match-only-valid-roman-numerals-with-a-regular-expression
+ roman = Regex(r"M{0,4}(CM|CD|D?{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})")
"""
compiledREtype = type(re.compile("[A-Z]"))
- def __init__( self, pattern, flags=0):
- """The parameters C{pattern} and C{flags} are passed to the C{re.compile()} function as-is. See the Python C{re} module for an explanation of the acceptable patterns and flags."""
+ def __init__( self, pattern, flags=0, asGroupList=False, asMatch=False):
+ """The parameters ``pattern`` and ``flags`` are passed
+ to the ``re.compile()`` function as-is. See the Python
+ `re module `_ module for an
+ explanation of the acceptable patterns and flags.
+ """
super(Regex,self).__init__()
if isinstance(pattern, basestring):
@@ -2801,7 +3024,7 @@ class Regex(Token):
self.pattern = \
self.reString = str(pattern)
self.flags = flags
-
+
else:
raise ValueError("Regex may only be constructed with a string or a compiled RE object")
@@ -2809,6 +3032,8 @@ class Regex(Token):
self.errmsg = "Expected " + self.name
self.mayIndexError = False
self.mayReturnEmpty = True
+ self.asGroupList = asGroupList
+ self.asMatch = asMatch
def parseImpl( self, instring, loc, doActions=True ):
result = self.re.match(instring,loc)
@@ -2816,11 +3041,16 @@ class Regex(Token):
raise ParseException(instring, loc, self.errmsg, self)
loc = result.end()
- d = result.groupdict()
- ret = ParseResults(result.group())
- if d:
- for k in d:
- ret[k] = d[k]
+ if self.asMatch:
+ ret = result
+ elif self.asGroupList:
+ ret = result.groups()
+ else:
+ ret = ParseResults(result.group())
+ d = result.groupdict()
+ if d:
+ for k, v in d.items():
+ ret[k] = v
return loc,ret
def __str__( self ):
@@ -2834,28 +3064,70 @@ class Regex(Token):
return self.strRepr
+ def sub(self, repl):
+ """
+ Return Regex with an attached parse action to transform the parsed
+ result as if called using `re.sub(expr, repl, string) `_.
+
+ Example::
+
+ make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2")
+ print(make_html.transformString("h1:main title:"))
+ # prints "
main title
"
+ """
+ if self.asGroupList:
+ warnings.warn("cannot use sub() with Regex(asGroupList=True)",
+ SyntaxWarning, stacklevel=2)
+ raise SyntaxError()
+
+ if self.asMatch and callable(repl):
+ warnings.warn("cannot use sub() with a callable with Regex(asMatch=True)",
+ SyntaxWarning, stacklevel=2)
+ raise SyntaxError()
+
+ if self.asMatch:
+ def pa(tokens):
+ return tokens[0].expand(repl)
+ else:
+ def pa(tokens):
+ return self.re.sub(repl, tokens[0])
+ return self.addParseAction(pa)
class QuotedString(Token):
r"""
Token for matching strings that are delimited by quoting characters.
-
+
Defined with the following parameters:
- - quoteChar - string of one or more characters defining the quote delimiting string
- - escChar - character to escape quotes, typically backslash (default=C{None})
- - escQuote - special quote sequence to escape an embedded quote string (such as SQL's "" to escape an embedded ") (default=C{None})
- - multiline - boolean indicating whether quotes can span multiple lines (default=C{False})
- - unquoteResults - boolean indicating whether the matched text should be unquoted (default=C{True})
- - endQuoteChar - string of one or more characters defining the end of the quote delimited string (default=C{None} => same as quoteChar)
- - convertWhitespaceEscapes - convert escaped whitespace (C{'\t'}, C{'\n'}, etc.) to actual whitespace (default=C{True})
+
+ - quoteChar - string of one or more characters defining the
+ quote delimiting string
+ - escChar - character to escape quotes, typically backslash
+ (default= ``None`` )
+ - escQuote - special quote sequence to escape an embedded quote
+ string (such as SQL's ``""`` to escape an embedded ``"``)
+ (default= ``None`` )
+ - multiline - boolean indicating whether quotes can span
+ multiple lines (default= ``False`` )
+ - unquoteResults - boolean indicating whether the matched text
+ should be unquoted (default= ``True`` )
+ - endQuoteChar - string of one or more characters defining the
+ end of the quote delimited string (default= ``None`` => same as
+ quoteChar)
+ - convertWhitespaceEscapes - convert escaped whitespace
+ (``'\t'``, ``'\n'``, etc.) to actual whitespace
+ (default= ``True`` )
Example::
+
qs = QuotedString('"')
print(qs.searchString('lsjdf "This is the quote" sldjf'))
complex_qs = QuotedString('{{', endQuoteChar='}}')
print(complex_qs.searchString('lsjdf {{This is the "quote"}} sldjf'))
sql_qs = QuotedString('"', escQuote='""')
print(sql_qs.searchString('lsjdf "This is the quote with ""embedded"" quotes" sldjf'))
+
prints::
+
[['This is the quote']]
[['This is the "quote"']]
[['This is the quote with "embedded" quotes']]
@@ -2973,19 +3245,23 @@ class QuotedString(Token):
class CharsNotIn(Token):
- """
- Token for matching words composed of characters I{not} in a given set (will
- include whitespace in matched characters if not listed in the provided exclusion set - see example).
- Defined with string containing all disallowed characters, and an optional
- minimum, maximum, and/or exact length. The default value for C{min} is 1 (a
- minimum value < 1 is not valid); the default values for C{max} and C{exact}
- are 0, meaning no maximum or exact length restriction.
+ """Token for matching words composed of characters *not* in a given
+ set (will include whitespace in matched characters if not listed in
+ the provided exclusion set - see example). Defined with string
+ containing all disallowed characters, and an optional minimum,
+ maximum, and/or exact length. The default value for ``min`` is
+ 1 (a minimum value < 1 is not valid); the default values for
+ ``max`` and ``exact`` are 0, meaning no maximum or exact
+ length restriction.
Example::
+
# define a comma-separated-value as anything that is not a ','
csv_value = CharsNotIn(',')
print(delimitedList(csv_value).parseString("dkls,lsdkjf,s12 34,@!#,213"))
+
prints::
+
['dkls', 'lsdkjf', 's12 34', '@!#', '213']
"""
def __init__( self, notChars, min=1, max=0, exact=0 ):
@@ -2994,7 +3270,9 @@ class CharsNotIn(Token):
self.notChars = notChars
if min < 1:
- raise ValueError("cannot specify a minimum length < 1; use Optional(CharsNotIn()) if zero-length char group is permitted")
+ raise ValueError(
+ "cannot specify a minimum length < 1; use " +
+ "Optional(CharsNotIn()) if zero-length char group is permitted")
self.minLen = min
@@ -3044,19 +3322,38 @@ class CharsNotIn(Token):
return self.strRepr
class White(Token):
- """
- Special matching class for matching whitespace. Normally, whitespace is ignored
- by pyparsing grammars. This class is included when some whitespace structures
- are significant. Define with a string containing the whitespace characters to be
- matched; default is C{" \\t\\r\\n"}. Also takes optional C{min}, C{max}, and C{exact} arguments,
- as defined for the C{L{Word}} class.
+ """Special matching class for matching whitespace. Normally,
+ whitespace is ignored by pyparsing grammars. This class is included
+ when some whitespace structures are significant. Define with
+ a string containing the whitespace characters to be matched; default
+ is ``" \\t\\r\\n"``. Also takes optional ``min``,
+ ``max``, and ``exact`` arguments, as defined for the
+ :class:`Word` class.
"""
whiteStrs = {
- " " : "",
- "\t": "",
- "\n": "",
- "\r": "",
- "\f": "",
+ ' ' : '',
+ '\t': '',
+ '\n': '',
+ '\r': '',
+ '\f': '',
+ 'u\00A0': '',
+ 'u\1680': '',
+ 'u\180E': '',
+ 'u\2000': '',
+ 'u\2001': '',
+ 'u\2002': '',
+ 'u\2003': '',
+ 'u\2004': '',
+ 'u\2005': '',
+ 'u\2006': '',
+ 'u\2007': '',
+ 'u\2008': '',
+ 'u\2009': '',
+ 'u\200A': '',
+ 'u\200B': '',
+ 'u\202F': '',
+ 'u\205F': '',
+ 'u\3000': '',
}
def __init__(self, ws=" \t\r\n", min=1, max=0, exact=0):
super(White,self).__init__()
@@ -3102,8 +3399,8 @@ class _PositionToken(Token):
self.mayIndexError = False
class GoToColumn(_PositionToken):
- """
- Token to advance to a specific column of input text; useful for tabular report scraping.
+ """Token to advance to a specific column of input text; useful for
+ tabular report scraping.
"""
def __init__( self, colno ):
super(GoToColumn,self).__init__()
@@ -3128,11 +3425,11 @@ class GoToColumn(_PositionToken):
class LineStart(_PositionToken):
- """
- Matches if current position is at the beginning of a line within the parse string
-
+ """Matches if current position is at the beginning of a line within
+ the parse string
+
Example::
-
+
test = '''\
AAA this line
AAA and this line
@@ -3142,10 +3439,11 @@ class LineStart(_PositionToken):
for t in (LineStart() + 'AAA' + restOfLine).searchString(test):
print(t)
-
- Prints::
+
+ prints::
+
['AAA', ' this line']
- ['AAA', ' and this line']
+ ['AAA', ' and this line']
"""
def __init__( self ):
@@ -3158,8 +3456,8 @@ class LineStart(_PositionToken):
raise ParseException(instring, loc, self.errmsg, self)
class LineEnd(_PositionToken):
- """
- Matches if current position is at the end of a line within the parse string
+ """Matches if current position is at the end of a line within the
+ parse string
"""
def __init__( self ):
super(LineEnd,self).__init__()
@@ -3178,8 +3476,8 @@ class LineEnd(_PositionToken):
raise ParseException(instring, loc, self.errmsg, self)
class StringStart(_PositionToken):
- """
- Matches if current position is at the beginning of the parse string
+ """Matches if current position is at the beginning of the parse
+ string
"""
def __init__( self ):
super(StringStart,self).__init__()
@@ -3193,8 +3491,7 @@ class StringStart(_PositionToken):
return loc, []
class StringEnd(_PositionToken):
- """
- Matches if current position is at the end of the parse string
+ """Matches if current position is at the end of the parse string
"""
def __init__( self ):
super(StringEnd,self).__init__()
@@ -3211,12 +3508,13 @@ class StringEnd(_PositionToken):
raise ParseException(instring, loc, self.errmsg, self)
class WordStart(_PositionToken):
- """
- Matches if the current position is at the beginning of a Word, and
- is not preceded by any character in a given set of C{wordChars}
- (default=C{printables}). To emulate the C{\b} behavior of regular expressions,
- use C{WordStart(alphanums)}. C{WordStart} will also match at the beginning of
- the string being parsed, or at the beginning of a line.
+ """Matches if the current position is at the beginning of a Word,
+ and is not preceded by any character in a given set of
+ ``wordChars`` (default= ``printables``). To emulate the
+ ``\b`` behavior of regular expressions, use
+ ``WordStart(alphanums)``. ``WordStart`` will also match at
+ the beginning of the string being parsed, or at the beginning of
+ a line.
"""
def __init__(self, wordChars = printables):
super(WordStart,self).__init__()
@@ -3231,12 +3529,12 @@ class WordStart(_PositionToken):
return loc, []
class WordEnd(_PositionToken):
- """
- Matches if the current position is at the end of a Word, and
- is not followed by any character in a given set of C{wordChars}
- (default=C{printables}). To emulate the C{\b} behavior of regular expressions,
- use C{WordEnd(alphanums)}. C{WordEnd} will also match at the end of
- the string being parsed, or at the end of a line.
+ """Matches if the current position is at the end of a Word, and is
+ not followed by any character in a given set of ``wordChars``
+ (default= ``printables``). To emulate the ``\b`` behavior of
+ regular expressions, use ``WordEnd(alphanums)``. ``WordEnd``
+ will also match at the end of the string being parsed, or at the end
+ of a line.
"""
def __init__(self, wordChars = printables):
super(WordEnd,self).__init__()
@@ -3254,8 +3552,8 @@ class WordEnd(_PositionToken):
class ParseExpression(ParserElement):
- """
- Abstract subclass of ParserElement, for combining and post-processing parsed tokens.
+ """Abstract subclass of ParserElement, for combining and
+ post-processing parsed tokens.
"""
def __init__( self, exprs, savelist = False ):
super(ParseExpression,self).__init__(savelist)
@@ -3286,7 +3584,7 @@ class ParseExpression(ParserElement):
return self
def leaveWhitespace( self ):
- """Extends C{leaveWhitespace} defined in base class, and also invokes C{leaveWhitespace} on
+ """Extends ``leaveWhitespace`` defined in base class, and also invokes ``leaveWhitespace`` on
all contained expressions."""
self.skipWhitespace = False
self.exprs = [ e.copy() for e in self.exprs ]
@@ -3347,7 +3645,7 @@ class ParseExpression(ParserElement):
self.mayIndexError |= other.mayIndexError
self.errmsg = "Expected " + _ustr(self)
-
+
return self
def setResultsName( self, name, listAllMatches=False ):
@@ -3359,7 +3657,7 @@ class ParseExpression(ParserElement):
for e in self.exprs:
e.validate(tmp)
self.checkRecursion( [] )
-
+
def copy(self):
ret = super(ParseExpression,self).copy()
ret.exprs = [e.copy() for e in self.exprs]
@@ -3367,12 +3665,14 @@ class ParseExpression(ParserElement):
class And(ParseExpression):
"""
- Requires all given C{ParseExpression}s to be found in the given order.
+ Requires all given :class:`ParseExpression` s to be found in the given order.
Expressions may be separated by whitespace.
- May be constructed using the C{'+'} operator.
- May also be constructed using the C{'-'} operator, which will suppress backtracking.
+ May be constructed using the ``'+'`` operator.
+ May also be constructed using the ``'-'`` operator, which will
+ suppress backtracking.
Example::
+
integer = Word(nums)
name_expr = OneOrMore(Word(alphas))
@@ -3394,6 +3694,11 @@ class And(ParseExpression):
self.skipWhitespace = self.exprs[0].skipWhitespace
self.callPreparse = True
+ def streamline(self):
+ super(And, self).streamline()
+ self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
+ return self
+
def parseImpl( self, instring, loc, doActions=True ):
# pass False as last arg to _parse for first element, since we already
# pre-parsed the string as part of our And pre-parsing
@@ -3442,17 +3747,20 @@ class And(ParseExpression):
class Or(ParseExpression):
- """
- Requires that at least one C{ParseExpression} is found.
- If two expressions match, the expression that matches the longest string will be used.
- May be constructed using the C{'^'} operator.
+ """Requires that at least one :class:`ParseExpression` is found. If
+ two expressions match, the expression that matches the longest
+ string will be used. May be constructed using the ``'^'``
+ operator.
Example::
+
# construct Or using '^' operator
-
+
number = Word(nums) ^ Combine(Word(nums) + '.' + Word(nums))
print(number.searchString("123 3.1416 789"))
+
prints::
+
[['123'], ['3.1416'], ['789']]
"""
def __init__( self, exprs, savelist = False ):
@@ -3462,6 +3770,11 @@ class Or(ParseExpression):
else:
self.mayReturnEmpty = True
+ def streamline(self):
+ super(Or, self).streamline()
+ self.saveAsList = any(e.saveAsList for e in self.exprs)
+ return self
+
def parseImpl( self, instring, loc, doActions=True ):
maxExcLoc = -1
maxException = None
@@ -3521,14 +3834,14 @@ class Or(ParseExpression):
class MatchFirst(ParseExpression):
- """
- Requires that at least one C{ParseExpression} is found.
- If two expressions match, the first one listed is the one that will match.
- May be constructed using the C{'|'} operator.
+ """Requires that at least one :class:`ParseExpression` is found. If
+ two expressions match, the first one listed is the one that will
+ match. May be constructed using the ``'|'`` operator.
Example::
+
# construct MatchFirst using '|' operator
-
+
# watch the order of expressions to match
number = Word(nums) | Combine(Word(nums) + '.' + Word(nums))
print(number.searchString("123 3.1416 789")) # Fail! -> [['123'], ['3'], ['1416'], ['789']]
@@ -3541,9 +3854,15 @@ class MatchFirst(ParseExpression):
super(MatchFirst,self).__init__(exprs, savelist)
if self.exprs:
self.mayReturnEmpty = any(e.mayReturnEmpty for e in self.exprs)
+ # self.saveAsList = any(e.saveAsList for e in self.exprs)
else:
self.mayReturnEmpty = True
+ def streamline(self):
+ super(MatchFirst, self).streamline()
+ self.saveAsList = any(e.saveAsList for e in self.exprs)
+ return self
+
def parseImpl( self, instring, loc, doActions=True ):
maxExcLoc = -1
maxException = None
@@ -3589,12 +3908,13 @@ class MatchFirst(ParseExpression):
class Each(ParseExpression):
- """
- Requires all given C{ParseExpression}s to be found, but in any order.
- Expressions may be separated by whitespace.
- May be constructed using the C{'&'} operator.
+ """Requires all given :class:`ParseExpression` s to be found, but in
+ any order. Expressions may be separated by whitespace.
+
+ May be constructed using the ``'&'`` operator.
Example::
+
color = oneOf("RED ORANGE YELLOW GREEN BLUE PURPLE BLACK WHITE BROWN")
shape_type = oneOf("SQUARE CIRCLE TRIANGLE STAR HEXAGON OCTAGON")
integer = Word(nums)
@@ -3603,7 +3923,7 @@ class Each(ParseExpression):
color_attr = "color:" + color("color")
size_attr = "size:" + integer("size")
- # use Each (using operator '&') to accept attributes in any order
+ # use Each (using operator '&') to accept attributes in any order
# (shape and posn are required, color and size are optional)
shape_spec = shape_attr & posn_attr & Optional(color_attr) & Optional(size_attr)
@@ -3613,7 +3933,9 @@ class Each(ParseExpression):
color:GREEN size:20 shape:TRIANGLE posn:20,40
'''
)
+
prints::
+
shape: SQUARE color: BLACK posn: 100, 120
['shape:', 'SQUARE', 'color:', 'BLACK', 'posn:', ['100', ',', '120']]
- color: BLACK
@@ -3647,6 +3969,12 @@ class Each(ParseExpression):
self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
self.skipWhitespace = True
self.initExprGroups = True
+ self.saveAsList = True
+
+ def streamline(self):
+ super(Each, self).streamline()
+ self.mayReturnEmpty = all(e.mayReturnEmpty for e in self.exprs)
+ return self
def parseImpl( self, instring, loc, doActions=True ):
if self.initExprGroups:
@@ -3713,8 +4041,8 @@ class Each(ParseExpression):
class ParseElementEnhance(ParserElement):
- """
- Abstract subclass of C{ParserElement}, for combining and post-processing parsed tokens.
+ """Abstract subclass of :class:`ParserElement`, for combining and
+ post-processing parsed tokens.
"""
def __init__( self, expr, savelist=False ):
super(ParseElementEnhance,self).__init__(savelist)
@@ -3790,20 +4118,25 @@ class ParseElementEnhance(ParserElement):
class FollowedBy(ParseElementEnhance):
- """
- Lookahead matching of the given parse expression. C{FollowedBy}
- does I{not} advance the parsing position within the input string, it only
- verifies that the specified parse expression matches at the current
- position. C{FollowedBy} always returns a null token list.
+ """Lookahead matching of the given parse expression.
+ ``FollowedBy`` does *not* advance the parsing position within
+ the input string, it only verifies that the specified parse
+ expression matches at the current position. ``FollowedBy``
+ always returns a null token list. If any results names are defined
+ in the lookahead expression, those *will* be returned for access by
+ name.
Example::
+
# use FollowedBy to match a label only if it is followed by a ':'
data_word = Word(alphas)
label = data_word + FollowedBy(':')
attr_expr = Group(label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
-
+
OneOrMore(attr_expr).parseString("shape: SQUARE color: BLACK posn: upper left").pprint()
+
prints::
+
[['shape', 'SQUARE'], ['color', 'BLACK'], ['posn', 'upper left']]
"""
def __init__( self, expr ):
@@ -3811,20 +4144,108 @@ class FollowedBy(ParseElementEnhance):
self.mayReturnEmpty = True
def parseImpl( self, instring, loc, doActions=True ):
- self.expr.tryParse( instring, loc )
- return loc, []
+ _, ret = self.expr._parse(instring, loc, doActions=doActions)
+ del ret[:]
+ return loc, ret
+
+
+class PrecededBy(ParseElementEnhance):
+ """Lookbehind matching of the given parse expression.
+ ``PrecededBy`` does not advance the parsing position within the
+ input string, it only verifies that the specified parse expression
+ matches prior to the current position. ``PrecededBy`` always
+ returns a null token list, but if a results name is defined on the
+ given expression, it is returned.
+
+ Parameters:
+
+ - expr - expression that must match prior to the current parse
+ location
+ - retreat - (default= ``None``) - (int) maximum number of characters
+ to lookbehind prior to the current parse location
+
+ If the lookbehind expression is a string, Literal, Keyword, or
+ a Word or CharsNotIn with a specified exact or maximum length, then
+ the retreat parameter is not required. Otherwise, retreat must be
+ specified to give a maximum number of characters to look back from
+ the current parse position for a lookbehind match.
+
+ Example::
+
+ # VB-style variable names with type prefixes
+ int_var = PrecededBy("#") + pyparsing_common.identifier
+ str_var = PrecededBy("$") + pyparsing_common.identifier
+
+ """
+ def __init__(self, expr, retreat=None):
+ super(PrecededBy, self).__init__(expr)
+ self.expr = self.expr().leaveWhitespace()
+ self.mayReturnEmpty = True
+ self.mayIndexError = False
+ self.exact = False
+ if isinstance(expr, str):
+ retreat = len(expr)
+ self.exact = True
+ elif isinstance(expr, (Literal, Keyword)):
+ retreat = expr.matchLen
+ self.exact = True
+ elif isinstance(expr, (Word, CharsNotIn)) and expr.maxLen != _MAX_INT:
+ retreat = expr.maxLen
+ self.exact = True
+ elif isinstance(expr, _PositionToken):
+ retreat = 0
+ self.exact = True
+ self.retreat = retreat
+ self.errmsg = "not preceded by " + str(expr)
+ self.skipWhitespace = False
+
+ def parseImpl(self, instring, loc=0, doActions=True):
+ if self.exact:
+ if loc < self.retreat:
+ raise ParseException(instring, loc, self.errmsg)
+ start = loc - self.retreat
+ _, ret = self.expr._parse(instring, start)
+ else:
+ # retreat specified a maximum lookbehind window, iterate
+ test_expr = self.expr + StringEnd()
+ instring_slice = instring[:loc]
+ last_expr = ParseException(instring, loc, self.errmsg)
+ for offset in range(1, min(loc, self.retreat+1)):
+ try:
+ _, ret = test_expr._parse(instring_slice, loc-offset)
+ except ParseBaseException as pbe:
+ last_expr = pbe
+ else:
+ break
+ else:
+ raise last_expr
+ # return empty list of tokens, but preserve any defined results names
+ del ret[:]
+ return loc, ret
class NotAny(ParseElementEnhance):
- """
- Lookahead to disallow matching with the given parse expression. C{NotAny}
- does I{not} advance the parsing position within the input string, it only
- verifies that the specified parse expression does I{not} match at the current
- position. Also, C{NotAny} does I{not} skip over leading whitespace. C{NotAny}
- always returns a null token list. May be constructed using the '~' operator.
+ """Lookahead to disallow matching with the given parse expression.
+ ``NotAny`` does *not* advance the parsing position within the
+ input string, it only verifies that the specified parse expression
+ does *not* match at the current position. Also, ``NotAny`` does
+ *not* skip over leading whitespace. ``NotAny`` always returns
+ a null token list. May be constructed using the '~' operator.
Example::
-
+
+ AND, OR, NOT = map(CaselessKeyword, "AND OR NOT".split())
+
+ # take care not to mistake keywords for identifiers
+ ident = ~(AND | OR | NOT) + Word(alphas)
+ boolean_term = Optional(NOT) + ident
+
+ # very crude boolean expression - to support parenthesis groups and
+ # operation hierarchy, use infixNotation
+ boolean_expr = boolean_term + ZeroOrMore((AND | OR) + boolean_term)
+
+ # integers that are followed by "." are actually floats
+ integer = Word(nums) + ~Char(".")
"""
def __init__( self, expr ):
super(NotAny,self).__init__(expr)
@@ -3862,7 +4283,7 @@ class _MultipleMatch(ParseElementEnhance):
check_ender = self.not_ender is not None
if check_ender:
try_not_ender = self.not_ender.tryParse
-
+
# must be at least one (but first see if we are the stopOn sentinel;
# if so, fail)
if check_ender:
@@ -3884,18 +4305,18 @@ class _MultipleMatch(ParseElementEnhance):
pass
return loc, tokens
-
+
class OneOrMore(_MultipleMatch):
- """
- Repetition of one or more of the given expression.
-
+ """Repetition of one or more of the given expression.
+
Parameters:
- expr - expression that must match one or more times
- - stopOn - (default=C{None}) - expression for a terminating sentinel
- (only required if the sentinel would ordinarily match the repetition
- expression)
+ - stopOn - (default= ``None``) - expression for a terminating sentinel
+ (only required if the sentinel would ordinarily match the repetition
+ expression)
Example::
+
data_word = Word(alphas)
label = data_word + FollowedBy(':')
attr_expr = Group(label + Suppress(':') + OneOrMore(data_word).setParseAction(' '.join))
@@ -3906,7 +4327,7 @@ class OneOrMore(_MultipleMatch):
# use stopOn attribute for OneOrMore to avoid reading label string as part of the data
attr_expr = Group(label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
OneOrMore(attr_expr).parseString(text).pprint() # Better -> [['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'BLACK']]
-
+
# could also be written as
(attr_expr * (1,)).parseString(text).pprint()
"""
@@ -3921,21 +4342,20 @@ class OneOrMore(_MultipleMatch):
return self.strRepr
class ZeroOrMore(_MultipleMatch):
- """
- Optional repetition of zero or more of the given expression.
-
+ """Optional repetition of zero or more of the given expression.
+
Parameters:
- expr - expression that must match zero or more times
- - stopOn - (default=C{None}) - expression for a terminating sentinel
- (only required if the sentinel would ordinarily match the repetition
- expression)
+ - stopOn - (default= ``None``) - expression for a terminating sentinel
+ (only required if the sentinel would ordinarily match the repetition
+ expression)
- Example: similar to L{OneOrMore}
+ Example: similar to :class:`OneOrMore`
"""
def __init__( self, expr, stopOn=None):
super(ZeroOrMore,self).__init__(expr, stopOn=stopOn)
self.mayReturnEmpty = True
-
+
def parseImpl( self, instring, loc, doActions=True ):
try:
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
@@ -3960,27 +4380,29 @@ class _NullToken(object):
_optionalNotMatched = _NullToken()
class Optional(ParseElementEnhance):
- """
- Optional matching of the given expression.
+ """Optional matching of the given expression.
Parameters:
- expr - expression that must match zero or more times
- default (optional) - value to be returned if the optional expression is not found.
Example::
+
# US postal code can be a 5-digit zip, plus optional 4-digit qualifier
zip = Combine(Word(nums, exact=5) + Optional('-' + Word(nums, exact=4)))
zip.runTests('''
# traditional ZIP code
12345
-
+
# ZIP+4 form
12101-0001
-
+
# invalid ZIP
98765-
''')
+
prints::
+
# traditional ZIP code
12345
['12345']
@@ -4024,20 +4446,21 @@ class Optional(ParseElementEnhance):
return self.strRepr
class SkipTo(ParseElementEnhance):
- """
- Token for skipping over all undefined text until the matched expression is found.
+ """Token for skipping over all undefined text until the matched
+ expression is found.
Parameters:
- expr - target expression marking the end of the data to be skipped
- - include - (default=C{False}) if True, the target expression is also parsed
+ - include - (default= ``False``) if True, the target expression is also parsed
(the skipped text and target expression are returned as a 2-element list).
- - ignore - (default=C{None}) used to define grammars (typically quoted strings and
+ - ignore - (default= ``None``) used to define grammars (typically quoted strings and
comments) that might contain false matches to the target expression
- - failOn - (default=C{None}) define expressions that are not allowed to be
- included in the skipped test; if found before the target expression is found,
+ - failOn - (default= ``None``) define expressions that are not allowed to be
+ included in the skipped test; if found before the target expression is found,
the SkipTo is not a match
Example::
+
report = '''
Outstanding Issues Report - 1 Jan 2000
@@ -4054,14 +4477,16 @@ class SkipTo(ParseElementEnhance):
# - parse action will call token.strip() for each matched token, i.e., the description body
string_data = SkipTo(SEP, ignore=quotedString)
string_data.setParseAction(tokenMap(str.strip))
- ticket_expr = (integer("issue_num") + SEP
- + string_data("sev") + SEP
- + string_data("desc") + SEP
+ ticket_expr = (integer("issue_num") + SEP
+ + string_data("sev") + SEP
+ + string_data("desc") + SEP
+ integer("days_open"))
-
+
for tkt in ticket_expr.searchString(report):
print tkt.dump()
+
prints::
+
['101', 'Critical', 'Intermittent system crash', '6']
- days_open: 6
- desc: Intermittent system crash
@@ -4084,7 +4509,7 @@ class SkipTo(ParseElementEnhance):
self.mayReturnEmpty = True
self.mayIndexError = False
self.includeMatch = include
- self.asList = False
+ self.saveAsList = False
if isinstance(failOn, basestring):
self.failOn = ParserElement._literalStringClass(failOn)
else:
@@ -4098,14 +4523,14 @@ class SkipTo(ParseElementEnhance):
expr_parse = self.expr._parse
self_failOn_canParseNext = self.failOn.canParseNext if self.failOn is not None else None
self_ignoreExpr_tryParse = self.ignoreExpr.tryParse if self.ignoreExpr is not None else None
-
+
tmploc = loc
while tmploc <= instrlen:
if self_failOn_canParseNext is not None:
# break if failOn expression matches
if self_failOn_canParseNext(instring, tmploc):
break
-
+
if self_ignoreExpr_tryParse is not None:
# advance past ignore expressions
while 1:
@@ -4113,7 +4538,7 @@ class SkipTo(ParseElementEnhance):
tmploc = self_ignoreExpr_tryParse(instring, tmploc)
except ParseBaseException:
break
-
+
try:
expr_parse(instring, tmploc, doActions=False, callPreParse=False)
except (ParseException, IndexError):
@@ -4131,7 +4556,7 @@ class SkipTo(ParseElementEnhance):
loc = tmploc
skiptext = instring[startloc:loc]
skipresult = ParseResults(skiptext)
-
+
if self.includeMatch:
loc, mat = expr_parse(instring,loc,doActions,callPreParse=False)
skipresult += mat
@@ -4139,23 +4564,31 @@ class SkipTo(ParseElementEnhance):
return loc, skipresult
class Forward(ParseElementEnhance):
- """
- Forward declaration of an expression to be defined later -
+ """Forward declaration of an expression to be defined later -
used for recursive grammars, such as algebraic infix notation.
- When the expression is known, it is assigned to the C{Forward} variable using the '<<' operator.
+ When the expression is known, it is assigned to the ``Forward``
+ variable using the '<<' operator.
+
+ Note: take care when assigning to ``Forward`` not to overlook
+ precedence of operators.
- Note: take care when assigning to C{Forward} not to overlook precedence of operators.
Specifically, '|' has a lower precedence than '<<', so that::
+
fwdExpr << a | b | c
+
will actually be evaluated as::
+
(fwdExpr << a) | b | c
+
thereby leaving b and c out as parseable alternatives. It is recommended that you
- explicitly group the values inserted into the C{Forward}::
+ explicitly group the values inserted into the ``Forward``::
+
fwdExpr << (a | b | c)
+
Converting to use the '<<=' operator instead will avoid this problem.
- See L{ParseResults.pprint} for an example of a recursive parser created using
- C{Forward}.
+ See :class:`ParseResults.pprint` for an example of a recursive
+ parser created using ``Forward``.
"""
def __init__( self, other=None ):
super(Forward,self).__init__( other, savelist=False )
@@ -4172,10 +4605,10 @@ class Forward(ParseElementEnhance):
self.saveAsList = self.expr.saveAsList
self.ignoreExprs.extend(self.expr.ignoreExprs)
return self
-
+
def __ilshift__(self, other):
return self << other
-
+
def leaveWhitespace( self ):
self.skipWhitespace = False
return self
@@ -4225,19 +4658,20 @@ class _ForwardNoRecurse(Forward):
class TokenConverter(ParseElementEnhance):
"""
- Abstract subclass of C{ParseExpression}, for converting parsed results.
+ Abstract subclass of :class:`ParseExpression`, for converting parsed results.
"""
def __init__( self, expr, savelist=False ):
super(TokenConverter,self).__init__( expr )#, savelist )
self.saveAsList = False
class Combine(TokenConverter):
- """
- Converter to concatenate all matching tokens to a single string.
- By default, the matching patterns must also be contiguous in the input string;
- this can be disabled by specifying C{'adjacent=False'} in the constructor.
+ """Converter to concatenate all matching tokens to a single string.
+ By default, the matching patterns must also be contiguous in the
+ input string; this can be disabled by specifying
+ ``'adjacent=False'`` in the constructor.
Example::
+
real = Word(nums) + '.' + Word(nums)
print(real.parseString('3.1416')) # -> ['3', '.', '1416']
# will also erroneously match the following
@@ -4276,10 +4710,11 @@ class Combine(TokenConverter):
return retToks
class Group(TokenConverter):
- """
- Converter to return the matched tokens as a list - useful for returning tokens of C{L{ZeroOrMore}} and C{L{OneOrMore}} expressions.
+ """Converter to return the matched tokens as a list - useful for
+ returning tokens of :class:`ZeroOrMore` and :class:`OneOrMore` expressions.
Example::
+
ident = Word(alphas)
num = Word(nums)
term = ident | num
@@ -4291,38 +4726,40 @@ class Group(TokenConverter):
"""
def __init__( self, expr ):
super(Group,self).__init__( expr )
- self.saveAsList = True
+ self.saveAsList = expr.saveAsList
def postParse( self, instring, loc, tokenlist ):
return [ tokenlist ]
class Dict(TokenConverter):
- """
- Converter to return a repetitive expression as a list, but also as a dictionary.
- Each element can also be referenced using the first token in the expression as its key.
- Useful for tabular report scraping when the first column can be used as a item key.
+ """Converter to return a repetitive expression as a list, but also
+ as a dictionary. Each element can also be referenced using the first
+ token in the expression as its key. Useful for tabular report
+ scraping when the first column can be used as a item key.
Example::
+
data_word = Word(alphas)
label = data_word + FollowedBy(':')
attr_expr = Group(label + Suppress(':') + OneOrMore(data_word).setParseAction(' '.join))
text = "shape: SQUARE posn: upper left color: light blue texture: burlap"
attr_expr = (label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
-
+
# print attributes as plain groups
print(OneOrMore(attr_expr).parseString(text).dump())
-
+
# instead of OneOrMore(expr), parse using Dict(OneOrMore(Group(expr))) - Dict will auto-assign names
result = Dict(OneOrMore(Group(attr_expr))).parseString(text)
print(result.dump())
-
- # access named fields as dict entries, or output as dict
- print(result['shape'])
- print(result.asDict())
- prints::
- ['shape', 'SQUARE', 'posn', 'upper left', 'color', 'light blue', 'texture', 'burlap']
+ # access named fields as dict entries, or output as dict
+ print(result['shape'])
+ print(result.asDict())
+
+ prints::
+
+ ['shape', 'SQUARE', 'posn', 'upper left', 'color', 'light blue', 'texture', 'burlap']
[['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'light blue'], ['texture', 'burlap']]
- color: light blue
- posn: upper left
@@ -4330,7 +4767,8 @@ class Dict(TokenConverter):
- texture: burlap
SQUARE
{'color': 'light blue', 'posn': 'upper left', 'texture': 'burlap', 'shape': 'SQUARE'}
- See more examples at L{ParseResults} of accessing fields by results name.
+
+ See more examples at :class:`ParseResults` of accessing fields by results name.
"""
def __init__( self, expr ):
super(Dict,self).__init__( expr )
@@ -4362,10 +4800,10 @@ class Dict(TokenConverter):
class Suppress(TokenConverter):
- """
- Converter for ignoring the results of a parsed expression.
+ """Converter for ignoring the results of a parsed expression.
Example::
+
source = "a, b, c,d"
wd = Word(alphas)
wd_list1 = wd + ZeroOrMore(',' + wd)
@@ -4375,10 +4813,13 @@ class Suppress(TokenConverter):
# way afterward - use Suppress to keep them out of the parsed output
wd_list2 = wd + ZeroOrMore(Suppress(',') + wd)
print(wd_list2.parseString(source))
+
prints::
+
['a', ',', 'b', ',', 'c', ',', 'd']
['a', 'b', 'c', 'd']
- (See also L{delimitedList}.)
+
+ (See also :class:`delimitedList`.)
"""
def postParse( self, instring, loc, tokenlist ):
return []
@@ -4388,8 +4829,7 @@ class Suppress(TokenConverter):
class OnlyOnce(object):
- """
- Wrapper for parse actions, to ensure they are only called once.
+ """Wrapper for parse actions, to ensure they are only called once.
"""
def __init__(self, methodCall):
self.callable = _trim_arity(methodCall)
@@ -4404,13 +4844,15 @@ class OnlyOnce(object):
self.called = False
def traceParseAction(f):
- """
- Decorator for debugging parse actions.
-
- When the parse action is called, this decorator will print C{">> entering I{method-name}(line:I{current_source_line}, I{parse_location}, I{matched_tokens})".}
- When the parse action completes, the decorator will print C{"<<"} followed by the returned value, or any exception that the parse action raised.
+ """Decorator for debugging parse actions.
+
+ When the parse action is called, this decorator will print
+ ``">> entering method-name(line:, , )"``.
+ When the parse action completes, the decorator will print
+ ``"<<"`` followed by the returned value, or any exception that the parse action raised.
Example::
+
wd = Word(alphas)
@traceParseAction
@@ -4419,7 +4861,9 @@ def traceParseAction(f):
wds = OneOrMore(wd).setParseAction(remove_duplicate_chars)
print(wds.parseString("slkdjs sld sldd sdlf sdljf"))
+
prints::
+
>>entering remove_duplicate_chars(line: 'slkdjs sld sldd sdlf sdljf', 0, (['slkdjs', 'sld', 'sldd', 'sdlf', 'sdljf'], {}))
< ['aa', 'bb', 'cc']
delimitedList(Word(hexnums), delim=':', combine=True).parseString("AA:BB:CC:DD:EE") # -> ['AA:BB:CC:DD:EE']
"""
@@ -4467,16 +4913,21 @@ def delimitedList( expr, delim=",", combine=False ):
return ( expr + ZeroOrMore( Suppress( delim ) + expr ) ).setName(dlName)
def countedArray( expr, intExpr=None ):
- """
- Helper to define a counted list of expressions.
+ """Helper to define a counted list of expressions.
+
This helper defines a pattern of the form::
+
integer expr expr expr...
+
where the leading integer tells how many expr expressions follow.
- The matched tokens returns the array of expr tokens as a list - the leading count token is suppressed.
-
- If C{intExpr} is specified, it should be a pyparsing expression that produces an integer value.
+ The matched tokens returns the array of expr tokens as a list - the
+ leading count token is suppressed.
+
+ If ``intExpr`` is specified, it should be a pyparsing expression
+ that produces an integer value.
Example::
+
countedArray(Word(alphas)).parseString('2 ab cd ef') # -> ['ab', 'cd']
# in this parser, the leading integer value is given in binary,
@@ -4507,17 +4958,19 @@ def _flatten(L):
return ret
def matchPreviousLiteral(expr):
- """
- Helper to define an expression that is indirectly defined from
- the tokens matched in a previous expression, that is, it looks
- for a 'repeat' of a previous expression. For example::
+ """Helper to define an expression that is indirectly defined from
+ the tokens matched in a previous expression, that is, it looks for
+ a 'repeat' of a previous expression. For example::
+
first = Word(nums)
second = matchPreviousLiteral(first)
matchExpr = first + ":" + second
- will match C{"1:1"}, but not C{"1:2"}. Because this matches a
- previous literal, will also match the leading C{"1:1"} in C{"1:10"}.
- If this is not desired, use C{matchPreviousExpr}.
- Do I{not} use with packrat parsing enabled.
+
+ will match ``"1:1"``, but not ``"1:2"``. Because this
+ matches a previous literal, will also match the leading
+ ``"1:1"`` in ``"1:10"``. If this is not desired, use
+ :class:`matchPreviousExpr`. Do *not* use with packrat parsing
+ enabled.
"""
rep = Forward()
def copyTokenToRepeater(s,l,t):
@@ -4535,18 +4988,19 @@ def matchPreviousLiteral(expr):
return rep
def matchPreviousExpr(expr):
- """
- Helper to define an expression that is indirectly defined from
- the tokens matched in a previous expression, that is, it looks
- for a 'repeat' of a previous expression. For example::
+ """Helper to define an expression that is indirectly defined from
+ the tokens matched in a previous expression, that is, it looks for
+ a 'repeat' of a previous expression. For example::
+
first = Word(nums)
second = matchPreviousExpr(first)
matchExpr = first + ":" + second
- will match C{"1:1"}, but not C{"1:2"}. Because this matches by
- expressions, will I{not} match the leading C{"1:1"} in C{"1:10"};
- the expressions are evaluated first, and then compared, so
- C{"1"} is compared with C{"10"}.
- Do I{not} use with packrat parsing enabled.
+
+ will match ``"1:1"``, but not ``"1:2"``. Because this
+ matches by expressions, will *not* match the leading ``"1:1"``
+ in ``"1:10"``; the expressions are evaluated first, and then
+ compared, so ``"1"`` is compared with ``"10"``. Do *not* use
+ with packrat parsing enabled.
"""
rep = Forward()
e2 = expr.copy()
@@ -4571,26 +5025,33 @@ def _escapeRegexRangeChars(s):
return _ustr(s)
def oneOf( strs, caseless=False, useRegex=True ):
- """
- Helper to quickly define a set of alternative Literals, and makes sure to do
- longest-first testing when there is a conflict, regardless of the input order,
- but returns a C{L{MatchFirst}} for best performance.
+ """Helper to quickly define a set of alternative Literals, and makes
+ sure to do longest-first testing when there is a conflict,
+ regardless of the input order, but returns
+ a :class:`MatchFirst` for best performance.
Parameters:
- - strs - a string of space-delimited literals, or a collection of string literals
- - caseless - (default=C{False}) - treat all literals as caseless
- - useRegex - (default=C{True}) - as an optimization, will generate a Regex
- object; otherwise, will generate a C{MatchFirst} object (if C{caseless=True}, or
- if creating a C{Regex} raises an exception)
+
+ - strs - a string of space-delimited literals, or a collection of
+ string literals
+ - caseless - (default= ``False``) - treat all literals as
+ caseless
+ - useRegex - (default= ``True``) - as an optimization, will
+ generate a Regex object; otherwise, will generate
+ a :class:`MatchFirst` object (if ``caseless=True``, or if
+ creating a :class:`Regex` raises an exception)
Example::
+
comp_oper = oneOf("< = > <= >= !=")
var = Word(alphas)
number = Word(nums)
term = var | number
comparison_expr = term + comp_oper + term
print(comparison_expr.searchString("B = 12 AA=23 B<=AA AA>12"))
+
prints::
+
[['B', '=', '12'], ['AA', '=', '23'], ['B', '<=', 'AA'], ['AA', '>', '12']]
"""
if caseless:
@@ -4644,19 +5105,21 @@ def oneOf( strs, caseless=False, useRegex=True ):
return MatchFirst(parseElementClass(sym) for sym in symbols).setName(' | '.join(symbols))
def dictOf( key, value ):
- """
- Helper to easily and clearly define a dictionary by specifying the respective patterns
- for the key and value. Takes care of defining the C{L{Dict}}, C{L{ZeroOrMore}}, and C{L{Group}} tokens
- in the proper order. The key pattern can include delimiting markers or punctuation,
- as long as they are suppressed, thereby leaving the significant key text. The value
- pattern can include named results, so that the C{Dict} results can include named token
- fields.
+ """Helper to easily and clearly define a dictionary by specifying
+ the respective patterns for the key and value. Takes care of
+ defining the :class:`Dict`, :class:`ZeroOrMore`, and
+ :class:`Group` tokens in the proper order. The key pattern
+ can include delimiting markers or punctuation, as long as they are
+ suppressed, thereby leaving the significant key text. The value
+ pattern can include named results, so that the :class:`Dict` results
+ can include named token fields.
Example::
+
text = "shape: SQUARE posn: upper left color: light blue texture: burlap"
attr_expr = (label + Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join))
print(OneOrMore(attr_expr).parseString(text).dump())
-
+
attr_label = label
attr_value = Suppress(':') + OneOrMore(data_word, stopOn=label).setParseAction(' '.join)
@@ -4666,7 +5129,9 @@ def dictOf( key, value ):
print(result['shape'])
print(result.shape) # object attribute access works too
print(result.asDict())
+
prints::
+
[['shape', 'SQUARE'], ['posn', 'upper left'], ['color', 'light blue'], ['texture', 'burlap']]
- color: light blue
- posn: upper left
@@ -4676,29 +5141,34 @@ def dictOf( key, value ):
SQUARE
{'color': 'light blue', 'shape': 'SQUARE', 'posn': 'upper left', 'texture': 'burlap'}
"""
- return Dict( ZeroOrMore( Group ( key + value ) ) )
+ return Dict(OneOrMore(Group(key + value)))
def originalTextFor(expr, asString=True):
- """
- Helper to return the original, untokenized text for a given expression. Useful to
- restore the parsed fields of an HTML start tag into the raw tag text itself, or to
- revert separate tokens with intervening whitespace back to the original matching
- input text. By default, returns astring containing the original parsed text.
-
- If the optional C{asString} argument is passed as C{False}, then the return value is a
- C{L{ParseResults}} containing any results names that were originally matched, and a
- single token containing the original matched text from the input string. So if
- the expression passed to C{L{originalTextFor}} contains expressions with defined
- results names, you must set C{asString} to C{False} if you want to preserve those
- results name values.
+ """Helper to return the original, untokenized text for a given
+ expression. Useful to restore the parsed fields of an HTML start
+ tag into the raw tag text itself, or to revert separate tokens with
+ intervening whitespace back to the original matching input text. By
+ default, returns astring containing the original parsed text.
+
+ If the optional ``asString`` argument is passed as
+ ``False``, then the return value is
+ a :class:`ParseResults` containing any results names that
+ were originally matched, and a single token containing the original
+ matched text from the input string. So if the expression passed to
+ :class:`originalTextFor` contains expressions with defined
+ results names, you must set ``asString`` to ``False`` if you
+ want to preserve those results name values.
Example::
+
src = "this is test bold text normal text "
for tag in ("b","i"):
opener,closer = makeHTMLTags(tag)
patt = originalTextFor(opener + SkipTo(closer) + closer)
print(patt.searchString(src)[0])
+
prints::
+
[' bold text']
['text']
"""
@@ -4715,29 +5185,33 @@ def originalTextFor(expr, asString=True):
matchExpr.ignoreExprs = expr.ignoreExprs
return matchExpr
-def ungroup(expr):
- """
- Helper to undo pyparsing's default grouping of And expressions, even
- if all but one are non-empty.
+def ungroup(expr):
+ """Helper to undo pyparsing's default grouping of And expressions,
+ even if all but one are non-empty.
"""
return TokenConverter(expr).setParseAction(lambda t:t[0])
def locatedExpr(expr):
- """
- Helper to decorate a returned token with its starting and ending locations in the input string.
+ """Helper to decorate a returned token with its starting and ending
+ locations in the input string.
+
This helper adds the following results names:
+
- locn_start = location where matched expression begins
- locn_end = location where matched expression ends
- value = the actual parsed results
- Be careful if the input text contains C{} characters, you may want to call
- C{L{ParserElement.parseWithTabs}}
+ Be careful if the input text contains ```` characters, you
+ may want to call :class:`ParserElement.parseWithTabs`
Example::
+
wd = Word(alphas)
for match in locatedExpr(wd).searchString("ljsdf123lksdjjf123lkkjj1222"):
print(match)
+
prints::
+
[[0, 'ljsdf', 5]]
[[8, 'lksdjjf', 15]]
[[18, 'lkkjj', 23]]
@@ -4761,22 +5235,30 @@ _charRange = Group(_singleChar + Suppress("-") + _singleChar)
_reBracketExpr = Literal("[") + Optional("^").setResultsName("negate") + Group( OneOrMore( _charRange | _singleChar ) ).setResultsName("body") + "]"
def srange(s):
- r"""
- Helper to easily define string ranges for use in Word construction. Borrows
- syntax from regexp '[]' string range definitions::
+ r"""Helper to easily define string ranges for use in Word
+ construction. Borrows syntax from regexp '[]' string range
+ definitions::
+
srange("[0-9]") -> "0123456789"
srange("[a-z]") -> "abcdefghijklmnopqrstuvwxyz"
srange("[a-z$_]") -> "abcdefghijklmnopqrstuvwxyz$_"
- The input string must be enclosed in []'s, and the returned string is the expanded
- character set joined into a single string.
- The values enclosed in the []'s may be:
+
+ The input string must be enclosed in []'s, and the returned string
+ is the expanded character set joined into a single string. The
+ values enclosed in the []'s may be:
+
- a single character
- - an escaped character with a leading backslash (such as C{\-} or C{\]})
- - an escaped hex character with a leading C{'\x'} (C{\x21}, which is a C{'!'} character)
- (C{\0x##} is also supported for backwards compatibility)
- - an escaped octal character with a leading C{'\0'} (C{\041}, which is a C{'!'} character)
- - a range of any of the above, separated by a dash (C{'a-z'}, etc.)
- - any combination of the above (C{'aeiouy'}, C{'a-zA-Z0-9_$'}, etc.)
+ - an escaped character with a leading backslash (such as ``\-``
+ or ``\]``)
+ - an escaped hex character with a leading ``'\x'``
+ (``\x21``, which is a ``'!'`` character) (``\0x##``
+ is also supported for backwards compatibility)
+ - an escaped octal character with a leading ``'\0'``
+ (``\041``, which is a ``'!'`` character)
+ - a range of any of the above, separated by a dash (``'a-z'``,
+ etc.)
+ - any combination of the above (``'aeiouy'``,
+ ``'a-zA-Z0-9_$'``, etc.)
"""
_expanded = lambda p: p if not isinstance(p,ParseResults) else ''.join(unichr(c) for c in range(ord(p[0]),ord(p[1])+1))
try:
@@ -4785,9 +5267,8 @@ def srange(s):
return ""
def matchOnlyAtCol(n):
- """
- Helper method for defining parse actions that require matching at a specific
- column in the input text.
+ """Helper method for defining parse actions that require matching at
+ a specific column in the input text.
"""
def verifyCol(strg,locn,toks):
if col(locn,strg) != n:
@@ -4795,24 +5276,26 @@ def matchOnlyAtCol(n):
return verifyCol
def replaceWith(replStr):
- """
- Helper method for common parse actions that simply return a literal value. Especially
- useful when used with C{L{transformString}()}.
+ """Helper method for common parse actions that simply return
+ a literal value. Especially useful when used with
+ :class:`transformString` ().
Example::
+
num = Word(nums).setParseAction(lambda toks: int(toks[0]))
na = oneOf("N/A NA").setParseAction(replaceWith(math.nan))
term = na | num
-
+
OneOrMore(term).parseString("324 234 N/A 234") # -> [324, 234, nan, 234]
"""
return lambda s,l,t: [replStr]
def removeQuotes(s,l,t):
- """
- Helper parse action for removing quotation marks from parsed quoted strings.
+ """Helper parse action for removing quotation marks from parsed
+ quoted strings.
Example::
+
# by default, quotation marks are included in parsed results
quotedString.parseString("'Now is the Winter of our Discontent'") # -> ["'Now is the Winter of our Discontent'"]
@@ -4823,18 +5306,20 @@ def removeQuotes(s,l,t):
return t[0][1:-1]
def tokenMap(func, *args):
- """
- Helper to define a parse action by mapping a function to all elements of a ParseResults list.If any additional
- args are passed, they are forwarded to the given function as additional arguments after
- the token, as in C{hex_integer = Word(hexnums).setParseAction(tokenMap(int, 16))}, which will convert the
- parsed data to an integer using base 16.
+ """Helper to define a parse action by mapping a function to all
+ elements of a ParseResults list. If any additional args are passed,
+ they are forwarded to the given function as additional arguments
+ after the token, as in
+ ``hex_integer = Word(hexnums).setParseAction(tokenMap(int, 16))``,
+ which will convert the parsed data to an integer using base 16.
+
+ Example (compare the last to example in :class:`ParserElement.transformString`::
- Example (compare the last to example in L{ParserElement.transformString}::
hex_ints = OneOrMore(Word(hexnums)).setParseAction(tokenMap(int, 16))
hex_ints.runTests('''
00 11 22 aa FF 0a 0d 1a
''')
-
+
upperword = Word(alphas).setParseAction(tokenMap(str.upper))
OneOrMore(upperword).runTests('''
my kingdom for a horse
@@ -4844,7 +5329,9 @@ def tokenMap(func, *args):
OneOrMore(wd).setParseAction(' '.join).runTests('''
now is the winter of our discontent made glorious summer by this sun of york
''')
+
prints::
+
00 11 22 aa FF 0a 0d 1a
[0, 17, 34, 170, 255, 10, 13, 26]
@@ -4858,7 +5345,7 @@ def tokenMap(func, *args):
return [func(tokn, *args) for tokn in t]
try:
- func_name = getattr(func, '__name__',
+ func_name = getattr(func, '__name__',
getattr(func, '__class__').__name__)
except Exception:
func_name = str(func)
@@ -4867,11 +5354,13 @@ def tokenMap(func, *args):
return pa
upcaseTokens = tokenMap(lambda t: _ustr(t).upper())
-"""(Deprecated) Helper parse action to convert tokens to upper case. Deprecated in favor of L{pyparsing_common.upcaseTokens}"""
+"""(Deprecated) Helper parse action to convert tokens to upper case.
+Deprecated in favor of :class:`pyparsing_common.upcaseTokens`"""
downcaseTokens = tokenMap(lambda t: _ustr(t).lower())
-"""(Deprecated) Helper parse action to convert tokens to lower case. Deprecated in favor of L{pyparsing_common.downcaseTokens}"""
-
+"""(Deprecated) Helper parse action to convert tokens to lower case.
+Deprecated in favor of :class:`pyparsing_common.downcaseTokens`"""
+
def _makeTags(tagStr, xml):
"""Internal helper to construct opening and closing tag expressions, given a tag name"""
if isinstance(tagStr,basestring):
@@ -4902,55 +5391,63 @@ def _makeTags(tagStr, xml):
return openTag, closeTag
def makeHTMLTags(tagStr):
- """
- Helper to construct opening and closing tag expressions for HTML, given a tag name. Matches
- tags in either upper or lower case, attributes with namespaces and with quoted or unquoted values.
+ """Helper to construct opening and closing tag expressions for HTML,
+ given a tag name. Matches tags in either upper or lower case,
+ attributes with namespaces and with quoted or unquoted values.
Example::
- text = '
}.
+ """Helper to create a validating parse action to be used with start
+ tags created with :class:`makeXMLTags` or
+ :class:`makeHTMLTags`. Use ``withAttribute`` to qualify
+ a starting tag with a required attribute value, to avoid false
+ matches on common tags such as ``
`` or ``
``.
- Call C{withAttribute} with a series of attribute names and values. Specify the list
- of filter attributes names and values as:
- - keyword arguments, as in C{(align="right")}, or
- - as an explicit dict with C{**} operator, when an attribute name is also a Python
- reserved word, as in C{**{"class":"Customer", "align":"right"}}
- - a list of name-value tuples, as in ( ("ns1:class", "Customer"), ("ns2:align","right") )
- For attribute names with a namespace prefix, you must use the second form. Attribute
- names are matched insensitive to upper/lower case.
-
- If just testing for C{class} (with or without a namespace), use C{L{withClass}}.
+ Call ``withAttribute`` with a series of attribute names and
+ values. Specify the list of filter attributes names and values as:
- To verify that the attribute exists, but without specifying a value, pass
- C{withAttribute.ANY_VALUE} as the value.
+ - keyword arguments, as in ``(align="right")``, or
+ - as an explicit dict with ``**`` operator, when an attribute
+ name is also a Python reserved word, as in ``**{"class":"Customer", "align":"right"}``
+ - a list of name-value tuples, as in ``(("ns1:class", "Customer"), ("ns2:align","right"))``
+
+ For attribute names with a namespace prefix, you must use the second
+ form. Attribute names are matched insensitive to upper/lower case.
+
+ If just testing for ``class`` (with or without a namespace), use
+ :class:`withClass`.
+
+ To verify that the attribute exists, but without specifying a value,
+ pass ``withAttribute.ANY_VALUE`` as the value.
Example::
+
html = '''
Some text
@@ -4958,7 +5455,7 @@ def withAttribute(*args,**attrDict):
1,3 2,3 1,1
this has no type
-
+
'''
div,div_end = makeHTMLTags("div")
@@ -4967,13 +5464,15 @@ def withAttribute(*args,**attrDict):
grid_expr = div_grid + SkipTo(div | div_end)("body")
for grid_header in grid_expr.searchString(html):
print(grid_header.body)
-
+
# construct a match with any div tag having a type attribute, regardless of the value
div_any_type = div().setParseAction(withAttribute(type=withAttribute.ANY_VALUE))
div_expr = div_any_type + SkipTo(div | div_end)("body")
for div_header in div_expr.searchString(html):
print(div_header.body)
+
prints::
+
1 4 0 1 0
1 4 0 1 0
@@ -4995,11 +5494,12 @@ def withAttribute(*args,**attrDict):
withAttribute.ANY_VALUE = object()
def withClass(classname, namespace=''):
- """
- Simplified version of C{L{withAttribute}} when matching on a div class - made
- difficult because C{class} is a reserved word in Python.
+ """Simplified version of :class:`withAttribute` when
+ matching on a div class - made difficult because ``class`` is
+ a reserved word in Python.
Example::
+
html = '''
Some text
@@ -5007,84 +5507,96 @@ def withClass(classname, namespace=''):
1,3 2,3 1,1
this <div> has no class
-
+
'''
div,div_end = makeHTMLTags("div")
div_grid = div().setParseAction(withClass("grid"))
-
+
grid_expr = div_grid + SkipTo(div | div_end)("body")
for grid_header in grid_expr.searchString(html):
print(grid_header.body)
-
+
div_any_type = div().setParseAction(withClass(withAttribute.ANY_VALUE))
div_expr = div_any_type + SkipTo(div | div_end)("body")
for div_header in div_expr.searchString(html):
print(div_header.body)
+
prints::
+
1 4 0 1 0
1 4 0 1 0
1,3 2,3 1,1
"""
classattr = "%s:class" % namespace if namespace else "class"
- return withAttribute(**{classattr : classname})
+ return withAttribute(**{classattr : classname})
-opAssoc = _Constants()
+opAssoc = SimpleNamespace()
opAssoc.LEFT = object()
opAssoc.RIGHT = object()
def infixNotation( baseExpr, opList, lpar=Suppress('('), rpar=Suppress(')') ):
- """
- Helper method for constructing grammars of expressions made up of
- operators working in a precedence hierarchy. Operators may be unary or
- binary, left- or right-associative. Parse actions can also be attached
- to operator expressions. The generated parser will also recognize the use
- of parentheses to override operator precedences (see example below).
-
- Note: if you define a deep operator list, you may see performance issues
- when using infixNotation. See L{ParserElement.enablePackrat} for a
- mechanism to potentially improve your parser performance.
+ """Helper method for constructing grammars of expressions made up of
+ operators working in a precedence hierarchy. Operators may be unary
+ or binary, left- or right-associative. Parse actions can also be
+ attached to operator expressions. The generated parser will also
+ recognize the use of parentheses to override operator precedences
+ (see example below).
+
+ Note: if you define a deep operator list, you may see performance
+ issues when using infixNotation. See
+ :class:`ParserElement.enablePackrat` for a mechanism to potentially
+ improve your parser performance.
Parameters:
- - baseExpr - expression representing the most basic element for the nested
- - opList - list of tuples, one for each operator precedence level in the
- expression grammar; each tuple is of the form
- (opExpr, numTerms, rightLeftAssoc, parseAction), where:
- - opExpr is the pyparsing expression for the operator;
- may also be a string, which will be converted to a Literal;
- if numTerms is 3, opExpr is a tuple of two expressions, for the
- two operators separating the 3 terms
- - numTerms is the number of terms for this operator (must
- be 1, 2, or 3)
- - rightLeftAssoc is the indicator whether the operator is
- right or left associative, using the pyparsing-defined
- constants C{opAssoc.RIGHT} and C{opAssoc.LEFT}.
+ - baseExpr - expression representing the most basic element for the
+ nested
+ - opList - list of tuples, one for each operator precedence level
+ in the expression grammar; each tuple is of the form ``(opExpr,
+ numTerms, rightLeftAssoc, parseAction)``, where:
+
+ - opExpr is the pyparsing expression for the operator; may also
+ be a string, which will be converted to a Literal; if numTerms
+ is 3, opExpr is a tuple of two expressions, for the two
+ operators separating the 3 terms
+ - numTerms is the number of terms for this operator (must be 1,
+ 2, or 3)
+ - rightLeftAssoc is the indicator whether the operator is right
+ or left associative, using the pyparsing-defined constants
+ ``opAssoc.RIGHT`` and ``opAssoc.LEFT``.
- parseAction is the parse action to be associated with
- expressions matching this operator expression (the
- parse action tuple member may be omitted); if the parse action
- is passed a tuple or list of functions, this is equivalent to
- calling C{setParseAction(*fn)} (L{ParserElement.setParseAction})
- - lpar - expression for matching left-parentheses (default=C{Suppress('(')})
- - rpar - expression for matching right-parentheses (default=C{Suppress(')')})
+ expressions matching this operator expression (the parse action
+ tuple member may be omitted); if the parse action is passed
+ a tuple or list of functions, this is equivalent to calling
+ ``setParseAction(*fn)``
+ (:class:`ParserElement.setParseAction`)
+ - lpar - expression for matching left-parentheses
+ (default= ``Suppress('(')``)
+ - rpar - expression for matching right-parentheses
+ (default= ``Suppress(')')``)
Example::
- # simple example of four-function arithmetic with ints and variable names
+
+ # simple example of four-function arithmetic with ints and
+ # variable names
integer = pyparsing_common.signed_integer
- varname = pyparsing_common.identifier
-
+ varname = pyparsing_common.identifier
+
arith_expr = infixNotation(integer | varname,
[
('-', 1, opAssoc.RIGHT),
(oneOf('* /'), 2, opAssoc.LEFT),
(oneOf('+ -'), 2, opAssoc.LEFT),
])
-
+
arith_expr.runTests('''
5+3*6
(5+3)*6
-2--11
''', fullDump=False)
+
prints::
+
5+3*6
[[5, '+', [3, '*', 6]]]
@@ -5094,6 +5606,12 @@ def infixNotation( baseExpr, opList, lpar=Suppress('('), rpar=Suppress(')') ):
-2--11
[[['-', 2], '-', ['-', 11]]]
"""
+ # captive version of FollowedBy that does not do parse actions or capture results names
+ class _FB(FollowedBy):
+ def parseImpl(self, instring, loc, doActions=True):
+ self.expr.tryParse(instring, loc)
+ return loc, []
+
ret = Forward()
lastExpr = baseExpr | ( lpar + ret + rpar )
for i,operDef in enumerate(opList):
@@ -5101,19 +5619,20 @@ def infixNotation( baseExpr, opList, lpar=Suppress('('), rpar=Suppress(')') ):
termName = "%s term" % opExpr if arity < 3 else "%s%s term" % opExpr
if arity == 3:
if opExpr is None or len(opExpr) != 2:
- raise ValueError("if numterms=3, opExpr must be a tuple or list of two expressions")
+ raise ValueError(
+ "if numterms=3, opExpr must be a tuple or list of two expressions")
opExpr1, opExpr2 = opExpr
thisExpr = Forward().setName(termName)
if rightLeftAssoc == opAssoc.LEFT:
if arity == 1:
- matchExpr = FollowedBy(lastExpr + opExpr) + Group( lastExpr + OneOrMore( opExpr ) )
+ matchExpr = _FB(lastExpr + opExpr) + Group( lastExpr + OneOrMore( opExpr ) )
elif arity == 2:
if opExpr is not None:
- matchExpr = FollowedBy(lastExpr + opExpr + lastExpr) + Group( lastExpr + OneOrMore( opExpr + lastExpr ) )
+ matchExpr = _FB(lastExpr + opExpr + lastExpr) + Group( lastExpr + OneOrMore( opExpr + lastExpr ) )
else:
- matchExpr = FollowedBy(lastExpr+lastExpr) + Group( lastExpr + OneOrMore(lastExpr) )
+ matchExpr = _FB(lastExpr+lastExpr) + Group( lastExpr + OneOrMore(lastExpr) )
elif arity == 3:
- matchExpr = FollowedBy(lastExpr + opExpr1 + lastExpr + opExpr2 + lastExpr) + \
+ matchExpr = _FB(lastExpr + opExpr1 + lastExpr + opExpr2 + lastExpr) + \
Group( lastExpr + opExpr1 + lastExpr + opExpr2 + lastExpr )
else:
raise ValueError("operator must be unary (1), binary (2), or ternary (3)")
@@ -5122,14 +5641,14 @@ def infixNotation( baseExpr, opList, lpar=Suppress('('), rpar=Suppress(')') ):
# try to avoid LR with this extra test
if not isinstance(opExpr, Optional):
opExpr = Optional(opExpr)
- matchExpr = FollowedBy(opExpr.expr + thisExpr) + Group( opExpr + thisExpr )
+ matchExpr = _FB(opExpr.expr + thisExpr) + Group( opExpr + thisExpr )
elif arity == 2:
if opExpr is not None:
- matchExpr = FollowedBy(lastExpr + opExpr + thisExpr) + Group( lastExpr + OneOrMore( opExpr + thisExpr ) )
+ matchExpr = _FB(lastExpr + opExpr + thisExpr) + Group( lastExpr + OneOrMore( opExpr + thisExpr ) )
else:
- matchExpr = FollowedBy(lastExpr + thisExpr) + Group( lastExpr + OneOrMore( thisExpr ) )
+ matchExpr = _FB(lastExpr + thisExpr) + Group( lastExpr + OneOrMore( thisExpr ) )
elif arity == 3:
- matchExpr = FollowedBy(lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr) + \
+ matchExpr = _FB(lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr) + \
Group( lastExpr + opExpr1 + thisExpr + opExpr2 + thisExpr )
else:
raise ValueError("operator must be unary (1), binary (2), or ternary (3)")
@@ -5146,7 +5665,8 @@ def infixNotation( baseExpr, opList, lpar=Suppress('('), rpar=Suppress(')') ):
return ret
operatorPrecedence = infixNotation
-"""(Deprecated) Former name of C{L{infixNotation}}, will be dropped in a future release."""
+"""(Deprecated) Former name of :class:`infixNotation`, will be
+dropped in a future release."""
dblQuotedString = Combine(Regex(r'"(?:[^"\n\r\\]|(?:"")|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*')+'"').setName("string enclosed in double quotes")
sglQuotedString = Combine(Regex(r"'(?:[^'\n\r\\]|(?:'')|(?:\\(?:[^x]|x[0-9a-fA-F]+)))*")+"'").setName("string enclosed in single quotes")
@@ -5155,28 +5675,33 @@ quotedString = Combine(Regex(r'"(?:[^"\n\r\\]|(?:"")|(?:\\(?:[^x]|x[0-9a-fA-F]+)
unicodeString = Combine(_L('u') + quotedString.copy()).setName("unicode string literal")
def nestedExpr(opener="(", closer=")", content=None, ignoreExpr=quotedString.copy()):
- """
- Helper method for defining nested lists enclosed in opening and closing
- delimiters ("(" and ")" are the default).
+ """Helper method for defining nested lists enclosed in opening and
+ closing delimiters ("(" and ")" are the default).
Parameters:
- - opener - opening character for a nested list (default=C{"("}); can also be a pyparsing expression
- - closer - closing character for a nested list (default=C{")"}); can also be a pyparsing expression
- - content - expression for items within the nested lists (default=C{None})
- - ignoreExpr - expression for ignoring opening and closing delimiters (default=C{quotedString})
+ - opener - opening character for a nested list
+ (default= ``"("``); can also be a pyparsing expression
+ - closer - closing character for a nested list
+ (default= ``")"``); can also be a pyparsing expression
+ - content - expression for items within the nested lists
+ (default= ``None``)
+ - ignoreExpr - expression for ignoring opening and closing
+ delimiters (default= :class:`quotedString`)
- If an expression is not provided for the content argument, the nested
- expression will capture all whitespace-delimited content between delimiters
- as a list of separate values.
+ If an expression is not provided for the content argument, the
+ nested expression will capture all whitespace-delimited content
+ between delimiters as a list of separate values.
- Use the C{ignoreExpr} argument to define expressions that may contain
- opening or closing characters that should not be treated as opening
- or closing characters for nesting, such as quotedString or a comment
- expression. Specify multiple expressions using an C{L{Or}} or C{L{MatchFirst}}.
- The default is L{quotedString}, but if no expressions are to be ignored,
- then pass C{None} for this argument.
+ Use the ``ignoreExpr`` argument to define expressions that may
+ contain opening or closing characters that should not be treated as
+ opening or closing characters for nesting, such as quotedString or
+ a comment expression. Specify multiple expressions using an
+ :class:`Or` or :class:`MatchFirst`. The default is
+ :class:`quotedString`, but if no expressions are to be ignored, then
+ pass ``None`` for this argument.
Example::
+
data_type = oneOf("void int short long char float double")
decl_data_type = Combine(data_type + Optional(Word('*')))
ident = Word(alphas+'_', alphanums+'_')
@@ -5186,29 +5711,31 @@ def nestedExpr(opener="(", closer=")", content=None, ignoreExpr=quotedString.cop
code_body = nestedExpr('{', '}', ignoreExpr=(quotedString | cStyleComment))
- c_function = (decl_data_type("type")
+ c_function = (decl_data_type("type")
+ ident("name")
- + LPAR + Optional(delimitedList(arg), [])("args") + RPAR
+ + LPAR + Optional(delimitedList(arg), [])("args") + RPAR
+ code_body("body"))
c_function.ignore(cStyleComment)
-
+
source_code = '''
- int is_odd(int x) {
- return (x%2);
+ int is_odd(int x) {
+ return (x%2);
}
-
- int dec_to_hex(char hchar) {
- if (hchar >= '0' && hchar <= '9') {
- return (ord(hchar)-ord('0'));
- } else {
+
+ int dec_to_hex(char hchar) {
+ if (hchar >= '0' && hchar <= '9') {
+ return (ord(hchar)-ord('0'));
+ } else {
return (10+ord(hchar)-ord('A'));
- }
+ }
}
'''
for func in c_function.searchString(source_code):
print("%(name)s (%(type)s) args: %(args)s" % func)
+
prints::
+
is_odd (int) args: [['int', 'x']]
dec_to_hex (int) args: [['char', 'hchar']]
"""
@@ -5226,7 +5753,7 @@ def nestedExpr(opener="(", closer=")", content=None, ignoreExpr=quotedString.cop
).setParseAction(lambda t:t[0].strip()))
else:
if ignoreExpr is not None:
- content = (Combine(OneOrMore(~ignoreExpr +
+ content = (Combine(OneOrMore(~ignoreExpr +
~Literal(opener) + ~Literal(closer) +
CharsNotIn(ParserElement.DEFAULT_WHITE_CHARS,exact=1))
).setParseAction(lambda t:t[0].strip()))
@@ -5245,23 +5772,24 @@ def nestedExpr(opener="(", closer=")", content=None, ignoreExpr=quotedString.cop
return ret
def indentedBlock(blockStatementExpr, indentStack, indent=True):
- """
- Helper method for defining space-delimited indentation blocks, such as
- those used to define block statements in Python source code.
+ """Helper method for defining space-delimited indentation blocks,
+ such as those used to define block statements in Python source code.
Parameters:
- - blockStatementExpr - expression defining syntax of statement that
- is repeated within the indented block
- - indentStack - list created by caller to manage indentation stack
- (multiple statementWithIndentedBlock expressions within a single grammar
- should share a common indentStack)
- - indent - boolean indicating whether block must be indented beyond the
- the current level; set to False for block of left-most statements
- (default=C{True})
- A valid block must contain at least one C{blockStatement}.
+ - blockStatementExpr - expression defining syntax of statement that
+ is repeated within the indented block
+ - indentStack - list created by caller to manage indentation stack
+ (multiple statementWithIndentedBlock expressions within a single
+ grammar should share a common indentStack)
+ - indent - boolean indicating whether block must be indented beyond
+ the the current level; set to False for block of left-most
+ statements (default= ``True``)
+
+ A valid block must contain at least one ``blockStatement``.
Example::
+
data = '''
def A(z):
A1
@@ -5302,7 +5830,9 @@ def indentedBlock(blockStatementExpr, indentStack, indent=True):
parseTree = module_body.parseString(data)
parseTree.pprint()
+
prints::
+
[['def',
'A',
['(', 'z', ')'],
@@ -5320,7 +5850,7 @@ def indentedBlock(blockStatementExpr, indentStack, indent=True):
'spam',
['(', 'x', 'y', ')'],
':',
- [[['def', 'eggs', ['(', 'z', ')'], ':', [['pass']]]]]]]
+ [[['def', 'eggs', ['(', 'z', ')'], ':', [['pass']]]]]]]
"""
def checkPeerIndent(s,l,t):
if l >= len(s): return
@@ -5370,51 +5900,61 @@ def replaceHTMLEntity(t):
# it's easy to get these comment structures wrong - they're very common, so may as well make them available
cStyleComment = Combine(Regex(r"/\*(?:[^*]|\*(?!/))*") + '*/').setName("C style comment")
-"Comment of the form C{/* ... */}"
+"Comment of the form ``/* ... */``"
htmlComment = Regex(r"").setName("HTML comment")
-"Comment of the form C{}"
+"Comment of the form ````"
restOfLine = Regex(r".*").leaveWhitespace().setName("rest of line")
dblSlashComment = Regex(r"//(?:\\\n|[^\n])*").setName("// comment")
-"Comment of the form C{// ... (to end of line)}"
+"Comment of the form ``// ... (to end of line)``"
cppStyleComment = Combine(Regex(r"/\*(?:[^*]|\*(?!/))*") + '*/'| dblSlashComment).setName("C++ style comment")
-"Comment of either form C{L{cStyleComment}} or C{L{dblSlashComment}}"
+"Comment of either form :class:`cStyleComment` or :class:`dblSlashComment`"
javaStyleComment = cppStyleComment
-"Same as C{L{cppStyleComment}}"
+"Same as :class:`cppStyleComment`"
pythonStyleComment = Regex(r"#.*").setName("Python style comment")
-"Comment of the form C{# ... (to end of line)}"
+"Comment of the form ``# ... (to end of line)``"
_commasepitem = Combine(OneOrMore(Word(printables, excludeChars=',') +
Optional( Word(" \t") +
~Literal(",") + ~LineEnd() ) ) ).streamline().setName("commaItem")
commaSeparatedList = delimitedList( Optional( quotedString.copy() | _commasepitem, default="") ).setName("commaSeparatedList")
-"""(Deprecated) Predefined expression of 1 or more printable words or quoted strings, separated by commas.
- This expression is deprecated in favor of L{pyparsing_common.comma_separated_list}."""
+"""(Deprecated) Predefined expression of 1 or more printable words or
+quoted strings, separated by commas.
+
+This expression is deprecated in favor of :class:`pyparsing_common.comma_separated_list`.
+"""
# some other useful expressions - using lower-case class name since we are really using this as a namespace
class pyparsing_common:
- """
- Here are some common low-level expressions that may be useful in jump-starting parser development:
- - numeric forms (L{integers}, L{reals}, L{scientific notation})
- - common L{programming identifiers}
- - network addresses (L{MAC}, L{IPv4}, L{IPv6})
- - ISO8601 L{dates} and L{datetime}
- - L{UUID}
- - L{comma-separated list}
+ """Here are some common low-level expressions that may be useful in
+ jump-starting parser development:
+
+ - numeric forms (:class:`integers`, :class:`reals`,
+ :class:`scientific notation`)
+ - common :class:`programming identifiers`
+ - network addresses (:class:`MAC`,
+ :class:`IPv4`, :class:`IPv6`)
+ - ISO8601 :class:`dates` and
+ :class:`datetime`
+ - :class:`UUID`
+ - :class:`comma-separated list`
+
Parse actions:
- - C{L{convertToInteger}}
- - C{L{convertToFloat}}
- - C{L{convertToDate}}
- - C{L{convertToDatetime}}
- - C{L{stripHTMLTags}}
- - C{L{upcaseTokens}}
- - C{L{downcaseTokens}}
+
+ - :class:`convertToInteger`
+ - :class:`convertToFloat`
+ - :class:`convertToDate`
+ - :class:`convertToDatetime`
+ - :class:`stripHTMLTags`
+ - :class:`upcaseTokens`
+ - :class:`downcaseTokens`
Example::
+
pyparsing_common.number.runTests('''
# any int or real number, returned as the appropriate type
100
@@ -5461,7 +6001,9 @@ class pyparsing_common:
# uuid
12345678-1234-5678-1234-567812345678
''')
+
prints::
+
# any int or real number, returned as the appropriate type
100
[100]
@@ -5563,7 +6105,8 @@ class pyparsing_common:
"""expression that parses a floating point number and returns a float"""
sci_real = Regex(r'[+-]?\d+([eE][+-]?\d+|\.\d*([eE][+-]?\d+)?)').setName("real number with scientific notation").setParseAction(convertToFloat)
- """expression that parses a floating point number with optional scientific notation and returns a float"""
+ """expression that parses a floating point number with optional
+ scientific notation and returns a float"""
# streamlining this expression makes the docs nicer-looking
number = (sci_real | real | signed_integer).streamline()
@@ -5571,12 +6114,12 @@ class pyparsing_common:
fnumber = Regex(r'[+-]?\d+\.?\d*([eE][+-]?\d+)?').setName("fnumber").setParseAction(convertToFloat)
"""any int or real number, returned as float"""
-
+
identifier = Word(alphas+'_', alphanums+'_').setName("identifier")
"""typical code identifier (leading alpha or '_', followed by 0 or more alphas, nums, or '_')"""
-
+
ipv4_address = Regex(r'(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})(\.(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})){3}').setName("IPv4 address")
- "IPv4 address (C{0.0.0.0 - 255.255.255.255})"
+ "IPv4 address (``0.0.0.0 - 255.255.255.255``)"
_ipv6_part = Regex(r'[0-9a-fA-F]{1,4}').setName("hex_integer")
_full_ipv6_address = (_ipv6_part + (':' + _ipv6_part)*7).setName("full IPv6 address")
@@ -5585,7 +6128,7 @@ class pyparsing_common:
_mixed_ipv6_address = ("::ffff:" + ipv4_address).setName("mixed IPv6 address")
ipv6_address = Combine((_full_ipv6_address | _mixed_ipv6_address | _short_ipv6_address).setName("IPv6 address")).setName("IPv6 address")
"IPv6 address (long, short, or mixed form)"
-
+
mac_address = Regex(r'[0-9a-fA-F]{2}([:.-])[0-9a-fA-F]{2}(?:\1[0-9a-fA-F]{2}){4}').setName("MAC address")
"MAC address xx:xx:xx:xx:xx (may also have '-' or '.' delimiters)"
@@ -5595,13 +6138,16 @@ class pyparsing_common:
Helper to create a parse action for converting parsed date string to Python datetime.date
Params -
- - fmt - format to be passed to datetime.strptime (default=C{"%Y-%m-%d"})
+ - fmt - format to be passed to datetime.strptime (default= ``"%Y-%m-%d"``)
Example::
+
date_expr = pyparsing_common.iso8601_date.copy()
date_expr.setParseAction(pyparsing_common.convertToDate())
print(date_expr.parseString("1999-12-31"))
+
prints::
+
[datetime.date(1999, 12, 31)]
"""
def cvt_fn(s,l,t):
@@ -5613,17 +6159,20 @@ class pyparsing_common:
@staticmethod
def convertToDatetime(fmt="%Y-%m-%dT%H:%M:%S.%f"):
- """
- Helper to create a parse action for converting parsed datetime string to Python datetime.datetime
+ """Helper to create a parse action for converting parsed
+ datetime string to Python datetime.datetime
Params -
- - fmt - format to be passed to datetime.strptime (default=C{"%Y-%m-%dT%H:%M:%S.%f"})
+ - fmt - format to be passed to datetime.strptime (default= ``"%Y-%m-%dT%H:%M:%S.%f"``)
Example::
+
dt_expr = pyparsing_common.iso8601_datetime.copy()
dt_expr.setParseAction(pyparsing_common.convertToDatetime())
print(dt_expr.parseString("1999-12-31T23:59:59.999"))
+
prints::
+
[datetime.datetime(1999, 12, 31, 23, 59, 59, 999000)]
"""
def cvt_fn(s,l,t):
@@ -5634,31 +6183,34 @@ class pyparsing_common:
return cvt_fn
iso8601_date = Regex(r'(?P\d{4})(?:-(?P\d\d)(?:-(?P\d\d))?)?').setName("ISO8601 date")
- "ISO8601 date (C{yyyy-mm-dd})"
+ "ISO8601 date (``yyyy-mm-dd``)"
iso8601_datetime = Regex(r'(?P\d{4})-(?P\d\d)-(?P\d\d)[T ](?P\d\d):(?P\d\d)(:(?P\d\d(\.\d*)?)?)?(?PZ|[+-]\d\d:?\d\d)?').setName("ISO8601 datetime")
- "ISO8601 datetime (C{yyyy-mm-ddThh:mm:ss.s(Z|+-00:00)}) - trailing seconds, milliseconds, and timezone optional; accepts separating C{'T'} or C{' '}"
+ "ISO8601 datetime (``yyyy-mm-ddThh:mm:ss.s(Z|+-00:00)``) - trailing seconds, milliseconds, and timezone optional; accepts separating ``'T'`` or ``' '``"
uuid = Regex(r'[0-9a-fA-F]{8}(-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}').setName("UUID")
- "UUID (C{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx})"
+ "UUID (``xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx``)"
_html_stripper = anyOpenTag.suppress() | anyCloseTag.suppress()
@staticmethod
def stripHTMLTags(s, l, tokens):
- """
- Parse action to remove HTML tags from web page HTML source
+ """Parse action to remove HTML tags from web page HTML source
Example::
- # strip HTML links from normal text
- text = '