Merge branch 'master' into bugfix/1002-allow-global-resolving

HISTORY.txt

@@ -1,5 +1,18 @@
-8.3.3
- - Added enhancement for pip-tools to resolve dependecies with specific versions of python
+9.0.1:
+ - Fixed issue with specifiers being treated as paths on Windows.
+ - Fixed regression causing development packages to always be installed.
+9.0.0: 
+ - Fixed bug where packages beginning with vcs names (e.g. git) weren't installed correctly.
+ - Fixed url parsing for <vcs>+<vcs>:// style urls.
+ - Pipenv can now install relative file paths.
+ - Better messaging around failed installs.
+ - More resilient network io when retrieving data from PyPI.
+ - Fixed bug with bad dependency pinning via pip-tools.
+ - Prompt user to destroy and recreate virtualenvironment if they are in a currently activated environment.
+ - Added enhancement for pip-tools to resolve dependencies with specific versions of python
+ - Fixed bug where newlines were not escaped in .env files when loaded
+ - Sequentially install all local and vcs dependencies to avoid write race conditions.
+ - Fixed accidental exclusion of files from some VCS installs.
 8.3.2:
  - Moved automated update check to once every 24 hours. 
  - Better default for PYENV_ROOT.
@@ -10,14 +23,14 @@
  - Updated vendored delegator.py.         
  - Changed --dev flag for the uninstall command to --all-dev to better represent what it does.
 8.3.0:
- - Add support for installation from remote requiments file.
+ - Add support for installation from remote requirements file.
  - Add --reverse to pipenv graph, displaying inverted dependency graph.
  - VCS dependencies now install sequentially to avoid write lock conflicts.
  - Allow PIPENV_IGNORE_VIRTUALENVS to work with pipenv shell on Windows.
  - Enforce newline termination of Pipfile.
  - More robust requirements.txt conversion experience.
  - Respect allow_prereleases in all locking scenarios.
- - Seperated default and development dependency output when using lock -r and lock -r -d respectively.
+ - Separated default and development dependency output when using lock -r and lock -r -d respectively.
  - Print whole help message with pipenv --help.
 8.2.7:
  - Add update --sequential.
@@ -26,7 +39,7 @@
  - Update vendored setuptools.
  - Improvements to check --unused.
  - Fix install for local sdist packages.
- - Updating the patched pip-tools with the wheel dependency bugifx.
+ - Updating the patched pip-tools with the wheel dependency bugfix.
  - Fix git remote address modified changing underscore to a hyphen.
  - Fix py2toml with dashes (dev-packages)
  - Fix for --dry-run, reporting backwards.
@@ -108,7 +121,7 @@
  - Fix argument parsing.
 7.9.7:
  - Fix help printout screen (and update it).
- - Use urllib3's warning supression directly.
+ - Use urllib3's warning suppression directly.
 7.9.6:
  - Did you mean?
 7.9.5:
@@ -173,17 +186,17 @@
  - Change --two, and --three to use --python 2 and --python 3 under the hood.
  - This restores --two / --three usage on windows.
 7.6.8:
- - `pipenv intall -r requirements.txt --dev` now works.
+ - `pipenv install -r requirements.txt --dev` now works.
 7.6.7:
  - New less-fancy progress bars (for linux users, specifically).
- - Suport --python 3.
+ - Support --python 3.
 7.6.6:
  - Packaging problem.
 7.6.5:
  - Patched vendored 'safety' package to remove yaml dependency — should work on all Pythons now.
 7.6.4:
  - Extensive integration test suite.
- - Don't suggest autocurrections as often.
+ - Don't suggest autocorrections as often.
  - Cleanups.
  - Don't depend on setuptools anymore.
 7.6.3:
@@ -223,13 +236,13 @@
 7.4.4:
  - PIPENV_PIPFILE environment variable support.
  - --site-packages flag, for the crazy at heart.
- - Installation conccurency on Windows.
- - make `graph --json` consistient with `graph`.
+ - Installation concurrency on Windows.
+ - make `graph --json` consistent with `graph`.
  - Much better support for suggesting package names.
  - Updated to pipfile spec 4, support for path= for relative package names.
  - Import sources from requirements files.
  - Cleanup stderr/stdout.
- - 'pipenv check' only reports saftey now for Python 3.
+ - 'pipenv check' only reports safety now for Python 3.
 7.4.3:
  - Download/install things concurrently.
 7.4.2:
@@ -405,7 +418,7 @@
 5.3.3:
  - Automatic notification of version updates.
 5.3.2:
- - Automatic locking afer install/uninstall (because it's fast now!)
+ - Automatic locking after install/uninstall (because it's fast now!)
 5.3.1:
  - Improvements for windows.
 5.3.0:
@@ -510,7 +523,7 @@
  - Removal of $ pipenv install --requirements.
  - Addition of $ pipenv lock -r.
 3.3.2:
- - User-configuraable max-depth for Pipfile searching.
+ - User-configurable max-depth for Pipfile searching.
  - Bugfix.
 3.3.1:
  - Bugfix for install.
@@ -531,11 +544,11 @@
 3.2.10:
  - Fix bugs.
 3.2.9:
- - Remove remporary requirements.txt after installation.
+ - Remove temporary requirements.txt after installation.
  - Add support for --python option, for specifying any version of Python.
  - Read source Pipfile.lock.
 3.2.8:
- - Lock before installing all depdendencies, if lockfile isn't present.
+ - Lock before installing all dependencies, if lockfile isn't present.
 3.2.7:
  - Cache proper names for great speed increases.
 3.2.6:

NOTICES

@@ -1,3 +1,3 @@
-The contents of the vendor and pactched directories are subject to different licenses
+The contents of the vendor and patched directories are subject to different licenses
 than the rest of this project. Their respective licenses can be looked
 up at pypi.python.org.

Pipfile

@@ -4,7 +4,11 @@
 pytest = "*"
 mock = "*"
 sphinx = "<=1.5.5"
-"-e ." = "*"
 twine = "*"
-"sphinx-click" = "*"
-"pytest-xdist" = "*"
+sphinx-click = "*"
+pytest-xdist = "*"
+
+
+[packages]
+
+"e1839a8" = {path = ".", editable = true}

README.rst

@@ -10,8 +10,8 @@ Pipenv: Python Development Workflow for Humans
 .. image:: https://img.shields.io/pypi/pyversions/pipenv.svg
     :target: https://pypi.python.org/pypi/pipenv
 
-.. image:: https://travis-ci.org/kennethreitz/pipenv.svg?branch=master
-    :target: https://travis-ci.org/kennethreitz/pipenv
+.. image:: https://travis-ci.org/pypa/pipenv.svg?branch=master
+    :target: https://travis-ci.org/pypa/pipenv
 
 .. image:: https://img.shields.io/appveyor/ci/kennethreitz/pipenv.svg
     :target: https://ci.appveyor.com/project/kennethreitz/pipenv/branch/master
@@ -21,7 +21,7 @@ Pipenv: Python Development Workflow for Humans
 
 ---------------
 
-**Pipenv** — the officially recommended Python packaging tool from `Python.org <https://packaging.python.org/new-tutorials/installing-and-using-packages/>`_, free (as in freedom).
+**Pipenv** — the officially recommended Python packaging tool from `Python.org <https://packaging.python.org/tutorials/managing-dependencies/#managing-dependencies>`_, free (as in freedom).
 
 Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world. *Windows is a first–class citizen, in our world.*
 
@@ -151,7 +151,7 @@ Fish is the best shell. You should use it.
        Show a graph of your installed dependencies:
        $ pipenv graph
 
-       Check your installed dependencies for security vulnerabilties:
+       Check your installed dependencies for security vulnerabilities:
        $ pipenv check
 
        Install a local setup.py into your virtual environment/Pipfile:

docs/_templates/hacks.html

@@ -1,4 +1,5 @@
-Alabaster (krTheme++) Hacks -->
+<script type="text/javascript">$('#searchbox').hide(0);</script>
+<!--Alabaster (krTheme++) Hacks -->
 
 <!-- CSS Adjustments (I'm very picky.) -->
 <style type="text/css">
@@ -6,8 +7,8 @@ Alabaster (krTheme++) Hacks -->
   /* Rezzy requires precise alignment. */
   img.logo {margin-left: -20px!important;}
 
-  /* "Quick Search" should be capitalized. */
-  div#searchbox h3 {text-transform: capitalize;}
+  /* "Quick Search" should be not be shown for now. */
+  div#searchbox h3 {display: none;}
 
   /* Make the document a little wider, less code is cut-off. */
   div.document {width: 1008px;}
@@ -69,4 +70,4 @@ Alabaster (krTheme++) Hacks -->
 <style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
 
 
-<!-- That was not a hack. That was art.
+<!-- That was not a hack. That was art.

docs/_templates/sidebarintro.html

@@ -17,7 +17,7 @@
 <h3>Stay Informed</h3>
 <p>Receive updates on new releases and upcoming projects.</p>
 
-<p><iframe src="http://ghbtns.com/github-btn.html?user=kennethreitz&type=follow&count=false"
+<p><iframe src="https://ghbtns.com/github-btn.html?user=kennethreitz&type=follow&count=false"
   allowtransparency="true" frameborder="0" scrolling="0" width="200" height="20"></iframe></p>
 
 <p><a href="https://twitter.com/kennethreitz" class="twitter-follow-button" data-show-count="false">Follow @kennethreitz</a> <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script></p>

docs/advanced.rst

@@ -284,7 +284,7 @@ will detect it.
 
     - ``PIPENV_NOSPIN`` — Disable terminal spinner, for cleaner logs. Automatically set in CI environments.
 
-    - ``PIPENV_MAX_DEPTH`` — Set to an integer for the maximum number of directories to resursively
+    - ``PIPENV_MAX_DEPTH`` — Set to an integer for the maximum number of directories to recursively
       search for a Pipfile.
 
     - ``PIPENV_TIMEOUT`` — Set to an integer for the max number of seconds Pipenv will

docs/basics.rst

@@ -270,7 +270,7 @@ $ pipenv uninstall
 //////////////////
 
 ``$ pipenv uninstall`` supports all of the parameters in `pipenv install <#pipenv-install>`_,
-as well as two additonal options, ``--all`` and ``--all-dev``.
+as well as two additional options, ``--all`` and ``--all-dev``.
 
     - ``--all`` — This parameter will purge all files from the virtual environment,
       but leave the Pipfile untouched.

docs/index.rst

@@ -20,7 +20,7 @@ Pipenv: Python Dev Workflow for Humans
 
 ---------------
 
-**Pipenv** — the officially recommended Python packaging tool from `Python.org <https://packaging.python.org/new-tutorials/installing-and-using-packages/>`_, free (as in freedom).
+**Pipenv** — the officially recommended Python packaging tool from `Python.org <https://packaging.python.org/tutorials/managing-dependencies/#managing-dependencies>`_, free (as in freedom).
 
 Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world. *Windows is a first–class citizen, in our world.*
 
@@ -43,6 +43,11 @@ The problems that Pipenv seeks to solve are multi-faceted:
 Install Pipenv Today!
 ---------------------
 
+
+.. note:: The use of **Python 3** is *highly* preferred over Python 2, when installing Pipenv. Compatibility with three virtualenvs is greatly improved when using Python 3 as the installation target.
+
+  —*Kenneth Reitz*
+
 Pipenv is a python package and so can be installed using ``pip`` as you would expect.
 ::
 

pipenv/__version__.py

@@ -3,4 +3,4 @@
 #  //___/ / / / //___/ / //       //   / /  || / /
 # //       / / //       ((____   //   / /   ||/ /
 
-__version__ = '8.3.2'
+__version__ = '9.0.1'

pipenv/cli.py

@@ -33,7 +33,7 @@ from click_didyoumean import DYMCommandCollection
 from .project import Project
 from .utils import (
     convert_deps_from_pip, convert_deps_to_pip, is_required_version,
-    proper_case, pep423_name, split_vcs, resolve_deps, shellquote, is_vcs,
+    proper_case, pep423_name, split_file, merge_deps, resolve_deps, shellquote, is_vcs,
     python_version, suggest_package, find_windows_executable, is_file,
     prepare_pip_source_args, temp_environ, is_valid_url, download_file,
     get_requirement, need_update_check, touch_update_stamp
@@ -123,7 +123,10 @@ project = Project()
 
 def load_dot_env():
     if not PIPENV_DONT_LOAD_ENV:
-        denv = dotenv.find_dotenv(PIPENV_DOTENV_LOCATION or os.sep.join([project.project_directory, '.env']))
+        # If the project doesn't exist yet, check current directory for a .env file
+        project_directory = project.project_directory or '.'
+
+        denv = dotenv.find_dotenv(PIPENV_DOTENV_LOCATION or os.sep.join([project_directory, '.env']))
         if os.path.isfile(denv):
             click.echo(crayons.normal('Loading .env environment variables…', bold=True), err=True)
         dotenv.load_dotenv(denv, override=True)
@@ -220,7 +223,7 @@ def ensure_latest_pip():
 
             windows = '-m' if os.name == 'nt' else ''
 
-            c = delegator.run('"{0}" install {1} pip --upgrade'.format(which_pip()), windows, block=False)
+            c = delegator.run('"{0}" install {1} pip --upgrade'.format(which_pip(), windows), block=False)
             click.echo(crayons.blue(c.out))
     except AttributeError:
         pass
@@ -544,6 +547,9 @@ def ensure_python(three=None, python=None):
 def ensure_virtualenv(three=None, python=None, site_packages=False):
     """Creates a virtualenv, if one doesn't exist."""
 
+    def abort():
+        sys.exit(1)
+
     global USING_DEFAULT_PYTHON
 
     if not project.virtualenv_exists:
@@ -580,6 +586,12 @@ def ensure_virtualenv(three=None, python=None, site_packages=False):
         ensure_python(three=three, python=python)
 
         click.echo(crayons.red('Virtualenv already exists!'), err=True)
+        # If VIRTUAL_ENV is set, there is a possibility that we are
+        # going to remove the active virtualenv that the user cares
+        # about, so confirm first.
+        if 'VIRTUAL_ENV' in os.environ:
+            if not (PIPENV_YES or click.confirm('Remove existing virtualenv?', default=True)):
+                abort()
         click.echo(crayons.normal(u'Removing existing virtualenv…', bold=True), err=True)
 
         # Remove the virtualenv.
@@ -756,10 +768,10 @@ def do_install_dependencies(
     if skip_lock or only or not project.lockfile_exists:
         if not bare:
             click.echo(crayons.normal(u'Installing dependencies from Pipfile…', bold=True))
-            lockfile = split_vcs(project._lockfile)
+            lockfile = split_file(project._lockfile)
     else:
         with open(project.lockfile_location) as f:
-            lockfile = split_vcs(simplejson.load(f))
+            lockfile = split_file(simplejson.load(f))
 
         if not bare:
             click.echo(
@@ -774,42 +786,16 @@ def do_install_dependencies(
     # Allow pip to resolve dependencies when in skip-lock mode.
     no_deps = (not skip_lock)
 
-    deps = {}
-    vcs_deps = {}
-
-    # Store dev only deps for a requirements output
-    dev_deps = {}
-    dev_vcs_deps = {}
-
-    # Add development deps if --dev was passed.
-    if dev:
-        deps.update(lockfile['develop'])
-        vcs_deps.update(lockfile.get('develop-vcs', {}))
-
-        # Add only dev deps if requirements was passed
-        if requirements:
-            dev_deps.update(lockfile['develop'])
-            dev_vcs_deps.update(lockfile.get('develop-vcs', {}))
-
-
-    # Install default dependencies, always.
-    deps.update(lockfile['default'] if not only else {})
-    vcs_deps.update(lockfile.get('default-vcs', {}))
-
-    if ignore_hashes:
-        # Remove hashes from generated requirements.
-        for k, v in deps.items():
-            if 'hash' in v:
-                del v['hash']
-
-    # Convert the deps to pip-compatible arguments.
-    deps_list = [(d, ignore_hashes, blocking) for d in convert_deps_to_pip(deps, project, r=False, include_index=True)]
+    deps_list, dev_deps_list = merge_deps(
+        lockfile,
+        project,
+        dev=dev,
+        requirements=requirements,
+        ignore_hashes=ignore_hashes,
+        blocking=blocking,
+        only=only
+    )
     failed_deps_list = []
-
-    if len(vcs_deps):
-        deps_list.extend((d, True, True) for d in convert_deps_to_pip(vcs_deps, project, r=False))
-
-    # --requirements was passed.
     if requirements:
         # Output only default dependencies
         if not dev:
@@ -818,14 +804,9 @@ def do_install_dependencies(
 
         # Output only dev dependencies
         if dev:
-            dev_deps_list = [(d, ignore_hashes, blocking) for d in convert_deps_to_pip(dev_deps, project, r=False, include_index=True)]
-            if len(dev_vcs_deps):
-                dev_deps_list.extend((d, True, True) for d in convert_deps_to_pip(dev_vcs_deps, project, r=False))
-
             click.echo('\n'.join(d[0] for d in dev_deps_list))
             sys.exit(0)
 
-
     procs = []
 
     deps_list_bar = progress.bar(deps_list, label=INSTALL_LABEL if os.name != 'nt' else '')
@@ -1164,7 +1145,7 @@ def do_lock(verbose=False, system=False, clear=False, pre=False):
     try:
         lockfile['_meta']['host-environment-markers'] = simplejson.loads(c.out)
     except ValueError:
-        click.echo(crayons.red("An unexpected error occured while accessing your virtualenv's python installation!"))
+        click.echo(crayons.red("An unexpected error occurred while accessing your virtualenv's python installation!"))
         click.echo('Please run $ {0} to re-create your environment.'.format(crayons.red('pipenv --rm')))
         sys.exit(1)
 
@@ -1537,7 +1518,7 @@ Usage Examples:
    Show a graph of your installed dependencies:
    $ {4}
 
-   Check your installed dependencies for security vulnerabilties:
+   Check your installed dependencies for security vulnerabilities:
    $ {7}
 
    Install a local setup.py into your virtual environment/Pipfile:
@@ -1818,26 +1799,29 @@ def install(
         remote = True
 
     if requirements:
+        error, e = None, None
         click.echo(crayons.normal(u'Requirements file provided! Importing into Pipfile…', bold=True), err=True)
         try:
             import_requirements(r=project.path_to(requirements), dev=dev)
         except (UnicodeDecodeError, pip.exceptions.PipError) as e:
             # Don't print the temp file path if remote since it will be deleted.
             req_path = requirements_url if remote else project.path_to(requirements)
-            click.echo(
-                crayons.red(
-                    u'Unexpected syntax in {0}. Are you sure this is a '
-                     'requirements.txt style file?'.format(req_path)
-                )
-            )
-            click.echo(crayons.blue(str(e)), err=True)
-            sys.exit(1)
+            error = (u'Unexpected syntax in {0}. Are you sure this is a '
+                      'requirements.txt style file?'.format(req_path))
+        except AssertionError as e:
+            error = (u'Requirements file doesn\'t appear to exist. Please ensure the file exists in your '
+                      'project directory or you provided the correct path.')
         finally:
             # If requirements file was provided by remote url delete the temporary file
             if remote:
                 os.close(fd)  # Close for windows to allow file cleanup.
                 os.remove(project.path_to(temp_reqs))
 
+            if error and e:
+                click.echo(crayons.red(error))
+                click.echo(crayons.blue(str(e)), err=True)
+                sys.exit(1)
+
     if code:
         click.echo(crayons.normal(u'Discovering imports from local codebase…', bold=True))
         for req in import_from_code(code):
@@ -2079,6 +2063,9 @@ def lock(three=None, python=False, verbose=False, requirements=False, dev=False,
 
 def do_shell(three=None, python=False, fancy=False, shell_args=None):
 
+    # Ensure that virtualenv is available.
+    ensure_project(three=three, python=python, validate=False)
+
     # Set an environment variable, so we know we're in the environment.
     os.environ['PIPENV_ACTIVE'] = '1'
 
@@ -2200,9 +2187,6 @@ def shell(three=None, python=False, fancy=False, shell_args=None, anyway=False):
 
             sys.exit(1)
 
-    # Ensure that virtualenv is available.
-    ensure_project(three=three, python=python, validate=False)
-
     # Load .env file.
     load_dot_env()
 
@@ -2573,7 +2557,7 @@ def update(ctx, dev=False, three=None, python=None, dry_run=False, bare=False, d
         do_purge()
 
         # Lock.
-        do_lock(pre=pre)
+        do_lock(clear=clear, pre=pre)
 
         # Install everything.
         do_init(dev=dev, verbose=verbose, concurrent=concurrent)

pipenv/environments.py

@@ -35,7 +35,7 @@ PIPENV_DONT_LOAD_ENV = bool(os.environ.get('PIPENV_DONT_LOAD_ENV'))
 PIPENV_YES = bool(os.environ.get('PIPENV_YES'))
 
 # Tells Pipenv how many subprocesses to use when installing.
-PIPENV_MAX_SUBPROCESS = int(os.environ.get('PIPENV_MAX_SUBPROCESS', '8'))
+PIPENV_MAX_SUBPROCESS = int(os.environ.get('PIPENV_MAX_SUBPROCESS', '16'))
 
 # User-configurable max-depth for Pipfile searching.
 # Note: +1 because of a temporary bug in Pipenv.

pipenv/patched/README.md

@@ -5,6 +5,7 @@
 - Pip is modified, to make it resolve deep extras links.
 - Pip-tools is modified, to make it resolve markers.
 - Pip-tools is modified, to make it do 12 rounds instead of 10, by default.
+- Pip-tools is modified, to use its own cache path with dependency resolution, and not conflict with the "real" pip-tools.
 - Crayons is upgraded.
 - Safety is hacked together to always work on any system.
 - TOML libraries are upgraded to... work.

pipenv/patched/dotenv/main.py

@@ -97,15 +97,7 @@ def dotenv_values(dotenv_path):
 
 def parse_dotenv(dotenv_path):
     with open(dotenv_path) as f:
-        for line in f:
-            line = line.strip()
-            if not line or line.startswith('#') or '=' not in line:
-                continue
-            k, v = line.split('=', 1)
-
-            # Remove any leading and trailing spaces in key, value
-            k, v = k.strip(), v.strip().encode('unicode-escape').decode('ascii')
-
+        for k, v in re.findall('^\s*(\w*)\s*=\s*("[^"]*"|[^\s]*)\s*$', f.read(), flags=re.MULTILINE):
             if len(v) > 0:
                 quoted = v[0] == v[len(v) - 1] in ['"', "'"]
 

pipenv/patched/pip/req/req_set.py

@@ -749,7 +749,7 @@ class RequirementSet(object):
         # print('\n')
         # print('\n')
 
-        return self.requirements.values() + more_reqs
+        return more_reqs
 
     def cleanup_files(self):
         """Clean up files, remove builds."""

pipenv/patched/piptools/locations.py

@@ -2,10 +2,13 @@ import os
 from shutil import rmtree
 
 from .click import secho
-from pip.utils.appdirs import user_cache_dir
+# Patch by vphilippon 2017-11-22: Use pipenv cache path.
+# from pip.utils.appdirs import user_cache_dir
+from pipenv.environments import PIPENV_CACHE_DIR
 
 # The user_cache_dir helper comes straight from pip itself
-CACHE_DIR = user_cache_dir('pip-tools')
+# CACHE_DIR = user_cache_dir(os.path.join('pip-tools'))
+CACHE_DIR = PIPENV_CACHE_DIR
 
 # NOTE
 # We used to store the cache dir under ~/.pip-tools, which is not the

pipenv/project.py

@@ -56,23 +56,34 @@ class Project(object):
     def _build_package_list(self, package_section):
         """Returns a list of packages for pip-tools to consume."""
         ps = {}
+        # TODO: Separate the logic for showing packages from the filters for supplying pip-tools
         for k, v in self.parsed_pipfile.get(package_section, {}).items():
             # Skip editable VCS deps.
             if hasattr(v, 'keys'):
                 # When a vcs url is gven without editable it only appears as a key
-                if is_vcs(v) or is_vcs(k):
-                    # Non-editable VCS entries can't be resolved by piptools
+                # Eliminate any vcs, path, or url entries which are not editable
+                # Since pip-tools can't do deep resolution on them, even setuptools-installable ones
+                if (is_vcs(v) or is_vcs(k) or (is_installable_file(k) or is_installable_file(v)) or
+                        any((prefix in v and
+                             (os.path.isfile(v[prefix]) or is_valid_url(v[prefix])))
+                            for prefix in ['path', 'file'])):
+                    # If they are editable, do resolve them
                     if 'editable' not in v:
                         continue
                     else:
                         ps.update({k: v})
                 else:
-                    if not (is_installable_file(k) or is_installable_file(v) or
-                            any(file_prefix in v for file_prefix in ['path', 'file'])):
-                        ps.update({k: v})
+                    ps.update({k: v})
             else:
+                # Since these entries have no attributes we know they are not editable
+                # So we can safely exclude things that need to be editable in order to be resolved
+                # First exclude anything that is a vcs entry either in the key or value
                 if not (any(is_vcs(i) for i in [k, v]) or
+                        # Then exclude any installable files that are not directories
+                        # Because pip-tools can resolve setup.py for example
                         any(is_installable_file(i) for i in [k, v]) or
+                        # Then exclude any URLs because they need to be editable also
+                        # Things that are excluded can only be 'shallow resolved'
                         any(is_valid_url(i) for i in [k, v])):
                     ps.update({k: v})
         return ps
@@ -98,7 +109,10 @@ class Project(object):
 
     @property
     def project_directory(self):
-        return os.path.abspath(os.path.join(self.pipfile_location, os.pardir))
+        if self.pipfile_location is not None:
+            return os.path.abspath(os.path.join(self.pipfile_location, os.pardir))
+        else:
+            return None
 
     @property
     def requirements_exists(self):
@@ -182,7 +196,7 @@ class Project(object):
     @property
     def proper_names_location(self):
         if self._proper_names_location is None:
-            loc = os.sep.join([self.virtualenv_location, 'pipenev-proper-names.txt'])
+            loc = os.sep.join([self.virtualenv_location, 'pipenv-proper-names.txt'])
             self._proper_names_location = loc
 
         # Create the database, if it doesn't exist.

pipenv/utils.py

@@ -36,7 +36,7 @@ from piptools.scripts.compile import get_pip_command
 from piptools import logging
 from piptools.exceptions import NoCandidateFound
 from pip.exceptions import DistributionNotFound
-from requests.exceptions import HTTPError
+from requests.exceptions import HTTPError, ConnectionError
 
 from .pep508checker import lookup
 from .environments import SESSION_IS_INTERACTIVE, PIPENV_MAX_ROUNDS, PIPENV_CACHE_DIR
@@ -277,16 +277,34 @@ def get_requirement(dep):
     remote URIs, and package names, and that we pass only valid requirement strings
     to the requirements parser. Performs necessary modifications to requirements
     object if the user input was a local relative path.
+    
+    :param str dep: A requirement line
+    :returns: :class:`requirements.Requirement` object
     """
     path = None
+    # Split out markers if they are present - similar to how pip does it
+    # See pip.req.req_install.InstallRequirement.from_line
+    if not any(dep.startswith(uri_prefix) for uri_prefix in SCHEME_LIST):
+        marker_sep = ';'
+    else:
+        marker_sep = '; '
+    if marker_sep in dep:
+        dep, markers = dep.split(marker_sep, 1)
+        markers = markers.strip()
+        if not markers:
+            markers = None
+    else:
+        markers = None
+    # Strip extras from the requirement so we can make a properly parseable req
+    dep, extras = pip.req.req_install._strip_extras(dep)
     # Only operate on local, existing, non-URI formatted paths
     if (is_file(dep) and isinstance(dep, six.string_types) and
             not any(dep.startswith(uri_prefix) for uri_prefix in SCHEME_LIST)):
         dep_path = Path(dep)
         # Only parse if it is a file or an installable dir
         if dep_path.is_file() or (dep_path.is_dir() and pip.utils.is_installable_dir(dep)):
-            if dep_path.is_absolute():
-                path = dep
+            if dep_path.is_absolute() or dep_path.as_posix() == '.':
+                path = dep_path.as_posix()
             else:
                 path = get_converted_relative_path(dep)
             dep = dep_path.resolve().as_uri()
@@ -296,6 +314,11 @@ def get_requirement(dep):
     if req.local_file and req.uri and not req.path and path:
         req.path = path
         req.uri = None
+    if markers:
+        req.markers = markers
+    if extras:
+        # Bizarrely this is also what pip does...
+        req.extras = [r for r in requirements.parse('fakepkg{0}'.format(extras))][0].extras
     return req
 
 
@@ -500,7 +523,8 @@ def actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources,
 
         raise RuntimeError
 
-    return resolved_tree
+    return resolved_tree, resolver
+
 
 def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, python=False, clear=False, pre=False, allow_global=False):
     """Given a list of dependencies, return a resolved list of dependencies,
@@ -519,7 +543,7 @@ def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, p
     with HackedPythonVersion(python_version=python, python_path=python_path):
 
         try:
-            resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
+            resolved_tree, resolver = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
         except RuntimeError:
             # Don't exit here, like usual.
             resolved_tree = None
@@ -531,12 +555,11 @@ def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, p
             try:
                 # Attempt to resolve again, with different Python version information,
                 # particularly for particularly particular packages.
-                resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
+                resolved_tree, resolver = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
             except RuntimeError:
                 sys.exit(1)
 
 
-
     for result in resolved_tree:
         if not result.editable:
             name = pep423_name(result.name)
@@ -568,7 +591,7 @@ def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, p
                     if not collected_hashes:
                         collected_hashes = list(list(resolver.resolve_hashes([result]).items())[0][1])
 
-                except (ValueError, KeyError):
+                except (ValueError, KeyError, ConnectionError):
                     if verbose:
                         print('Error fetching {}'.format(name))
 
@@ -611,28 +634,26 @@ def convert_deps_from_pip(dep):
         hashable_path = req.uri if req.uri else req.path
         req.name = hashlib.sha256(hashable_path.encode('utf-8')).hexdigest()
         req.name = req.name[len(req.name) - 7:]
-
         # {path: uri} TOML (spec 4 I guess...)
         if req.uri:
             dependency[req.name] = {'file': hashable_path}
         else:
             dependency[req.name] = {'path': hashable_path}
 
+        if req.extras:
+            dependency[req.name].update(extras)
+
         # Add --editable if applicable
         if req.editable:
             dependency[req.name].update({'editable': True})
 
     # VCS Installs. Extra check for unparsed git over SSH
-    if req.vcs or is_vcs(req.path):
+    elif req.vcs or is_vcs(req.path):
         if req.name is None:
             raise ValueError('pipenv requires an #egg fragment for version controlled '
                              'dependencies. Please install remote dependency '
                              'in the form {0}#egg=<package-name>.'.format(req.uri))
 
-        # Extras: e.g. #egg=requests[security]
-        if req.extras:
-            dependency[req.name] = extras
-
         # Set up this requirement as a proper VCS requirement if it was not
         if not req.vcs and req.path.startswith(VCS_LIST):
             req.vcs = [vcs for vcs in VCS_LIST if req.path.startswith(vcs)][0]
@@ -640,7 +661,9 @@ def convert_deps_from_pip(dep):
             req.path = None
 
         # Crop off the git+, etc part.
-        dependency.setdefault(req.name, {}).update({req.vcs: req.uri[len(req.vcs) + 1:]})
+        if req.uri.startswith('{0}+'.format(req.vcs)):
+            req.uri = req.uri[len(req.vcs) + 1:]
+        dependency.setdefault(req.name, {}).update({req.vcs: req.uri})
 
         # Add --editable, if it's there.
         if req.editable:
@@ -654,6 +677,10 @@ def convert_deps_from_pip(dep):
         if req.revision:
             dependency[req.name].update({'ref': req.revision})
 
+        # Extras: e.g. #egg=requests[security]
+        if req.extras:
+            dependency[req.name].update({'extras': req.extras})
+
     elif req.extras or req.specs:
 
         specs = None
@@ -679,7 +706,6 @@ def convert_deps_from_pip(dep):
         for key in dependency.copy():
             if not hasattr(dependency[key], 'keys'):
                 del dependency[key]
-
     return dependency
 
 
@@ -843,6 +869,16 @@ def is_installable_file(path):
         path = urlparse(path['file']).path if 'file' in path else path['path']
     if not isinstance(path, six.string_types) or path == '*':
         return False
+    # If the string starts with a valid specifier operator, test if it is a valid
+    # specifier set before making a path object (to avoid breaking windows)
+    if any(path.startswith(spec) for spec in '!=<>~'):
+        try:
+            pip.utils.packaging.specifiers.SpecifierSet(path)
+        # If this is not a valid specifier, just move on and try it as a path
+        except pip.utils.packaging.specifiers.InvalidSpecifier:
+            pass
+        else:
+            return False
     lookup_path = Path(path)
     return lookup_path.is_file() or (lookup_path.is_dir() and
             pip.utils.is_installable_dir(lookup_path.resolve().as_posix()))
@@ -893,21 +929,102 @@ def proper_case(package_name):
     return good_name
 
 
-def split_vcs(split_file):
-    """Split VCS dependencies out from file."""
+def split_section(input_file, section_suffix, test_function):
+    """
+    Split a pipfile or a lockfile section out by section name and test function
+    
+        :param dict input_file: A dictionary containing either a pipfile or lockfile
+        :param str section_suffix: A string of the name of the section
+        :param func test_function: A test function to test against the value in the key/value pair
+    
+    >>> split_section(my_lockfile, 'vcs', is_vcs)
+    {
+        'default': {
+            "six": {
+                "hashes": [
+                    "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb",
+                    "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"
+                ],
+                "version": "==1.11.0"
+            }
+        },
+        'default-vcs': {
+            "e1839a8": {
+                "editable": true,
+                "path": "."
+            }
+        }
+    }
+    """
+    pipfile_sections = ('packages', 'dev-packages')
+    lockfile_sections = ('default', 'develop')
+    if any(section in input_file for section in pipfile_sections):
+        sections = pipfile_sections
+    elif any(section in input_file for section in lockfile_sections):
+        sections = lockfile_sections
+    else:
+        # return the original file if we can't find any pipfile or lockfile sections
+        return input_file
 
-    if 'packages' in split_file or 'dev-packages' in split_file:
-        sections = ('packages', 'dev-packages')
-    elif 'default' in split_file or 'develop' in split_file:
-        sections = ('default', 'develop')
-
-    # For each vcs entry in a given section, move it to section-vcs.
     for section in sections:
-        entries = split_file.get(section, {})
-        vcs_dict = dict((k, entries.pop(k)) for k in list(entries.keys()) if is_vcs(entries[k]))
-        split_file[section + '-vcs'] = vcs_dict
+        split_dict = {}
+        entries = input_file.get(section, {})
+        for k in list(entries.keys()):
+            if test_function(entries.get(k)):
+                split_dict[k] = entries.pop(k)
+        input_file['-'.join([section, section_suffix])] = split_dict
+    return input_file
 
-    return split_file
+
+def split_file(file_dict):
+    """Split VCS and editable dependencies out from file."""
+    sections = {
+        'vcs': is_vcs,
+        'editable': lambda x: hasattr(x, 'keys') and x.get('editable')
+    }
+    for k, func in sections.items():
+        file_dict = split_section(file_dict, k, func)
+    return file_dict
+
+
+def merge_deps(file_dict, project, dev=False, requirements=False, ignore_hashes=False, blocking=False, only=False):
+    """
+    Given a file_dict, merges dependencies and converts them to pip dependency lists.
+        :param dict file_dict: The result of calling :func:`pipenv.utils.split_file`
+        :param :class:`pipenv.project.Project` project: Pipenv project
+        :param bool dev=False: Flag indicating whether dev dependencies are to be installed 
+        :param bool requirements=False: Flag indicating whether to use a requirements file
+        :param bool ignore_hashes=False:
+        :param bool blocking=False:
+        :param bool only=False:
+        :return: Pip-converted 3-tuples of [deps, requirements_deps]
+    """
+    deps = []
+    requirements_deps = []
+
+    for section in list(file_dict.keys()):
+        # Turn develop-vcs into ['develop', 'vcs']
+        section_name, suffix = section.rsplit('-', 1) if '-' in section and not section == 'dev-packages' else (section, None)
+        if not file_dict[section] or section_name not in ('dev-packages', 'packages', 'default', 'develop'):
+            continue
+        is_dev = section_name in ('dev-packages', 'develop')
+        if is_dev and not dev:
+            continue
+
+        if ignore_hashes:
+            for k, v in file_dict[section]:
+                if 'hash' in v:
+                    del v['hash']
+
+        # Block and ignore hashes for all suffixed sections (vcs/editable)
+        no_hashes = True if suffix else ignore_hashes
+        block = True if suffix else blocking
+        include_index = True if not suffix else False
+        converted = convert_deps_to_pip(file_dict[section], project, r=False, include_index=include_index)
+        deps.extend((d, no_hashes, block) for d in converted)
+        if dev and is_dev and requirements:
+            requirements_deps.extend((d, no_hashes, block) for d in converted)
+    return deps, requirements_deps
 
 
 def recase_file(file_dict):

setup.py

@@ -79,7 +79,7 @@ setup(
     long_description=long_description,
     author='Kenneth Reitz',
     author_email='me@kennethreitz.org',
-    url='https://github.com/kennethreitz/pipenv',
+    url='https://github.com/pypa/pipenv',
     packages=find_packages(exclude=['tests']),
     entry_points={
         'console_scripts': ['pipenv=pipenv:cli'],

tests/test_pipenv.py

@@ -1,4 +1,5 @@
 import os
+from pkg_resources import parse_version
 import re
 import tempfile
 import shutil
@@ -242,6 +243,32 @@ class TestPipenv:
             c = p.pipenv('run python -m requests.help')
             assert c.return_code == 0
 
+    @pytest.mark.dev
+    @pytest.mark.install
+    def test_install_without_dev(self):
+        """Ensure that running `pipenv install` doesn't install dev packages"""
+        with PipenvInstance() as p:
+            with open(p.pipfile_path, 'w') as f:
+                contents = """
+[packages]
+tablib = "*"
+
+[dev-packages]
+records = "*"
+                """.strip()
+                f.write(contents)
+            c = p.pipenv('install')
+            assert c.return_code == 0
+            assert 'tablib' in p.pipfile['packages']
+            assert 'records' in p.pipfile['dev-packages']
+            assert 'tablib' in p.lockfile['default']
+            assert 'records' in p.lockfile['develop']
+            c = p.pipenv('run python -c "import records"')
+            assert c.return_code != 0
+            c = p.pipenv('run python -c "import tablib"')
+            assert c.return_code == 0
+                
+
     @pytest.mark.run
     @pytest.mark.uninstall
     def test_uninstall(self):
@@ -334,6 +361,39 @@ class TestPipenv:
             assert 'urllib3' in p.lockfile['default']
             assert 'pysocks' in p.lockfile['default']
 
+    @pytest.mark.extras
+    @pytest.mark.install
+    @pytest.mark.local
+    def test_local_extras_install(self):
+        with PipenvInstance() as p:
+            setup_py = os.path.join(p.path, 'setup.py')
+            with open(setup_py, 'w') as fh:
+                contents = """
+from setuptools import setup, find_packages
+
+setup(
+    name='test_pipenv',
+    version='0.1',
+    description='Pipenv Test Package',
+    author='Pipenv Test',
+    author_email='test@pipenv.package',
+    license='PIPENV',
+    packages=find_packages(),
+    install_requires=['tablib'],
+    extras_require={'dev': ['flake8', 'pylint']},
+    zip_safe=False
+)
+                """.strip()
+                fh.write(contents)
+            c = p.pipenv('install .[dev]')
+            assert c.return_code == 0
+            key = [k for k in p.pipfile['packages'].keys()][0]
+            dep = p.pipfile['packages'][key]
+            assert dep['path'] == '.'
+            assert dep['extras'] == ['dev']
+            assert key in p.lockfile['default']
+            assert 'dev' in p.lockfile['default'][key]['extras']
+
     @pytest.mark.vcs
     @pytest.mark.install
     def test_basic_vcs_install(self):
@@ -364,6 +424,22 @@ class TestPipenv:
             assert 'urllib3' in p.lockfile['default']
             assert 'certifi' in p.lockfile['default']
 
+    @pytest.mark.install
+    @pytest.mark.pin
+    def test_windows_pinned_pipfile(self):
+        with PipenvInstance() as p:
+            with open(p.pipfile_path, 'w') as f:
+                contents = """
+[packages]
+tablib = "<0.12"
+                """.strip()
+                f.write(contents)
+            c = p.pipenv('install')
+            assert c.return_code == 0
+            assert 'tablib' in p.pipfile['packages']
+            assert 'tablib' in p.lockfile['default']
+
+
     @pytest.mark.run
     @pytest.mark.install
     def test_multiprocess_bug_and_install(self):
@@ -422,6 +498,23 @@ tpfd = "*"
             c = p.pipenv('run python -c "import requests; import idna; import certifi; import records; import tpfd; import parse;"')
             assert c.return_code == 0
 
+    @pytest.mark.install
+    @pytest.mark.resolver
+    @pytest.mark.backup_resolver
+    def test_backup_resolver(self):
+        with PipenvInstance() as p:
+            with open(p.pipfile_path, 'w') as f:
+                contents = """
+[packages]
+"ibm-db-sa-py3" = "==0.3.1-1"
+                """.strip()
+                f.write(contents)
+
+            c = p.pipenv('install')
+            assert c.return_code == 0
+            assert 'ibm-db-sa-py3' in p.lockfile['default']
+
+
     @pytest.mark.sequential
     @pytest.mark.install
     @pytest.mark.update
@@ -504,6 +597,19 @@ requests = {version = "*", os_name = "== 'splashwear'"}
             c = p.pipenv('run python -c "import requests;"')
             assert c.return_code == 1
 
+    @pytest.mark.install
+    @pytest.mark.vcs
+    @pytest.mark.tablib
+    def test_install_editable_git_tag(self):
+        with PipenvInstance() as p:
+            c = p.pipenv('install -e git+git://github.com/kennethreitz/tablib.git@v0.12.1#egg=tablib')
+            assert c.return_code == 0
+            assert 'tablib' in p.pipfile['packages']
+            assert 'tablib' in p.lockfile['default']
+            assert 'git' in p.lockfile['default']['tablib']
+            assert p.lockfile['default']['tablib']['git'] == 'git://github.com/kennethreitz/tablib.git'
+            assert 'ref' in p.lockfile['default']['tablib']            
+
     @pytest.mark.run
     @pytest.mark.alt
     @pytest.mark.install
@@ -816,6 +922,49 @@ maya = "*"
                     if os.path.exists(path):
                         os.unlink(path)
 
+    @pytest.mark.requirements
+    @pytest.mark.complex
+    def test_complex_lock_changing_candidate(self):
+        # The requests candidate will change from latest to <2.12.
+
+        with PipenvInstance() as p:
+            with open(p.pipfile_path, 'w') as f:
+                contents = """
+[packages]
+"docker-compose" = "==1.16.0"
+requests = "*"
+                """.strip()
+                f.write(contents)
+
+            c = p.pipenv('lock')
+            assert c.return_code == 0
+            assert parse_version(p.lockfile['default']['requests']['version'][2:]) < parse_version('2.12')
+
+            c = p.pipenv('install')
+            assert c.return_code == 0
+
+    @pytest.mark.extras
+    @pytest.mark.lock
+    @pytest.mark.requirements
+    @pytest.mark.complex
+    def test_complex_lock_deep_extras(self):
+        # records[pandas] requires tablib[pandas] which requires pandas.
+
+        with PipenvInstance() as p:
+            with open(p.pipfile_path, 'w') as f:
+                contents = """
+[packages]
+records = {extras = ["pandas"], version = "==0.5.2"}
+                """.strip()
+                f.write(contents)
+
+            c = p.pipenv('lock')
+            assert c.return_code == 0
+            assert 'tablib' in p.lockfile['default']
+            assert 'pandas' in p.lockfile['default']
+
+            c = p.pipenv('install')
+            assert c.return_code == 0
 
     @pytest.mark.lock
     @pytest.mark.deploy
@@ -863,6 +1012,29 @@ requests = "==2.14.0"
 
             assert 'file' in dep
 
+    @pytest.mark.install
+    @pytest.mark.files
+    @pytest.mark.resolver
+    def test_local_package(self):
+        """This test ensures that local packages (directories with a setup.py)
+        installed in editable mode have their dependencies resolved as well"""
+        file_name = 'tablib-0.12.1.tar.gz'
+        package = 'tablib-0.12.1'
+        # Not sure where travis/appveyor run tests from
+        test_dir = os.path.dirname(os.path.abspath(__file__))
+        source_path = os.path.abspath(os.path.join(test_dir, 'test_artifacts', file_name))
+        with PipenvInstance() as p:
+            # This tests for a bug when installing a zipfile in the current dir
+            copy_to = os.path.join(p.path, file_name)
+            shutil.copy(source_path, copy_to)
+            import tarfile
+            with tarfile.open(copy_to, 'r:gz') as tgz:
+                tgz.extractall(path=p.path)
+            c = p.pipenv('install -e {0}'.format(package))
+            assert c.return_code == 0
+            assert all(pkg in p.lockfile['default'] for pkg in ['xlrd', 'xlwt', 'pyyaml', 'odfpy'])
+
+
     @pytest.mark.install
     @pytest.mark.files
     def test_local_zipfiles(self):
@@ -891,7 +1063,7 @@ requests = "==2.14.0"
     @pytest.mark.install
     @pytest.mark.files
     @pytest.mark.urls
-    def test_install_remote_requirments(self):
+    def test_install_remote_requirements(self):
         with PipenvInstance() as p:
             # using a github hosted requirements.txt file
             c = p.pipenv('install -r https://raw.githubusercontent.com/kennethreitz/pipenv/3688148ac7cfecefb085c474b092c31d791952c1/tests/test_artifacts/requirements.txt')

tests/test_utils.py

@@ -131,11 +131,12 @@ class TestUtils:
     def test_is_vcs(self, entry, expected):
         assert pipenv.utils.is_vcs(entry) is expected
 
-    def test_split_vcs(self):
+    def test_split_file(self):
         pipfile_dict = {
             'packages': {
                 'requests': {'git': 'https://github.com/kennethreitz/requests.git'},
-                'Flask': '*'
+                'Flask': '*',
+                'tablib': {'path': '.', 'editable': True}
             },
             'dev-packages': {
                 'Django': '==1.10',
@@ -143,10 +144,11 @@ class TestUtils:
                 'crayons': {'hg': 'https://hg.alsonotreal.com/crayons'}
             }
         }
-        split_dict = pipenv.utils.split_vcs(pipfile_dict)
+        split_dict = pipenv.utils.split_file(pipfile_dict)
 
         assert list(split_dict['packages'].keys()) == ['Flask']
         assert split_dict['packages-vcs'] == {'requests': {'git': 'https://github.com/kennethreitz/requests.git'}}
+        assert split_dict['packages-editable'] == {'tablib': {'path': '.', 'editable': True}}
         assert list(split_dict['dev-packages'].keys()) == ['Django']
         assert 'click' in split_dict['dev-packages-vcs']
         assert 'crayons' in split_dict['dev-packages-vcs']