Pipenv uninstall, and other improvements (#650)

* Update pipenv-python-version

* use latest versions in pipenv script

Signed-off-by: Kenneth Reitz <me@kennethreitz.org>

* Update pipenv-python-version

* 3.6.4

Signed-off-by: Kenneth Reitz <me@kennethreitz.org>

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Python Buildpack Changelog
 
+# 126
+
+Skip installs if Pipfile.lock hasn't changed, and uninstall stale dependencies with Pipenv.
+
+- No longer warn if there is no `Procfile`.
+
+# 125
+
+Set `PYTHONPATH` during collectstatic runs, other updates.
+
+- Update Set `PYTHONPATH` during collectstatic runs.
+- Update Pipenv "3.6" to "3.6.4".
+
+# 124
+
+Update buildpack to automatically install [dev-packages] (Pipenv) during Heroku CI builds.
+
+# 123
+
+Update gunicorn init.d script to allow overrides.
+
+# 122
+
+Update default Python to v3.6.4.
+
 # 121
 
 Update default Python to v3.6.3.

README.md

@@ -4,11 +4,11 @@
 
 [![Build Status](https://travis-ci.org/heroku/heroku-buildpack-python.svg?branch=master)](https://travis-ci.org/heroku/heroku-buildpack-python)
 
-This is the official [Heroku buildpack](https://devcenter.heroku.com/articles/buildpacks) for Python apps, powered by [Pipenv](http://docs.pipenv.org/en/latest/), [pip](https://pip.pypa.io/) and other excellent software.
+This is the official [Heroku buildpack](https://devcenter.heroku.com/articles/buildpacks) for Python apps, powered by [Pipenv](http://docs.pipenv.org/), [pip](https://pip.pypa.io/) and other excellent software.
 
 Recommended web frameworks include **Django** and **Flask**. The recommended webserver is **Gunicorn**. There are no restrictions around what software can be used (as long as it's pip-installable). Web processes must bind to `$PORT`, and only the HTTP protocol is permitted for incoming connections.
 
-Some Python packages with obscure C dependencies are [not compatible](https://devcenter.heroku.com/articles/python-c-deps).
+Python packages with C dependencies that are not [available on the stack image](https://devcenter.heroku.com/articles/stack-packages) are generally not supported, unless `manylinux` wheels are provided by the package maintainers (common). For recommended solutions, check out [this article](https://devcenter.heroku.com/articles/python-c-deps) for more information. 
 
 See it in Action
 ----------------
@@ -16,14 +16,14 @@ See it in Action
 Deploying a Python application couldn't be easier:
 
     $ ls
-    Pipfile		Procfile	web.py
+    Pipfile		Pipfile.lock	Procfile	web.py
 
     $ heroku create --buildpack heroku/python
 
     $ git push heroku master
     …
     -----> Python app detected
-    -----> Installing python-3.6.3
+    -----> Installing python-3.6.4
     -----> Installing pip
     -----> Installing requirements with latest pipenv…
            ...
@@ -41,12 +41,7 @@ You can also specify the latest production release of this buildpack for upcomin
 Specify a Python Runtime
 ------------------------
 
-Specific versions of the Python runtime can be specified with a `runtime.txt` file:
-
-    $ cat runtime.txt
-    python-2.7.14
-
-Or, with a `Pipfile.lock` (generated from the following `Pipfile`):
+Specific versions of the Python runtime can be specified in your `Pipfile`:
 
     [requires]
     python_version = "2.7"
@@ -56,9 +51,12 @@ Or, more specifically:
     [requires]
     python_full_version = "2.7.14"
 
+Or, with a `runtime.txt` file:
+
+    $ cat runtime.txt
+    python-2.7.14
+
 Runtime options include:
 
-- `python-3.6.3`
+- `python-3.6.4`
 - `python-2.7.14`
-- `pypy-5.7.1` (unsupported, experimental)
-- `pypy3-5.5.1` (unsupported, experimental)

bin/compile

@@ -33,6 +33,12 @@ ENV_DIR=$3
 
 export BUILD_DIR CACHE_DIR ENV_DIR
 
+VENDOR_URL="https://lang-python.s3.amazonaws.com/$STACK"
+if [[ -n ${BUILDPACK_VENDOR_URL:-} ]]; then
+    VENDOR_URL="$BUILDPACK_VENDOR_URL"
+fi
+export VENDOR_URL
+
 # Python defaults
 DEFAULT_PYTHON_VERSION="python-3.6.4"
 LATEST_3="python-3.6.4"
@@ -94,12 +100,6 @@ export PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/li
 # Switch to the repo's context.
 cd "$BUILD_DIR"
 
-# Warn for lack of Procfile.
-if [[ ! -f Procfile ]]; then
-  puts-warn 'Warning: Your application is missing a Procfile. This file tells Heroku how to run your application.'
-  puts-warn 'Learn more: https://devcenter.heroku.com/articles/procfile'
-fi
-
 # Prepare the cache.
 mkdir -p "$CACHE_DIR"
 
@@ -160,7 +160,14 @@ mtime "python.install.time" "${start}"
 
 # Pipenv support.
 # shellcheck source=bin/steps/pipenv
-source "$BIN_DIR/steps/pipenv"
+sub_env "$BIN_DIR/steps/pipenv"
+
+# Uninstall removed dependencies with Pip.
+let start=$(nowms)
+# shellcheck source=bin/steps/pip-uninstall
+source "$BIN_DIR/steps/pip-uninstall"
+mtime "pip.uninstall.time" "${start}"
+
 
 # If no requirements.txt file given, assume `setup.py develop` is intended.
 if [ ! -f requirements.txt ] && [ ! -f Pipfile ]; then
@@ -191,12 +198,6 @@ sub_env "$BIN_DIR/steps/geo-libs"
 # shellcheck source=bin/steps/gdal
 source "$BIN_DIR/steps/gdal"
 
-# Uninstall removed dependencies with Pip.
-let start=$(nowms)
-# shellcheck source=bin/steps/pip-uninstall
-source "$BIN_DIR/steps/pip-uninstall"
-mtime "pip.uninstall.time" "${start}"
-
 # Install dependencies with Pip (where the magic happens).
 let start=$(nowms)
 # shellcheck source=bin/steps/pip-install
@@ -224,14 +225,24 @@ mtime "collectstatic.time" "${start}"
 # Create .profile script for application runtime environment variables.
 set_env PATH "\$HOME/.heroku/python/bin:\$PATH"
 set_env PYTHONUNBUFFERED true
-set_env PYTHONHOME /app/.heroku/python
+set_env PYTHONHOME "\$HOME/.heroku/python"
 
-set_env LIBRARY_PATH "/app/.heroku/vendor/lib:/app/.heroku/python/lib:\$LIBRARY_PATH"
-set_env LD_LIBRARY_PATH "/app/.heroku/vendor/lib:/app/.heroku/python/lib:\$LD_LIBRARY_PATH"
+set_env LIBRARY_PATH "\$HOME/.heroku/vendor/lib:\$HOME/.heroku/python/lib:\$LIBRARY_PATH"
+set_env LD_LIBRARY_PATH "\$HOME/.heroku/vendor/lib:\$HOME/.heroku/python/lib:\$LD_LIBRARY_PATH"
 
 set_default_env LANG en_US.UTF-8
 set_default_env PYTHONHASHSEED random
-set_default_env PYTHONPATH /app/
+set_default_env PYTHONPATH "\$HOME"
+
+# python expects to be in /app, if at runtime, it is not, set
+# up symlinks... this can occur when the subdir buildpack is used
+cat <<EOT >> "$PROFILE_PATH"
+if [[ \$HOME != "/app" ]]; then
+    mkdir -p /app/.heroku
+    ln -nsf "\$HOME/.heroku/python" /app/.heroku/python
+    ln -nsf "\$HOME/.heroku/vendor" /app/.heroku/vendor
+fi
+EOT
 
 # Install sane-default script for $WEB_CONCURRENCY and $FORWARDED_ALLOW_IPS.
 cp "$ROOT_DIR/vendor/WEB_CONCURRENCY.sh" "$WEB_CONCURRENCY_PROFILE_PATH"

bin/steps/collectstatic

@@ -30,6 +30,7 @@ if [ ! "$DISABLE_COLLECTSTATIC" ] && [ -f "$MANAGE_FILE" ] && [ "$DJANGO_INSTALL
     puts-step "$ python $MANAGE_FILE collectstatic --noinput"
 
     # Run collectstatic, cleanup some of the noisy output.
+    export PYTHONPATH=.
     python "$MANAGE_FILE" collectstatic --noinput --traceback 2>&1 | sed '/^Post-processed/d;/^Copying/d;/^$/d' | indent
     COLLECTSTATIC_STATUS="${PIPESTATUS[0]}"
 

bin/steps/cryptography

@@ -15,7 +15,7 @@ if [[ "$STACK" == "heroku-16" ]]; then
 fi
 
 # The location of the pre-compiled libffi binary.
-VENDORED_LIBFFI="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/libffi.tar.gz"
+VENDORED_LIBFFI="${VENDOR_URL}/libraries/vendor/libffi.tar.gz"
 
 PKG_CONFIG_PATH="/app/.heroku/vendor/lib/pkgconfig:$PKG_CONFIG_PATH"
 

bin/steps/gdal

@@ -10,7 +10,7 @@
 # This script is invoked by [`bin/compile`](/).
 
 # The location of the pre-compiled cryptography binary.
-VENDORED_GDAL="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/gdal.tar.gz"
+VENDORED_GDAL="${VENDOR_URL}/libraries/vendor/gdal.tar.gz"
 
 PKG_CONFIG_PATH="/app/.heroku/vendor/lib/pkgconfig:$PKG_CONFIG_PATH"
 

bin/steps/geo-libs

@@ -10,9 +10,9 @@
 # This script is invoked by [`bin/compile`](/).
 
 # The location of the pre-compiled cryptography binary.
-VENDORED_GDAL="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/gdal.tar.gz"
-VENDORED_GEOS="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/geos.tar.gz"
-VENDORED_PROJ="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/proj.tar.gz"
+VENDORED_GDAL="${VENDOR_URL}/libraries/vendor/gdal.tar.gz"
+VENDORED_GEOS="${VENDOR_URL}/libraries/vendor/geos.tar.gz"
+VENDORED_PROJ="${VENDOR_URL}/libraries/vendor/proj.tar.gz"
 
 PKG_CONFIG_PATH="/app/.heroku/vendor/lib/pkgconfig:$PKG_CONFIG_PATH"
 

bin/steps/pip-install

@@ -8,6 +8,12 @@ if [ ! "$SKIP_PIP_INSTALL" ]; then
     # Install dependencies with Pip.
     puts-step "Installing requirements with pip"
 
+    # Set PIP_EXTRA_INDEX_URL
+    if [[ -r $ENV_DIR/PIP_EXTRA_INDEX_URL ]]; then
+        PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")"
+        export PIP_EXTRA_INDEX_URL
+    fi
+
     set +e
     
     # Measure that we're using pip.
@@ -29,4 +35,12 @@ if [ ! "$SKIP_PIP_INSTALL" ]; then
     /app/.heroku/python/bin/pip freeze --disable-pip-version-check > .heroku/python/requirements-installed.txt
 
     echo
+    
+    # Install test dependencies, for CI. 
+    if [ "$INSTALL_TEST" ]; then
+        if [[ -f "$1/requirements-test.txt" ]]; then
+            puts-step "Installing test dependencies…"
+            /app/.heroku/python/bin/pip install -r "$1/requirements-test.txt" --exists-action=w --src=./.heroku/src --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent
+        fi
+    fi
 fi

bin/steps/pip-uninstall

@@ -2,19 +2,24 @@
 
 set +e
 # Install dependencies with Pip.
+# shellcheck source=bin/utils
+source $BIN_DIR/utils
 
-if [[ -f .heroku/python/requirements-declared.txt ]]; then
+if [ ! "$SKIP_PIP_INSTALL" ]; then
 
-  cp .heroku/python/requirements-declared.txt requirements-declared.txt
+  if [[ -f .heroku/python/requirements-declared.txt ]]; then
 
-  pip-diff --stale requirements-declared.txt requirements.txt --exclude setuptools pip wheel > .heroku/python/requirements-stale.txt
+    cp .heroku/python/requirements-declared.txt requirements-declared.txt
 
-  rm -fr requirements-declared.txt
+    pip-diff --stale requirements-declared.txt requirements.txt --exclude setuptools pip wheel > .heroku/python/requirements-stale.txt
 
-  if [[ -s .heroku/python/requirements-stale.txt ]]; then
-    puts-step "Uninstalling stale dependencies"
-    /app/.heroku/python/bin/pip uninstall -r .heroku/python/requirements-stale.txt -y --exists-action=w | cleanup | indent
+    rm -fr requirements-declared.txt
+
+    if [[ -s .heroku/python/requirements-stale.txt ]]; then
+      puts-step "Uninstalling stale dependencies"
+      /app/.heroku/python/bin/pip uninstall -r .heroku/python/requirements-stale.txt -y --exists-action=w | cleanup | indent
+    fi
   fi
-fi
 
+fi
 set -e

bin/steps/pipenv

@@ -5,30 +5,72 @@
 # shellcheck source=bin/utils
 source $BIN_DIR/utils
 
-# Pipenv support (Generate requriements.txt with pipenv).
-if [[ -f Pipfile ]]; then
-    if [[ ! -f requirements.txt ]]; then
-        puts-step "Installing requirements with latest Pipenv…"
 
-        # Measure that we're using Pipenv.
-        mcount "tool.pipenv"
+if [[ -f Pipfile.lock ]]; then
+    if [[ -f .heroku/python/Pipfile.lock.sha256 ]]; then
+        if [[ $(openssl dgst -sha256 Pipfile.lock) == $(cat .heroku/python/Pipfile.lock.sha256) ]]; then
+            if [[ ! "$PIPENV_ALWAYS_INSTALL" ]]; then
+                echo "Skipping installation, as Pipfile.lock hasn't changed since last deploy." | indent
+                echo "To disable this functionality, run the following command:"
+                echo ""
+                echo "    $ heroku config:set PIPENV_ALWAYS_INSTALL=1" | indent
 
-        # Install pipenv.
-        /app/.heroku/python/bin/pip install pipenv --upgrade &> /dev/null
-
-        if [[ ! -f Pipfile.lock ]]; then
-            /app/.heroku/python/bin/pipenv install --system --skip-lock 2>&1 | indent
-        else
-            /app/.heroku/python/bin/pipenv install --system --deploy 2>&1 | indent
+                SKIP_PIPENV_INSTALL=1
+            fi
         fi
-        # Install the dependencies.
-
-
-        # Skip pip install, later.
-        export SKIP_PIP_INSTALL=1
-
-        # Pip freeze, for compatibility.
-        /app/.heroku/python/bin/pip freeze > requirements.txt
-
     fi
 fi
+
+
+if [ ! "$SKIP_PIPENV_INSTALL" ]; then
+
+    # Pipenv support (Generate requriements.txt with pipenv).
+    if [[ -f Pipfile ]]; then
+        if [[ ! -f requirements.txt ]]; then
+            puts-step "Installing requirements with latest Pipenv…"
+
+            # Measure that we're using Pipenv.
+            mcount "tool.pipenv"
+
+            # Set PIP_EXTRA_INDEX_URL
+            if [[ -r $ENV_DIR/PIP_EXTRA_INDEX_URL ]]; then
+                PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")"
+                export PIP_EXTRA_INDEX_URL
+            fi
+
+            # if [[ -f .heroku/python/requirements-declared.txt ]]; then
+            #     cp .heroku/python/requirements-declared.txt requirements.txt
+            # fi
+
+            # Install pipenv.
+            /app/.heroku/python/bin/pip install pipenv --upgrade &> /dev/null
+
+            # Install the dependencies.
+            if [[ ! -f Pipfile.lock ]]; then
+                /app/.heroku/python/bin/pipenv install --system --skip-lock 2>&1 | indent
+            else
+                pipenv-to-pip Pipfile.lock > requirements.txt
+                "$BIN_DIR/steps/pip-uninstall"
+                cp requirements.txt .heroku/python/requirements-declared.txt
+                openssl dgst -sha256 Pipfile.lock > .heroku/python/Pipfile.lock.sha256
+
+                /app/.heroku/python/bin/pipenv install --system --deploy 2>&1 | indent
+            fi
+
+            # Install the test dependencies, for CI.
+            if [ "$INSTALL_TEST" ]; then
+                puts-step "Installing test dependencies…"
+                /app/.heroku/python/bin/pipenv install --dev --system --deploy 2>&1 | cleanup | indent
+            fi
+
+            # Skip pip install, later.
+            export SKIP_PIP_INSTALL=1
+
+            # Pip freeze, for compatibility.
+            pip freeze > requirements.txt
+        fi
+    fi
+else
+    pipenv-to-pip Pipfile.lock > requirements.txt
+    export SKIP_PIP_INSTALL=1
+fi

bin/steps/pipenv-python-version

@@ -19,10 +19,10 @@ if [[ -f $BUILD_DIR/Pipfile ]]; then
             if [[ "$PYTHON" == "null" ]]; then
                 PYTHON=$(jq -r '._meta.requires.python_version' "$BUILD_DIR/Pipfile.lock")
                 if [ "$PYTHON" = 2.7 ]; then
-                    echo "python-2.7.14" > "$BUILD_DIR/runtime.txt"
+                    echo "$LATEST_2" > "$BUILD_DIR/runtime.txt"
                 fi
                 if [ "$PYTHON" = 3.6 ]; then
-                    echo "python-3.6.3" > "$BUILD_DIR/runtime.txt"
+                    echo "$LATEST_3" > "$BUILD_DIR/runtime.txt"
                 fi
             fi
 

bin/steps/pylibmc

@@ -15,7 +15,7 @@ if [[ "$STACK" == "heroku-16" ]]; then
 fi
 
 # The location of the pre-compiled libmemcached binary.
-VENDORED_MEMCACHED="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/libmemcache.tar.gz"
+VENDORED_MEMCACHED="${VENDOR_URL}/libraries/vendor/libmemcache.tar.gz"
 
 # Syntax sugar.
 # shellcheck source=bin/utils

bin/steps/python

@@ -4,6 +4,9 @@ set +e
 runtime-fixer runtime.txt
 PYTHON_VERSION=$(cat runtime.txt)
 
+# The location of the pre-compiled python binary.
+VENDORED_PYTHON="${VENDOR_URL}/runtimes/$PYTHON_VERSION.tar.gz"
+
 if [[ $PYTHON_VERSION =~ ^python-2 ]]; then
   if [[ "$PYTHON_VERSION" != "$LATEST_2" ]]; then
     puts-warn "The latest version of Python 2 is $LATEST_2 (you are using $PYTHON_VERSION, which is unsupported)."
@@ -41,7 +44,7 @@ if [ ! "$SKIP_INSTALL" ]; then
 
     mcount "version.python.$PYTHON_VERSION"
 
-    if ! curl "https://lang-python.s3.amazonaws.com/$STACK/runtimes/$PYTHON_VERSION.tar.gz" -s | tar zxv -C .heroku/python &> /dev/null; then
+    if ! curl "${VENDORED_PYTHON}" -s | tar zxv -C .heroku/python &> /dev/null; then
       puts-warn "Requested runtime ($PYTHON_VERSION) is not available for this stack ($STACK)."
       puts-warn "Aborting.  More info: https://devcenter.heroku.com/articles/python-support"
       exit 1

bin/test-compile

@@ -6,8 +6,8 @@ BIN_DIR=$(cd "$(dirname "$0")" || return; pwd) # absolute path
 # shellcheck source=bin/utils
 source "$BIN_DIR/utils"
 
-DISABLE_COLLECTSTATIC=1 "$(dirname "${0:-}")/compile" "$1" "$2" "$3"
+# Locale support for Pipenv.
+export LC_ALL=C.UTF-8
+export LANG=C.UTF-8
 
-if [[ -f "$1/requirements-test.txt" ]]; then
-    /app/.heroku/python/bin/pip install -r "$1/requirements-test.txt" --exists-action=w --src=./.heroku/src --disable-pip-version-check --no-cache-dir 2>&1 | cleanup | indent
-fi
+DISABLE_COLLECTSTATIC=1 INSTALL_TEST=1 "$(dirname "${0:-}")/compile" "$1" "$2" "$3"

vendor/pip-pop/pip-diff

@@ -41,6 +41,7 @@ class Requirements(object):
                 if not getattr(requirement.req, 'name', None):
                     # Prior to pip 8.1.2 the attribute `name` did not exist.
                     requirement.req.name = requirement.req.project_name
+                    requirement.req.name = requirement.req.name.lower()
                 self.requirements.append(requirement.req)
 
 

vendor/pipenv-to-pip

@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+
+import json
+import sys
+
+
+def main():
+    INFILE = sys.argv[1]
+
+    with open(INFILE, 'rb') as f:
+        lockfile = json.load(f)
+
+    packages = []
+    for package in lockfile.get('default', {}):
+        try:
+            packages.append('{0}{1}'.format(package, lockfile['default'][package]['version']))
+        except KeyError:
+            pass
+
+    print('\n'.join(packages))
+
+
+try:
+    main()
+except Exception:
+    pass