Install pip using itself rather than get-pip.py (#1007)

`get-pip.py` is no longer used, since:
- It uses `--force-reinstall`, which is unnecessary here and slows down
  repeat builds (given we call pip install every time now). Trying to
  work around this by using `get-pip.py` only for the initial install,
  and real pip for subsequent updates would mean we lose protection
  against cached broken installs, plus significantly increase the
  version combinations test matrix.
- It means downloading pip twice (once embedded in `get-pip.py`, and
  again during the install, since `get-pip.py` can't install the
  embedded version directly).
- We would still have to manage several versions of `get-pip.py`, to
  support older Pythons (once we upgrade to newer pip).

We don't use `ensurepip` since:
- not all of the previously generated Python runtimes on S3 include it.
- we would still have to upgrade pip/setuptools afterwards.
- the versions of pip/setuptools bundled with ensurepip differ greatly
  depending on Python version, and we could easily start using a CLI
  flag for the first pip install before upgrade that isn't supported on
  all versions, without even knowing it (unless we test against hundreds
  of Python archives).

Instead we install pip using itself in wheel form. See:
https://github.com/pypa/pip/issues/2351#issuecomment-69994524

The new pip wheel assets on S3 were generated using:

```
$ pip download --no-cache pip==19.1.1
Collecting pip==19.1.1
  Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB)
  Saved ./pip-19.1.1-py2.py3-none-any.whl
Successfully downloaded pip

$ pip download --no-cache pip==20.0.2
Collecting pip==20.0.2
  Downloading pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
  Saved ./pip-20.0.2-py2.py3-none-any.whl
Successfully downloaded pip

$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
(dryrun) upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl

$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
```

CHANGELOG.md

@@ -5,6 +5,7 @@
 - For repeat builds, also manage the installed versions of setuptools/wheel, rather than just that of pip (#1007).
 - Install an explicit version of wheel rather than the latest release at the time (#1007).
 - Output the installed version of pip, setuptools and wheel in the build log (#1007).
+- Install pip using itself rather than `get-pip.py` (#1007).
 - Install setuptools from PyPI rather than a vendored copy (#1007).
 - Reduce the number of environment variables exposed to `bin/{pre,post}_compile` and other subprocesses (#1011)
 

bin/steps/README.MD

@@ -2,13 +2,6 @@
 
 TODO: Add context on Python install steps, such as why symlinking vs copying
 
-## Installing the Pip tool
-
-The Python Buildpack uses a tool called `get-pip` to install the pip tool. This
-is done in the `python` script.
-
-This is in part because Python historically did not come with pip by default.
-
 ## Installing Python packages using Pip
 
 ### Convention: Use `python` process to invoke Pip

bin/steps/python

@@ -145,25 +145,36 @@ if [[ "${PYTHON_VERSION}" == ${PY34}* ]]; then
   WHEEL_VERSION='0.33.6'
 fi
 
-if [[ -f "$BUILD_DIR/Pipfile" ]]; then
-  # The buildpack is pinned to old pipenv, which requires older pip.
-  PIP_VERSION='9.0.2'
+# We don't use get-pip.py, since:
+#  - it uses `--force-reinstall`, which is unnecessary here and slows down repeat builds
+#  - it means downloading pip twice (once embedded in get-pip.py, and again during
+#    the install, since get-pip.py can't install the embedded version directly)
+#  - we would still have to manage several versions of get-pip.py, to support older Pythons.
+# Instead, we use the pip wheel to install itself, using the method described here:
+# https://github.com/pypa/pip/issues/2351#issuecomment-69994524
+PIP_WHEEL_FILENAME="pip-${PIP_VERSION}-py2.py3-none-any.whl"
+PIP_WHEEL_URL="https://lang-python.s3.amazonaws.com/common/${PIP_WHEEL_FILENAME}"
+PIP_WHEEL="${TMPDIR:-/tmp}/${PIP_WHEEL_FILENAME}"
+
+if ! curl -sSf "${PIP_WHEEL_URL}" -o "$PIP_WHEEL"; then
+  mcount "failure.python.download-pip"
+  puts-warn "Failed to download pip"
+  exit 1
 fi
 
-# Heroku uses the get-pip utility maintained by the Python community to vendor Pip.
-# https://github.com/pypa/get-pip
-GETPIP_URL="https://lang-python.s3.amazonaws.com/etc/get-pip.py"
-GETPIP_PY="${TMPDIR:-/tmp}/get-pip.py"
-
-if ! curl -s "${GETPIP_URL}" -o "$GETPIP_PY" &> /dev/null; then
-    mcount "failure.python.get-pip"
-    echo "Failed to pull down get-pip"
-    exit 1
+if [[ -f "$BUILD_DIR/Pipfile" ]]; then
+  # The buildpack is pinned to old pipenv, which requires older pip.
+  # Pip 9.0.2 doesn't support installing itself from a wheel, so we have to use split
+  # versions here (ie: installer pip version different from target pip version).
+  PIP_VERSION='9.0.2'
+  PIP_TO_INSTALL="pip==${PIP_VERSION}"
+else
+  PIP_TO_INSTALL="${PIP_WHEEL}"
 fi
 
 puts-step "Installing pip ${PIP_VERSION}, setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}"
 
-/app/.heroku/python/bin/python "$GETPIP_PY" pip=="${PIP_VERSION}" "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" &> /dev/null
+/app/.heroku/python/bin/python "${PIP_WHEEL}/pip" install "${PIP_TO_INSTALL}" "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}" &> /dev/null
 
 set -e
 hash -r