Upgrades setuptools from 39.0.1 to:
- 44.1.1 for Python 2.7 (since it's the last supported version)
- 43.0.0 for Python 3.4 (since it's the last supported version)
- 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed)
https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1Fixes#949.
Closes#973.
They are now displayed in the build output (instead of being sent to
`/dev/null`) and fail the build early instead of failing later in
`bin/steps/pip-install`.
Fixes#1002.
Since the version check is redundant given we control/choose the version.
The pip cache is redundant since we instead cache site-packages. The pip
cache also ends up in `/app` so isn't included in the build cache anyway.
`get-pip.py` is no longer used, since:
- It uses `--force-reinstall`, which is unnecessary here and slows down
repeat builds (given we call pip install every time now). Trying to
work around this by using `get-pip.py` only for the initial install,
and real pip for subsequent updates would mean we lose protection
against cached broken installs, plus significantly increase the
version combinations test matrix.
- It means downloading pip twice (once embedded in `get-pip.py`, and
again during the install, since `get-pip.py` can't install the
embedded version directly).
- We would still have to manage several versions of `get-pip.py`, to
support older Pythons (once we upgrade to newer pip).
We don't use `ensurepip` since:
- not all of the previously generated Python runtimes on S3 include it.
- we would still have to upgrade pip/setuptools afterwards.
- the versions of pip/setuptools bundled with ensurepip differ greatly
depending on Python version, and we could easily start using a CLI
flag for the first pip install before upgrade that isn't supported on
all versions, without even knowing it (unless we test against hundreds
of Python archives).
Instead we install pip using itself in wheel form. See:
https://github.com/pypa/pip/issues/2351#issuecomment-69994524
The new pip wheel assets on S3 were generated using:
```
$ pip download --no-cache pip==19.1.1
Collecting pip==19.1.1
Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-19.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ pip download --no-cache pip==20.0.2
Collecting pip==20.0.2
Downloading pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-20.0.2-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
(dryrun) upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
```
Previously the pip/setuptools/wheel install step was skipped so long
as Python hadn't just been clean installed (ie so long as not a new app,
emptied cache, Python upgrade, stack change) and pip was the expected
version.
This meant that setuptool/wheel could be the wrong version (or even just
not installed at all), and this would not be corrected.
Now, we now use pip itself to determine whether the installed packages
are up to date, since parsing pip's output is fragile (eg #1003) and
would be tedious given there would be three packages to check.
Unfortunately `get-pip.py` uses `--force-reinstall` which means
performing this step every time is not the no-op it would otherwise be,
but this will be resolved by switching away from `get-pip.py` in the
next commit.
Fixes#1000.
Fixes#1003.
Closes#999.
Since `get-pip.py` / pip will automatically detect and remove old
pip/setuptools versions if needed, so removing them manually is both not
necessary and slows down the build in the case where the pip version
changed, but setuptools remained the same.
Before:
- if `wheel` was not already installed, then `get-pip.py` would
automatically install the latest version on PyPI, which is `0.34.2`
(or `0.33.6` for Python 3.4).
- if `wheel` was already installed, then it was left unchanged
regardless of the version installed.
Now:
- if `wheel` is not already installed, then the same versions will be
installed as before, except these versions are pinned and will now not
change unexpectedly after future `wheel` releases.
- if `wheel` is already installed, then it's upgraded/downgraded to the
target version as needed.
Partly addresses #1000, though this change only helps builds where the
pip/setuptools/wheel install flow is triggered (currently only new apps
or ones where Python was purged or pip was not the correct version).
Since the wheel version is now known, it's output to the build log to
ease debugging and for parity with pip/setuptools.
The rest of #1000 will be fixed in later commits.
Since:
* we'll be updating setuptools soon, and newer setuptools has dropped
support for Python versions this buildpack needs to support. As such
if we continued to vendor setuptools, we would need to vendor at
least three different versions.
* we want to try and update setuptools more frequently than we have
in the past, which will mean more repo bloat from binary churn.
* we're still pinning to a specific version, meaning vendoring doesn't
have determinism benefits.
* setuptools is only fetched from PyPI for new installs (or where
versions have changed), so this doesn't increase build time, load on
PyPI, or reliance on PyPI in the common case.
* setuptools is already being inadvertently installed from PyPI prior to
being installed from the vendored copy (see #1001), so we're in effect
already using/depending on PyPI here.
* switching to storing setuptools on S3 wouldn't help reliability as
much as it would appear at first glance, since the later `pip install`
of customer dependencies will fail if PyPI is down anyway.
Since:
* "explicit is better than implicit"
* we'll soon be upgrading setuptools, and debugging breakage caused by
upgrades will be easier if versions are visible in the build log
Since:
* "explicit is better than implicit"
* we'll soon be upgrading pip, and debugging breakage caused by upgrades
will be easier if versions are visible in the build log
Closes#939.
* Don't clear the cache on first app deploy
* Add output for debugging cache behavior
* Debug output of changes, clean up whitespace
* Update hatchet to use latest getting started guide
* Clean up caching output logs
This output was confusing and unhelptul to most users
* Changelog
* Test if we need these lines
* dang fi
* Remove unnecessary code
* Remove confusing output of change
* Update log output
* Update test to match new expected log output
* Update changelog
With inspiration from @KevinBrolly, this patch uses the stack image
SQLite3 package but also still providing the dev headers and binary that
users may still be using today. The benefit is that we won't need to
rebuild all the python binaries for this to take affect. We can just
stop shipping SQLite3 from future binaries. In addition, we don't need
to worry about what version and when to update SQLite3 and maintaining
the packages ourselves.
This also includes updates to Python 2.7.15 and Python 3.6.6 so they can
rebuilt with the stack image dev headers instead of building our own
vendored SQLite3.
Python 3.7.0 is supported but not preferred given how new it is. As a
result, we don't want it to be the default, but we also don't want users
to be confused when upgrading to it.
Closes gh-728
Ed Morley
kennethreitz