Previously the pip/setuptools/wheel install step was skipped so long
as Python hadn't just been clean installed (ie so long as not a new app,
emptied cache, Python upgrade, stack change) and pip was the expected
version.
This meant that setuptool/wheel could be the wrong version (or even just
not installed at all), and this would not be corrected.
Now, we now use pip itself to determine whether the installed packages
are up to date, since parsing pip's output is fragile (eg #1003) and
would be tedious given there would be three packages to check.
Unfortunately `get-pip.py` uses `--force-reinstall` which means
performing this step every time is not the no-op it would otherwise be,
but this will be resolved by switching away from `get-pip.py` in the
next commit.
Fixes#1000.
Fixes#1003.
Closes#999.
Since `get-pip.py` / pip will automatically detect and remove old
pip/setuptools versions if needed, so removing them manually is both not
necessary and slows down the build in the case where the pip version
changed, but setuptools remained the same.
Before:
- if `wheel` was not already installed, then `get-pip.py` would
automatically install the latest version on PyPI, which is `0.34.2`
(or `0.33.6` for Python 3.4).
- if `wheel` was already installed, then it was left unchanged
regardless of the version installed.
Now:
- if `wheel` is not already installed, then the same versions will be
installed as before, except these versions are pinned and will now not
change unexpectedly after future `wheel` releases.
- if `wheel` is already installed, then it's upgraded/downgraded to the
target version as needed.
Partly addresses #1000, though this change only helps builds where the
pip/setuptools/wheel install flow is triggered (currently only new apps
or ones where Python was purged or pip was not the correct version).
Since the wheel version is now known, it's output to the build log to
ease debugging and for parity with pip/setuptools.
The rest of #1000 will be fixed in later commits.
Since:
* we'll be updating setuptools soon, and newer setuptools has dropped
support for Python versions this buildpack needs to support. As such
if we continued to vendor setuptools, we would need to vendor at
least three different versions.
* we want to try and update setuptools more frequently than we have
in the past, which will mean more repo bloat from binary churn.
* we're still pinning to a specific version, meaning vendoring doesn't
have determinism benefits.
* setuptools is only fetched from PyPI for new installs (or where
versions have changed), so this doesn't increase build time, load on
PyPI, or reliance on PyPI in the common case.
* setuptools is already being inadvertently installed from PyPI prior to
being installed from the vendored copy (see #1001), so we're in effect
already using/depending on PyPI here.
* switching to storing setuptools on S3 wouldn't help reliability as
much as it would appear at first glance, since the later `pip install`
of customer dependencies will fail if PyPI is down anyway.
Since:
* "explicit is better than implicit"
* we'll soon be upgrading setuptools, and debugging breakage caused by
upgrades will be easier if versions are visible in the build log
Since:
* "explicit is better than implicit"
* we'll soon be upgrading pip, and debugging breakage caused by upgrades
will be easier if versions are visible in the build log
Closes#939.
The following env vars are no longer exposed to subprocesses run by the
buildpack (such as the `bin/pre_compile` and `bin/post_compile` hooks):
* `BPLOG_PREFIX`
* `CACHED_PYTHON_STACK`
* `DEFAULT_PYTHON_STACK`
* `DEFAULT_PYTHON_VERSION`
* `LATEST_27`
* `LATEST_34`
* `LATEST_35`
* `LATEST_36`
* `LATEST_37`
* `LATEST_38`
* `PIP_UPDATE`
* `PY27`
* `PY34`
* `PY35`
* `PY36`
* `PY37`
* `PYPY_27`
* `PYPY_36`
* `RECOMMENDED_PYTHON_VERSION`
* `WARNINGS_LOG`
There were previously no tests at all for the pre/post-compile hooks,
so I've added some now.
Fixes#1010.
This change (along with #1021, which skips an unnecessary docker build)
reduces the wall clock time from ~22 minutes to ~6 minutes. Even with
the additional overhead from increased parallelism, the combined job
duration (~50 minutes) has not increased due to the other time savings.
Changes:
- for the unit tests, each stack is now tested in its own job and so
tested in parallel
- the use of Travis stages has been removed, since by design it blocks
later tasks on earlier stages having completed - reducing parallelism
unnecessarily for this use case
- all jobs except for the Hatchet job now use Travis' `minimal` image,
and no longer install redundant Ruby + bundler
- the `sudo: {required,false}` references have been removed, since
Travis no longer supports its non-sudo container infrastructure so
ignores that option
Fixes#1018.
[skip changelog]
Previously `make test` ran all unit test suites against all stacks, which
would take up to an hour locally. This could be sped up by using one of
the stack-specific targets (such as `make test-heroku-18`), however
there was still no way to only run one of the test suites.
Now `make test` can be controlled more precisely using optional `STACK`
and `TEST_CMD` arguments, eg:
`make test STACK=heroku-16 TEST_CMD=test/versions`
Travis has now been made to use this feature, which unblocks future
Travis speedups (such as splitting the jobs up further in #1018) and
means on Travis the correct Docker image is now used (see #958).
The `tests.sh` script has been removed since it's unused after #839 and
redundant given the make targets.
Fixes#958.
Fixes#1020.
To prevent external environment variables from leaking into the tests,
which otherwise causes problems trying to write tests for #1011.
Several tests which were relying on this leak had to be fixed, so that
the env vars they were using are set using `ENV_DIR`, as happens in
production.
Fixes#1014.
Fixes#1015.
Adds support for:
* CPython 2.7.18, 3.5.9, 3.7.7 and 3.8.3
* PyPy 2.7 and 3.6, version 7.3.1
The binaries will need generating and uploading before CI will pass.
Note: Whilst the build script for CPython 3.8.3 did already exist in the
repository, it appears to have been accidentally created in #920, which
predated the existence of that version of Python - so the binaries do
not exist on S3.
The Heroku-18 Docker image tag has also been unpinned, since the new
libssl version is now available at runtime in all environments, so we
don't need to force building against the older version of the headers.
Fixes W-7582174.
Previously when a PR title was edited, the "check for changelog" script would not fire. This means if someone edited it to add a "[changelog skip]" that the check would still show to be failing. This PR adds additional triggers for the check so that when the PR is edited, the check will re-run.
* Update travis conditional for hatchet to check for HEROKU_API_ env vars
[changelog skip]
* remove check to skip hatchet in travis in favor of better travis config
The compile-time cryptography step that used to use the libffi archives
on S3 was removed in 2018:
https://github.com/heroku/heroku-buildpack-python/commit/c373e80c1285260e5adcbc855f54bbeb6999005c
...since the `cryptography` Python package now ships wheels.
The script is also incorrect, since similar to #964 it only skips builds
for Heroku-16, whereas all stacks since Cedar-14 include libffi-dev in
the build image, so don't need it built/uploaded for later vendoring.
Refs W-7485877.
The `libmemcached` package is available in the base stack image for all
stacks newer than `cedar-14`, so at buildpack compile time the vendor
step is skipped for those stacks:
https://github.com/heroku/heroku-buildpack-python/blob/106f2997fa124852a2a35ee8bfa604ad20c47988/bin/steps/pylibmc#L12-L15
As such, it is not necessary to run the libmemcached bob-builder formula
on newer stacks. The conditional has been updated so it correctly handles
heroku-18 and also the upcoming heroku-20.
An exit code of 1 has been used, otherwise `bob upload` will build and
then upload a zero byte archive to S3, which will go unused.
(This is in comparison to bob formulas that are nested, where an exit
code of 0 is actually desirable, since it allows skipping steps.)
Refs W-7485877.
Co-authored-by: Joe Kutner <jpkutner@gmail.com>
These aren't referenced anywhere in the repository, and contain older
versions of the dependencies than are declared in `requirements.txt`
(which itself is used).
Co-authored-by: Joe Kutner <jpkutner@gmail.com>
* Update travis config to only setup hatchet when running hatchet [changelog skip]
Fix a bug in the hatchet tests, and allow previous builds to finish before running the next test
* Add logging when skipping hatchet tests
Only skip hatchet tests on a forked PR
* Build on Travis only for master branch
* Upgrade from trusty to bionic on Travis
* Add support for Python 3.8 latest version
If the pip lock file only specifies `3.8` and no bug fix version, it should use Python LATEST_38.
* Update CHANGELOG.md
* Update changelog
Co-authored-by: Johannes Hoppe <info@johanneshoppe.com>
Co-authored-by: Casey <caseylfaist@gmail.com>
* Add a Hatchet test for python 3.8.2
* update changelog
* Update test to match build output
* Fix formatting and a syntax error in tests
* Fix syntax error in hatchet spec
* Don't clear the cache on first app deploy
* Add output for debugging cache behavior
* Debug output of changes, clean up whitespace
* Update hatchet to use latest getting started guide
* Clean up caching output logs
This output was confusing and unhelptul to most users
* Changelog
* Test if we need these lines
* dang fi
* Remove unnecessary code
* Remove confusing output of change
* Update log output
* Update test to match new expected log output
* Update changelog
Ed Morley