The existing Python 3.4.10 archive actually contained Python 3.7.2,
since the version in the source URL was not updated when the file was
created in #813.
The build formula now uses the shared build script approach like all of
the other build scripts, which ensures the version can never get out of
sync (since it's extracted from the formula filename).
The build for Heroku-18 failed to compile `_ssl` properly (even though
the build exited zero) since Python 3.4.10 is old enough it doesn't work
well with libssl1.1. Installing `libssl1.0-dev` in the build image
locally resolved the issue - however we don't want to use that in the
future for newer Python, so I've not updated the `heroku-18.Dockerfile`.
In addition, with the rebuilt archives the tests now pass on Cedar-14,
so no longer need to be marked as failing.
Closes @W-7947035@.
The Hatchet run requires a valid Heroku login, the credentials for which
are set via Travis secure environment variables, which by design are not
revealed to PRs from forks.
The previous conditional wasn't working as intended - the Hatchet job
was still being triggered for forks from PRs.
The new conditional fixes this, and also means that forks could set
their own credentials via Travis environment variables if they wanted
a way to run the tests in CI on their own repo.
See:
https://docs.travis-ci.com/user/conditions-v1Fixes#1019.
Closes @W-7918482@.
[skip changelog]
Since they are now set via the Travis repository level secrets feature
instead. This both works around the Travis bug seen in #1045, and also
means its easier to set up Travis on forks, since otherwise the
`.travis.yml` secrets would overwrite the global secrets.
As part of this move the test account used has also been changed, and
will be documented here:
https://github.com/heroku/languages-team/blob/main/guides/create_test_users_for_ci.md#known-usernames
Closes @W-7949880@.
[skip changelog]
Since these variables refer to the latest version of PyPy, compared to
the similarly named `PYPY27` and `PYPY36` variables (ie same name except
without the underscore) which refer to the major/minor version only.
The similar names caused me to use the wrong one locally whilst working
on another PR, which was caught by tests but demonstrates why we should
rename them.
Closes @W-7935256@.
[skip changelog]
Since the unit tests instead use the utilities in this separate file:
https://github.com/heroku/heroku-buildpack-python/blob/419ef479969c4d5945f2c0620292229ef464f4c8/test/utils
A changelog entry has been added since whilst this file is for internal
testing only, the buildpack's `vendor/` directory is put on `PATH`, so
in theory it could have been called outside the buildpack (though this
seems extremely unlikely since the script isn't very useful externally).
Fixes#1027.
Closes @W-7918496@.
Since we don't use that tool (<https://pre-commit.com>), and there are
better alternatives should we want to expand coverage of these kind of
things.
Closes @W-7923935@.
[skip changelog]
It stopped being used as of #781.
```
$ rg ci-setup.sh --stats
0 matches
$ git-content-search ci-setup.sh
e7da63f update to newer hatchet integration
M .travis.yml
576def4 fix travis dependency blocker
M .travis.yml
$ git show e7da63f | rg ci-setup.sh -C 1
-before_install:
- - sudo bash etc/ci-setup.sh
+ - bundle exec hatchet ci:setup
```
Hatchet embeds its own setup script, which is called via the rake task:
https://github.com/heroku/hatchet/blob/v6.0.0/etc/ci_setup.rb
Closes @W-7923930@.
[skip changelog]
Previously if an app was using an older version of PyPy, the buildpack
would show a confusing "Could not find that version" message (even
though the version was found), when it really meant to warn about there
being a newer release available.
It looks like the version check messages were perhaps copied and pasted
from something else, but the message wording not updated at the time.
I've also added tests since there were none for this feature.
Fixes#1004.
Closes @W-7918745@.
So that any failures during `hatchet ci:setup` cause the build to fail
early, rather than try to proceed with running the Hatchet tests.
@W-7929878@
[skip changelog]
This will allow for one-way sync of GitHub issues in this repository
into our internal issues tracker, GUS. Issues are only synced when the
specified GitHub label is added.
In the future I may switch the chosen label to just be the standard
`t: bug` type labels, but for now I'm choosing a separate label so that
we have more control over what is synced.
See:
https://lwc-gus-bot.herokuapp.com/#getting-started
@W-7918433@
- Switches releases to always using H2 (`##`) rather than a mixture of
H1s (meaning multiple H1s in the same document) and H2s.
- Always refers to the versions as `vNNN` rather than sometimes without
the `v` prefix.
- Other formatting and typo fixes.
@W-7905079@
Updates pip from 20.0.2 to 20.1.1 for Python 2.7 and Python 3.5+:
https://pip.pypa.io/en/stable/news/#id40
The version used for Python 3.4 remains unchanged at 19.1.1, since it's
the last version of pip that supports it.
Pip has been updated to 20.1.1 rather than the recently released 20.2,
since the latter has a few regressions and even though these will be
fixed shortly in 20.2.1, we should let the changes soak for longer
before picking them up.
The `PIP_NO_PYTHON_VERSION_WARNING` environment variable has been set
(equivalent to passing `--no-python-version-warning`) to prevent the
Python 2.7 EOL warnings added in pip 20.1 from spamming the build log:
https://github.com/pypa/pip/blob/20.1.1/src/pip/_internal/cli/base_command.py#L139-L154
This was set via environment variable rather than CLI flag, since:
* otherwise we'd have to pass it to every pip invocation
* older pip (such as the 19.1.1 used by Python 3.4) doesn't support this
option and would error out due to an unknown CLI flag being passed,
unless we added conditional flags throughout.
The new pip wheel was uploaded to S3 using:
```
$ pip download --no-cache pip==20.1.1
Collecting pip==20.1.1
Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Saved ./pip-20.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
```
Fixes#1005.
@W-7659489@
Upgrades setuptools from 39.0.1 to:
- 44.1.1 for Python 2.7 (since it's the last supported version)
- 43.0.0 for Python 3.4 (since it's the last supported version)
- 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed)
https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1Fixes#949.
Closes#973.
They are now displayed in the build output (instead of being sent to
`/dev/null`) and fail the build early instead of failing later in
`bin/steps/pip-install`.
Fixes#1002.
Since the version check is redundant given we control/choose the version.
The pip cache is redundant since we instead cache site-packages. The pip
cache also ends up in `/app` so isn't included in the build cache anyway.
`get-pip.py` is no longer used, since:
- It uses `--force-reinstall`, which is unnecessary here and slows down
repeat builds (given we call pip install every time now). Trying to
work around this by using `get-pip.py` only for the initial install,
and real pip for subsequent updates would mean we lose protection
against cached broken installs, plus significantly increase the
version combinations test matrix.
- It means downloading pip twice (once embedded in `get-pip.py`, and
again during the install, since `get-pip.py` can't install the
embedded version directly).
- We would still have to manage several versions of `get-pip.py`, to
support older Pythons (once we upgrade to newer pip).
We don't use `ensurepip` since:
- not all of the previously generated Python runtimes on S3 include it.
- we would still have to upgrade pip/setuptools afterwards.
- the versions of pip/setuptools bundled with ensurepip differ greatly
depending on Python version, and we could easily start using a CLI
flag for the first pip install before upgrade that isn't supported on
all versions, without even knowing it (unless we test against hundreds
of Python archives).
Instead we install pip using itself in wheel form. See:
https://github.com/pypa/pip/issues/2351#issuecomment-69994524
The new pip wheel assets on S3 were generated using:
```
$ pip download --no-cache pip==19.1.1
Collecting pip==19.1.1
Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-19.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ pip download --no-cache pip==20.0.2
Collecting pip==20.0.2
Downloading pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-20.0.2-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
(dryrun) upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
```
Previously the pip/setuptools/wheel install step was skipped so long
as Python hadn't just been clean installed (ie so long as not a new app,
emptied cache, Python upgrade, stack change) and pip was the expected
version.
This meant that setuptool/wheel could be the wrong version (or even just
not installed at all), and this would not be corrected.
Now, we now use pip itself to determine whether the installed packages
are up to date, since parsing pip's output is fragile (eg #1003) and
would be tedious given there would be three packages to check.
Unfortunately `get-pip.py` uses `--force-reinstall` which means
performing this step every time is not the no-op it would otherwise be,
but this will be resolved by switching away from `get-pip.py` in the
next commit.
Fixes#1000.
Fixes#1003.
Closes#999.
Since `get-pip.py` / pip will automatically detect and remove old
pip/setuptools versions if needed, so removing them manually is both not
necessary and slows down the build in the case where the pip version
changed, but setuptools remained the same.
Before:
- if `wheel` was not already installed, then `get-pip.py` would
automatically install the latest version on PyPI, which is `0.34.2`
(or `0.33.6` for Python 3.4).
- if `wheel` was already installed, then it was left unchanged
regardless of the version installed.
Now:
- if `wheel` is not already installed, then the same versions will be
installed as before, except these versions are pinned and will now not
change unexpectedly after future `wheel` releases.
- if `wheel` is already installed, then it's upgraded/downgraded to the
target version as needed.
Partly addresses #1000, though this change only helps builds where the
pip/setuptools/wheel install flow is triggered (currently only new apps
or ones where Python was purged or pip was not the correct version).
Since the wheel version is now known, it's output to the build log to
ease debugging and for parity with pip/setuptools.
The rest of #1000 will be fixed in later commits.
Since:
* we'll be updating setuptools soon, and newer setuptools has dropped
support for Python versions this buildpack needs to support. As such
if we continued to vendor setuptools, we would need to vendor at
least three different versions.
* we want to try and update setuptools more frequently than we have
in the past, which will mean more repo bloat from binary churn.
* we're still pinning to a specific version, meaning vendoring doesn't
have determinism benefits.
* setuptools is only fetched from PyPI for new installs (or where
versions have changed), so this doesn't increase build time, load on
PyPI, or reliance on PyPI in the common case.
* setuptools is already being inadvertently installed from PyPI prior to
being installed from the vendored copy (see #1001), so we're in effect
already using/depending on PyPI here.
* switching to storing setuptools on S3 wouldn't help reliability as
much as it would appear at first glance, since the later `pip install`
of customer dependencies will fail if PyPI is down anyway.
Since:
* "explicit is better than implicit"
* we'll soon be upgrading setuptools, and debugging breakage caused by
upgrades will be easier if versions are visible in the build log
Since:
* "explicit is better than implicit"
* we'll soon be upgrading pip, and debugging breakage caused by upgrades
will be easier if versions are visible in the build log
Closes#939.
The following env vars are no longer exposed to subprocesses run by the
buildpack (such as the `bin/pre_compile` and `bin/post_compile` hooks):
* `BPLOG_PREFIX`
* `CACHED_PYTHON_STACK`
* `DEFAULT_PYTHON_STACK`
* `DEFAULT_PYTHON_VERSION`
* `LATEST_27`
* `LATEST_34`
* `LATEST_35`
* `LATEST_36`
* `LATEST_37`
* `LATEST_38`
* `PIP_UPDATE`
* `PY27`
* `PY34`
* `PY35`
* `PY36`
* `PY37`
* `PYPY_27`
* `PYPY_36`
* `RECOMMENDED_PYTHON_VERSION`
* `WARNINGS_LOG`
There were previously no tests at all for the pre/post-compile hooks,
so I've added some now.
Fixes#1010.
This change (along with #1021, which skips an unnecessary docker build)
reduces the wall clock time from ~22 minutes to ~6 minutes. Even with
the additional overhead from increased parallelism, the combined job
duration (~50 minutes) has not increased due to the other time savings.
Changes:
- for the unit tests, each stack is now tested in its own job and so
tested in parallel
- the use of Travis stages has been removed, since by design it blocks
later tasks on earlier stages having completed - reducing parallelism
unnecessarily for this use case
- all jobs except for the Hatchet job now use Travis' `minimal` image,
and no longer install redundant Ruby + bundler
- the `sudo: {required,false}` references have been removed, since
Travis no longer supports its non-sudo container infrastructure so
ignores that option
Fixes#1018.
[skip changelog]
Previously `make test` ran all unit test suites against all stacks, which
would take up to an hour locally. This could be sped up by using one of
the stack-specific targets (such as `make test-heroku-18`), however
there was still no way to only run one of the test suites.
Now `make test` can be controlled more precisely using optional `STACK`
and `TEST_CMD` arguments, eg:
`make test STACK=heroku-16 TEST_CMD=test/versions`
Travis has now been made to use this feature, which unblocks future
Travis speedups (such as splitting the jobs up further in #1018) and
means on Travis the correct Docker image is now used (see #958).
The `tests.sh` script has been removed since it's unused after #839 and
redundant given the make targets.
Fixes#958.
Fixes#1020.
To prevent external environment variables from leaking into the tests,
which otherwise causes problems trying to write tests for #1011.
Several tests which were relying on this leak had to be fixed, so that
the env vars they were using are set using `ENV_DIR`, as happens in
production.
Fixes#1014.
Fixes#1015.
Ed Morley