Since:
* We want the S3 bucket to be owned by a different AWS account and it's
not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
(for example to improve the sqlite3 handling, or to tweak the compile
flags used), and it will be easier to reason about the change if we
can guarantee only recent buildpack versions are using the assets
rather than several year old unmaintained forks.
The assets were synced from the old bucket using (minus the `--dryrun`):
```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
--dryrun \
--metadata-directive REPLACE \
--exclude "*" \
--include 'common/*' \
--include 'heroku-*/runtimes/*' \
--include 'heroku-*/libraries/vendor/gdal.tar.gz' \
--include 'heroku-*/libraries/vendor/geos.tar.gz' \
--include 'heroku-*/libraries/vendor/proj.tar.gz' \
--exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
--exclude '*/runtimes/*-opt.tar.gz' \
--exclude '*/runtimes/sqlite-free/*'
```
The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.
The Cedar-14 assets were not migrated since it's EOL next month.
The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.
Closes @W-8060097@.
The tests use `test/shunit2` so the copy under `vendor/` is redundant.
I've added to the CHANGELOG since `vendor/` is added by the buildpack to
`PATH` (something long term we should stop doing), increasing the chance
an app was accidentally relying upon `shunit2` (albeit still unlikely).
Closes @W-8145517@.
Previously the buildpack's S3 bucket was defined in two places - once
in `VENDOR_URL` and again during the pip installation step. This
duplication was necessary since `VENDOR_URL` also contained the stack's
name, whereas the pip use-case used a non-stack-specific S3 key prefix.
In order to:
* reduce this duplication
* simplify this buildpack's S3 bucket migration (where we'll soon be
needing the vary the bucket name and wouldn't want to have to
duplicate that logic in multiple places)
* allow overriding of the URL for the pip use-case
...the `VENDOR_URL` variable has been replaced with `S3_BASE_URL` which
no longer contains the stack name.
The user-configurable override has similarly been renamed from
`BUILDPACK_VENDOR_URL` to `BUILDPACK_S3_BASE_URL`. Note: As before,
this override cannot be set via standard app variables (see #989).
The unused `USE_STAGING_BINARIES` environment variable has been
removed, since it's a leftover from the project to stand up a staging S3 bucket.
It's redundant given the `BUILDPACK_S3_BASE_URL` variable.
Closes @W-8142401@.
* Moves all manual build steps to make targets, to simplify
the commands run, and reduce chance for error.
* Removes the need to remember to rebuild the builder
image by building it automatically prior to launching.
* Adds a new make target for deploying multiple runtime
versions at once to speed up the common case.
* Reduces repetition/superfluous content in documentation.
* Removes unused `S3_REGION` from `dockerenv.default`
(the contents of S3 buckets inherit the region of the bucket).
* Documents build dependencies in `requirements.txt`.
Closes @W-8119717@.
[skip changelog]
* Switches from using Python 2 to Python 3 for `bob-builder`
* Adds `--no-install-recommends` to reduce unnecessary packages
* Removes unnecessary pinning of pip for Heroku-18 build
* Moves `DEBIAN_FRONTEND` to RUN level env var, given:
https://serverfault.com/a/797318
Refs @W-8119717@.
[skip changelog]
There were previously 6 virtually identical tests checking the handling of
a non-existent Python version being specified in `runtime.txt`.
Only one is necessary - removing the rest will improve CI run time.
Closes @W-8110383@.
[skip changelog]
Previously the test for Python 3.8 version warnings was named the same
as an earlier test for Python 3.7, meaning the earlier test definition
was overwritten and so never run.
The later test has now been renamed to the correct version, and the
test ordering adjusted for consistency with the rest of the file.
Closes @W-8110123@.
[skip changelog]
Since the `tool.pipenv` event is being emitted twice per pipenv build,
inflating its usage.
This whole file could do with a massive refactor (4 levels deep of
conditionals is never a good sign), but that can wait until a later PR.
In the future it would also be great to have testing of metrics events.
Closes @W-8094963@.
Currently an app's Python version can be set via a few different means:
- explicitly by the user (via `runtime.txt` or `Pipfile.lock`)
- implicitly via the sticky versions feature (for existing apps)
- implicitly via default version for new apps / those with empty cache
In order to determine the priority of features like automatic Python
patch version upgrades for sticky-versioned apps, it's useful to have
metrics for these.
There were previously no tests for either the sticky versions feature,
or changing the Python version by updating the `runtime.txt` file, so
I've added some now (given that I updated the conditional to add the
metrics, so useful to have coverage).
I've also removed the confusing overwrite of `DEFAULT_PYTHON_VERSION`
with the cached version, and kept them as two separate variables.
Closes @W-8099632@.
Closes @W-8099645@.
Previously the metric events describing the chosen Python version were
only emitted when that Python version was installed, and not when it
was being used from the build cache (the common case).
Now the version is emitted for all builds, improving visibility into
the distribution of Python usage, and helping determine the priority
of features like opt-in automatic Python patch updates.
Closes @W-8059668@.
The existing Python 3.4.10 archive actually contained Python 3.7.2,
since the version in the source URL was not updated when the file was
created in #813.
The build formula now uses the shared build script approach like all of
the other build scripts, which ensures the version can never get out of
sync (since it's extracted from the formula filename).
The build for Heroku-18 failed to compile `_ssl` properly (even though
the build exited zero) since Python 3.4.10 is old enough it doesn't work
well with libssl1.1. Installing `libssl1.0-dev` in the build image
locally resolved the issue - however we don't want to use that in the
future for newer Python, so I've not updated the `heroku-18.Dockerfile`.
In addition, with the rebuilt archives the tests now pass on Cedar-14,
so no longer need to be marked as failing.
Closes @W-7947035@.
The Hatchet run requires a valid Heroku login, the credentials for which
are set via Travis secure environment variables, which by design are not
revealed to PRs from forks.
The previous conditional wasn't working as intended - the Hatchet job
was still being triggered for forks from PRs.
The new conditional fixes this, and also means that forks could set
their own credentials via Travis environment variables if they wanted
a way to run the tests in CI on their own repo.
See:
https://docs.travis-ci.com/user/conditions-v1Fixes#1019.
Closes @W-7918482@.
[skip changelog]
Since they are now set via the Travis repository level secrets feature
instead. This both works around the Travis bug seen in #1045, and also
means its easier to set up Travis on forks, since otherwise the
`.travis.yml` secrets would overwrite the global secrets.
As part of this move the test account used has also been changed, and
will be documented here:
https://github.com/heroku/languages-team/blob/main/guides/create_test_users_for_ci.md#known-usernames
Closes @W-7949880@.
[skip changelog]
Since these variables refer to the latest version of PyPy, compared to
the similarly named `PYPY27` and `PYPY36` variables (ie same name except
without the underscore) which refer to the major/minor version only.
The similar names caused me to use the wrong one locally whilst working
on another PR, which was caught by tests but demonstrates why we should
rename them.
Closes @W-7935256@.
[skip changelog]
Since the unit tests instead use the utilities in this separate file:
https://github.com/heroku/heroku-buildpack-python/blob/419ef479969c4d5945f2c0620292229ef464f4c8/test/utils
A changelog entry has been added since whilst this file is for internal
testing only, the buildpack's `vendor/` directory is put on `PATH`, so
in theory it could have been called outside the buildpack (though this
seems extremely unlikely since the script isn't very useful externally).
Fixes#1027.
Closes @W-7918496@.
Since we don't use that tool (<https://pre-commit.com>), and there are
better alternatives should we want to expand coverage of these kind of
things.
Closes @W-7923935@.
[skip changelog]
It stopped being used as of #781.
```
$ rg ci-setup.sh --stats
0 matches
$ git-content-search ci-setup.sh
e7da63f update to newer hatchet integration
M .travis.yml
576def4 fix travis dependency blocker
M .travis.yml
$ git show e7da63f | rg ci-setup.sh -C 1
-before_install:
- - sudo bash etc/ci-setup.sh
+ - bundle exec hatchet ci:setup
```
Hatchet embeds its own setup script, which is called via the rake task:
https://github.com/heroku/hatchet/blob/v6.0.0/etc/ci_setup.rb
Closes @W-7923930@.
[skip changelog]
Previously if an app was using an older version of PyPy, the buildpack
would show a confusing "Could not find that version" message (even
though the version was found), when it really meant to warn about there
being a newer release available.
It looks like the version check messages were perhaps copied and pasted
from something else, but the message wording not updated at the time.
I've also added tests since there were none for this feature.
Fixes#1004.
Closes @W-7918745@.
So that any failures during `hatchet ci:setup` cause the build to fail
early, rather than try to proceed with running the Hatchet tests.
@W-7929878@
[skip changelog]
Ed Morley