The Check Changelog GitHub action now:
* is skipped for dependabot PRs
* uses a more descriptive job name than `check`
Closes @W-8052460@.
Closes @W-8098931@.
[skip changelog]
It's often useful to be able to run a compile outside of the unit or
Hatchet tests when developing the buildpack or debugging.
Whilst running the compile locally inside Docker won't fully replicate
a compile on the platform, for many use-cases it's close enough and
importantly gives a faster development feedback loop, since buildpack
changes don't have to be committed/pushed prior to triggering a compile.
Closes @W-8436406@.
So that it links to the changelog post now that it exists:
https://devcenter.heroku.com/changelog-items/1947
And to adjust the emphasis towards the migration guide
over the Geo buildpack repository link.
Closes @W-8391415@.
These variables were used by Anvil and/or the legacy `slug_compiler`
that predated cytokine.
I've not added a changelog entry since none were exported, so wouldn't
have been available to subprocesses anyway.
Closes @W-8387606@.
[skip changelog]
Switches the last consumers of it to a simpler utility function that
uses `pkgutil.find_loader()`:
https://docs.python.org/3/library/pkgutil.html#pkgutil.find_loader
Both of these consumers are covered by existing tests.
Then removes `sp-grep` and the remaining parts of `pip-pop`.
Closes @W-8208817@.
Since the last usage was removed in #1117.
Whilst this tool was intended for internal buildpack usage only, the
vendor directory is on `PATH`, so I've documented the removal in
CHANGELOG just in case.
Refs @W-8208817@.
Since the last usage was removed in #1113, and pip-grep is actually
broken on newer pip anyway (though helpfully silently ignores
exceptions, so one wouldn't know).
This is the first of the pip-pop removals, the rest will follow as the
last usages are switched over.
Whilst this tool was intended for internal buildpack usage only, the
vendor directory is on `PATH`, so I've documented the removal in
CHANGELOG just in case.
Refs @W-8208817@.
The pip-uninstall step has been unused for pip-using apps since #925,
since the buildpack now invalidates the entire package cache instead.
Whilst the step appears to still be used for pipenv-using apps, the
code is not run, since `SKIP_PIP_INSTALL=1` is set too early.
This bug was introduced in a334672a1a
which landed straight to `master` two days after the feature was
introduced in #650.
Longer term we should likely get pipenv installs to do something similar
to pip (invalidate the whole cache based on checksum of the lockfile),
however for now I'm removing this deadcode since it's the last consumer
of the `pip-diff` script which we want to remove.
Closes @W-8386830@.
[skip changelog]
The `BUILD_WITH_GEO_LIBRARIES` feature was removed in the previous PR,
and replaced with a warning that the feature is no longer supported.
After further thought I believe it would be best to make this warning
fatal, to prevent unexpected failures at runtime, if consumers of the
library either aren't invoked during the build, or were previously
installed/cached and were dynamically linked against the library.
Continuation of #1113 / @W-7654424@.
The standalone Geo buildpack offers more modern GDAL/GEOS/PROJ library
versions, and can be used by apps in all languages, not just Python:
https://github.com/heroku/heroku-geo-buildpack
As such the Python buildpack's undocumented built-in support was
deprecated back in April 2020, with a scheduled removal date of
6th October 2020:
https://devcenter.heroku.com/changelog-items/1759https://help.heroku.com/D5INLB1A/python-s-build_with_geo_libraries-legacy-feature-is-now-deprecated
Metrics show very few builds continuing to use the built-in support.
Apps with the `BUILD_WITH_GEO_LIBRARIES` env var set will now be shown a
warning directing them to the standalone buildpack, as well as apps that
hit GDAL related pip install errors but aren't using the env var.
This also moves us one step closer to being able to remove
the vendored copy of pip-pop (which is partially broken on
newer pip).
Closes @W-7654424@.
Since the stack is end of life and builds have been disabled:
https://devcenter.heroku.com/changelog-items/1943
There are only two temporarily exempted customers using Python, who
can switch to the Cedar-14 support branch if they still need to build
their Python apps (most of which haven't been built recently).
Closes @W-8054727@.
Now `make deploy-runtimes` will build binaries for Heroku-20 by default
too, without the need to pass it in via an explicit `STACKS=...`list.
Closes @W-8233698@.
[skip changelog]
Since fetching buildpack-stdlib from S3 has a number of disadvantages:
- it's not possible to grep the repo when trying to work out what
something from the stdlib is doing
- shellcheck can't fully scan the code, since it similarly doesn't
have the source
- another compile-time HTTP request that can fail due to transient
network issues and so reduce reliability
- dependency on the security/release-process of an additional bucket
Since the stdlib is small, doesn't often change, and is not a binary,
it's a great fit for just vendoring in the repo.
The `BIN_DIR` calculation added to `bin/utils` is based on the approach
mentioned here:
https://www.ostricher.com/2014/10/the-right-way-to-get-the-directory-of-a-bash-script/
...rather than copying the version in `bin/compile`, since the latter uses
`$0` so doesn't work with sourced scripts (such as `bin/utils`).
Closes W-8094463.
In #1011 the number of buildpack variables that are exported (and so
exposed to subprocesses) was reduced, since in general we don't want
to leak buildpack internals into end-user steps such as the pre/post
compile hooks.
However since this change, any buildpack metric emitted from within
a `sub_env` wrapper (which is a buildpack-stdlib utility function)
is missing its `buildpack.python` prefix.
This is because buildpack-stdlib:
* lazy-loads `BPLOG_PREFIX` (rather than doing so when initially
sourced, which is the approach used for `BUILDPACK_LOG_FILE`)
* doesn't check whether `BPLOG_PREFIX` is set before emitting metrics
See:
https://github.com/heroku/buildpack-stdlib/blob/v8/stdlib.sh
As a stop-gap until we either fix this in buildpack-stdlib (W-8095466),
or remove usages of the `sub_env` wrapper (since I think they are
counter-productive in this buildpack), I've added back the export
for `BPLOG_PREFIX`.
Fixes @W-8095436@.
* Fixes the "Installing <version>" assertions so that they don't false
positive against the "please upgrade to <version>" output.
* Removes modification of test fixtures during tests, since it can lead
to failures depending on test order, and confusion when debugging.
* Updates the PyPy version warning tests to use a slightly newer (but
still not latest) PyPy version, which means that the test now passes
on Cedar-14 and can be unskipped.
* Switches to using an empty requirements file for version tests that
duplicate the main test, to save spending time installing dependencies
unnecessarily.
* Switches the NLTK test to using the default buildpack Python version,
rather than an ancient Python 3.6.
* Skips the Python 3.7/3.8 tests on Cedar-14 rather than asserting
failure, since we know they'll never pass due to Cedar-14's libssl being
older than required.
* Removes redundant `testSqliteInstall` test since it duplicates the
Python version install tests.
Longer term I'll be moving many of the unit tests to Hatchet, however
this at least makes the tests more dependable in the meantime.
Closes @W-8060219@.
Closes @W-8176779@.
[skip changelog]
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
(for example to improve the sqlite3 handling, or to tweak the compile
flags used), and it will be easier to reason about the change if we
can guarantee only recent buildpack versions are using the assets
rather than several year old unmaintained forks.
The assets were synced from the old bucket using (minus the `--dryrun`):
```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
--dryrun \
--metadata-directive REPLACE \
--exclude "*" \
--include 'common/*' \
--include 'heroku-*/runtimes/*' \
--include 'heroku-*/libraries/vendor/gdal.tar.gz' \
--include 'heroku-*/libraries/vendor/geos.tar.gz' \
--include 'heroku-*/libraries/vendor/proj.tar.gz' \
--exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
--exclude '*/runtimes/*-opt.tar.gz' \
--exclude '*/runtimes/sqlite-free/*'
```
The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.
The Cedar-14 assets were not migrated since it's EOL next month.
The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.
Closes @W-8060097@.
The tests use `test/shunit2` so the copy under `vendor/` is redundant.
I've added to the CHANGELOG since `vendor/` is added by the buildpack to
`PATH` (something long term we should stop doing), increasing the chance
an app was accidentally relying upon `shunit2` (albeit still unlikely).
Closes @W-8145517@.
Previously the buildpack's S3 bucket was defined in two places - once
in `VENDOR_URL` and again during the pip installation step. This
duplication was necessary since `VENDOR_URL` also contained the stack's
name, whereas the pip use-case used a non-stack-specific S3 key prefix.
In order to:
* reduce this duplication
* simplify this buildpack's S3 bucket migration (where we'll soon be
needing the vary the bucket name and wouldn't want to have to
duplicate that logic in multiple places)
* allow overriding of the URL for the pip use-case
...the `VENDOR_URL` variable has been replaced with `S3_BASE_URL` which
no longer contains the stack name.
The user-configurable override has similarly been renamed from
`BUILDPACK_VENDOR_URL` to `BUILDPACK_S3_BASE_URL`. Note: As before,
this override cannot be set via standard app variables (see #989).
The unused `USE_STAGING_BINARIES` environment variable has been
removed, since it's a leftover from the project to stand up a staging S3 bucket.
It's redundant given the `BUILDPACK_S3_BASE_URL` variable.
Closes @W-8142401@.
* Moves all manual build steps to make targets, to simplify
the commands run, and reduce chance for error.
* Removes the need to remember to rebuild the builder
image by building it automatically prior to launching.
* Adds a new make target for deploying multiple runtime
versions at once to speed up the common case.
* Reduces repetition/superfluous content in documentation.
* Removes unused `S3_REGION` from `dockerenv.default`
(the contents of S3 buckets inherit the region of the bucket).
* Documents build dependencies in `requirements.txt`.
Closes @W-8119717@.
[skip changelog]
* Switches from using Python 2 to Python 3 for `bob-builder`
* Adds `--no-install-recommends` to reduce unnecessary packages
* Removes unnecessary pinning of pip for Heroku-18 build
* Moves `DEBIAN_FRONTEND` to RUN level env var, given:
https://serverfault.com/a/797318
Refs @W-8119717@.
[skip changelog]
There were previously 6 virtually identical tests checking the handling of
a non-existent Python version being specified in `runtime.txt`.
Only one is necessary - removing the rest will improve CI run time.
Closes @W-8110383@.
[skip changelog]
Previously the test for Python 3.8 version warnings was named the same
as an earlier test for Python 3.7, meaning the earlier test definition
was overwritten and so never run.
The later test has now been renamed to the correct version, and the
test ordering adjusted for consistency with the rest of the file.
Closes @W-8110123@.
[skip changelog]
Since the `tool.pipenv` event is being emitted twice per pipenv build,
inflating its usage.
This whole file could do with a massive refactor (4 levels deep of
conditionals is never a good sign), but that can wait until a later PR.
In the future it would also be great to have testing of metrics events.
Closes @W-8094963@.
Currently an app's Python version can be set via a few different means:
- explicitly by the user (via `runtime.txt` or `Pipfile.lock`)
- implicitly via the sticky versions feature (for existing apps)
- implicitly via default version for new apps / those with empty cache
In order to determine the priority of features like automatic Python
patch version upgrades for sticky-versioned apps, it's useful to have
metrics for these.
There were previously no tests for either the sticky versions feature,
or changing the Python version by updating the `runtime.txt` file, so
I've added some now (given that I updated the conditional to add the
metrics, so useful to have coverage).
I've also removed the confusing overwrite of `DEFAULT_PYTHON_VERSION`
with the cached version, and kept them as two separate variables.
Closes @W-8099632@.
Closes @W-8099645@.
Previously the metric events describing the chosen Python version were
only emitted when that Python version was installed, and not when it
was being used from the build cache (the common case).
Now the version is emitted for all builds, improving visibility into
the distribution of Python usage, and helping determine the priority
of features like opt-in automatic Python patch updates.
Closes @W-8059668@.
Ed Morley