Since fetching buildpack-stdlib from S3 has a number of disadvantages:
- it's not possible to grep the repo when trying to work out what
something from the stdlib is doing
- shellcheck can't fully scan the code, since it similarly doesn't
have the source
- another compile-time HTTP request that can fail due to transient
network issues and so reduce reliability
- dependency on the security/release-process of an additional bucket
Since the stdlib is small, doesn't often change, and is not a binary,
it's a great fit for just vendoring in the repo.
The `BIN_DIR` calculation added to `bin/utils` is based on the approach
mentioned here:
https://www.ostricher.com/2014/10/the-right-way-to-get-the-directory-of-a-bash-script/
...rather than copying the version in `bin/compile`, since the latter uses
`$0` so doesn't work with sourced scripts (such as `bin/utils`).
Closes W-8094463.
In #1011 the number of buildpack variables that are exported (and so
exposed to subprocesses) was reduced, since in general we don't want
to leak buildpack internals into end-user steps such as the pre/post
compile hooks.
However since this change, any buildpack metric emitted from within
a `sub_env` wrapper (which is a buildpack-stdlib utility function)
is missing its `buildpack.python` prefix.
This is because buildpack-stdlib:
* lazy-loads `BPLOG_PREFIX` (rather than doing so when initially
sourced, which is the approach used for `BUILDPACK_LOG_FILE`)
* doesn't check whether `BPLOG_PREFIX` is set before emitting metrics
See:
https://github.com/heroku/buildpack-stdlib/blob/v8/stdlib.sh
As a stop-gap until we either fix this in buildpack-stdlib (W-8095466),
or remove usages of the `sub_env` wrapper (since I think they are
counter-productive in this buildpack), I've added back the export
for `BPLOG_PREFIX`.
Fixes @W-8095436@.
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
(for example to improve the sqlite3 handling, or to tweak the compile
flags used), and it will be easier to reason about the change if we
can guarantee only recent buildpack versions are using the assets
rather than several year old unmaintained forks.
The assets were synced from the old bucket using (minus the `--dryrun`):
```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
--dryrun \
--metadata-directive REPLACE \
--exclude "*" \
--include 'common/*' \
--include 'heroku-*/runtimes/*' \
--include 'heroku-*/libraries/vendor/gdal.tar.gz' \
--include 'heroku-*/libraries/vendor/geos.tar.gz' \
--include 'heroku-*/libraries/vendor/proj.tar.gz' \
--exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
--exclude '*/runtimes/*-opt.tar.gz' \
--exclude '*/runtimes/sqlite-free/*'
```
The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.
The Cedar-14 assets were not migrated since it's EOL next month.
The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.
Closes @W-8060097@.
Previously the buildpack's S3 bucket was defined in two places - once
in `VENDOR_URL` and again during the pip installation step. This
duplication was necessary since `VENDOR_URL` also contained the stack's
name, whereas the pip use-case used a non-stack-specific S3 key prefix.
In order to:
* reduce this duplication
* simplify this buildpack's S3 bucket migration (where we'll soon be
needing the vary the bucket name and wouldn't want to have to
duplicate that logic in multiple places)
* allow overriding of the URL for the pip use-case
...the `VENDOR_URL` variable has been replaced with `S3_BASE_URL` which
no longer contains the stack name.
The user-configurable override has similarly been renamed from
`BUILDPACK_VENDOR_URL` to `BUILDPACK_S3_BASE_URL`. Note: As before,
this override cannot be set via standard app variables (see #989).
The unused `USE_STAGING_BINARIES` environment variable has been
removed, since it's a leftover from the project to stand up a staging S3 bucket.
It's redundant given the `BUILDPACK_S3_BASE_URL` variable.
Closes @W-8142401@.
Currently an app's Python version can be set via a few different means:
- explicitly by the user (via `runtime.txt` or `Pipfile.lock`)
- implicitly via the sticky versions feature (for existing apps)
- implicitly via default version for new apps / those with empty cache
In order to determine the priority of features like automatic Python
patch version upgrades for sticky-versioned apps, it's useful to have
metrics for these.
There were previously no tests for either the sticky versions feature,
or changing the Python version by updating the `runtime.txt` file, so
I've added some now (given that I updated the conditional to add the
metrics, so useful to have coverage).
I've also removed the confusing overwrite of `DEFAULT_PYTHON_VERSION`
with the cached version, and kept them as two separate variables.
Closes @W-8099632@.
Closes @W-8099645@.
Updates pip from 20.0.2 to 20.1.1 for Python 2.7 and Python 3.5+:
https://pip.pypa.io/en/stable/news/#id40
The version used for Python 3.4 remains unchanged at 19.1.1, since it's
the last version of pip that supports it.
Pip has been updated to 20.1.1 rather than the recently released 20.2,
since the latter has a few regressions and even though these will be
fixed shortly in 20.2.1, we should let the changes soak for longer
before picking them up.
The `PIP_NO_PYTHON_VERSION_WARNING` environment variable has been set
(equivalent to passing `--no-python-version-warning`) to prevent the
Python 2.7 EOL warnings added in pip 20.1 from spamming the build log:
https://github.com/pypa/pip/blob/20.1.1/src/pip/_internal/cli/base_command.py#L139-L154
This was set via environment variable rather than CLI flag, since:
* otherwise we'd have to pass it to every pip invocation
* older pip (such as the 19.1.1 used by Python 3.4) doesn't support this
option and would error out due to an unknown CLI flag being passed,
unless we added conditional flags throughout.
The new pip wheel was uploaded to S3 using:
```
$ pip download --no-cache pip==20.1.1
Collecting pip==20.1.1
Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Saved ./pip-20.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
```
Fixes#1005.
@W-7659489@
The following env vars are no longer exposed to subprocesses run by the
buildpack (such as the `bin/pre_compile` and `bin/post_compile` hooks):
* `BPLOG_PREFIX`
* `CACHED_PYTHON_STACK`
* `DEFAULT_PYTHON_STACK`
* `DEFAULT_PYTHON_VERSION`
* `LATEST_27`
* `LATEST_34`
* `LATEST_35`
* `LATEST_36`
* `LATEST_37`
* `LATEST_38`
* `PIP_UPDATE`
* `PY27`
* `PY34`
* `PY35`
* `PY36`
* `PY37`
* `PYPY_27`
* `PYPY_36`
* `RECOMMENDED_PYTHON_VERSION`
* `WARNINGS_LOG`
There were previously no tests at all for the pre/post-compile hooks,
so I've added some now.
Fixes#1010.
* Don't clear the cache on first app deploy
* Add output for debugging cache behavior
* Debug output of changes, clean up whitespace
* Update hatchet to use latest getting started guide
* Clean up caching output logs
This output was confusing and unhelptul to most users
* Changelog
* Test if we need these lines
* dang fi
* Remove unnecessary code
* Remove confusing output of change
* Update log output
* Update test to match new expected log output
* Update changelog
This addresses an issue raised by @CaseyFeist during code review:
Updating pip for pipenv users or requiring them to update without a
heads up won't be a good experience (our version is old enough that
they'll need to uninstall and reinstall pipenv locally to successfully
update). If you can refactor this to stay pinned to current version for
pipenv users only, I should be able to accept this (and the related
project updates).
https://github.com/heroku/heroku-buildpack-python/pull/833#issuecomment-537758441
* new recipe for new runtime
* add new runtime formula
* add test updates for new runtime release
* wrangle tests into submission
* update tests to use default_pythons
* delete commented code
With inspiration from @KevinBrolly, this patch uses the stack image
SQLite3 package but also still providing the dev headers and binary that
users may still be using today. The benefit is that we won't need to
rebuild all the python binaries for this to take affect. We can just
stop shipping SQLite3 from future binaries. In addition, we don't need
to worry about what version and when to update SQLite3 and maintaining
the packages ourselves.
This also includes updates to Python 2.7.15 and Python 3.6.6 so they can
rebuilt with the stack image dev headers instead of building our own
vendored SQLite3.
Python 3.7.0 is supported but not preferred given how new it is. As a
result, we don't want it to be the default, but we also don't want users
to be confused when upgrading to it.
Closes gh-728
Ed Morley
kennethreitz