The `BUILD_WITH_GEO_LIBRARIES` feature was removed in the previous PR,
and replaced with a warning that the feature is no longer supported.
After further thought I believe it would be best to make this warning
fatal, to prevent unexpected failures at runtime, if consumers of the
library either aren't invoked during the build, or were previously
installed/cached and were dynamically linked against the library.
Continuation of #1113 / @W-7654424@.
The standalone Geo buildpack offers more modern GDAL/GEOS/PROJ library
versions, and can be used by apps in all languages, not just Python:
https://github.com/heroku/heroku-geo-buildpack
As such the Python buildpack's undocumented built-in support was
deprecated back in April 2020, with a scheduled removal date of
6th October 2020:
https://devcenter.heroku.com/changelog-items/1759https://help.heroku.com/D5INLB1A/python-s-build_with_geo_libraries-legacy-feature-is-now-deprecated
Metrics show very few builds continuing to use the built-in support.
Apps with the `BUILD_WITH_GEO_LIBRARIES` env var set will now be shown a
warning directing them to the standalone buildpack, as well as apps that
hit GDAL related pip install errors but aren't using the env var.
This also moves us one step closer to being able to remove
the vendored copy of pip-pop (which is partially broken on
newer pip).
Closes @W-7654424@.
Since the stack is end of life and builds have been disabled:
https://devcenter.heroku.com/changelog-items/1943
There are only two temporarily exempted customers using Python, who
can switch to the Cedar-14 support branch if they still need to build
their Python apps (most of which haven't been built recently).
Closes @W-8054727@.
Since fetching buildpack-stdlib from S3 has a number of disadvantages:
- it's not possible to grep the repo when trying to work out what
something from the stdlib is doing
- shellcheck can't fully scan the code, since it similarly doesn't
have the source
- another compile-time HTTP request that can fail due to transient
network issues and so reduce reliability
- dependency on the security/release-process of an additional bucket
Since the stdlib is small, doesn't often change, and is not a binary,
it's a great fit for just vendoring in the repo.
The `BIN_DIR` calculation added to `bin/utils` is based on the approach
mentioned here:
https://www.ostricher.com/2014/10/the-right-way-to-get-the-directory-of-a-bash-script/
...rather than copying the version in `bin/compile`, since the latter uses
`$0` so doesn't work with sourced scripts (such as `bin/utils`).
Closes W-8094463.
In #1011 the number of buildpack variables that are exported (and so
exposed to subprocesses) was reduced, since in general we don't want
to leak buildpack internals into end-user steps such as the pre/post
compile hooks.
However since this change, any buildpack metric emitted from within
a `sub_env` wrapper (which is a buildpack-stdlib utility function)
is missing its `buildpack.python` prefix.
This is because buildpack-stdlib:
* lazy-loads `BPLOG_PREFIX` (rather than doing so when initially
sourced, which is the approach used for `BUILDPACK_LOG_FILE`)
* doesn't check whether `BPLOG_PREFIX` is set before emitting metrics
See:
https://github.com/heroku/buildpack-stdlib/blob/v8/stdlib.sh
As a stop-gap until we either fix this in buildpack-stdlib (W-8095466),
or remove usages of the `sub_env` wrapper (since I think they are
counter-productive in this buildpack), I've added back the export
for `BPLOG_PREFIX`.
Fixes @W-8095436@.
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
(for example to improve the sqlite3 handling, or to tweak the compile
flags used), and it will be easier to reason about the change if we
can guarantee only recent buildpack versions are using the assets
rather than several year old unmaintained forks.
The assets were synced from the old bucket using (minus the `--dryrun`):
```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
--dryrun \
--metadata-directive REPLACE \
--exclude "*" \
--include 'common/*' \
--include 'heroku-*/runtimes/*' \
--include 'heroku-*/libraries/vendor/gdal.tar.gz' \
--include 'heroku-*/libraries/vendor/geos.tar.gz' \
--include 'heroku-*/libraries/vendor/proj.tar.gz' \
--exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
--exclude '*/runtimes/*-opt.tar.gz' \
--exclude '*/runtimes/sqlite-free/*'
```
The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.
The Cedar-14 assets were not migrated since it's EOL next month.
The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.
Closes @W-8060097@.
Previously the buildpack's S3 bucket was defined in two places - once
in `VENDOR_URL` and again during the pip installation step. This
duplication was necessary since `VENDOR_URL` also contained the stack's
name, whereas the pip use-case used a non-stack-specific S3 key prefix.
In order to:
* reduce this duplication
* simplify this buildpack's S3 bucket migration (where we'll soon be
needing the vary the bucket name and wouldn't want to have to
duplicate that logic in multiple places)
* allow overriding of the URL for the pip use-case
...the `VENDOR_URL` variable has been replaced with `S3_BASE_URL` which
no longer contains the stack name.
The user-configurable override has similarly been renamed from
`BUILDPACK_VENDOR_URL` to `BUILDPACK_S3_BASE_URL`. Note: As before,
this override cannot be set via standard app variables (see #989).
The unused `USE_STAGING_BINARIES` environment variable has been
removed, since it's a leftover from the project to stand up a staging S3 bucket.
It's redundant given the `BUILDPACK_S3_BASE_URL` variable.
Closes @W-8142401@.
Since the `tool.pipenv` event is being emitted twice per pipenv build,
inflating its usage.
This whole file could do with a massive refactor (4 levels deep of
conditionals is never a good sign), but that can wait until a later PR.
In the future it would also be great to have testing of metrics events.
Closes @W-8094963@.
Currently an app's Python version can be set via a few different means:
- explicitly by the user (via `runtime.txt` or `Pipfile.lock`)
- implicitly via the sticky versions feature (for existing apps)
- implicitly via default version for new apps / those with empty cache
In order to determine the priority of features like automatic Python
patch version upgrades for sticky-versioned apps, it's useful to have
metrics for these.
There were previously no tests for either the sticky versions feature,
or changing the Python version by updating the `runtime.txt` file, so
I've added some now (given that I updated the conditional to add the
metrics, so useful to have coverage).
I've also removed the confusing overwrite of `DEFAULT_PYTHON_VERSION`
with the cached version, and kept them as two separate variables.
Closes @W-8099632@.
Closes @W-8099645@.
Previously the metric events describing the chosen Python version were
only emitted when that Python version was installed, and not when it
was being used from the build cache (the common case).
Now the version is emitted for all builds, improving visibility into
the distribution of Python usage, and helping determine the priority
of features like opt-in automatic Python patch updates.
Closes @W-8059668@.
Since these variables refer to the latest version of PyPy, compared to
the similarly named `PYPY27` and `PYPY36` variables (ie same name except
without the underscore) which refer to the major/minor version only.
The similar names caused me to use the wrong one locally whilst working
on another PR, which was caught by tests but demonstrates why we should
rename them.
Closes @W-7935256@.
[skip changelog]
Previously if an app was using an older version of PyPy, the buildpack
would show a confusing "Could not find that version" message (even
though the version was found), when it really meant to warn about there
being a newer release available.
It looks like the version check messages were perhaps copied and pasted
from something else, but the message wording not updated at the time.
I've also added tests since there were none for this feature.
Fixes#1004.
Closes @W-7918745@.
Updates pip from 20.0.2 to 20.1.1 for Python 2.7 and Python 3.5+:
https://pip.pypa.io/en/stable/news/#id40
The version used for Python 3.4 remains unchanged at 19.1.1, since it's
the last version of pip that supports it.
Pip has been updated to 20.1.1 rather than the recently released 20.2,
since the latter has a few regressions and even though these will be
fixed shortly in 20.2.1, we should let the changes soak for longer
before picking them up.
The `PIP_NO_PYTHON_VERSION_WARNING` environment variable has been set
(equivalent to passing `--no-python-version-warning`) to prevent the
Python 2.7 EOL warnings added in pip 20.1 from spamming the build log:
https://github.com/pypa/pip/blob/20.1.1/src/pip/_internal/cli/base_command.py#L139-L154
This was set via environment variable rather than CLI flag, since:
* otherwise we'd have to pass it to every pip invocation
* older pip (such as the 19.1.1 used by Python 3.4) doesn't support this
option and would error out due to an unknown CLI flag being passed,
unless we added conditional flags throughout.
The new pip wheel was uploaded to S3 using:
```
$ pip download --no-cache pip==20.1.1
Collecting pip==20.1.1
Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Saved ./pip-20.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
```
Fixes#1005.
@W-7659489@
Upgrades setuptools from 39.0.1 to:
- 44.1.1 for Python 2.7 (since it's the last supported version)
- 43.0.0 for Python 3.4 (since it's the last supported version)
- 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed)
https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1Fixes#949.
Closes#973.
They are now displayed in the build output (instead of being sent to
`/dev/null`) and fail the build early instead of failing later in
`bin/steps/pip-install`.
Fixes#1002.
Since the version check is redundant given we control/choose the version.
The pip cache is redundant since we instead cache site-packages. The pip
cache also ends up in `/app` so isn't included in the build cache anyway.
`get-pip.py` is no longer used, since:
- It uses `--force-reinstall`, which is unnecessary here and slows down
repeat builds (given we call pip install every time now). Trying to
work around this by using `get-pip.py` only for the initial install,
and real pip for subsequent updates would mean we lose protection
against cached broken installs, plus significantly increase the
version combinations test matrix.
- It means downloading pip twice (once embedded in `get-pip.py`, and
again during the install, since `get-pip.py` can't install the
embedded version directly).
- We would still have to manage several versions of `get-pip.py`, to
support older Pythons (once we upgrade to newer pip).
We don't use `ensurepip` since:
- not all of the previously generated Python runtimes on S3 include it.
- we would still have to upgrade pip/setuptools afterwards.
- the versions of pip/setuptools bundled with ensurepip differ greatly
depending on Python version, and we could easily start using a CLI
flag for the first pip install before upgrade that isn't supported on
all versions, without even knowing it (unless we test against hundreds
of Python archives).
Instead we install pip using itself in wheel form. See:
https://github.com/pypa/pip/issues/2351#issuecomment-69994524
The new pip wheel assets on S3 were generated using:
```
$ pip download --no-cache pip==19.1.1
Collecting pip==19.1.1
Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-19.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ pip download --no-cache pip==20.0.2
Collecting pip==20.0.2
Downloading pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-20.0.2-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
(dryrun) upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
```
Previously the pip/setuptools/wheel install step was skipped so long
as Python hadn't just been clean installed (ie so long as not a new app,
emptied cache, Python upgrade, stack change) and pip was the expected
version.
This meant that setuptool/wheel could be the wrong version (or even just
not installed at all), and this would not be corrected.
Now, we now use pip itself to determine whether the installed packages
are up to date, since parsing pip's output is fragile (eg #1003) and
would be tedious given there would be three packages to check.
Unfortunately `get-pip.py` uses `--force-reinstall` which means
performing this step every time is not the no-op it would otherwise be,
but this will be resolved by switching away from `get-pip.py` in the
next commit.
Fixes#1000.
Fixes#1003.
Closes#999.
Since `get-pip.py` / pip will automatically detect and remove old
pip/setuptools versions if needed, so removing them manually is both not
necessary and slows down the build in the case where the pip version
changed, but setuptools remained the same.
Before:
- if `wheel` was not already installed, then `get-pip.py` would
automatically install the latest version on PyPI, which is `0.34.2`
(or `0.33.6` for Python 3.4).
- if `wheel` was already installed, then it was left unchanged
regardless of the version installed.
Now:
- if `wheel` is not already installed, then the same versions will be
installed as before, except these versions are pinned and will now not
change unexpectedly after future `wheel` releases.
- if `wheel` is already installed, then it's upgraded/downgraded to the
target version as needed.
Partly addresses #1000, though this change only helps builds where the
pip/setuptools/wheel install flow is triggered (currently only new apps
or ones where Python was purged or pip was not the correct version).
Since the wheel version is now known, it's output to the build log to
ease debugging and for parity with pip/setuptools.
The rest of #1000 will be fixed in later commits.
Since:
* we'll be updating setuptools soon, and newer setuptools has dropped
support for Python versions this buildpack needs to support. As such
if we continued to vendor setuptools, we would need to vendor at
least three different versions.
* we want to try and update setuptools more frequently than we have
in the past, which will mean more repo bloat from binary churn.
* we're still pinning to a specific version, meaning vendoring doesn't
have determinism benefits.
* setuptools is only fetched from PyPI for new installs (or where
versions have changed), so this doesn't increase build time, load on
PyPI, or reliance on PyPI in the common case.
* setuptools is already being inadvertently installed from PyPI prior to
being installed from the vendored copy (see #1001), so we're in effect
already using/depending on PyPI here.
* switching to storing setuptools on S3 wouldn't help reliability as
much as it would appear at first glance, since the later `pip install`
of customer dependencies will fail if PyPI is down anyway.
Since:
* "explicit is better than implicit"
* we'll soon be upgrading setuptools, and debugging breakage caused by
upgrades will be easier if versions are visible in the build log
Since:
* "explicit is better than implicit"
* we'll soon be upgrading pip, and debugging breakage caused by upgrades
will be easier if versions are visible in the build log
Closes#939.
The following env vars are no longer exposed to subprocesses run by the
buildpack (such as the `bin/pre_compile` and `bin/post_compile` hooks):
* `BPLOG_PREFIX`
* `CACHED_PYTHON_STACK`
* `DEFAULT_PYTHON_STACK`
* `DEFAULT_PYTHON_VERSION`
* `LATEST_27`
* `LATEST_34`
* `LATEST_35`
* `LATEST_36`
* `LATEST_37`
* `LATEST_38`
* `PIP_UPDATE`
* `PY27`
* `PY34`
* `PY35`
* `PY36`
* `PY37`
* `PYPY_27`
* `PYPY_36`
* `RECOMMENDED_PYTHON_VERSION`
* `WARNINGS_LOG`
There were previously no tests at all for the pre/post-compile hooks,
so I've added some now.
Fixes#1010.
Adds support for:
* CPython 2.7.18, 3.5.9, 3.7.7 and 3.8.3
* PyPy 2.7 and 3.6, version 7.3.1
The binaries will need generating and uploading before CI will pass.
Note: Whilst the build script for CPython 3.8.3 did already exist in the
repository, it appears to have been accidentally created in #920, which
predated the existence of that version of Python - so the binaries do
not exist on S3.
The Heroku-18 Docker image tag has also been unpinned, since the new
libssl version is now available at runtime in all environments, so we
don't need to force building against the older version of the headers.
Fixes W-7582174.
* Add support for Python 3.8 latest version
If the pip lock file only specifies `3.8` and no bug fix version, it should use Python LATEST_38.
* Update CHANGELOG.md
* Update changelog
Co-authored-by: Johannes Hoppe <info@johanneshoppe.com>
Co-authored-by: Casey <caseylfaist@gmail.com>
* Don't clear the cache on first app deploy
* Add output for debugging cache behavior
* Debug output of changes, clean up whitespace
* Update hatchet to use latest getting started guide
* Clean up caching output logs
This output was confusing and unhelptul to most users
* Changelog
* Test if we need these lines
* dang fi
* Remove unnecessary code
* Remove confusing output of change
* Update log output
* Update test to match new expected log output
* Update changelog
Ed Morley