It's often useful to be able to run a compile outside of the unit or
Hatchet tests when developing the buildpack or debugging.
Whilst running the compile locally inside Docker won't fully replicate
a compile on the platform, for many use-cases it's close enough and
importantly gives a faster development feedback loop, since buildpack
changes don't have to be committed/pushed prior to triggering a compile.
Closes @W-8436406@.
So that it links to the changelog post now that it exists:
https://devcenter.heroku.com/changelog-items/1947
And to adjust the emphasis towards the migration guide
over the Geo buildpack repository link.
Closes @W-8391415@.
These variables were used by Anvil and/or the legacy `slug_compiler`
that predated cytokine.
I've not added a changelog entry since none were exported, so wouldn't
have been available to subprocesses anyway.
Closes @W-8387606@.
[skip changelog]
Switches the last consumers of it to a simpler utility function that
uses `pkgutil.find_loader()`:
https://docs.python.org/3/library/pkgutil.html#pkgutil.find_loader
Both of these consumers are covered by existing tests.
Then removes `sp-grep` and the remaining parts of `pip-pop`.
Closes @W-8208817@.
Since the last usage was removed in #1117.
Whilst this tool was intended for internal buildpack usage only, the
vendor directory is on `PATH`, so I've documented the removal in
CHANGELOG just in case.
Refs @W-8208817@.
Since the last usage was removed in #1113, and pip-grep is actually
broken on newer pip anyway (though helpfully silently ignores
exceptions, so one wouldn't know).
This is the first of the pip-pop removals, the rest will follow as the
last usages are switched over.
Whilst this tool was intended for internal buildpack usage only, the
vendor directory is on `PATH`, so I've documented the removal in
CHANGELOG just in case.
Refs @W-8208817@.
The pip-uninstall step has been unused for pip-using apps since #925,
since the buildpack now invalidates the entire package cache instead.
Whilst the step appears to still be used for pipenv-using apps, the
code is not run, since `SKIP_PIP_INSTALL=1` is set too early.
This bug was introduced in a334672a1a
which landed straight to `master` two days after the feature was
introduced in #650.
Longer term we should likely get pipenv installs to do something similar
to pip (invalidate the whole cache based on checksum of the lockfile),
however for now I'm removing this deadcode since it's the last consumer
of the `pip-diff` script which we want to remove.
Closes @W-8386830@.
[skip changelog]
The `BUILD_WITH_GEO_LIBRARIES` feature was removed in the previous PR,
and replaced with a warning that the feature is no longer supported.
After further thought I believe it would be best to make this warning
fatal, to prevent unexpected failures at runtime, if consumers of the
library either aren't invoked during the build, or were previously
installed/cached and were dynamically linked against the library.
Continuation of #1113 / @W-7654424@.
The standalone Geo buildpack offers more modern GDAL/GEOS/PROJ library
versions, and can be used by apps in all languages, not just Python:
https://github.com/heroku/heroku-geo-buildpack
As such the Python buildpack's undocumented built-in support was
deprecated back in April 2020, with a scheduled removal date of
6th October 2020:
https://devcenter.heroku.com/changelog-items/1759https://help.heroku.com/D5INLB1A/python-s-build_with_geo_libraries-legacy-feature-is-now-deprecated
Metrics show very few builds continuing to use the built-in support.
Apps with the `BUILD_WITH_GEO_LIBRARIES` env var set will now be shown a
warning directing them to the standalone buildpack, as well as apps that
hit GDAL related pip install errors but aren't using the env var.
This also moves us one step closer to being able to remove
the vendored copy of pip-pop (which is partially broken on
newer pip).
Closes @W-7654424@.
Since the stack is end of life and builds have been disabled:
https://devcenter.heroku.com/changelog-items/1943
There are only two temporarily exempted customers using Python, who
can switch to the Cedar-14 support branch if they still need to build
their Python apps (most of which haven't been built recently).
Closes @W-8054727@.
Now `make deploy-runtimes` will build binaries for Heroku-20 by default
too, without the need to pass it in via an explicit `STACKS=...`list.
Closes @W-8233698@.
[skip changelog]
Since fetching buildpack-stdlib from S3 has a number of disadvantages:
- it's not possible to grep the repo when trying to work out what
something from the stdlib is doing
- shellcheck can't fully scan the code, since it similarly doesn't
have the source
- another compile-time HTTP request that can fail due to transient
network issues and so reduce reliability
- dependency on the security/release-process of an additional bucket
Since the stdlib is small, doesn't often change, and is not a binary,
it's a great fit for just vendoring in the repo.
The `BIN_DIR` calculation added to `bin/utils` is based on the approach
mentioned here:
https://www.ostricher.com/2014/10/the-right-way-to-get-the-directory-of-a-bash-script/
...rather than copying the version in `bin/compile`, since the latter uses
`$0` so doesn't work with sourced scripts (such as `bin/utils`).
Closes W-8094463.
In #1011 the number of buildpack variables that are exported (and so
exposed to subprocesses) was reduced, since in general we don't want
to leak buildpack internals into end-user steps such as the pre/post
compile hooks.
However since this change, any buildpack metric emitted from within
a `sub_env` wrapper (which is a buildpack-stdlib utility function)
is missing its `buildpack.python` prefix.
This is because buildpack-stdlib:
* lazy-loads `BPLOG_PREFIX` (rather than doing so when initially
sourced, which is the approach used for `BUILDPACK_LOG_FILE`)
* doesn't check whether `BPLOG_PREFIX` is set before emitting metrics
See:
https://github.com/heroku/buildpack-stdlib/blob/v8/stdlib.sh
As a stop-gap until we either fix this in buildpack-stdlib (W-8095466),
or remove usages of the `sub_env` wrapper (since I think they are
counter-productive in this buildpack), I've added back the export
for `BPLOG_PREFIX`.
Fixes @W-8095436@.
* Fixes the "Installing <version>" assertions so that they don't false
positive against the "please upgrade to <version>" output.
* Removes modification of test fixtures during tests, since it can lead
to failures depending on test order, and confusion when debugging.
* Updates the PyPy version warning tests to use a slightly newer (but
still not latest) PyPy version, which means that the test now passes
on Cedar-14 and can be unskipped.
* Switches to using an empty requirements file for version tests that
duplicate the main test, to save spending time installing dependencies
unnecessarily.
* Switches the NLTK test to using the default buildpack Python version,
rather than an ancient Python 3.6.
* Skips the Python 3.7/3.8 tests on Cedar-14 rather than asserting
failure, since we know they'll never pass due to Cedar-14's libssl being
older than required.
* Removes redundant `testSqliteInstall` test since it duplicates the
Python version install tests.
Longer term I'll be moving many of the unit tests to Hatchet, however
this at least makes the tests more dependable in the meantime.
Closes @W-8060219@.
Closes @W-8176779@.
[skip changelog]
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
(for example to improve the sqlite3 handling, or to tweak the compile
flags used), and it will be easier to reason about the change if we
can guarantee only recent buildpack versions are using the assets
rather than several year old unmaintained forks.
The assets were synced from the old bucket using (minus the `--dryrun`):
```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
--dryrun \
--metadata-directive REPLACE \
--exclude "*" \
--include 'common/*' \
--include 'heroku-*/runtimes/*' \
--include 'heroku-*/libraries/vendor/gdal.tar.gz' \
--include 'heroku-*/libraries/vendor/geos.tar.gz' \
--include 'heroku-*/libraries/vendor/proj.tar.gz' \
--exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
--exclude '*/runtimes/*-opt.tar.gz' \
--exclude '*/runtimes/sqlite-free/*'
```
The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.
The Cedar-14 assets were not migrated since it's EOL next month.
The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.
Closes @W-8060097@.
The tests use `test/shunit2` so the copy under `vendor/` is redundant.
I've added to the CHANGELOG since `vendor/` is added by the buildpack to
`PATH` (something long term we should stop doing), increasing the chance
an app was accidentally relying upon `shunit2` (albeit still unlikely).
Closes @W-8145517@.
Previously the buildpack's S3 bucket was defined in two places - once
in `VENDOR_URL` and again during the pip installation step. This
duplication was necessary since `VENDOR_URL` also contained the stack's
name, whereas the pip use-case used a non-stack-specific S3 key prefix.
In order to:
* reduce this duplication
* simplify this buildpack's S3 bucket migration (where we'll soon be
needing the vary the bucket name and wouldn't want to have to
duplicate that logic in multiple places)
* allow overriding of the URL for the pip use-case
...the `VENDOR_URL` variable has been replaced with `S3_BASE_URL` which
no longer contains the stack name.
The user-configurable override has similarly been renamed from
`BUILDPACK_VENDOR_URL` to `BUILDPACK_S3_BASE_URL`. Note: As before,
this override cannot be set via standard app variables (see #989).
The unused `USE_STAGING_BINARIES` environment variable has been
removed, since it's a leftover from the project to stand up a staging S3 bucket.
It's redundant given the `BUILDPACK_S3_BASE_URL` variable.
Closes @W-8142401@.
* Moves all manual build steps to make targets, to simplify
the commands run, and reduce chance for error.
* Removes the need to remember to rebuild the builder
image by building it automatically prior to launching.
* Adds a new make target for deploying multiple runtime
versions at once to speed up the common case.
* Reduces repetition/superfluous content in documentation.
* Removes unused `S3_REGION` from `dockerenv.default`
(the contents of S3 buckets inherit the region of the bucket).
* Documents build dependencies in `requirements.txt`.
Closes @W-8119717@.
[skip changelog]
* Switches from using Python 2 to Python 3 for `bob-builder`
* Adds `--no-install-recommends` to reduce unnecessary packages
* Removes unnecessary pinning of pip for Heroku-18 build
* Moves `DEBIAN_FRONTEND` to RUN level env var, given:
https://serverfault.com/a/797318
Refs @W-8119717@.
[skip changelog]
There were previously 6 virtually identical tests checking the handling of
a non-existent Python version being specified in `runtime.txt`.
Only one is necessary - removing the rest will improve CI run time.
Closes @W-8110383@.
[skip changelog]
Previously the test for Python 3.8 version warnings was named the same
as an earlier test for Python 3.7, meaning the earlier test definition
was overwritten and so never run.
The later test has now been renamed to the correct version, and the
test ordering adjusted for consistency with the rest of the file.
Closes @W-8110123@.
[skip changelog]
Since the `tool.pipenv` event is being emitted twice per pipenv build,
inflating its usage.
This whole file could do with a massive refactor (4 levels deep of
conditionals is never a good sign), but that can wait until a later PR.
In the future it would also be great to have testing of metrics events.
Closes @W-8094963@.
Currently an app's Python version can be set via a few different means:
- explicitly by the user (via `runtime.txt` or `Pipfile.lock`)
- implicitly via the sticky versions feature (for existing apps)
- implicitly via default version for new apps / those with empty cache
In order to determine the priority of features like automatic Python
patch version upgrades for sticky-versioned apps, it's useful to have
metrics for these.
There were previously no tests for either the sticky versions feature,
or changing the Python version by updating the `runtime.txt` file, so
I've added some now (given that I updated the conditional to add the
metrics, so useful to have coverage).
I've also removed the confusing overwrite of `DEFAULT_PYTHON_VERSION`
with the cached version, and kept them as two separate variables.
Closes @W-8099632@.
Closes @W-8099645@.
Previously the metric events describing the chosen Python version were
only emitted when that Python version was installed, and not when it
was being used from the build cache (the common case).
Now the version is emitted for all builds, improving visibility into
the distribution of Python usage, and helping determine the priority
of features like opt-in automatic Python patch updates.
Closes @W-8059668@.
The existing Python 3.4.10 archive actually contained Python 3.7.2,
since the version in the source URL was not updated when the file was
created in #813.
The build formula now uses the shared build script approach like all of
the other build scripts, which ensures the version can never get out of
sync (since it's extracted from the formula filename).
The build for Heroku-18 failed to compile `_ssl` properly (even though
the build exited zero) since Python 3.4.10 is old enough it doesn't work
well with libssl1.1. Installing `libssl1.0-dev` in the build image
locally resolved the issue - however we don't want to use that in the
future for newer Python, so I've not updated the `heroku-18.Dockerfile`.
In addition, with the rebuilt archives the tests now pass on Cedar-14,
so no longer need to be marked as failing.
Closes @W-7947035@.
The Hatchet run requires a valid Heroku login, the credentials for which
are set via Travis secure environment variables, which by design are not
revealed to PRs from forks.
The previous conditional wasn't working as intended - the Hatchet job
was still being triggered for forks from PRs.
The new conditional fixes this, and also means that forks could set
their own credentials via Travis environment variables if they wanted
a way to run the tests in CI on their own repo.
See:
https://docs.travis-ci.com/user/conditions-v1Fixes#1019.
Closes @W-7918482@.
[skip changelog]
Since they are now set via the Travis repository level secrets feature
instead. This both works around the Travis bug seen in #1045, and also
means its easier to set up Travis on forks, since otherwise the
`.travis.yml` secrets would overwrite the global secrets.
As part of this move the test account used has also been changed, and
will be documented here:
https://github.com/heroku/languages-team/blob/main/guides/create_test_users_for_ci.md#known-usernames
Closes @W-7949880@.
[skip changelog]
Since these variables refer to the latest version of PyPy, compared to
the similarly named `PYPY27` and `PYPY36` variables (ie same name except
without the underscore) which refer to the major/minor version only.
The similar names caused me to use the wrong one locally whilst working
on another PR, which was caught by tests but demonstrates why we should
rename them.
Closes @W-7935256@.
[skip changelog]
Since the unit tests instead use the utilities in this separate file:
https://github.com/heroku/heroku-buildpack-python/blob/419ef479969c4d5945f2c0620292229ef464f4c8/test/utils
A changelog entry has been added since whilst this file is for internal
testing only, the buildpack's `vendor/` directory is put on `PATH`, so
in theory it could have been called outside the buildpack (though this
seems extremely unlikely since the script isn't very useful externally).
Fixes#1027.
Closes @W-7918496@.
Since we don't use that tool (<https://pre-commit.com>), and there are
better alternatives should we want to expand coverage of these kind of
things.
Closes @W-7923935@.
[skip changelog]
It stopped being used as of #781.
```
$ rg ci-setup.sh --stats
0 matches
$ git-content-search ci-setup.sh
e7da63f update to newer hatchet integration
M .travis.yml
576def4 fix travis dependency blocker
M .travis.yml
$ git show e7da63f | rg ci-setup.sh -C 1
-before_install:
- - sudo bash etc/ci-setup.sh
+ - bundle exec hatchet ci:setup
```
Hatchet embeds its own setup script, which is called via the rake task:
https://github.com/heroku/hatchet/blob/v6.0.0/etc/ci_setup.rb
Closes @W-7923930@.
[skip changelog]
Previously if an app was using an older version of PyPy, the buildpack
would show a confusing "Could not find that version" message (even
though the version was found), when it really meant to warn about there
being a newer release available.
It looks like the version check messages were perhaps copied and pasted
from something else, but the message wording not updated at the time.
I've also added tests since there were none for this feature.
Fixes#1004.
Closes @W-7918745@.
So that any failures during `hatchet ci:setup` cause the build to fail
early, rather than try to proceed with running the Hatchet tests.
@W-7929878@
[skip changelog]
This will allow for one-way sync of GitHub issues in this repository
into our internal issues tracker, GUS. Issues are only synced when the
specified GitHub label is added.
In the future I may switch the chosen label to just be the standard
`t: bug` type labels, but for now I'm choosing a separate label so that
we have more control over what is synced.
See:
https://lwc-gus-bot.herokuapp.com/#getting-started
@W-7918433@
- Switches releases to always using H2 (`##`) rather than a mixture of
H1s (meaning multiple H1s in the same document) and H2s.
- Always refers to the versions as `vNNN` rather than sometimes without
the `v` prefix.
- Other formatting and typo fixes.
@W-7905079@
Updates pip from 20.0.2 to 20.1.1 for Python 2.7 and Python 3.5+:
https://pip.pypa.io/en/stable/news/#id40
The version used for Python 3.4 remains unchanged at 19.1.1, since it's
the last version of pip that supports it.
Pip has been updated to 20.1.1 rather than the recently released 20.2,
since the latter has a few regressions and even though these will be
fixed shortly in 20.2.1, we should let the changes soak for longer
before picking them up.
The `PIP_NO_PYTHON_VERSION_WARNING` environment variable has been set
(equivalent to passing `--no-python-version-warning`) to prevent the
Python 2.7 EOL warnings added in pip 20.1 from spamming the build log:
https://github.com/pypa/pip/blob/20.1.1/src/pip/_internal/cli/base_command.py#L139-L154
This was set via environment variable rather than CLI flag, since:
* otherwise we'd have to pass it to every pip invocation
* older pip (such as the 19.1.1 used by Python 3.4) doesn't support this
option and would error out due to an unknown CLI flag being passed,
unless we added conditional flags throughout.
The new pip wheel was uploaded to S3 using:
```
$ pip download --no-cache pip==20.1.1
Collecting pip==20.1.1
Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Saved ./pip-20.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
```
Fixes#1005.
@W-7659489@
Upgrades setuptools from 39.0.1 to:
- 44.1.1 for Python 2.7 (since it's the last supported version)
- 43.0.0 for Python 3.4 (since it's the last supported version)
- 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed)
https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1Fixes#949.
Closes#973.
They are now displayed in the build output (instead of being sent to
`/dev/null`) and fail the build early instead of failing later in
`bin/steps/pip-install`.
Fixes#1002.
Since the version check is redundant given we control/choose the version.
The pip cache is redundant since we instead cache site-packages. The pip
cache also ends up in `/app` so isn't included in the build cache anyway.
`get-pip.py` is no longer used, since:
- It uses `--force-reinstall`, which is unnecessary here and slows down
repeat builds (given we call pip install every time now). Trying to
work around this by using `get-pip.py` only for the initial install,
and real pip for subsequent updates would mean we lose protection
against cached broken installs, plus significantly increase the
version combinations test matrix.
- It means downloading pip twice (once embedded in `get-pip.py`, and
again during the install, since `get-pip.py` can't install the
embedded version directly).
- We would still have to manage several versions of `get-pip.py`, to
support older Pythons (once we upgrade to newer pip).
We don't use `ensurepip` since:
- not all of the previously generated Python runtimes on S3 include it.
- we would still have to upgrade pip/setuptools afterwards.
- the versions of pip/setuptools bundled with ensurepip differ greatly
depending on Python version, and we could easily start using a CLI
flag for the first pip install before upgrade that isn't supported on
all versions, without even knowing it (unless we test against hundreds
of Python archives).
Instead we install pip using itself in wheel form. See:
https://github.com/pypa/pip/issues/2351#issuecomment-69994524
The new pip wheel assets on S3 were generated using:
```
$ pip download --no-cache pip==19.1.1
Collecting pip==19.1.1
Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-19.1.1-py2.py3-none-any.whl
Successfully downloaded pip
$ pip download --no-cache pip==20.0.2
Collecting pip==20.0.2
Downloading pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Saved ./pip-20.0.2-py2.py3-none-any.whl
Successfully downloaded pip
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
(dryrun) upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl
upload: ./pip-20.0.2-py2.py3-none-any.whl to s3://lang-python/common/pip-20.0.2-py2.py3-none-any.whl
```
Previously the pip/setuptools/wheel install step was skipped so long
as Python hadn't just been clean installed (ie so long as not a new app,
emptied cache, Python upgrade, stack change) and pip was the expected
version.
This meant that setuptool/wheel could be the wrong version (or even just
not installed at all), and this would not be corrected.
Now, we now use pip itself to determine whether the installed packages
are up to date, since parsing pip's output is fragile (eg #1003) and
would be tedious given there would be three packages to check.
Unfortunately `get-pip.py` uses `--force-reinstall` which means
performing this step every time is not the no-op it would otherwise be,
but this will be resolved by switching away from `get-pip.py` in the
next commit.
Fixes#1000.
Fixes#1003.
Closes#999.
Since `get-pip.py` / pip will automatically detect and remove old
pip/setuptools versions if needed, so removing them manually is both not
necessary and slows down the build in the case where the pip version
changed, but setuptools remained the same.
Before:
- if `wheel` was not already installed, then `get-pip.py` would
automatically install the latest version on PyPI, which is `0.34.2`
(or `0.33.6` for Python 3.4).
- if `wheel` was already installed, then it was left unchanged
regardless of the version installed.
Now:
- if `wheel` is not already installed, then the same versions will be
installed as before, except these versions are pinned and will now not
change unexpectedly after future `wheel` releases.
- if `wheel` is already installed, then it's upgraded/downgraded to the
target version as needed.
Partly addresses #1000, though this change only helps builds where the
pip/setuptools/wheel install flow is triggered (currently only new apps
or ones where Python was purged or pip was not the correct version).
Since the wheel version is now known, it's output to the build log to
ease debugging and for parity with pip/setuptools.
The rest of #1000 will be fixed in later commits.
Since:
* we'll be updating setuptools soon, and newer setuptools has dropped
support for Python versions this buildpack needs to support. As such
if we continued to vendor setuptools, we would need to vendor at
least three different versions.
* we want to try and update setuptools more frequently than we have
in the past, which will mean more repo bloat from binary churn.
* we're still pinning to a specific version, meaning vendoring doesn't
have determinism benefits.
* setuptools is only fetched from PyPI for new installs (or where
versions have changed), so this doesn't increase build time, load on
PyPI, or reliance on PyPI in the common case.
* setuptools is already being inadvertently installed from PyPI prior to
being installed from the vendored copy (see #1001), so we're in effect
already using/depending on PyPI here.
* switching to storing setuptools on S3 wouldn't help reliability as
much as it would appear at first glance, since the later `pip install`
of customer dependencies will fail if PyPI is down anyway.
Since:
* "explicit is better than implicit"
* we'll soon be upgrading setuptools, and debugging breakage caused by
upgrades will be easier if versions are visible in the build log
Since:
* "explicit is better than implicit"
* we'll soon be upgrading pip, and debugging breakage caused by upgrades
will be easier if versions are visible in the build log
Closes#939.
The following env vars are no longer exposed to subprocesses run by the
buildpack (such as the `bin/pre_compile` and `bin/post_compile` hooks):
* `BPLOG_PREFIX`
* `CACHED_PYTHON_STACK`
* `DEFAULT_PYTHON_STACK`
* `DEFAULT_PYTHON_VERSION`
* `LATEST_27`
* `LATEST_34`
* `LATEST_35`
* `LATEST_36`
* `LATEST_37`
* `LATEST_38`
* `PIP_UPDATE`
* `PY27`
* `PY34`
* `PY35`
* `PY36`
* `PY37`
* `PYPY_27`
* `PYPY_36`
* `RECOMMENDED_PYTHON_VERSION`
* `WARNINGS_LOG`
There were previously no tests at all for the pre/post-compile hooks,
so I've added some now.
Fixes#1010.
This change (along with #1021, which skips an unnecessary docker build)
reduces the wall clock time from ~22 minutes to ~6 minutes. Even with
the additional overhead from increased parallelism, the combined job
duration (~50 minutes) has not increased due to the other time savings.
Changes:
- for the unit tests, each stack is now tested in its own job and so
tested in parallel
- the use of Travis stages has been removed, since by design it blocks
later tasks on earlier stages having completed - reducing parallelism
unnecessarily for this use case
- all jobs except for the Hatchet job now use Travis' `minimal` image,
and no longer install redundant Ruby + bundler
- the `sudo: {required,false}` references have been removed, since
Travis no longer supports its non-sudo container infrastructure so
ignores that option
Fixes#1018.
[skip changelog]
Previously `make test` ran all unit test suites against all stacks, which
would take up to an hour locally. This could be sped up by using one of
the stack-specific targets (such as `make test-heroku-18`), however
there was still no way to only run one of the test suites.
Now `make test` can be controlled more precisely using optional `STACK`
and `TEST_CMD` arguments, eg:
`make test STACK=heroku-16 TEST_CMD=test/versions`
Travis has now been made to use this feature, which unblocks future
Travis speedups (such as splitting the jobs up further in #1018) and
means on Travis the correct Docker image is now used (see #958).
The `tests.sh` script has been removed since it's unused after #839 and
redundant given the make targets.
Fixes#958.
Fixes#1020.
To prevent external environment variables from leaking into the tests,
which otherwise causes problems trying to write tests for #1011.
Several tests which were relying on this leak had to be fixed, so that
the env vars they were using are set using `ENV_DIR`, as happens in
production.
Fixes#1014.
Fixes#1015.
Adds support for:
* CPython 2.7.18, 3.5.9, 3.7.7 and 3.8.3
* PyPy 2.7 and 3.6, version 7.3.1
The binaries will need generating and uploading before CI will pass.
Note: Whilst the build script for CPython 3.8.3 did already exist in the
repository, it appears to have been accidentally created in #920, which
predated the existence of that version of Python - so the binaries do
not exist on S3.
The Heroku-18 Docker image tag has also been unpinned, since the new
libssl version is now available at runtime in all environments, so we
don't need to force building against the older version of the headers.
Fixes W-7582174.
Previously when a PR title was edited, the "check for changelog" script would not fire. This means if someone edited it to add a "[changelog skip]" that the check would still show to be failing. This PR adds additional triggers for the check so that when the PR is edited, the check will re-run.
* Update travis conditional for hatchet to check for HEROKU_API_ env vars
[changelog skip]
* remove check to skip hatchet in travis in favor of better travis config
The compile-time cryptography step that used to use the libffi archives
on S3 was removed in 2018:
https://github.com/heroku/heroku-buildpack-python/commit/c373e80c1285260e5adcbc855f54bbeb6999005c
...since the `cryptography` Python package now ships wheels.
The script is also incorrect, since similar to #964 it only skips builds
for Heroku-16, whereas all stacks since Cedar-14 include libffi-dev in
the build image, so don't need it built/uploaded for later vendoring.
Refs W-7485877.
The `libmemcached` package is available in the base stack image for all
stacks newer than `cedar-14`, so at buildpack compile time the vendor
step is skipped for those stacks:
https://github.com/heroku/heroku-buildpack-python/blob/106f2997fa124852a2a35ee8bfa604ad20c47988/bin/steps/pylibmc#L12-L15
As such, it is not necessary to run the libmemcached bob-builder formula
on newer stacks. The conditional has been updated so it correctly handles
heroku-18 and also the upcoming heroku-20.
An exit code of 1 has been used, otherwise `bob upload` will build and
then upload a zero byte archive to S3, which will go unused.
(This is in comparison to bob formulas that are nested, where an exit
code of 0 is actually desirable, since it allows skipping steps.)
Refs W-7485877.
Co-authored-by: Joe Kutner <jpkutner@gmail.com>
These aren't referenced anywhere in the repository, and contain older
versions of the dependencies than are declared in `requirements.txt`
(which itself is used).
Co-authored-by: Joe Kutner <jpkutner@gmail.com>
* Update travis config to only setup hatchet when running hatchet [changelog skip]
Fix a bug in the hatchet tests, and allow previous builds to finish before running the next test
* Add logging when skipping hatchet tests
Only skip hatchet tests on a forked PR
* Build on Travis only for master branch
* Upgrade from trusty to bionic on Travis
* Add support for Python 3.8 latest version
If the pip lock file only specifies `3.8` and no bug fix version, it should use Python LATEST_38.
* Update CHANGELOG.md
* Update changelog
Co-authored-by: Johannes Hoppe <info@johanneshoppe.com>
Co-authored-by: Casey <caseylfaist@gmail.com>
* Add a Hatchet test for python 3.8.2
* update changelog
* Update test to match build output
* Fix formatting and a syntax error in tests
* Fix syntax error in hatchet spec
* Don't clear the cache on first app deploy
* Add output for debugging cache behavior
* Debug output of changes, clean up whitespace
* Update hatchet to use latest getting started guide
* Clean up caching output logs
This output was confusing and unhelptul to most users
* Changelog
* Test if we need these lines
* dang fi
* Remove unnecessary code
* Remove confusing output of change
* Update log output
* Update test to match new expected log output
* Update changelog
This addresses an issue raised by @CaseyFeist during code review:
Updating pip for pipenv users or requiring them to update without a
heads up won't be a good experience (our version is old enough that
they'll need to uninstall and reinstall pipenv locally to successfully
update). If you can refactor this to stay pinned to current version for
pipenv users only, I should be able to accept this (and the related
project updates).
https://github.com/heroku/heroku-buildpack-python/pull/833#issuecomment-537758441
The pip-diff and pip-grep tools from the vendorized `pip-pop` package
import internal modules from pip. In pip >= 10, internal modules were
moved under `pip._internal`, breaking the imports. Use `try...except
ImportError` to handle both import paths.
Also, the interface of the `PackageFinder` class from one of these
modules changed. Provide a wrapper function to allow creating objects of
this type using the old interface.
This reverts commit c410fd36a6.
This is a temporary revert in order to release new python binaries first
and then roll this change out in a separate release. This keeps releases
smaller and better organized in case we need to roll back.
The goal of this PR is to add a github action that checks for the presence of a changelog entry.
It is better to add entries as a PR is merged instead of having to remember what was merged and generate a changelog at release time.
By automating this check, it's one less thing the maintainer has to remember, and it's one less thing a change might be blocked on i.e. "Looks good, but please add a changelog entry".
Let me know if you have any questions and Happy Friday!
The pip-diff tool from vendor/pip-pop is used to determine stale
requirements. When pip-diff encounters an unexpected failure, a count is
logged using mcount from heroku/buildpack-stdlib.
Due to a typo, mount(8) was invoked instead of mcount, with an invalid
argument.
* new recipe for new runtime
* add new runtime formula
* add test updates for new runtime release
* wrangle tests into submission
* update tests to use default_pythons
* delete commented code
Due to how the version checks work - via sorting, not actual comparison - this previously resulted in _always_ installing sqlite3, even though it was already bundled for lower versions of python. The second version check also encompasses 3.7.0+, so there is no need to respecify the check.
It's occasionally useful to see at a glance when releases were pushed to understand what release might have caused a customer's issues. This back-fills dates for the last few months of releases and sets a precedent that can be followed in the future.
With inspiration from @KevinBrolly, this patch uses the stack image
SQLite3 package but also still providing the dev headers and binary that
users may still be using today. The benefit is that we won't need to
rebuild all the python binaries for this to take affect. We can just
stop shipping SQLite3 from future binaries. In addition, we don't need
to worry about what version and when to update SQLite3 and maintaining
the packages ourselves.
This also includes updates to Python 2.7.15 and Python 3.6.6 so they can
rebuilt with the stack image dev headers instead of building our own
vendored SQLite3.
- Add stage to Travis CI config and update tests.sh script to recognize
it
- Update tests to assert there is no Python 2 on Heroku-18
- Update nltk fixture to use Python 3.6 so we can test it on all stacks
Closes gh-730
Python 3.7.0 is supported but not preferred given how new it is. As a
result, we don't want it to be the default, but we also don't want users
to be confused when upgrading to it.
Closes gh-728
* Since pipenv is being pinned now, output pinned version in steps
Signed-off-by: Yury V. Zaytsev <yury.zaytsev@moneymeets.com>
* Update README.md example output to match new reality
Signed-off-by: Yury V. Zaytsev <yury.zaytsev@moneymeets.com>
* Update pipenv
Set GDAL and GEOS library locaiton explicitly in environment
variables:
* GDAL_LIBRARY_PATH "/app/.heroku/vendor/lib/libgdal.so"
* GEOS_LIBRARY_PATH "/app/.heroku/vendor/lib/libgeos_c.so"
Django has to settings with the same name. The setup now works as
described here:
https://devcenter.heroku.com/articles/postgis#geodjango-setup
* only do this on heroku-16
* history
Signed-off-by: Kenneth Reitz <me@kennethreitz.org>
* code comment
Signed-off-by: Kenneth Reitz <me@kennethreitz.org>
Found by diffing the published buildpack archives and looking at the
git log to see what changed. It looks like the v108 entry was
accidentally added in #405, when the release being published there
was actually v107. The setuptools update mentioned there never
happened, since the `SETUPTOOLS_VERSION` variable was found to be
unused and instead removed.
Previously the contents of the file being searched was not output,
making it hard to know how to fix a test so that it would pass.
Before:
```
$ make test-heroku-16
...
testCffi
ASSERT:Expected </tmp/shunit.ayVGAO/tmp/output.6gMs/stdout> to contain <ZZZZZZZ>
testPylibmc
```
After:
```
$ make test-heroku-16
...
testCffi
ASSERT:Expected </tmp/shunit.FXWeuI/tmp/output.7Vfx/stdout> to contain <ZZZZZZZ>
! Warning: Your application is missing a Procfile. This file tells Heroku how to run your application.
! Learn more: https://devcenter.heroku.com/articles/procfile
-----> Installing python-3.6.2
-----> Installing pip
-----> Installing requirements with pip
Collecting cffi (from -r /tmp/testBoLG0/requirements.txt (line 1))
Downloading cffi-1.10.0-cp36-cp36m-manylinux1_x86_64.whl (406kB)
Collecting pycparser (from cffi->-r /tmp/testBoLG0/requirements.txt (line 1))
Downloading pycparser-2.18.tar.gz (245kB)
Installing collected packages: pycparser, cffi
Running setup.py install for pycparser: started
Running setup.py install for pycparser: finished with status 'done'
Successfully installed cffi-1.10.0 pycparser-2.18
testPylibmc
```
* NLTK support: Update test to use multiple corpora
So that the incorrect handling of multiple IDs seen in #444 would
have been caught.
Also switches to some of the smaller corpora, to reduce time spent
downloading during tests (see sizes on http://www.nltk.org/nltk_data/).
* NLTK support: Fix passing of multiple corpora identifiers
As part of fixing the shellcheck warnigns in #438, double quotes had
been placed around `$nltk_packages` passed to the `nltk.downloader`,
which causes multiple identifiers to be treated as though it were just
one identifier that contains spaces.
The docs for the shellcheck warning in question recommend using arrays
if the intended behaviour really is to split on spaces:
https://github.com/koalaman/shellcheck/wiki/SC2086#exceptions
As such, `readarray` has been used, which is present in bash >=4.
The `[*]` array form is used in the log message, to prevent shellcheck
warning SC2145, whereas `[@]` is used when passed to `nltk.downloader`
to ensure the array elements are unpacked as required.
Note: Both before and after this fix, using anything but unix line
endings in `nltk.txt` will also cause breakage.
Reduces the impact on Travis runtime caused by #438, by:
* Making the shellcheck job run on the container infra (`sudo: false`)
which has faster boot times.
* Running the heroku-16 and cedar-14 stack tests in parallel, since
previously they were separate stages (which are run in series by
design).
Reduces Travis end-to-end time from ~10 minutes to ~6 minutes.
* Test that the cache is invalidated when the stack changes
* Improve UX when clearing cache due to the stack changing
Now outputs a message informing that the cache was cleared, and
clears the cache first to avoid a redundant message about removing
an old Python version.
This copies the official Docker Python images approach of stripping
the Python test suite after building from source.
For Python 3.6.2 this reduces the output from 184MB to 121MB and
removes the following directories:
```
.heroku/python/lib/python3.6/tkinter/test
.heroku/python/lib/python3.6/ctypes/test
.heroku/python/lib/python3.6/unittest/test
.heroku/python/lib/python3.6/distutils/tests
.heroku/python/lib/python3.6/sqlite3/test
.heroku/python/lib/python3.6/lib2to3/tests
.heroku/python/lib/python3.6/test
```
For Python 2.7.13 this reduces the output from 127MB to 91MB and
removes the following directories:
```
.heroku/python/lib/python2.7/ctypes/test
.heroku/python/lib/python2.7/unittest/test
.heroku/python/lib/python2.7/distutils/tests
.heroku/python/lib/python2.7/bsddb/test
.heroku/python/lib/python2.7/lib-tk/test
.heroku/python/lib/python2.7/sqlite3/test
.heroku/python/lib/python2.7/lib2to3/tests
.heroku/python/lib/python2.7/json/tests
.heroku/python/lib/python2.7/email/test
.heroku/python/lib/python2.7/test
```
Fixes#424.
This buildpack includes some vendorized packages to ease installation.
jq license
----------
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
shunit2 license
---------------
@@ -85,53 +63,3 @@ The Free Software Foundation may publish revised and/or new versions of the GNU
Each version is given a distinguishing version number. If the Library as you received it specifies that a certain numbered version of the GNU Lesser General Public License “or any later version” applies to it, you have the option of following the terms and conditions either of that published version or of any later version published by the Free Software Foundation. If the Library as you received it does not specify a version number of the GNU Lesser General Public License, you may choose any version of the GNU Lesser General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide whether future versions of the GNU Lesser General Public License shall apply, that proxy's public statement of acceptance of any version is permanent authorization for you to choose that version for the Library.
get-pip.py license
------------------
Copyright (c) 2008-2016 The pip developers (see AUTHORS.txt file)
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
pip-pop license
---------------
The MIT License (MIT)
Copyright (c) 2014 Kenneth Reitz.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
This is the official [Heroku buildpack](https://devcenter.heroku.com/articles/buildpacks) for Python apps, powered by [pip](https://pip.pypa.io/) and other excellent software.
This is the official [Heroku buildpack](https://devcenter.heroku.com/articles/buildpacks) for Python apps.
Recommended web frameworks include **Django** and **Flask**. The recommended webserver is **Gunicorn**. There are no restrictions around what software can be used (as long as it's pip-installable). Web processes must bind to `$PORT`, and only the HTTP protocol is permitted for incoming connections.
Recommended web frameworks include **Django** and **Flask**, among others. The recommended webserver is **Gunicorn**. There are no restrictions around what software can be used (as long as it's pip-installable). Web processes must bind to `$PORT`, and only the HTTP protocol is permitted for incoming connections.
Some Python packages with obscure C dependencies (e.g. scipy) are [not compatible](https://devcenter.heroku.com/articles/python-c-deps).
Python packages with C dependencies that are not [available on the stack image](https://devcenter.heroku.com/articles/stack-packages) are generally not supported, unless `manylinux` wheels are provided by the package maintainers (common). For recommended solutions, check out [this article](https://devcenter.heroku.com/articles/python-c-deps) for more information.
See it in Action
----------------
```
$ ls
my-application requirements.txt runtime.txt
Deploying a Python application couldn't be easier:
Alternatively, you can provide a `setup.py` file, or a `Pipfile`.
Using `pipenv` will generate `runtime.txt` at build time if one of the field `python_version` or `python_full_version` is specified in the `requires` section of your `Pipfile`.
A `requirements.txt` file must be present at the root of your application's repository.
Specify a Buildpack Version
---------------------------
You can also specify the latest production release of this buildpack for upcoming builds of an existing application:
You can specify the latest production release of this buildpack for upcoming builds of an existing application:
$ heroku buildpacks:set heroku/python
@@ -43,14 +60,37 @@ You can also specify the latest production release of this buildpack for upcomin
Specify a Python Runtime
------------------------
Specific versions of the Python runtime can be specified with a `runtime.txt` file:
Supported runtime options include:
$ cat runtime.txt
python-3.6.1
-`python-3.9.0`
-`python-3.8.7`
-`python-3.7.9`
-`python-3.6.12`
-`python-2.7.18`
Runtime options include:
## Tests
-`python-2.7.13`
-`python-3.6.1`
-`pypy-5.7.1` (unsupported, experimental)
-`pypy3-5.5.1` (unsupported, experimental)
The buildpack tests use [Docker](https://www.docker.com/) to simulate
curl -L "${dep_url}" | tar jx -C "${OUT_PREFIX}" --strip-components 1 # extract to $OUT_PREFIX, drop the first directory level, which is the archive name
curl -fL "${dep_url}" | tar jx -C "${OUT_PREFIX}" --strip-components 1 # extract to $OUT_PREFIX, drop the first directory level, which is the archive name
curl -L "${dep_url}" | tar jx -C "${OUT_PREFIX}" --strip-components 1 # extract to $OUT_PREFIX, drop the first directory level, which is the archive name
curl -fL "${dep_url}" | tar jx -C "${OUT_PREFIX}" --strip-components 1 # extract to $OUT_PREFIX, drop the first directory level, which is the archive name
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ed Morley
kennethreitz