Commit Graph

39 Commits

Author SHA1 Message Date
Ed Morley 452443d420 Remove support for the Cedar-14 stack (#1110)
Since the stack is end of life and builds have been disabled:
https://devcenter.heroku.com/changelog-items/1943

There are only two temporarily exempted customers using Python, who
can switch to the Cedar-14 support branch if they still need to build
their Python apps (most of which haven't been built recently).

Closes @W-8054727@.
2020-11-10 13:58:33 +00:00
Ed Morley fcf696b835 Add support for Heroku-20 (#968)
This adds support for the upcoming Heroku-20 stack.

The Heroku-20 Dockerfile is identical to that for Heroku-18, other
than the base image, and stack-related env var changes.

The initial Python versions made available will be those in:
https://devcenter.heroku.com/articles/python-support#supported-runtimes
https://devcenter.heroku.com/articles/python-support#supported-pypy-runtimes

...minus CPython 2.7, since it's EOL.

Which are:
* `python-3.6.12`
* `python-3.7.9`
* `python-3.8.6`
* `python-3.9.0`
* `pypy2.7-7.3.2`
* `pypy3.6-7.3.2`

Note: Unlike CPython 2.7, the PyPy 2.7 branch is still supported:
https://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-support-python2

In addition, I've generated binaries for each patch release immediately
prior to the latest versions (with the exception of 3.9.0, since there
isn't one), otherwise it's not possible to run the "out of date Python"
warning tests.

The binaries were generated using the process here:
https://github.com/heroku/heroku-buildpack-python/blob/main/builds/README.md

Specifically:

```
make deploy-runtimes STACKS='heroku-20' \
  RUNTIMES='python-3.6.11 python-3.6.12 python-3.7.8 python-3.7.9 python-3.8.5 python-3.8.6 python-3.9.0 pypy2.7-7.3.1 pypy2.7-7.3.2 pypy3.6-7.3.1 pypy3.6-7.3.2' \
  ENV_FILE=...
```

Binaries for the GDAL/GEOS/PROJ feature have not been generated, since
it's deprecated and due for removal shortly:
https://help.heroku.com/D5INLB1A/python-s-build_with_geo_libraries-legacy-feature-is-now-deprecated

Note: Like the Python 3.9.0 release, this uses the new S3 bucket, so
apps will need to be using a recent version of the buildpack in order
to build on Heroku-20:
https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version

Closes @W-7485877@.
2020-10-07 19:44:33 +01:00
Ed Morley a98ef91566 Tests: Clean up the Python version unit tests (#1092)
* Fixes the "Installing <version>" assertions so that they don't false
  positive against the "please upgrade to <version>" output.
* Removes modification of test fixtures during tests, since it can lead
  to failures depending on test order, and confusion when debugging.
* Updates the PyPy version warning tests to use a slightly newer (but
  still not latest) PyPy version, which means that the test now passes
  on Cedar-14 and can be unskipped.
* Switches to using an empty requirements file for version tests that
  duplicate the main test, to save spending time installing dependencies
  unnecessarily.
* Switches the NLTK test to using the default buildpack Python version,
  rather than an ancient Python 3.6.
* Skips the Python 3.7/3.8 tests on Cedar-14 rather than asserting
  failure, since we know they'll never pass due to Cedar-14's libssl being
  older than required.
* Removes redundant `testSqliteInstall` test since it duplicates the
  Python version install tests.

Longer term I'll be moving many of the unit tests to Hatchet, however
this at least makes the tests more dependable in the meantime.

Closes @W-8060219@.
Closes @W-8176779@.

[skip changelog]
2020-10-07 15:10:20 +01:00
Ed Morley b1690e9f47 Add support for Python 3.9.0 (#1090)
https://pythoninsider.blogspot.com/2020/10/python-390-is-now-available-and-you-can.html
https://www.python.org/downloads/release/python-390/
https://docs.python.org/release/3.9.0/whatsnew/3.9.html

Binaries generated using:

```
make deploy-runtimes RUNTIMES='python-3.9.0' STACKS='heroku-16 heroku-18' ENV_FILE=...
```

Closes @W-7791272@.
2020-10-06 09:35:36 +01:00
Ed Morley ff8945c0c2 Tests: Make assertCapturedSuccess check stderr is empty again (#1087)
`assertCapturedSuccess` used to check that `stderr` was empty, until:
https://github.com/heroku/heroku-buildpack-python/commit/797652a75d69a1fece96a26bf4514fe9e9e1c020#diff-65c067a6f0a3aef292fb54ec21a1fe8cR98

Adding back the assertion exposes some new bugs, which I'll fix in
later PRs. In the meantime affected tests have been adjusted to use
a new `assertCapturedSuccessWithStdErr`.

The test harness output for the failing case has also been improved,
to ease debugging.

Closes @W-7918492@.

[skip changelog]
2020-10-01 17:29:13 +01:00
Ed Morley 4505968fcb Tests: Remove redundant Python version tests (#1075)
There were previously 6 virtually identical tests checking the handling of
a non-existent Python version being specified in `runtime.txt`. 

Only one is necessary - removing the rest will improve CI run time.

Closes @W-8110383@.

[skip changelog]
2020-09-22 18:48:31 +01:00
Ed Morley 37d1474bee Tests: Prevent shadowing of the Python 3.7 version warning test (#1074)
Previously the test for Python 3.8 version warnings was named the same
as an earlier test for Python 3.7, meaning the earlier test definition
was overwritten and so never run.

The later test has now been renamed to the correct version, and the
test ordering adjusted for consistency with the rest of the file.

Closes @W-8110123@.

[skip changelog]
2020-09-22 18:04:11 +01:00
Ed Morley eb6ee49dfe Emit metrics for how the Python version was chosen (#1069)
Currently an app's Python version can be set via a few different means:
- explicitly by the user (via `runtime.txt` or `Pipfile.lock`)
- implicitly via the sticky versions feature (for existing apps)
- implicitly via default version for new apps / those with empty cache

In order to determine the priority of features like automatic Python
patch version upgrades for sticky-versioned apps, it's useful to have
metrics for these.

There were previously no tests for either the sticky versions feature,
or changing the Python version by updating the `runtime.txt` file, so
I've added some now (given that I updated the conditional to add the
metrics, so useful to have coverage).

I've also removed the confusing overwrite of `DEFAULT_PYTHON_VERSION`
with the cached version, and kept them as two separate variables.

Closes @W-8099632@.
Closes @W-8099645@.
2020-09-18 18:48:57 +01:00
Ed Morley 3fa3f15d35 Remove unnecessary date conditional for Python 2 EOL warning (#1053)
Since we're past the end of 2019, so the conditional is always true.

Closes @W-7952394@.

[skip changelog]
2020-08-12 22:49:31 +01:00
Ed Morley eabe71d578 Update the Python 3.4.10 build script to use the correct Python version (#1048)
The existing Python 3.4.10 archive actually contained Python 3.7.2,
since the version in the source URL was not updated when the file was
created in #813.

The build formula now uses the shared build script approach like all of
the other build scripts, which ensures the version can never get out of
sync (since it's extracted from the formula filename).

The build for Heroku-18 failed to compile `_ssl` properly (even though
the build exited zero) since Python 3.4.10 is old enough it doesn't work
well with libssl1.1. Installing `libssl1.0-dev` in the build image
locally resolved the issue - however we don't want to use that in the
future for newer Python, so I've not updated the `heroku-18.Dockerfile`.

In addition, with the rebuilt archives the tests now pass on Cedar-14,
so no longer need to be marked as failing.

Closes @W-7947035@.
2020-08-12 15:19:31 +01:00
Ed Morley cba3201ac2 Rename confusing PYPY_* variables (#1044)
Since these variables refer to the latest version of PyPy, compared to
the similarly named `PYPY27` and `PYPY36` variables (ie same name except
without the underscore) which refer to the major/minor version only.

The similar names caused me to use the wrong one locally whilst working
on another PR, which was caught by tests but demonstrates why we should
rename them.

Closes @W-7935256@.

[skip changelog]
2020-08-11 19:31:53 +01:00
Ed Morley f508bd538d Fix the security update version check message for PyPy (#1040)
Previously if an app was using an older version of PyPy, the buildpack
would show a confusing "Could not find that version" message (even
though the version was found), when it really meant to warn about there
being a newer release available.

It looks like the version check messages were perhaps copied and pasted
from something else, but the message wording not updated at the time.

I've also added tests since there were none for this feature.

Fixes #1004.
Closes @W-7918745@.
2020-08-11 19:15:16 +01:00
Ed Morley fc6698e597 Update pip to 20.1.1 (#1030)
Updates pip from 20.0.2 to 20.1.1 for Python 2.7 and Python 3.5+:
https://pip.pypa.io/en/stable/news/#id40

The version used for Python 3.4 remains unchanged at 19.1.1, since it's
the last version of pip that supports it.

Pip has been updated to 20.1.1 rather than the recently released 20.2,
since the latter has a few regressions and even though these will be
fixed shortly in 20.2.1, we should let the changes soak for longer
before picking them up.

The `PIP_NO_PYTHON_VERSION_WARNING` environment variable has been set
(equivalent to passing `--no-python-version-warning`) to prevent the
Python 2.7 EOL warnings added in pip 20.1 from spamming the build log:
https://github.com/pypa/pip/blob/20.1.1/src/pip/_internal/cli/base_command.py#L139-L154

This was set via environment variable rather than CLI flag, since:
* otherwise we'd have to pass it to every pip invocation
* older pip (such as the 19.1.1 used by Python 3.4) doesn't support this
  option and would error out due to an unknown CLI flag being passed,
  unless we added conditional flags throughout.

The new pip wheel was uploaded to S3 using:

```
$ pip download --no-cache pip==20.1.1
Collecting pip==20.1.1
  Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
  Saved ./pip-20.1.1-py2.py3-none-any.whl
Successfully downloaded pip

$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun
(dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl

$ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read
upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl
```

Fixes #1005.
@W-7659489@
2020-08-03 19:45:41 +01:00
Ed Morley 6fa6feb75d Update setuptools (#1024)
Upgrades setuptools from 39.0.1 to:
- 44.1.1 for Python 2.7 (since it's the last supported version)
- 43.0.0 for Python 3.4 (since it's the last supported version)
- 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed)

https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1

Fixes #949.
Closes #973.
2020-08-03 18:36:01 +01:00
Ed Morley 2097eab028 Install an explicit version of wheel rather than latest (#1007)
Before:
- if `wheel` was not already installed, then `get-pip.py` would
  automatically install the latest version on PyPI, which is `0.34.2`
  (or `0.33.6` for Python 3.4).
- if `wheel` was already installed, then it was left unchanged
  regardless of the version installed.

Now:
- if `wheel` is not already installed, then the same versions will be
  installed as before, except these versions are pinned and will now not
  change unexpectedly after future `wheel` releases.
- if `wheel` is already installed, then it's upgraded/downgraded to the
  target version as needed.

Partly addresses #1000, though this change only helps builds where the
pip/setuptools/wheel install flow is triggered (currently only new apps
or ones where Python was purged or pip was not the correct version).

Since the wheel version is now known, it's output to the build log to
ease debugging and for parity with pip/setuptools.

The rest of #1000 will be fixed in later commits.
2020-07-29 19:11:35 +01:00
Ed Morley 47a8b4b3b9 Output the installed version of setuptools in the build log (#1007)
Since:
* "explicit is better than implicit"
* we'll soon be upgrading setuptools, and debugging breakage caused by
  upgrades will be easier if versions are visible in the build log
2020-07-29 19:11:35 +01:00
Ed Morley 157ce25694 Output the installed version of pip in the build log (#1007)
Since:
* "explicit is better than implicit"
* we'll soon be upgrading pip, and debugging breakage caused by upgrades
  will be easier if versions are visible in the build log

Closes #939.
2020-07-29 19:11:35 +01:00
Casey Faist e68f556e63 Build and release Python 3.8.2 2020-02-27 15:26:33 -07:00
Casey Faist 0057d19082 Typo in bash if-else block 2019-12-23 12:28:31 -05:00
Casey Faist 0419921227 bash is not python, so use bash block closers 2019-12-23 10:56:14 -05:00
Casey Faist 7d743e7998 Don't test for pypy on cedar 14 2019-12-23 09:57:07 -05:00
Casey Faist 0e1ac6217b correct test expectations 2019-12-23 01:22:30 -05:00
Casey Faist 848c846a3d Add tests for beta Pypy support 2019-12-23 00:17:22 -05:00
Casey Faist f189df5415 Warn about EOL on Python 2 install 2019-12-23 00:13:54 -05:00
Casey Faist 3c193bd475 update 3.4 warn test 2019-10-29 07:42:50 -07:00
Casey Faist a7f62496b1 correct python35 tests 2019-10-28 15:16:14 -07:00
Casey Faist f1b0f8c154 correct the pass/fail metric for python 34 2019-10-24 12:48:04 -07:00
Casey Faist f90dfbc12e update 3.4 tests to fail on heroku 18 2019-10-24 09:40:31 -07:00
Casey Faist 9ca0c4eb84 check for 3.5 on all stacks 2019-10-24 09:36:29 -07:00
Casey Faist dc694d4712 test for 3.5 warn should pass when using staged binaries 2019-10-24 08:52:33 -07:00
Casey 22e6239222 Merge branch 'master' into deprecated-pythons-test 2019-10-21 10:56:23 -07:00
Casey Faist 011f1e7e60 wardn case should succeed on all stacks 2019-10-18 09:57:09 -07:00
Casey Faist 0b804b186c correct the capture version too 2019-10-18 08:53:40 -07:00
Casey Faist 7ff7fa4462 add 3.8 tests 2019-10-17 15:01:32 -07:00
Casey Faist 47c6dbab32 add var to handle staged binaries 2019-10-14 11:15:06 -07:00
Casey Faist 950572132c remove test for sqlite on older versions of python 3.4 2019-10-09 13:57:06 -07:00
Casey Faist 0c78978e90 don't check python 3.4 2019-10-08 14:57:15 -07:00
Casey Faist 24e0858902 add sqlite tests 2019-10-08 14:08:04 -07:00
Casey Faist 0ffbf60882 split out tests to avoid timeouts 2019-09-10 11:53:33 -07:00