v79

Add PyPy 5.0.1

CHANGELOG.md

@@ -1,5 +1,27 @@
 # Python Buildpack Changelog
 
+## v79 (2016-03-22)
+
+Compatibility improvements with heroku-apt-buildpack.
+
+## v78 (2016-03-18)
+
+Added automatic configuration of Gunicorn's `FORWARDED_ALLOW_IPS` setting.
+
+Improved detection of libffi dependency when using bcrypt via `Django[bcrypt]`.
+
+Improved GDAL support.
+
+- GDAL dependency detection now checks for pygdal and is case-insensitive.
+- The vendored GDAL library has been updated to 1.11.1.
+- GDAL bootstrapping now also installs the GEOS and Proj.4 libraries.
+
+Updated pip to 8.1.1 and setuptools to 20.3.
+
+## v77 (2016-02-10)
+
+Improvements to warnings and minor bugfix.
+
 ## v76 (2016-02-08)
 
 Improved Django collectstatic support.

LICENSE

@@ -1,6 +1,6 @@
 MIT License:
 
-Copyright (C) 2013 Heroku, Inc.
+Copyright (C) 2016 Heroku, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

Readme.md

@@ -1,54 +1,54 @@
 # Heroku Buildpack: Python
-![python-banner](https://cloud.githubusercontent.com/assets/51578/8914205/ecf2047c-346b-11e5-98c5-42547f9f4410.jpg)
+![python](https://cloud.githubusercontent.com/assets/51578/13712821/b68a42ce-e793-11e5-96b0-d8eb978137ba.png)
 
-This is a [Heroku buildpack](http://devcenter.heroku.com/articles/buildpacks) for Python apps, powered by [pip](http://www.pip-installer.org/).
+This is the official [Heroku buildpack](https://devcenter.heroku.com/articles/buildpacks) for Python apps, powered by [pip](https://pip.pypa.io/) and other excellent software.
 
-This buildpack supports running Django and Flask apps.
+Recommended web frameworks include **Django** and **Flask**. The recommended webserver is **Gunicorn**. There are no restrictions around what software can be used (as long as it's pip-installable). Web processes must bind to `$PORT`, and only the HTTP protocol is permitted for incoming connections.
 
+Some Python packages with obscure C dependencies (e.g. scipy) are [not compatible](https://devcenter.heroku.com/articles/python-c-deps). 
 
-Usage
------
+See it in Action
+----------------
 
-Example usage:
+Deploying a Python application couldn't be easier:
 
     $ ls
     Procfile  requirements.txt  web.py
 
-    $ heroku create --buildpack git://github.com/heroku/heroku-buildpack-python.git
+    $ heroku create --buildpack heroku/python
 
     $ git push heroku master
     ...
     -----> Python app detected
-    -----> Installing runtime (python-2.7.11)
-    -----> Installing dependencies using pip
-           Downloading/unpacking requests (from -r requirements.txt (line 1))
+    -----> Installing python-2.7.11
+         $ pip install -r requirements.txt
+           Collecting requests (from -r requirements.txt (line 1))
+             Downloading requests-2.9.1-py2.py3-none-any.whl (501kB)
            Installing collected packages: requests
-           Successfully installed requests
-           Cleaning up...
+           Successfully installed requests-2.9.1
+           
     -----> Discovering process types
            Procfile declares types -> (none)
 
-You can also add it to upcoming builds of an existing application:
+A `requirements.txt` file must be present at the root of your application's repository.
 
-    $ heroku buildpacks:set git://github.com/heroku/heroku-buildpack-python.git
+You can also specify the latest production release of this buildpack for upcoming builds of an existing application:
 
-The buildpack will detect your app as Python if it has the file `requirements.txt` in the root.
+    $ heroku buildpacks:set heroku/python
 
-It will use Pip to install your dependencies, vendoring a copy of the Python runtime into your slug.
 
-Specify a Runtime
------------------
+Specify a Python Runtime
+------------------------
 
-You can also provide arbitrary releases Python with a `runtime.txt` file.
+Specific versions of the Python runtime can be specified with a `runtime.txt` file:
 
     $ cat runtime.txt
     python-3.5.1
 
 Runtime options include:
 
-- python-2.7.11
-- python-3.5.1
-- pypy-4.0.1 (unsupported, experimental)
-- pypy3-2.4.0 (unsupported, experimental)
+- `python-2.7.11`
+- `python-3.5.1`
+- `pypy-5.0.1` (unsupported, experimental)
 
-Other [unsupported runtimes](https://github.com/heroku/heroku-buildpack-python/tree/master/builds/runtimes) are available as well.
+Other [unsupported runtimes](https://github.com/heroku/heroku-buildpack-python/tree/master/builds/runtimes) are available as well. Use at your own risk. 

bin/compile

@@ -1,10 +1,10 @@
 #!/usr/bin/env bash
 
-# The Heroku Python Buildpack. This script accepts paramaters for a build
+# The Heroku Python Buildpack. This script accepts parameters for a build
 # directory, a cache directory, and a directory for app environment variables.
 
-# Warning: there are a few hacks in this script to accomidate excellent builds
-# on Heroku. No gaurentee for external compatibility is made. However,
+# Warning: there are a few hacks in this script to accommodate excellent builds
+# on Heroku. No guarantee for external compatibility is made. However,
 # everything should work fine outside of the Heroku environment, if the
 # environment is setup correctly.
 
@@ -32,14 +32,14 @@ VIRTUALENV_LOC=".heroku/venv"
 LEGACY_TRIGGER="lib/python2.7"
 
 DEFAULT_PYTHON_VERSION="python-2.7.11"
-DEFAULT_PYTHON_STACK="cedar"
+DEFAULT_PYTHON_STACK="cedar-14"
 PYTHON_EXE="/app/.heroku/python/bin/python"
-PIP_VERSION="8.0.2"
-SETUPTOOLS_VERSION="19.6"
+PIP_VERSION="8.1.1"
+SETUPTOOLS_VERSION="20.3"
 
 # Common Problem Warnings
 export WARNINGS_LOG=$(mktemp)
-export DEFAULT_PYTHON_VERSION
+export RECOMMENDED_PYTHON_VERSION=$DEFAULT_PYTHON_VERSION
 
 # Setup bpwatch
 export PATH=$PATH:$ROOT_DIR/vendor/bpwatch
@@ -57,8 +57,8 @@ export PATH=$PATH:$ROOT_DIR/vendor/pip-pop
 
 # Sanitizing environment variables.
 unset GIT_DIR PYTHONHOME PYTHONPATH LD_LIBRARY_PATH LIBRARY_PATH
-unset RECEIVE_DATA SOURCE_VERSION RUN_KEY BUILD_INFO DEPLOY
-unset LOG_TOKEN DYNO CYTOKINE_LOG_FILE GEM_PATH
+unset RECEIVE_DATA RUN_KEY BUILD_INFO DEPLOY LOG_TOKEN DYNO
+unset CYTOKINE_LOG_FILE GEM_PATH
 
 # Setup buildpack instrumentation.
 bpwatch init $LOGPLEX_KEY
@@ -73,7 +73,7 @@ source $BIN_DIR/utils
 # Import collection of warnings.
 source $BIN_DIR/warnings
 
-# Directory Hacks for path consistiency.
+# Directory Hacks for path consistency.
 APP_DIR='/app'
 TMP_APP_DIR=$CACHE_DIR/tmp_app_dir
 
@@ -103,7 +103,7 @@ BUILD_DIR=$APP_DIR
 
 # Set up outputs under new context
 PROFILE_PATH="$BUILD_DIR/.profile.d/python.sh"
-WEBCONCURRENCY_PROFILE_PATH="$BUILD_DIR/.profile.d/python.webconcurrency.sh"
+GUNICORN_PROFILE_PATH="$BUILD_DIR/.profile.d/python.gunicorn.sh"
 
 # We'll need to send these statics to other scripts we `source`.
 export BUILD_DIR CACHE_DIR BIN_DIR PROFILE_PATH
@@ -112,11 +112,11 @@ export BUILD_DIR CACHE_DIR BIN_DIR PROFILE_PATH
 export PATH=$BUILD_DIR/.heroku/python/bin:$BUILD_DIR/.heroku/vendor/bin:$PATH
 export PYTHONUNBUFFERED=1
 export LANG=en_US.UTF-8
-export C_INCLUDE_PATH=/app/.heroku/vendor/include:$BUILD_DIR/.heroku/vendor/include:/app/.heroku/python/include
-export CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:$BUILD_DIR/.heroku/vendor/include:/app/.heroku/python/include
-export LIBRARY_PATH=/app/.heroku/vendor/lib:$BUILD_DIR/.heroku/vendor/lib:/app/.heroku/python/lib
-export LD_LIBRARY_PATH=/app/.heroku/vendor/lib:$BUILD_DIR/.heroku/vendor/lib:/app/.heroku/python/lib
-export PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:$BUILD_DIR/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config
+export C_INCLUDE_PATH=/app/.heroku/vendor/include:$BUILD_DIR/.heroku/vendor/include:/app/.heroku/python/include:$C_INCLUDE_PATH
+export CPLUS_INCLUDE_PATH=/app/.heroku/vendor/include:$BUILD_DIR/.heroku/vendor/include:/app/.heroku/python/include:$CPLUS_INCLUDE_PATH
+export LIBRARY_PATH=/app/.heroku/vendor/lib:$BUILD_DIR/.heroku/vendor/lib:/app/.heroku/python/lib:$LIBRARY_PATH
+export LD_LIBRARY_PATH=/app/.heroku/vendor/lib:$BUILD_DIR/.heroku/vendor/lib:/app/.heroku/python/lib:$LD_LIBRARY_PATH
+export PKG_CONFIG_PATH=/app/.heroku/vendor/lib/pkg-config:$BUILD_DIR/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config:$PKG_CONFIG_PATH
 
 # Switch to the repo's context.
 cd $BUILD_DIR
@@ -147,7 +147,7 @@ fi
 if [ -f $CACHE_DIR/.heroku/python-stack ]; then
   CACHED_PYTHON_STACK=$(cat $CACHE_DIR/.heroku/python-stack)
 else
-  CACHED_PYTHON_STACK=$DEFAULT_PYTHON_STACK
+  CACHED_PYTHON_STACK=$STACK
 fi
 
 # If no runtime given, assume default version.
@@ -201,6 +201,9 @@ source $BIN_DIR/steps/pylibmc
 # Libffi support.
 source $BIN_DIR/steps/cryptography
 
+# Support for Geo libraries.
+sub-env $BIN_DIR/steps/geo-libs
+
 # GDAL support.
 source $BIN_DIR/steps/gdal
 
@@ -221,8 +224,8 @@ set-default-env LANG en_US.UTF-8
 set-default-env PYTHONHASHSEED random
 set-default-env PYTHONPATH /app/
 
-# Install sane-default script for WEB_CONCURRENCY environment variable.
-cp $ROOT_DIR/vendor/python.webconcurrency.sh $WEBCONCURRENCY_PROFILE_PATH
+# Install sane-default script for $WEB_CONCURRENCY and $FORWARDED_ALLOW_IPS.
+cp $ROOT_DIR/vendor/python.gunicorn.sh $GUNICORN_PROFILE_PATH
 
 
 # Experimental post_compile hook.

bin/detect

@@ -4,7 +4,7 @@
 # [**Python Buildpack**](https://github.com/heroku/heroku-buildpack-python)
 # detector.
 #
-# A [buildpack](http://devcenter.heroku.com/articles/buildpacks) is an
+# A [buildpack](https://devcenter.heroku.com/articles/buildpacks) is an
 # adapter between a Python application and Heroku's runtime.
 
 # ## Usage

bin/steps/collectstatic

@@ -19,7 +19,7 @@ MANAGE_FILE=${MANAGE_FILE:-fakepath}
 # Legacy file-based support for $DISABLE_COLLECTSTATIC
 [ -f .heroku/collectstatic_disabled ] && DISABLE_COLLECTSTATIC=1
 
-# Ensure that Django is explicitily specified in requirments.txt
+# Ensure that Django is explicitly specified in requirements.txt
 pip-grep -s requirements.txt django Django && DJANGO_INSTALLED=1
 
 bpwatch start collectstatic  # metrics collection
@@ -47,7 +47,7 @@ if [ ! "$DISABLE_COLLECTSTATIC" ] && [ -f "$MANAGE_FILE" ] && [ "$DJANGO_INSTALL
         echo
         echo "          $ heroku config:set DISABLE_COLLECTSTATIC=1"
         echo
-        echo "       http://devcenter.heroku.com/articles/django-assets"
+        echo "       https://devcenter.heroku.com/articles/django-assets"
 
         # Additionally, dump out the environment, if debug mode is on.
         if [ "$DEBUG_COLLECTSTATIC" ]; then

bin/steps/cryptography

@@ -4,12 +4,12 @@
 # [**Python Buildpack**](https://github.com/heroku/heroku-buildpack-python)
 # compiler.
 #
-# A [buildpack](http://devcenter.heroku.com/articles/buildpacks) is an
+# A [buildpack](https://devcenter.heroku.com/articles/buildpacks) is an
 # adapter between a Python application and Heroku's runtime.
 #
 # This script is invoked by [`bin/compile`](/).
 
-# The location of the pre-compiled cryptography binary.
+# The location of the pre-compiled libffi binary.
 VENDORED_LIBFFI="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/libffi.tar.gz"
 
 PKG_CONFIG_PATH="/app/.heroku/vendor/lib/pkgconfig:$PKG_CONFIG_PATH"
@@ -19,15 +19,15 @@ source $BIN_DIR/utils
 
 bpwatch start libffi_install
 
-# If pylibmc exists within requirements, use vendored cryptography.
-if (pip-grep -s requirements.txt bcrypt cffi cryptography pyOpenSSL PyOpenSSL requests[security] &> /dev/null) then
+# If a package using cffi exists within requirements, use vendored libffi.
+if (pip-grep -s requirements.txt bcrypt cffi cryptography django[bcrypt] Django[bcrypt] pyOpenSSL PyOpenSSL requests[security] &> /dev/null) then
 
   if [ -d ".heroku/vendor/lib/libffi-3.1.1" ]; then
     export LIBFFI=$(pwd)/vendor
   else
     echo "-----> Noticed cffi. Bootstrapping libffi."
     mkdir -p .heroku/vendor
-    # Download and extract cryptography into target vendor directory.
+    # Download and extract libffi into target vendor directory.
     curl $VENDORED_LIBFFI -s | tar zxv -C .heroku/vendor &> /dev/null
 
     export LIBFFI=$(pwd)/vendor

bin/steps/gdal

@@ -4,7 +4,7 @@
 # [**Python Buildpack**](https://github.com/heroku/heroku-buildpack-python)
 # compiler.
 #
-# A [buildpack](http://devcenter.heroku.com/articles/buildpacks) is an
+# A [buildpack](https://devcenter.heroku.com/articles/buildpacks) is an
 # adapter between a Python application and Heroku's runtime.
 #
 # This script is invoked by [`bin/compile`](/).
@@ -20,7 +20,7 @@ source $BIN_DIR/utils
 bpwatch start gdal_install
 
 # If GDAL exists within requirements, use vendored gdal.
-if (pip-grep -s requirements.txt GDAL &> /dev/null) then
+if (pip-grep -s requirements.txt GDAL gdal pygdal &> /dev/null) then
 
   if [ -f ".heroku/vendor/bin/gdalserver" ]; then
     export GDAL=$(pwd)/vendor

bin/steps/geo-libs

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+# This script serves as the GDAL build step of the
+# [**Python Buildpack**](https://github.com/heroku/heroku-buildpack-python)
+# compiler.
+#
+# A [buildpack](https://devcenter.heroku.com/articles/buildpacks) is an
+# adapter between a Python application and Heroku's runtime.
+#
+# This script is invoked by [`bin/compile`](/).
+
+# The location of the pre-compiled cryptography binary.
+VENDORED_GDAL="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/gdal.tar.gz"
+VENDORED_GEOS="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/geos.tar.gz"
+VENDORED_PROJ="https://lang-python.s3.amazonaws.com/$STACK/libraries/vendor/proj.tar.gz"
+
+PKG_CONFIG_PATH="/app/.heroku/vendor/lib/pkgconfig:$PKG_CONFIG_PATH"
+
+# Syntax sugar.
+source $BIN_DIR/utils
+
+bpwatch start geo_libs_install
+
+# If GDAL exists within requirements, use vendored gdal.
+if [[ "$BUILD_WITH_GEO_LIBRARIES" ]]; then
+
+  if [ -f ".heroku/vendor/bin/gdalserver" ]; then
+    export GDAL=$(pwd)/vendor
+  else
+    echo "-----> Bootstrapping gdal, geos, proj."
+    mkdir -p .heroku/vendor
+    # Download and extract cryptography into target vendor directory.
+    curl $VENDORED_GDAL -s | tar zxv -C .heroku/vendor &> /dev/null
+    curl $VENDORED_GEOS -s | tar zxv -C .heroku/vendor &> /dev/null
+    curl $VENDORED_PROJ -s | tar zxv -C .heroku/vendor &> /dev/null
+
+    export GDAL=$(pwd)/vendor
+  fi
+fi
+
+bpwatch stop geo_libs_install

bin/steps/pylibmc

@@ -4,7 +4,7 @@
 # [**Python Buildpack**](https://github.com/heroku/heroku-buildpack-python)
 # compiler.
 #
-# A [buildpack](http://devcenter.heroku.com/articles/buildpacks) is an
+# A [buildpack](https://devcenter.heroku.com/articles/buildpacks) is an
 # adapter between a Python application and Heroku's runtime.
 #
 # This script is invoked by [`bin/compile`](/).

bin/warnings

@@ -5,7 +5,7 @@ old-platform() {
     echo
     puts-warn "Hello! It looks like your application is using an outdated version of Python."
     puts-warn "This caused the security warning you saw above during the 'pip install' step."
-    puts-warn "We recommend '$DEFAULT_PYTHON_VERSION', which you can specify in a 'runtime.txt' file."
+    puts-warn "We recommend '$RECOMMENDED_PYTHON_VERSION', which you can specify in a 'runtime.txt' file."
     puts-warn "  -- Much Love, Heroku."
   fi
 }

builds/libraries/libffi

@@ -1,23 +0,0 @@
-#!/usr/bin/env bash
-# Build Path: /app/.heroku/python/
-
-OUT_PREFIX=$1
-
-# Use new path, containing autoconf.
-export PATH="/app/.heroku/python/bin/:$PATH"
-hash -r
-
-
-echo "Building libffi..."
-
-SOURCE_TARBALL='https://cl.ly/2s1t1u3v0N0I/download/libffi-3.1.tar'
-
-curl -L $SOURCE_TARBALL | tar x
-
-cd libffi-3.1
-./configure --prefix=$OUT_PREFIX --disable-static &&
-make
-make install
-
-# Cleanup
-cd ..

builds/libraries/vendor/gdal

@@ -10,11 +10,11 @@ hash -r
 
 echo "Building gdal..."
 
-SOURCE_TARBALL='http://download.osgeo.org/gdal/1.11.0/gdal-1.11.0.tar.gz'
+SOURCE_TARBALL='http://download.osgeo.org/gdal/1.11.1/gdal-1.11.1.tar.gz'
 
 curl -L $SOURCE_TARBALL | tar zx
 
-cd gdal-1.11.0
+cd gdal-1.11.1
 ./configure --prefix=$OUT_PREFIX  &&
 make
 make install

builds/libraries/vendor/geos

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Build Path: /app/.heroku/vendor/
+
+OUT_PREFIX=$1
+
+# Use new path, containing autoconf.
+export PATH="/app/.heroku/python/bin/:$PATH"
+hash -r
+
+
+echo "Building geos..."
+
+SOURCE_TARBALL='http://download.osgeo.org/geos/geos-3.4.2.tar.bz2'
+
+curl -L $SOURCE_TARBALL | tar xj
+
+cd geos-3.4.2
+./configure --prefix=$OUT_PREFIX  &&
+make
+make install
+
+# Cleanup
+cd ..

builds/libraries/vendor/proj

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Build Path: /app/.heroku/vendor/
+
+OUT_PREFIX=$1
+
+# Use new path, containing autoconf.
+export PATH="/app/.heroku/python/bin/:$PATH"
+hash -r
+
+
+echo "Building gdal..."
+
+SOURCE_TARBALL='http://download.osgeo.org/proj/proj-4.8.0.tar.gz'
+
+curl -L $SOURCE_TARBALL | tar zx
+
+cd proj-4.8.0
+./configure --prefix=$OUT_PREFIX  &&
+make
+make install
+
+# Cleanup
+cd ..

builds/runtimes/pypy-5.0.0

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Build Path: /app/.heroku/python/
+# Build Deps: libraries/sqlite
+
+# NOTICE: This formula only works for the cedar-14 stack, not cedar.
+
+OUT_PREFIX=$1
+
+echo "Building PyPy..."
+SOURCE_TARBALL='https://bitbucket.org/pypy/pypy/downloads/pypy-5.0.0-linux64.tar.bz2'
+curl -L $SOURCE_TARBALL | tar jx
+cp -R pypy-5.0.0-linux64/* $OUT_PREFIX
+
+ln $OUT_PREFIX/bin/pypy $OUT_PREFIX/bin/python

builds/runtimes/pypy-5.0.1

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Build Path: /app/.heroku/python/
+# Build Deps: libraries/sqlite
+
+# NOTICE: This formula only works for the cedar-14 stack, not cedar.
+
+OUT_PREFIX=$1
+
+echo "Building PyPy..."
+SOURCE_TARBALL='https://bitbucket.org/pypy/pypy/downloads/pypy-5.0.1-linux64.tar.bz2'
+curl -L $SOURCE_TARBALL | tar jx
+cp -R pypy-5.0.1-linux64/* $OUT_PREFIX
+
+ln $OUT_PREFIX/bin/pypy $OUT_PREFIX/bin/python

builds/runtimes/python-3.5.1

@@ -10,7 +10,7 @@ curl -L $SOURCE_TARBALL | tar xz
 mv Python-3.5.1 src
 cd src
 
-./configure --prefix=$OUT_PREFIX --with-ensurepip=no --enable-shared
+./configure --prefix=$OUT_PREFIX --with-ensurepip=no
 make
 make install
 

vendor/python.gunicorn.sh

@@ -1,27 +1,32 @@
 case $(ulimit -u) in
 
-# 1X DYNO
+# Automatic configuration for Gunicorn's Workers setting.
+
+# Standard-1X (+Free, +Hobby) Dyno
 256)
   export DYNO_RAM=512
   export WEB_CONCURRENCY=${WEB_CONCURRENCY:-2}
   ;;
 
-# 2X DYNO
+# Standard-2X Dyno
 512)
   export DYNO_RAM=1024
   export WEB_CONCURRENCY=${WEB_CONCURRENCY:-4}
   ;;
 
-# IX DYNO
+# Performance-M Dyno
 16384)
   export DYNO_RAM=2560
   export WEB_CONCURRENCY=${WEB_CONCURRENCY:-8}
   ;;
 
-# PX DYNO
+# Performance-L Dyno
 32768)
   export DYNO_RAM=6656
   export WEB_CONCURRENCY=${WEB_CONCURRENCY:-11}
   ;;
 
 esac
+
+# Automatic configuration for Gunicorn's ForwardedAllowIPS setting.
+export FORWARDED_ALLOW_IPS='*'