diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 481f3f6..83a9cd7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,6 +6,28 @@ see: http://docs.python-guide.org/en/latest/notes/contribute/ +How to test your changes +------------------------ + +The html version of this guide is built with [sphinx](http://www.sphinx-doc.org/en/stable/). You may test your revisions locally by having sphinx installed. You can do this easily with pip (as described in the link). + +``` bash +pip install --user sphinx +``` + +Then navigate to the directory of the makefile and ```make build``` or ```make html```. Sphinx will then generate the html in a folder called _build/html + +After navigating to this folder, you can then use python's built in webserver to view your changes locally: + +``` bash +python3 -m http.server +``` + +By default, http.server listens on every ip address bound on your host on port 8000. To bind to a specific one, say, localhost on port 8005: + +``` bash +python3 -m http.server 8005 --bind 127.0.0.1 +``` Style Guide ----------- diff --git a/docs/_templates/sidebarintro.html b/docs/_templates/sidebarintro.html index b52228c..5a49dc0 100644 --- a/docs/_templates/sidebarintro.html +++ b/docs/_templates/sidebarintro.html @@ -20,6 +20,13 @@

Join Mailing List.

+

O'Reilly Book

+ +

This guide is now available in tangible book form!

+ + + +

All proceeds are being directly donated to the DjangoGirls organization.

Other Projects

@@ -53,7 +60,8 @@

Translations

\ No newline at end of file +
  • French
    • +
  • Chinese
    • +
  • Japanese
    • +
  • Korean
    • + diff --git a/docs/_templates/sidebarlogo.html b/docs/_templates/sidebarlogo.html index 9ef1f0a..bea79f7 100644 --- a/docs/_templates/sidebarlogo.html +++ b/docs/_templates/sidebarlogo.html @@ -19,7 +19,13 @@

    Join Mailing List.

    +

    O'Reilly Book

    +

    This guide is now available in tangible book form!

    + + + +

    All proceeds are being directly donated to the DjangoGirls organization.

    Other Projects

    @@ -36,7 +42,8 @@

    Translations

    \ No newline at end of file +
  • French
    • +
  • Chinese
    • +
  • Japanese
    • +
  • Korean
    • + diff --git a/docs/contents.rst.inc b/docs/contents.rst.inc index c2c603d..22cbdc9 100644 --- a/docs/contents.rst.inc +++ b/docs/contents.rst.inc @@ -64,6 +64,7 @@ different scenarios. scenarios/xml scenarios/json scenarios/crypto + scenarios/ml scenarios/clibs diff --git a/docs/dev/env.rst b/docs/dev/env.rst index 41ec2d7..d587027 100644 --- a/docs/dev/env.rst +++ b/docs/dev/env.rst @@ -150,6 +150,15 @@ features can be brought to IntelliJ with the free versions of PyCharm: Professional Edition (Free 30-day trial) and Community Edition(Apache 2.0 License) with fewer features. +Python (on Visual Studio Code) +----------------------------- + +`Python for Visual Studio `_ is an extension for the `Visual Studio Code IDE `_. +This is a free, light weight, open source IDE, with support for Mac, Windows, and Linux. +Built using open source technologies such as Node.js and Python, with compelling features such as Intellisense (autocompletion), local and remote debugging, linting, and the like. + +MIT licensed. + Enthought Canopy ---------------- `Enthought Canopy `_ is a Python diff --git a/docs/dev/virtualenvs.rst b/docs/dev/virtualenvs.rst index fa0bbfc..d8d746c 100644 --- a/docs/dev/virtualenvs.rst +++ b/docs/dev/virtualenvs.rst @@ -8,7 +8,11 @@ projects in separate places, by creating virtual Python environments for them. It solves the "Project X depends on version 1.x but, Project Y needs 4.x" dilemma, and keeps your global site-packages directory clean and manageable. +<<<<<<< HEAD For example, you can work on a project which requires Django 1.9 while also +======= +For example, you can work on a project which requires Django 1.10 while also +>>>>>>> upstream/master maintaining a project which requires Django 1.8. virtualenv @@ -127,7 +131,7 @@ control by adding it to the ignore list. virtualenvwrapper ----------------- -`virtualenvwrapper `_ +`virtualenvwrapper `_ provides a set of commands which makes working with virtual environments much more pleasant. It also places all your virtual environments in one place. @@ -139,7 +143,7 @@ To install (make sure **virtualenv** is already installed): $ export WORKON_HOME=~/Envs $ source /usr/local/bin/virtualenvwrapper.sh -(`Full virtualenvwrapper install instructions `_.) +(`Full virtualenvwrapper install instructions `_.) For Windows, you can use the `virtualenvwrapper-win `_. @@ -211,7 +215,7 @@ Other useful commands ``lssitepackages`` Shows contents of :file:`site-packages` directory. -`Full list of virtualenvwrapper commands `_. +`Full list of virtualenvwrapper commands `_. virtualenv-burrito ------------------ diff --git a/docs/intro/learning.rst b/docs/intro/learning.rst index 8ab6608..823da5b 100644 --- a/docs/intro/learning.rst +++ b/docs/intro/learning.rst @@ -38,7 +38,7 @@ without having to install Python locally. If you want a more traditional book, *Python For You and Me* is an excellent resource for learning all aspects of the language. - `Python for You and Me `_ + `Python for You and Me `_ Online Python Tutor ~~~~~~~~~~~~~~~~~~~ diff --git a/docs/intro/news.rst b/docs/intro/news.rst index 4297490..54f8ff7 100644 --- a/docs/intro/news.rst +++ b/docs/intro/news.rst @@ -47,3 +47,10 @@ Weekly Python Newsletter containing Python Articles, Projects, Videos, Tweets delivered in your inbox. Keep Your Python Programming Skills Updated. `Import Python Weekly Newsletter `_ + +Awesome Python Newsletter +~~~~~~~~~~~~~~~~~~~~ + +A weekly overview of the most popular Python news, articles and packages. + + `Awesome Python Newsletter `_ diff --git a/docs/scenarios/admin.rst b/docs/scenarios/admin.rst index f0b56a6..8e738d9 100644 --- a/docs/scenarios/admin.rst +++ b/docs/scenarios/admin.rst @@ -190,7 +190,7 @@ Ansible `Ansible `_ is an open source system automation tool. The biggest advantage over Puppet or Chef is it does not require an agent on the client machine. Playbooks are Ansible’s configuration, deployment, and -orchestration language and are written in in YAML with Jinja2 for templating. +orchestration language and are written in YAML with Jinja2 for templating. Ansible supports Python versions 2.6 and 2.7 and can be installed via pip: @@ -262,7 +262,7 @@ To create a simple cookbook the `knife `_ comma `Getting started with Chef `_ is a good starting point for Chef Beginners and many community maintained cookbooks that can -serve as a good reference or tweaked to serve your infrustructure configuration needs can be +serve as a good reference or tweaked to serve your infrastructure configuration needs can be found on the `Chef Supermarket `_. - `Chef Documentation `_ @@ -378,5 +378,14 @@ Buildout is primarily used to download and set up dependencies in Python eggs format of the software being developed or deployed. Recipes for build tasks in any environment can be created, and many are already available. -Buidout is written in Python. +Shinken +------- +`Shinken `_ is a modern, Nagios compatible +monitoring framework written in Python. Its main goal is to give users a flexible +architecture for their monitoring system that is designed to scale to large +environments. + +Shinken is backwards-compatible with the Nagios configuration standard, and +plugins.It works on any operating system, and architecture that supports Python +which includes Windows, GNU/Linux, and FreeBSD. diff --git a/docs/scenarios/ci.rst b/docs/scenarios/ci.rst index aa8d196..16e0ea0 100644 --- a/docs/scenarios/ci.rst +++ b/docs/scenarios/ci.rst @@ -35,7 +35,7 @@ automate the compile/test cycle to validate code changes. Tox --- -`tox `_ is an automation tool providing +`tox `_ is an automation tool providing packaging, testing and deployment of Python software right from the console or CI server. It is a generic virtualenv management and test command line tool which provides the following features: diff --git a/docs/scenarios/cli.rst b/docs/scenarios/cli.rst index c8cdcec..ab357c2 100644 --- a/docs/scenarios/cli.rst +++ b/docs/scenarios/cli.rst @@ -16,7 +16,7 @@ Some popular command-line applications include: * `httpie `_ - A command line HTTP client, a user-friendly cURL replacement * `git `_ - A distributed version control system -* `mercurial `_ - A distributed version control +* `mercurial `_ - A distributed version control system primarily written in Python Clint @@ -62,3 +62,13 @@ subcommands, output formatters, and other extensions. The framework is meant to be used to create multi-level commands such as subversion and git, where the main program handles some basic argument parsing and then invokes a sub-command to do the work. + +Cement +------ + +`Cement `_ is an advanced CLI Application Framework. +Its goal is to introduce a standard, and feature-full platform +for both simple and complex command line applications as well +as support rapid development needs without sacrificing quality. +Cement is flexible, and it's use cases span from the simplicity of a micro-framework +to the complexity of a meg-framework. diff --git a/docs/scenarios/clibs.rst b/docs/scenarios/clibs.rst index 290638e..82e041f 100644 --- a/docs/scenarios/clibs.rst +++ b/docs/scenarios/clibs.rst @@ -4,7 +4,7 @@ Interfacing with C/C++ Libraries C Foreign Function Interface ---------------------------- -`CFFI `_ provides a simple to use +`CFFI `_ provides a simple to use mechanism for interfacing with C from both CPython and PyPy. It supports two modes: an inline ABI compatibility mode (example provided below), which allows you to dynamically load and run functions from executable modules (essentially @@ -39,7 +39,7 @@ types, such as structs and unions, and allows you to modify things such as padding and alignment, if needed. It can be a bit crufty to use, but in conjunction with the `struct `_ module, you are essentially provided full control over how your data types get -translated into something something usable by a pure C(++) method. +translated into something usable by a pure C(++) method. Struct Equivalents ~~~~~~~~~~~~~~~~~~ diff --git a/docs/scenarios/crypto.rst b/docs/scenarios/crypto.rst index 94bd58d..e4b103a 100644 --- a/docs/scenarios/crypto.rst +++ b/docs/scenarios/crypto.rst @@ -6,7 +6,7 @@ Cryptography `Cryptography `_ is an actively developed library that provides cryptographic recipes and primitives. It supports -Python 2.6-2.7, Python 3.2+ and PyPy. +Python 2.6-2.7, Python 3.3+ and PyPy. Cryptography is divided into two layers of recipes and hazardous materials diff --git a/docs/scenarios/db.rst b/docs/scenarios/db.rst index a2de7fe..58d9cd2 100644 --- a/docs/scenarios/db.rst +++ b/docs/scenarios/db.rst @@ -29,7 +29,7 @@ Records `Records `_ is minimalist SQL library, designed for sending raw SQL queries to various databases. Data can be used -programatically, or exported to a number of useful data formats. +programmatically, or exported to a number of useful data formats. .. code-block:: console @@ -60,11 +60,11 @@ peewee `peewee `_ is another ORM with a focus on being lightweight with support for Python 2.6+ and 3.2+ which supports SQLite, MySQL and Postgres by default. The -`model layer `_ +`model layer `_ is similar to that of the Django ORM and it has -`SQL-like methods `_ +`SQL-like methods `_ to query data. While SQLite, MySQL and Postgres are supported out-of-the-box, -there is a `collection of add-ons `_ +there is a `collection of add-ons `_ available. PonyORM diff --git a/docs/scenarios/gui.rst b/docs/scenarios/gui.rst index 854fb82..f56b212 100644 --- a/docs/scenarios/gui.rst +++ b/docs/scenarios/gui.rst @@ -28,7 +28,7 @@ applications be ported from PyGTK to PyGObject. PyGObject aka (PyGi) -------------------- `PyGObject `_ provides Python bindings, which gives access to the entire GNOME software platform. -It is fully compatible with GTK+ 3. Here is a tutorial to get started with `Python GTK+ 3 Tutorial `_. +It is fully compatible with GTK+ 3. Here is a tutorial to get started with `Python GTK+ 3 Tutorial `_. `API Reference `_ diff --git a/docs/scenarios/imaging.rst b/docs/scenarios/imaging.rst index 23ecece..49d3af1 100644 --- a/docs/scenarios/imaging.rst +++ b/docs/scenarios/imaging.rst @@ -24,7 +24,7 @@ Installation Before installing Pillow, you'll have to install Pillow's prerequisites. Find the instructions for your platform in the -`Pillow installation instructions `_. +`Pillow installation instructions `_. After that, it's straightforward: @@ -57,7 +57,7 @@ Example exif_data There are more examples of the Pillow library in the -`Pillow tutorial `_. +`Pillow tutorial `_. OpenSource Computer Vision @@ -104,4 +104,4 @@ Example There are more Python-implemented examples of OpenCV in this `collection of tutorials -`_. +`_. diff --git a/docs/scenarios/json.rst b/docs/scenarios/json.rst index b5145fc..860e794 100644 --- a/docs/scenarios/json.rst +++ b/docs/scenarios/json.rst @@ -47,7 +47,7 @@ simplejson The JSON library was added to Python in version 2.6. If you're using an earlier version of Python, the -`simplejson `_ library is +`simplejson `_ library is available via PyPI. simplejson mimics the json standard library. It is available so that developers diff --git a/docs/scenarios/ml.rst b/docs/scenarios/ml.rst new file mode 100644 index 0000000..25cab80 --- /dev/null +++ b/docs/scenarios/ml.rst @@ -0,0 +1,117 @@ +================ +Machine Learning +================ + +Python has a vast number of libraries for data analysis, statistics and Machine Learning itself, making it a language of choice for many data scientists. + +Some widely used packages for Machine Learning and other Data Science applications are enlisted below. + +Scipy Stack +----------- + +The Scipy stack consists of a bunch of core helper packages used in data science, for statistical analysis and visualising data. Because of its huge number of functionalities and ease of use, the Stack is considered a must-have for most data science applications. + +The Stack consists of the following packages (link to documentation given): + +1. `NumPy `_ +2. `SciPy library `_ +3. `Matplotlib `_ +4. `IPython `_ +5. `pandas `_ +6. `Sympy `_ +7. `nose `_ + +The stack also comes with Python bundled in, but has been excluded from the above list. + +Installation +~~~~~~~~~~~~ + +For installing the full stack, or individual packages, you can refer to the instructions given `here `_. + +**NB:** `Anaconda `_ is highly preferred and recommended for installing and maintaining data science packages seamlessly. + +scikit-learn +------------ + +Scikit is a free and open-source machine learning library for Python. It offers off-the-shelf functions to implement many algorithms like linear regression, classifiers, SVMs, k-means, Neural Networks etc. It also has a few sample datasets which can be directly used for training and testing. + +Because of its speed, robustness and easiness to use, it's one of the most widely-used libraries for many Machine Learning applications. + +Installation +~~~~~~~~~~~~ + +Through PyPI: + +.. code-block:: python + + pip install -U scikit-learn + +Through conda: + +.. code-block:: python + + conda install scikit-learn + +scikit-learn also comes in shipped with Anaconda (mentioned above). For more installation instructions, refer to `this link `_. + +Example +~~~~~~~ + +For this example, we train a simple classifier on the `Iris dataset `_, which comes bundled in with scikit-learn. + +The dataset takes four features of flowers: sepal length, sepal width, petal length and petal width, and classifies them into three flower species (labels): setosa, versicolor or virginica. The labels have been represented as numbers in the dataset: 0 (setosa), 1 (versicolor) and 2 (virginica). + +We shuffle the Iris dataset, and divide it into separate training and testing sets: keeping the last 10 data points for testing and rest for training. We then train the classifier on the training set, and predict on the testing set. + +.. code-block:: python + + from sklearn.datasets import load_iris + from sklearn import tree + from sklearn.metrics import accuracy_score + import numpy as np + + #loading the iris dataset + iris = load_iris() + + x = iris.data #array of the data + y = iris.target #array of labels (i.e answers) of each data entry + + #getting label names i.e the three flower species + y_names = iris.target_names + + #taking random indices to split the dataset into train and test + test_ids = np.random.permutation(len(x)) + + #splitting data and labels into train and test + #keeping last 10 entries for testing, rest for training + + x_train = x[test_ids[:-10]] + x_test = x[test_ids[-10:]] + + y_train = y[test_ids[:-10]] + y_test = y[test_ids[-10:]] + + #classifying using decision tree + clf = tree.DecisionTreeClassifier() + + #training (fitting) the classifier with the training set + clf.fit(x_train, y_train) + + #predictions on the test dataset + pred = clf.predict(x_test) + + print pred #predicted labels i.e flower species + print y_test #actual labels + print (accuracy_score(pred, y_test))*100 #prediction accuracy + +Since we're splitting randomly and the classifier trains on every iteration, the accuracy may vary. Running the above code gives: + +.. code-block:: python + + [0 1 1 1 0 2 0 2 2 2] + [0 1 1 1 0 2 0 2 2 2] + 100.0 + +The first line contains the labels (i.e flower species) of the testing data as predicted by our classifier, and the second line contains the actual flower species as given in the dataset. We thus get an accuracy of 100% this time. + +More on scikit-learn can be read in the `documentation `_. \ No newline at end of file diff --git a/docs/scenarios/scientific.rst b/docs/scenarios/scientific.rst index 8600bc9..92c8c80 100644 --- a/docs/scenarios/scientific.rst +++ b/docs/scenarios/scientific.rst @@ -94,7 +94,7 @@ auto-alignment of data. Rpy2 ---- -`Rpy2 `_ is a Python binding for the R +`Rpy2 `_ is a Python binding for the R statistical package allowing the execution of R functions from Python and passing data back and forth between the two environments. Rpy2 is the object oriented implementation of the `Rpy `_ diff --git a/docs/scenarios/speed.rst b/docs/scenarios/speed.rst index 6050c98..2c775b8 100644 --- a/docs/scenarios/speed.rst +++ b/docs/scenarios/speed.rst @@ -108,7 +108,7 @@ keywords compared to the next one, which is implemented in pure Python: def primes(kmax): """Calculation of prime numbers in standard Python syntax""" - p= range(1000) + p = range(1000) result = [] if kmax > 1000: kmax = 1000 @@ -145,7 +145,7 @@ to be compiled into C types while also creating a Python list: def primes(kmax): """Calculation of prime numbers in standard Python syntax""" - p= range(1000) + p = range(1000) result = [] What is the difference? In the upper Cython version you can see the @@ -222,17 +222,212 @@ Pyrex Shedskin? --------- -Numba ------ -.. todo:: Write about Numba and the autojit compiler for NumPy +Concurrency +::::::::::: -Threading -::::::::: +Concurrent.futures +------------------ + +The `concurrent.futures`_ module is a module in the standard library that +provides a "high-level interface for asynchronously executing callables". It +abstracts away a lot of the more complicated details about using multiple +threads or processes for concurrency, and allows the user to focus on +accomplishing the task at hand. + +The `concurrent.futures`_ module exposes two main classes, the +`ThreadPoolExecutor` and the `ProcessPoolExecutor`. The ThreadPoolExecutor +will create a pool of worker threads that a user can submit jobs to. These jobs +will then be executed in another thread when the next worker thread becomes +available. + +The ProcessPoolExecutor works in the same way, except instead of using multiple +threads for its workers, it will use multiple processes. This makes it possible +to side-step the GIL, however because of the way things are passed to worker +processes, only picklable objects can be executed and returned. + +Because of the way the GIL works, a good rule of thumb is to use a +ThreadPoolExecutor when the task being executed involves a lot of blocking +(i.e. making requests over the network) and to use a ProcessPoolExecutor +executor when the task is computationally expensive. + +There are two main ways of executing things in parallel using the two +Executors. One way is with the `map(func, iterables)` method. This works +almost exactly like the builtin `map()` function, except it will execute +everything in parallel. : + +.. code-block:: python + + from concurrent.futures import ThreadPoolExecutor + import requests + + def get_webpage(url): + page = requests.get(url) + return page + + pool = ThreadPoolExecutor(max_workers=5) + + my_urls = ['http://google.com/']*10 # Create a list of urls + + for page in pool.map(get_webpage, my_urls): + # Do something with the result + print(page.text) + +For even more control, the `submit(func, *args, **kwargs)` method will schedule +a callable to be executed ( as `func(*args, **kwargs)`) and returns a `Future`_ +object that represents the execution of the callable. + +The Future object provides various methods that can be used to check on the +progress of the scheduled callable. These include: + +cancel() + Attempt to cancel the call. +cancelled() + Return True if the call was successfully cancelled. +running() + Return True if the call is currently being executed and cannot be + cancelled. +done() + Return True if the call was successfully cancelled or finished running. +result() + Return the value returned by the call. Note that this call will block until + the scheduled callable returns by default. +exception() + Return the exception raised by the call. If no exception was raised then + this returns `None`. Note that this will block just like `result()`. +add_done_callback(fn) + Attach a callback function that will be executed (as `fn(future)`) when the + scheduled callable returns. + + +.. code-block:: python + + from concurrent.futures import ProcessPoolExecutor, as_completed + + def is_prime(n): + if n % 2 == 0: + return n, False + + sqrt_n = int(n**0.5) + for i in range(3, sqrt_n + 1, 2): + if n % i == 0: + return n, False + return n, True + + PRIMES = [ + 112272535095293, + 112582705942171, + 112272535095293, + 115280095190773, + 115797848077099, + 1099726899285419] + + futures = [] + with ProcessPoolExecutor(max_workers=4) as pool: + # Schedule the ProcessPoolExecutor to check if a number is prime + # and add the returned Future to our list of futures + for p in PRIMES: + fut = pool.submit(is_prime, p) + futures.append(fut) + + # As the jobs are completed, print out the results + for number, result in as_completed(futures): + if result: + print("{} is prime".format(number)) + else: + print("{} is not prime".format(number)) + +The `concurrent.futures`_ module contains two helper functions for working with +Futures. The `as_completed(futures)` function returns an iterator over the list +of futures, yielding the futures as they complete. + +The `wait(futures)` function will simply block until all futures in the list of +futures provided have completed. + +For more information, on using the `concurrent.futures`_ module, consult the +official documentation. Threading --------- +The standard library comes with a `threading`_ module that allows a user to +work with multiple threads manually. + +Running a function in another thread is as simple as passing a callable and +it's arguments to `Thread`'s constructor and then calling `start()`: + +.. code-block:: python + + from threading import Thread + import requests + + def get_webpage(url): + page = requests.get(url) + return page + + some_thread = Thread(get_webpage, 'http://google.com/') + some_thread.start() + +To wait until the thread has terminated, call `join()`: + +.. code-block:: python + + some_thread.join() + +After calling `join()`, it is always a good idea to check whether the thread is +still alive (because the join call timed out): + +.. code-block:: python + + if some_thread.is_alive(): + print("join() must have timed out.") + else: + print("Our thread has terminated.") + +Because multiple threads have access to the same section of memory, sometimes +there might be situations where two or more threads are trying to write to the +same resource at the same time or where the output is dependent on the sequence +or timing of certain events. This is called a `data race`_ or race condition. +When this happens, the output will be garbled or you may encounter problems +which are difficult to debug. A good example is this `stackoverflow post`_. + +The way this can be avoided is by using a `Lock`_ that each thread needs to +acquire before writing to a shared resource. Locks can be acquired and released +through either the contextmanager protocol (`with` statement), or by using +`acquire()` and `release()` directly. Here is a (rather contrived) example: + + +.. code-block:: python + + from threading import Lock, Thread + + file_lock = Lock() + + def log(msg): + with file_lock: + open('website_changes.log', 'w') as f: + f.write(changes) + + def monitor_website(some_website): + """ + Monitor a website and then if there are any changes, + log them to disk. + """ + while True: + changes = check_for_changes(some_website) + if changes: + log(changes) + + websites = ['http://google.com/', ... ] + for website in websites: + t = Thread(monitor_website, website) + t.start() + +Here, we have a bunch of threads checking for changes on a list of sites and +whenever there are any changes, they attempt to write those changes to a file +by calling `log(changes)`. When `log()` is called, it will wait to acquire +the lock with `with file_lock:`. This ensures that at any one time, only one +thread is writing to the file. Spawning Processes ------------------ @@ -248,3 +443,8 @@ Multiprocessing .. _`New GIL`: http://www.dabeaz.com/python/NewGIL.pdf .. _`Special care`: http://docs.python.org/c-api/init.html#threads .. _`David Beazley's`: http://www.dabeaz.com/GIL/gilvis/measure2.py +.. _`concurrent.futures`: https://docs.python.org/3/library/concurrent.futures.html +.. _`Future`: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future +.. _`threading`: https://docs.python.org/3/library/threading.html +.. _`stackoverflow post`: http://stackoverflow.com/questions/26688424/python-threads-are-printing-at-the-same-time-messing-up-the-text-output +.. _`data race`: https://en.wikipedia.org/wiki/Race_condition diff --git a/docs/scenarios/web.rst b/docs/scenarios/web.rst index 9dd3928..9e07a59 100644 --- a/docs/scenarios/web.rst +++ b/docs/scenarios/web.rst @@ -102,10 +102,9 @@ I do not recommend using Tornado unless you think you need it. Pyramid -------- -`Pyramid `_ is a very flexible -framework with a heavy focus on modularity. It comes with a small number -of libraries ("batteries") built-in, and encourages users to extend its -base functionality. +`Pyramid `_ is a very flexible framework with a heavy +focus on modularity. It comes with a small number of libraries ("batteries") +built-in, and encourages users to extend its base functionality. Pyramid does not have a large user base, unlike Django and Flask. It's a capable framework, but not a very popular choice for new Python web @@ -155,7 +154,7 @@ Gunicorn is the recommended choice for new Python web applications today. Waitress -------- -`Waitress `_ is a pure-python WSGI server +`Waitress `_ is a pure-python WSGI server that claims "very acceptable performance". Its documentation is not very detailed, but it does offer some nice functionality that Gunicorn doesn't have (e.g. HTTP request buffering). @@ -167,18 +166,18 @@ Waitress is gaining popularity within the Python web development community. uWSGI ----- -`uWSGI `_ is a full stack for building +`uWSGI `_ is a full stack for building hosting services. In addition to process management, process monitoring, and other functionality, uWSGI acts as an application server for various programming languages and protocols - including Python and WSGI. uWSGI can either be run as a stand-alone web router, or be run behind a full web server (such as Nginx or Apache). In the latter case, a web server can configure uWSGI and an application's operation over the -`uwsgi protocol `_. +`uwsgi protocol `_. uWSGI's web server support allows for dynamically configuring Python, passing environment variables and further tuning. For full details, see `uWSGI magic -variables `_. +variables `_. I do not recommend using uWSGI unless you know why you need it. @@ -394,10 +393,10 @@ Jinja2 is the recommended templating library for new Python web applications. Chameleon --------- -`Chameleon `_ Page Templates are an HTML/XML template +`Chameleon `_ Page Templates are an HTML/XML template engine implementation of the `Template Attribute Language (TAL) `_, -`TAL Expression Syntax (TALES) `_, -and `Macro Expansion TAL (Metal) `_ syntaxes. +`TAL Expression Syntax (TALES) `_, +and `Macro Expansion TAL (Metal) `_ syntaxes. Chameleon is available for Python 2.5 and up (including 3.x and pypy), and is commonly used by the `Pyramid Framework `_. diff --git a/docs/shipping/freezing.rst b/docs/shipping/freezing.rst index d42f2d6..4e2de50 100644 --- a/docs/shipping/freezing.rst +++ b/docs/shipping/freezing.rst @@ -8,7 +8,7 @@ Freezing Your Code to end-users, that contains all of your application code as well as the Python interpreter. -Applications such as 'Dropbox', 'Eve Online', 'Civilisation IV', and +Applications such as 'Dropbox', 'Eve Online', 'Civilization IV', and BitTorrent clients do this. The advantage of distributing this way is that your application will "just work", diff --git a/docs/shipping/packaging.rst b/docs/shipping/packaging.rst index 1c65961..c679575 100644 --- a/docs/shipping/packaging.rst +++ b/docs/shipping/packaging.rst @@ -25,7 +25,7 @@ and being able and willing to use tools such as pip to install your code's other dependencies. This is fine when distributing to other developers, but makes this method unsuitable for distributing applications to end-users. -The `Python Packaging Guide `_ +The `Python Packaging Guide `_ provides an extensive guide on creating and maintaining Python packages. Alternatives to Packaging @@ -184,3 +184,4 @@ Useful Tools - `fpm `_ - `alien `_ +- `dh-virtualenv `_ (for APT/DEB omnibus packaging) diff --git a/docs/starting/install/linux.rst b/docs/starting/install/linux.rst index e68c5d0..aa56ec0 100644 --- a/docs/starting/install/linux.rst +++ b/docs/starting/install/linux.rst @@ -55,8 +55,8 @@ in separate places, by creating virtual Python environments for them. It solves "Project X depends on version 1.x but, Project Y needs 4.x" dilemma, and keeps your global site-packages directory clean and manageable. -For example, you can work on a project which requires Django 1.3 while also -maintaining a project which requires Django 1.0. +For example, you can work on a project which requires Django 1.10 while also +maintaining a project which requires Django 1.8. To start using this and see more information: :ref:`Virtual Environments ` docs. diff --git a/docs/starting/install/osx.rst b/docs/starting/install/osx.rst index 4ffba86..bc5e539 100644 --- a/docs/starting/install/osx.rst +++ b/docs/starting/install/osx.rst @@ -24,13 +24,16 @@ Doing it Right Let's install a real version of Python. -Before installing Python, you'll need to install GCC. GCC can be obtained -by downloading `XCode `_, the smaller -`Command Line Tools `_ (must have an -Apple account) or the even smaller `OSX-GCC-Installer `_ +Before installing Python, you'll need to install a C compiler. The fastest way +is to install the Xcode Command Line Tools by running +``xcode-select --install``. You can also download the full version of +`Xcode `_ from the Mac App Store, or the +minimal but unofficial +`OSX-GCC-Installer `_ package. .. note:: +<<<<<<< HEAD If you already have XCode installed, do not install OSX-GCC-Installer. In combination, the software can cause issues that are difficult to diagnose. @@ -38,6 +41,11 @@ package. .. note:: If you perform a fresh install of XCode, you will also need to add the commandline tools by running ``xcode-select --install`` on the terminal. +======= + If you already have Xcode installed or plan to use Homebrew, do not install + OSX-GCC-Installer. In combination, the software can cause issues that are + difficult to diagnose. +>>>>>>> upstream/master While OS X comes with a large number of UNIX utilities, those familiar with Linux systems will notice one key component missing: a decent package manager. @@ -66,7 +74,17 @@ Now, we can install Python 2.7: $ brew install python +<<<<<<< HEAD This will take a minute or two. +======= +or Python 3: + +.. code-block:: console + + $ brew install python3 + +This will take a minute or two. +>>>>>>> upstream/master Setuptools and Pip @@ -75,6 +93,7 @@ Setuptools and Pip Homebrew installs Setuptools and ``pip`` for you. Setuptools enables you to download and install any compliant Python +<<<<<<< HEAD -software over a network (usually the Internet) with a single command -(``easy_install``). It also enables you to add this network installation -capability to your own Python software with very little work. @@ -83,6 +102,14 @@ Setuptools enables you to download and install any compliant Python recommended over ``easy_install``. It is superior to ``easy_install`` in `several ways `_, +======= +software over a network (usually the Internet) with a single command +(``easy_install``). It also enables you to add this network installation +capability to your own Python software with very little work. + +``pip`` is a tool for easily installing and managing Python packages, +that is recommended over ``easy_install``. It is superior to ``easy_install`` in `several ways `_, +>>>>>>> upstream/master and is actively maintained. @@ -95,7 +122,13 @@ in separate places, by creating virtual Python environments for them. It solves your global site-packages directory clean and manageable. For example, you can work on a project which requires Django 1.10 while also +<<<<<<< HEAD maintaining a project which requires Django 1.7. +======= +maintaining a project which requires Django 1.8. + +To start using this and see more information: :ref:`Virtual Environments ` docs. +>>>>>>> upstream/master To start using this and see more information: :ref:`Virtual Environments ` docs. diff --git a/docs/starting/install/win.rst b/docs/starting/install/win.rst index a34566e..131ace6 100644 --- a/docs/starting/install/win.rst +++ b/docs/starting/install/win.rst @@ -3,7 +3,7 @@ Installing Python on Windows ============================ -First, download the `latest version `_ +First, download the `latest version `_ of Python 2.7 from the official Website. If you want to be sure you are installing a fully up-to-date version, click the Downloads > Windows link from the home page of the `Python.org web site `_ . @@ -74,8 +74,8 @@ in separate places, by creating virtual Python environments for them. It solves "Project X depends on version 1.x but, Project Y needs 4.x" dilemma, and keeps your global site-packages directory clean and manageable. -For example, you can work on a project which requires Django 1.3 while also -maintaining a project which requires Django 1.0. +For example, you can work on a project which requires Django 1.10 while also +maintaining a project which requires Django 1.8. To start using this and see more information: :ref:`Virtual Environments ` docs. diff --git a/docs/starting/installation.rst b/docs/starting/installation.rst index 46b9fe4..55114b2 100644 --- a/docs/starting/installation.rst +++ b/docs/starting/installation.rst @@ -16,7 +16,7 @@ Installation Guides These guides go over the proper installation of :ref:`Python ` for development purposes, as well as setuptools, pip and virtualenv. -- :ref:`Python 3 on Mac OS X `. -- :ref:`Python 2 on Mac OS X `. +- :ref:`Python 3 on MacOS `. +- :ref:`Python 2 on MacOS `. - :ref:`Python 2 on Microsoft Windows `. - :ref:`Python 2 on Ubuntu Linux `. diff --git a/docs/starting/which-python.rst b/docs/starting/which-python.rst index b07b634..26732e3 100644 --- a/docs/starting/which-python.rst +++ b/docs/starting/which-python.rst @@ -137,12 +137,12 @@ installation with the .NET Common Language Runtime (CLR). This is the inverse approach to that taken by IronPython (see above), to which it is more complementary than competing with. -In conjunction with Mono, PythonNet enables native Python +In conjunction with Mono, pythonnet enables native Python installations on non-Windows operating systems, such as OS X and Linux, to operate within the .NET framework. It can be run in addition to IronPython without conflict. -PythonNet supports from Python 2.3 up to Python 2.7. [#pythonnet_ver]_ +Pythonnet supports from Python 2.6 up to Python 3.5. [#pythonnet_ver1]_ [#pythonnet_ver2]_ .. [#pypy_ver] http://pypy.org/compat.html @@ -150,6 +150,8 @@ PythonNet supports from Python 2.3 up to Python 2.7. [#pythonnet_ver]_ .. [#iron_ver] http://ironpython.codeplex.com/releases/view/81726 -.. [#pythonnet_ver] http://pythonnet.github.io/readme.html +.. [#pythonnet_ver1] https://travis-ci.org/pythonnet/pythonnet + +.. [#pythonnet_ver2] https://ci.appveyor.com/project/TonyRoberts/pythonnet-480xs .. [#pep373_eol] https://www.python.org/dev/peps/pep-0373/#id2 diff --git a/docs/writing/documentation.rst b/docs/writing/documentation.rst index c6905e3..bdd39e1 100644 --- a/docs/writing/documentation.rst +++ b/docs/writing/documentation.rst @@ -65,6 +65,11 @@ There is also **great**, **free** hosting for your Sphinx_ docs: your source repository so that rebuilding your documentation will happen automatically. +When run, Sphinx_ will import your code and using Python's introspection +features it will extract all function, method and class signatures. It will +also extract the accompanying docstrings, and compile it all into well +structured and easily readable documentation for your project. + .. note:: Sphinx is famous for its API generation, but it also works well @@ -104,11 +109,11 @@ In Python, *docstrings* describe modules, classes, and functions: .. code-block:: python def square_and_rooter(x): - """Returns the square root of self times self.""" + """Return the square root of self times self.""" ... In general, follow the comment section of :pep:`8#comments` (the "Python Style -Guide"). +Guide"). More information about docstrings can be found at :pep:`0257#specification` (The Docstring Conventions Guide). Commenting Sections of Code ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -127,6 +132,30 @@ Some tools use docstrings to embed more-than-documentation behavior, such as unit test logic. Those can be nice, but you won't ever go wrong with vanilla "here's what this does." +Tools like Sphinx_ will parse your docstrings as reStructuredText and render it +correctly as HTML. This makes it very easy to embed snippets of example code in +a project's documentation. + +Additionally, Doctest_ will read all embedded docstrings that look like input +from the Python commandline (prefixed with ">>>") and run them, checking to see +if the output of the command matches the text on the following line. This +allows developers to embed real examples and usage of functions alongside +their source code, and as a side effect, it also ensures that their code is +tested and works. + +:: + + def my_function(a, b): + """ + >>> my_function(2, 3) + 6 + >>> my_function('a', 3) + 'aaa' + """ + return a * b + +.. _Doctest: https://docs.python.org/3/library/doctest.html + Docstrings versus Block comments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -141,8 +170,82 @@ comment block is a programmer's note. The docstring describes the """Returns the square root of self times self.""" ... -.. see also:: Further reading on docstrings: :pep:`257` +Unlike block comments, docstrings are built into the Python language itself. +This means you can use all of Python's powerful introspection capabilities to +access docstrings at runtime, compared with comments which are optimised out. +Docstrings are accessible from both the `__doc__` dunder attribute for almost +every Python object, as well as with the built in `help()` function. +While block comments are usually used to explain *what* a section of code is +doing, or the specifics of an algorithm, docstrings are more intended for +explaining to other users of your code (or you in 6 months time) *how* a +particular function can be used and the general purpose of a function, class, +or module. + +Writing Docstrings +~~~~~~~~~~~~~~~~~~ + +Depending on the complexity of the function, method, or class being written, a +one-line docstring may be perfectly appropriate. These are generally used for +really obvious cases, such as:: + + def add(a, b): + """Add two numbers and return the result.""" + return a + b + +The docstring should describe the function in a way that is easy to understand. +For simple cases like trivial functions and classes, simply embedding the +function's signature (i.e. `add(a, b) -> result`) in the docstring is +unnecessary. This is because with Python's `inspect` module, it is already +quite easy to find this information if needed, and it is also readily available +by reading the source code. + +In larger or more complex projects however, it is often a good idea to give +more information about a function, what it does, any exceptions it may raise, +what it returns, or relevant details about the parameters. + +For more detailed documentation of code a popular style is the one used for the +Numpy project, often called `Numpy style`_ docstrings. While it can take up a +few more lines the previous example, it allows the developer to include a lot +more information about a method, function, or class. :: + + def random_number_generator(arg1, arg2): + """ + Summary line. + + Extended description of function. + + Parameters + ---------- + arg1 : int + Description of arg1 + arg2 : str + Description of arg2 + + Returns + ------- + int + Description of return value + + """ + return 42 + +The `sphinx.ext.napoleon`_ plugin allows Sphinx to parse this style of +docstrings, making it easy to incorporate NumPy style docstrings into your +project. + +At the end of the day, it doesn't really matter what style is used for writing +docstrings, their purpose is to serve as documentation for anyone who may need +to read or make changes to your code. As long as it is correct, understandable +and gets the relevant points across then it has done the job it was designed to +do. + + +For further reading on docstrings, feel free to consult :pep:`257` + +.. _thomas-cokelaer.info: http://thomas-cokelaer.info/tutorials/sphinx/docstring_python.html +.. _sphinx.ext.napoleon: https://sphinxcontrib-napoleon.readthedocs.io/ +.. _`NumPy style`: http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html Other Tools ----------- diff --git a/docs/writing/gotchas.rst b/docs/writing/gotchas.rst index 62704c5..f8e2e23 100644 --- a/docs/writing/gotchas.rst +++ b/docs/writing/gotchas.rst @@ -224,7 +224,7 @@ Removing Bytecode (.pyc) Files Here's nice trick for removing all of these files, if they already exist:: - $ find . -name "*.pyc" -exec rm -rf {} \; + $ find . -type f -name "*.py[co]" -delete -or -type d -name "__pycache__" -delete Run that from the root directory of your project, and all ``.pyc`` files will suddenly vanish. Much better. diff --git a/docs/writing/logging.rst b/docs/writing/logging.rst index 8763e8d..9b41aba 100644 --- a/docs/writing/logging.rst +++ b/docs/writing/logging.rst @@ -163,10 +163,10 @@ the configuration dictionary. 'formatter': 'f', 'level': logging.DEBUG} }, - loggers = { - 'root': {'handlers': ['h'], - 'level': logging.DEBUG} - } + root = { + 'handlers': ['h'], + 'level': logging.DEBUG, + }, ) dictConfig(logging_config) diff --git a/docs/writing/reading.rst b/docs/writing/reading.rst index 39b0888..9734225 100644 --- a/docs/writing/reading.rst +++ b/docs/writing/reading.rst @@ -1,7 +1,7 @@ Reading Great Code ================== -One of the core tenants behind the design of Python is creating +One of the core tenets behind the design of Python is creating readable code. The motivation behind this design is simple: The number one thing that Python programmers do is read code. diff --git a/docs/writing/structure.rst b/docs/writing/structure.rst index 6448dba..a156fba 100644 --- a/docs/writing/structure.rst +++ b/docs/writing/structure.rst @@ -254,6 +254,8 @@ your project. test: py.test tests + .PHONY: init test + Other generic management scripts (e.g. ``manage.py`` or ``fabfile.py``) belong at the root of the repository as well. @@ -269,7 +271,7 @@ following, as they always have: :: - $ django-admin.py start-project samplesite + $ django-admin.py startproject samplesite The resulting repository structure looks like this: @@ -291,7 +293,7 @@ Let's do it properly: :: - $ django-admin.py start-project samplesite . + $ django-admin.py startproject samplesite . Note the "``.``". @@ -600,7 +602,7 @@ clearer and thus preferred. This mechanism is useful for separating concerns and avoiding external un-related logic 'polluting' the core logic of the function or method. A good example of a piece of functionality that is better handled -with decoration is memoization or caching: you want to store the results of an +with decoration is `memoization `__ or caching: you want to store the results of an expensive function in a table and use them directly instead of recomputing them when they have already been computed. This is clearly not part of the function logic. diff --git a/docs/writing/style.rst b/docs/writing/style.rst index b916569..7454e94 100644 --- a/docs/writing/style.rst +++ b/docs/writing/style.rst @@ -3,17 +3,17 @@ Code Style ========== -If you ask Python programmers what they like most in Python, they will -often say its high readability. Indeed, a high level of readability +If you ask Python programmers what they like most about Python, they will +often cite its high readability. Indeed, a high level of readability is at the heart of the design of the Python language, following the recognized fact that code is read much more often than it is written. -One reason for Python code to be easily read and understood is its relatively +One reason for the high readability of Python code is its relatively complete set of Code Style guidelines and "Pythonic" idioms. -Moreover, when a veteran Python developer (a Pythonista) points to portions of -code and says they are not "Pythonic", it usually means that these lines -of code do not follow the common guidelines and fail to express the intent in +When a veteran Python developer (a Pythonista) calls portions of +code not "Pythonic", they usually mean that these lines +of code do not follow the common guidelines and fail to express its intent in what is considered the best (hear: most readable) way. On some border cases, no best way has been agreed upon on how to express @@ -53,7 +53,7 @@ One statement per line While some compound statements such as list comprehensions are allowed and appreciated for their brevity and their expressiveness, -it is bad practice to have two disjoint statements on the same line of code. +it is bad practice to have two disjointed statements on the same line of code. **Bad** @@ -124,13 +124,13 @@ inside the function) that was added "just in case" and is seemingly never used, than to add a new optional argument and its logic when needed. 3. The **arbitrary argument list** is the third way to pass arguments to a -function. If the function intention is better expressed by a signature with an -extensible number of positional arguments, it can be defined with the ``*args`` -constructs. In the function body, ``args`` will be a tuple of all the -remaining positional arguments. For example, ``send(message, *args)`` can be -called with each recipient as an argument: ``send('Hello', 'God', 'Mom', -'Cthulhu')``, and in the function body ``args`` will be equal to ``('God', -'Mom', 'Cthulhu')``. + function. If the function intention is better expressed by a signature with + an extensible number of positional arguments, it can be defined with the + ``*args`` constructs. In the function body, ``args`` will be a tuple of all + the remaining positional arguments. For example, ``send(message, *args)`` + can be called with each recipient as an argument:``send('Hello', 'God', + 'Mom', 'Cthulhu')``, and in the function body ``args`` will be equal to + ``('God', 'Mom', 'Cthulhu')``. However, this construct has some drawbacks and should be used with caution. If a function receives a list of arguments of the same nature, it is often more @@ -455,9 +455,9 @@ PEP 8 easy-to-read version of PEP 8 is also available at `pep8.org `_. This is highly recommended reading. The entire Python community does their -best to adhere to the guidelines laidout within this document. Some project +best to adhere to the guidelines laid out within this document. Some project may sway from it from time to time, while others may -`ammend its recommendations `_. +`amend its recommendations `_. That being said, conforming your Python code to PEP 8 is generally a good idea and helps make code more consistent when working on projects with other diff --git a/docs/writing/tests.rst b/docs/writing/tests.rst index 84e5d27..b8b7ca0 100644 --- a/docs/writing/tests.rst +++ b/docs/writing/tests.rst @@ -3,8 +3,8 @@ Testing Your Code Testing your code is very important. -Getting used to writing the testing code and the running code in parallel is -now considered a good habit. Used wisely, this method helps you define more +Getting used to writing testing code and running this code in parallel is now +considered a good habit. Used wisely, this method helps you define more precisely your code's intent and have a more decoupled architecture. Some general rules of testing: @@ -12,8 +12,8 @@ Some general rules of testing: - A testing unit should focus on one tiny bit of functionality and prove it correct. -- Each test unit must be fully independent. Each of them must be able to run - alone, and also within the test suite, regardless of the order they are +- Each test unit must be fully independent. Each test must be able to run + alone, and also within the test suite, regardless of the order that they are called. The implication of this rule is that each test must be loaded with a fresh dataset and may have to do some cleanup afterwards. This is usually handled by :meth:`setUp()` and :meth:`tearDown()` methods. @@ -27,8 +27,8 @@ Some general rules of testing: tests as often as needed. - Learn your tools and learn how to run a single test or a test case. Then, - when developing a function inside a module, run this function's tests very - often, ideally automatically when you save the code. + when developing a function inside a module, run this function's tests + frequently, ideally automatically when you save the code. - Always run the full test suite before a coding session, and run it again after. This will give you more confidence that you did not break anything @@ -63,11 +63,11 @@ Some general rules of testing: - Another use of the testing code is as an introduction to new developers. When someone will have to work on the code base, running and reading the related - testing code is often the best they can do. They will or should discover the - hot spots, where most difficulties arise, and the corner cases. If they have - to add some functionality, the first step should be to add a test and, by this - means, ensure the new functionality is not already a working path that has not - been plugged into the interface. + testing code is often the best thing that they can do to start. They will + or should discover the hot spots, where most difficulties arise, and the + corner cases. If they have to add some functionality, the first step should + be to add a test to ensure that the new functionality is not already a + working path that has not been plugged into the interface. The Basics :::::::::: @@ -202,7 +202,7 @@ nose provides automatic test discovery to save you the hassle of manually creating test suites. It also provides numerous plugins for features such as xUnit-compatible test output, coverage reporting, and test selection. - `nose `_ + `nose `_ tox