kennethreitz/python-guide
1
0
Fork 0
mirror of https://github.com/kennethreitz/python-guide.git synced 2026-06-05 23:00:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #258 from rgbkrk/master

Browse Source 
Tiny bit of grammar fixes and plenty of spacing fixes
This commit is contained in:
Kenneth Reitz
2013-03-22 22:12:24 -07:00
parent 581284da5f 7a1077a464
commit 8483ae30a9
9 changed files with 644 additions and 645 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Readme.rst
+6 -2
View File
@@ -5,9 +5,13 @@ Hitchhiker's Guide to Python
-----------
**Work in progress. If you'd like to help, please do. There's a lot of work to be done.**
**Work in progress. If you'd like to help, please do. There's a lot of work to
be done.**
This guide is currently under heavy development. This opinionated guide exists to provide both novice and expert Python developers a best-practice handbook to the installation, configuration, and usage of Python on a daily basis.
This guide is currently under heavy development. This opinionated guide
exists to provide both novice and expert Python developers a best-practice
handbook to the installation, configuration, and usage of Python on a daily
basis.
Topics include:
docs/dev/env.rst
+19 -24
View File
@@ -64,21 +64,17 @@ to do that which also shows status and warning messages in the statusbar would b
Python-mode
^^^^^^^^^^^
Python-mode_ is complex solution in VIM for work with python code.
Python-mode_ is a complex solution in VIM for working with python code.
It has:
- Async python code checking (pylint, pyflakes, pep8, mccabe) in any combination;
- Code refactoring and autocompletion with Rope;
- Fastest python folding;
- Nice and powered python syntax;
- Virtual env support;
- Search by python documentation and run python code;
- More other things like auto pep8 error fixes;
- Very customizable and documented as well;
- Have all required libraries in self;
And more stuff.
- Async python code checking (pylint, pyflakes, pep8, mccabe) in any combination
- Code refactoring and autocompletion with Rope
- Fast python folding
- Virtualenv support
- Search by python documentation and run python code
- Auto pep8 error fixes
And more.
.. _indent: http://www.vim.org/scripts/script.php?script_id=974
.. _syntax: http://www.vim.org/scripts/script.php?script_id=790
@@ -101,8 +97,8 @@ already an Emacs user is `Python Programming in Emacs`_ at EmacsWiki.
1. Emacs itself comes with a python mode.
2. Python ships with an alternate version:
`python-mode.el <https://launchpad.net/python-mode>`_
3. Fabián Ezequiel Gallina's provides nice functionality and
behavior out of the box: `python.el <https://github.com/fgallina/python.el>`_
3. Fabián Ezequiel Gallina's `python.el <https://github.com/fgallina/python.el>`_
provides nice functionality and behavior out of the box
.. _Python Programming in Emacs: http://emacswiki.org/emacs/PythonProgrammingInEmacs
@@ -145,16 +141,15 @@ The most popular Eclipse plugin for Python development is Aptana's
Komodo IDE
----------
`Komodo IDE <http://www.activestate.com/komodo-ide>`_ is developed by
ActiveState and is a commercial IDE for Windows, Mac
and Linux.
ActiveState and is a commercial IDE for Windows, Mac and Linux.
Spyder
------
`Spyder <http://code.google.com/p/spyderlib/>`_ an IDE specifically geared
`Spyder <http://code.google.com/p/spyderlib/>`_ is an IDE specifically geared
toward working with scientific python libraries (namely `Scipy <http://www.scipy.org/>`_).
Includes integration with pyflakes_, `pylint <http://www.logilab.org/857>`_,
It includes integration with pyflakes_, `pylint <http://www.logilab.org/857>`_,
and `rope <http://rope.sourceforge.net/>`_.
Spyder is open-source (free), offers code completion, syntax highlighting,
@@ -164,7 +159,7 @@ class and function browser, and object inspection.
WingIDE
-------
`WingIDE <http://wingware.com/>`_ a python specific IDE. Runs for Linux,
`WingIDE <http://wingware.com/>`_ is a python specific IDE. It runs on Linux,
Windows, and Mac (as an X11 application, which frustrates some Mac users).
@@ -172,7 +167,7 @@ NINJA-IDE
---------
`NINJA-IDE <http://www.ninja-ide.org/>`_ (from the recursive acronym: "Ninja-IDE
Is Not Just Another IDE", is a cross-platform IDE, specially designed to build
Is Not Just Another IDE") is a cross-platform IDE, specially designed to build
Python applications, and runs on Linux/X11, Mac OS X and Windows desktop operating
systems. Installers for these platforms can be downloaded from the website.
@@ -211,7 +206,7 @@ library which you can use to install other packages. The name of the
virtual environment (in this case, it was ``venv``) can be anything;
omitting the name will place the files in the current directory instead.
In order the start using the virtual environment, run::
To start using the virtual environment, run::
$ source venv/bin/activate
@@ -288,12 +283,12 @@ IDLE
`IDLE <http://docs.python.org/library/idle.html>`_ is an integrated
development environment that is part of Python standard library. It is
completely written in Python and uses Tkinter GUI toolkit. Though IDLE
is not suited for full-blown development using Python , it is quite
completely written in Python and uses the Tkinter GUI toolkit. Though IDLE
is not suited for full-blown development using Python, it is quite
helpful to try out small Python snippets and experiment with different
features in Python.
It provides following features:
It provides the following features:
* Python Shell Window (interpreter)
* Multi window text editor that colorizes Python code
docs/intro/duction.rst
+88 -88
View File
@@ -1,88 +1,88 @@
Introduction
============
From the `official Python website <http://python.org/about/>`_:
Python is a general-purpose, high-level programming language similar
to Tcl, Perl, Ruby, Scheme, or Java. Some of its main key features
include:
* very clear, readable syntax
Python's philosophy focuses on readability, from code blocks
delineated with significant whitespace to intuitive keywords in
place of inscrutable punctuation
* extensive standard libraries and third party modules for virtually
any task
Python is sometimes described with the words "batteries included"
for its extensive
`standard library <http://docs.python.org/library/>`_, which can
includes modules for regular expressions, file IO, fraction handling,
object serialization, and much more.
Additionally, the
`Python Package Index <http://pypi.python.org/pypi/>`_ is available
for users to submit their packages for widespread use, similar to
Perl's `CPAN <http://www.cpan.org>`_. There is a thriving community
of very powerful Python frameworks and tools like
the `Django <http://www.djangoproject.com>`_ web framework and the
`NumPy <http://numpy.scipy.org>`_ set of math routines.
* integration with other systems
Python can integrate with `Java libraries <http://www.jython.org>`_,
enabling it to be used with the rich Java environment that corporate
programmers are used to. It can also be
`extended by C or C++ modules <http://docs.python.org/extending/>`_
when speed is of the essence.
* ubiquity on computers
Python is available on Windows, \*nix, and Mac. It runs wherever the
Java virtual machine runs, and the reference implementation CPython
can help bring Python to wherever there is a working C compiler.
* friendly community
Python has a vibrant and large :ref:`community <the-community>`
which maintains wikis, conferences, countless repositories,
mailing lists, IRC channels, and so much more. Heck, the Python
community is even helping to write this guide!
.. _about-ref:
About This Guide
----------------
Purpose
~~~~~~~
The Hitchhiker's Guide to Python exists to provide both novice and expert
Python developers a best-practice handbook to the installation, configuration,
and usage of Python on a daily basis.
By the Community
~~~~~~~~~~~~~~~~
This guide is architected and maintained by `Kenneth Reitz
<https://github.com/kennethreitz>`_ in an open fashion. This is a
community-driven effort that serves one purpose: to serve the community.
For the Community
~~~~~~~~~~~~~~~~~
All contributions to the Guide are welcome, from Pythonistas of all levels.
If you think there's a gap in what the Guide covers, fork the Guide on
GitHub and submit a pull request. Contributions are welcome from everyone,
whether they're an old hand or a first-time Pythonista, and the authors to
the Guide will gladly help if you have any questions about the
appropriateness, completeness, or accuracy of a contribution.
To get started working on The Hitchhiker's Guide, see
the: doc:`/notes/contribute` page.
Introduction
============
From the `official Python website <http://python.org/about/>`_:
Python is a general-purpose, high-level programming language similar
to Tcl, Perl, Ruby, Scheme, or Java. Some of its main key features
include:
* very clear, readable syntax
Python's philosophy focuses on readability, from code blocks
delineated with significant whitespace to intuitive keywords in
place of inscrutable punctuation
* extensive standard libraries and third party modules for virtually
any task
Python is sometimes described with the words "batteries included"
for its extensive
`standard library <http://docs.python.org/library/>`_, which includes
modules for regular expressions, file IO, fraction handling,
object serialization, and much more.
Additionally, the
`Python Package Index <http://pypi.python.org/pypi/>`_ is available
for users to submit their packages for widespread use, similar to
Perl's `CPAN <http://www.cpan.org>`_. There is a thriving community
of very powerful Python frameworks and tools like
the `Django <http://www.djangoproject.com>`_ web framework and the
`NumPy <http://numpy.scipy.org>`_ set of math routines.
* integration with other systems
Python can integrate with `Java libraries <http://www.jython.org>`_,
enabling it to be used with the rich Java environment that corporate
programmers are used to. It can also be
`extended by C or C++ modules <http://docs.python.org/extending/>`_
when speed is of the essence.
* ubiquity on computers
Python is available on Windows, \*nix, and Mac. It runs wherever the
Java virtual machine runs, and the reference implementation CPython
can help bring Python to wherever there is a working C compiler.
* friendly community
Python has a vibrant and large :ref:`community <the-community>`
which maintains wikis, conferences, countless repositories,
mailing lists, IRC channels, and so much more. Heck, the Python
community is even helping to write this guide!
.. _about-ref:
About This Guide
----------------
Purpose
~~~~~~~
The Hitchhiker's Guide to Python exists to provide both novice and expert
Python developers a best-practice handbook to the installation, configuration,
and usage of Python on a daily basis.
By the Community
~~~~~~~~~~~~~~~~
This guide is architected and maintained by `Kenneth Reitz
<https://github.com/kennethreitz>`_ in an open fashion. This is a
community-driven effort that serves one purpose: to serve the community.
For the Community
~~~~~~~~~~~~~~~~~
All contributions to the Guide are welcome, from Pythonistas of all levels.
If you think there's a gap in what the Guide covers, fork the Guide on
GitHub and submit a pull request. Contributions are welcome from everyone,
whether they're an old hand or a first-time Pythonista, and the authors to
the Guide will gladly help if you have any questions about the
appropriateness, completeness, or accuracy of a contribution.
To get started working on The Hitchhiker's Guide, see
the: doc:`/notes/contribute` page.
docs/notes/styleguide.rst
+1 -1
View File
@@ -4,7 +4,7 @@
The Guide Style Guide
=====================
As with all documentation, having a consistent formating helps make the
As with all documentation, having a consistent format helps make the
document more understandable. In order to make The Guide easier to digest,
all contributions should fit within the rules of this style guide where
appropriate.
docs/shipping/freezing.rst
+6 -6
View File
@@ -20,19 +20,19 @@ Comparison
Solutions and platforms/features supported:
=========== ======= ===== ==== ======== ======= ============= ============== ==== =====================
Solution Windows Linux OS X Python 3 Licence One-file mode Zipfile import Eggs pkg_resources support
Solution Windows Linux OS X Python 3 License One-file mode Zipfile import Eggs pkg_resources support
=========== ======= ===== ==== ======== ======= ============= ============== ==== =====================
bbFreeze yes yes yes no MIT no yes yes yes
py2exe yes no no no MIT yes yes no no
pyInstaller yes yes yes no GPL yes no yes no
cx_Freeze yes yes yes yes PSF no yes yes no
bbFreeze yes yes yes no MIT no yes yes yes
py2exe yes no no no MIT yes yes no no
pyInstaller yes yes yes no GPL yes no yes no
cx_Freeze yes yes yes yes PSF no yes yes no
=========== ======= ===== ==== ======== ======= ============= ============== ==== =====================
.. todo:: Add other solutions: py2app
.. note::
Freezing Python code on Linux into a Windows executable was only once
supported in PyInstaller, `but later dropped
supported in PyInstaller `and later dropped.
<http://stackoverflow.com/questions/2950971/cross-compiling-a-python-script-on-linux-into-a-windows-executable#comment11890276_2951046>`_.
.. note::
docs/starting/install/win.rst
+1 -1
View File
@@ -4,7 +4,7 @@ Installing Python on Windows
============================
First, download the `latest version <http://python.org/ftp/python/2.7.3/python-2.7.3.msi>`_
of Python 2 from the official Website. If you want to be sure you are installing a fully
of Python 2.7 from the official Website. If you want to be sure you are installing a fully
up-to-date version then use the "Windows Installer" link from the home page of the
`Python.org web site <http://python.org>`_ .
docs/starting/which-python.rst
+1 -1
View File
@@ -18,7 +18,7 @@ Which Python to use?
Today
-----
If you're choosing a Python interpreter to use, I *highly* recommend you Use
If you're choosing a Python interpreter to use, I *highly* recommend you use
Python 2.7.x, unless you have a strong reason not to.
docs/writing/reading.rst
+43 -43
View File
@@ -1,44 +1,44 @@
Reading Great Code
==================
One of the core tenants behind the design of Python is creating
readable code. The motivation behind this design is simple: The number
one thing that Python programmers do is read code.
One of the secrets of becoming a great Python programmer is to read,
understand, and comprehend excellent code.
Excellent code typically follows the guidelines outlined in :ref:`code_style`,
and does its best to express a clear and concise intent to the reader.
Included below is a list of recommended Python projects for reading. Each of
these projects are paragons of excellent Python code.
- `Howdoi <https://github.com/gleitz/howdoi>`_
Howdoi is a code search tool, written in Python.
- `Flask <https://github.com/mitsuhiko/flask>`_
Flask is a microframework for Python based on Werkzeug and Jinja2.
It's intended for getting started very quickly and was developed with
best intentions in mind.
- `Werkzeug <https://github.com/mitsuhiko/werkzeug>`_
Werkzeug started as simple collection of various utilities for WSGI
applications and has become one of the most advanced WSGI utility modules.
It includes a powerful debugger, full-featured request and response objects,
HTTP utilities to handle entity tags, cache control headers, HTTP dates,
cookie handling, file uploads, a powerful URL routing system and a bunch
of community-contributed addon modules.
- `Requests <https://github.com/kennethreitz/requests>`_
Requests is an Apache2 Licensed HTTP library, written in Python,
for human beings.
- `Tablib <https://github.com/kennethreitz/tablib>`_
Tablib is a format-agnostic tabular dataset library, written in Python.
.. todo:: Embed and explain YouTube video showing python code reading: http://www.youtube.com/watch?v=Jc8M9-LoEuo This may require installing a Sphinx plugin. https://bitbucket.org/birkenfeld/sphinx-contrib/src/a09f29fc16970f34350ca36ac7f229e00b1b1674/youtube?at=default
.. todo:: Include code examples of exemplary code from each of the projects listed. Explain why it is excellent code. Use complex examples.
Reading Great Code
==================
One of the core tenants behind the design of Python is creating
readable code. The motivation behind this design is simple: The number
one thing that Python programmers do is read code.
One of the secrets of becoming a great Python programmer is to read,
understand, and comprehend excellent code.
Excellent code typically follows the guidelines outlined in :ref:`code_style`,
and does its best to express a clear and concise intent to the reader.
Included below is a list of recommended Python projects for reading. Each of
these projects are paragons of excellent Python code.
- `Howdoi <https://github.com/gleitz/howdoi>`_
Howdoi is a code search tool, written in Python.
- `Flask <https://github.com/mitsuhiko/flask>`_
Flask is a microframework for Python based on Werkzeug and Jinja2.
It's intended for getting started very quickly and was developed with
best intentions in mind.
- `Werkzeug <https://github.com/mitsuhiko/werkzeug>`_
Werkzeug started as simple collection of various utilities for WSGI
applications and has become one of the most advanced WSGI utility modules.
It includes a powerful debugger, full-featured request and response objects,
HTTP utilities to handle entity tags, cache control headers, HTTP dates,
cookie handling, file uploads, a powerful URL routing system and a bunch
of community-contributed addon modules.
- `Requests <https://github.com/kennethreitz/requests>`_
Requests is an Apache2 Licensed HTTP library, written in Python,
for human beings.
- `Tablib <https://github.com/kennethreitz/tablib>`_
Tablib is a format-agnostic tabular dataset library, written in Python.
.. todo:: Embed and explain YouTube video showing python code reading: http://www.youtube.com/watch?v=Jc8M9-LoEuo This may require installing a Sphinx plugin. https://bitbucket.org/birkenfeld/sphinx-contrib/src/a09f29fc16970f34350ca36ac7f229e00b1b1674/youtube?at=default
.. todo:: Include code examples of exemplary code from each of the projects listed. Explain why it is excellent code. Use complex examples.
.. todo:: Explain techniques to rapidly identify data structures, algorithms and determine what the code is doing.
docs/writing/structure.rst
+479 -479
View File
@@ -1,479 +1,479 @@
Structuring Your Project
========================
By "structure" we mean the decisions you make concerning
how your project best meets its objective. We need to consider how to
best leverage Python's features to create clean, effective code.
In practical terms, "structure" means making clean code whose logic and
dependencies are clear as well as how the files and folders are organized
in the filesystem.
Which functions should go into which modules? How does data flow through
the project? What features and functions can be grouped together and
isolated? By answering questions like these you can begin to plan, in
a broad sense, what your finished product will look like.
In this section we take a closer look at Python's module and import
systems as they are the central element to enforcing structure in your
project. We then discuss various perspectives on how to build code which
can be extended and tested reliably.
Structure is Key
----------------
Thanks to the way imports and modules are handled in Python, it is
relatively easy to structure a Python project. Easy, here, means
that you do not have many constraints and that the module
importing model is easy to grasp. Therefore, you are left with the
pure architectural task of crafting the different parts of your
project and their interactions.
Easy structuring of a project means it is also easy
to do it poorly. Some signs of a poorly structured project
include:
- Multiple and messy circular dependencies: if your classes
Table and Chair in furn.py need to import Carpenter from workers.py
to answer a question such as table.isdoneby(),
and if conversely the class Carpenter needs to import Table and Chair,
to answer the question carpenter.whatdo(), then you
have a circular dependency. In this case you will have to resort to
fragile hacks such as using import statements inside
methods or functions.
- Hidden coupling: each and every change in Table's implementation
breaks 20 tests in unrelated test cases because it breaks Carpenter's code,
which requires very careful surgery to adapt the change. This means
you have too many assumptions about Table in Carpenter's code or the
reverse.
- Heavy usage of global state or context: instead of explicitly
passing ``(height, width, type, wood)`` to each other, Table
and Carpenter rely on global variables that can be modified
and are modified on the fly by different agents. You need to
scrutinize all access to these global variables to understand why
a rectangular table became a square, and discover that remote
template code is also modifying this context, messing with
table dimensions.
- Spaghetti code: multiple pages of nested if clauses and for loops
with a lot of copy-pasted procedural code and no
proper segmentation are known as spaghetti code. Python's
meaningful indentation (one of its most controversial features) make
it very hard to maintain this kind of code. So the good news is that
you might not see too much of it.
- Ravioli code is more likely in Python: it consists of hundreds of
similar little pieces of logic, often classes or objects, without
proper structure. If you never can remember if you have to use
FurnitureTable, AssetTable or Table, or even TableNew for your
task at hand, you might be swimming in ravioli code.
Modules
-------
Python modules are one of the main abstraction layers available and probably the
most natural one. Abstraction layers allow separating code into parts holding
related data and functionality.
For example, a layer of a project can handle interfacing with user actions,
while another would handle low-level manipulation of data. The most natural way
to separate these two layers is to regroup all interfacing functionality
in one file, and all low-level operations in another file. In this case,
the interface file needs to import the low-level file. This is done with the
`import` and `from ... import` statements.
As soon as you use `import` statements you use modules. These can be either built-in
modules such as `os` and `sys`, third-party modules you have installed in your
environment, or your project's internal modules.
To keep in line with the style guide, keep module names short, lowercase, and
be sure to avoid using special symbols like the dot (.) or question mark (?).
So a file name like `my.spam.py` is one you should avoid! Naming this way
will interfere with the way python looks for modules.
In this example python expects to find a "spam.py" file in a folder named "my"
which is not the case. There is an
`example <http://docs.python.org/tutorial/modules.html#packages>`_ of how the
dot notation should be used in the python docs.
If you'd like you could name it as `my_spam.py` but even our friend the
underscore should not be seen often in module names.
Aside for some naming restrictions, nothing special is required for a Python file
to be a module, but the import mechanism needs to be understood in order to use
this concept properly and avoid some issues.
Concretely, the `import modu` statement will look for the proper file, which is
`modu.py` in the same directory as the caller if it exists. If it is not
found, the Python interpreter will search for `modu.py` in the "path"
recursively and raise an ImportError exception if it is not found.
Once `modu.py` is found, the Python interpreter will execute the module in an
isolated scope. Any top-level statement in `modu.py` will be executed,
including other imports if any. Function and class definitions are stored in
the module's dictionary.
Then, the module's variables, functions, and classes will be available to the caller
through the module's namespace, a central concept in programming that is
particularly helpful and powerful in Python.
In many languages, an `include file` directive is used by the preprocessor to
take all code found in the file and 'copy' it into the caller's code. It is
different in Python: the included code is isolated in a module namespace, which
means that you generally don't have to worry that the included code could have
unwanted effects, e.g. override an existing function with the same name.
It is possible to simulate the more standard behavior by using a special syntax
of the import statement: `from modu import *`. This is generally considered bad
practice. **Using `import *` makes code harder to read and makes dependencies less
compartmentalized**.
Using `from modu import func` is a way to pinpoint the function you want to
import and put it in the global namespace. While much less harmful than `import
*` because it shows explicitly what is imported in the global namespace, its
advantage over a simpler `import modu` is only that it will save some typing.
**Very bad**
.. code-block:: python
[...]
from modu import *
[...]
x = sqrt(4) # Is sqrt part of modu? A builtin? Defined above?
**Better**
.. code-block:: python
from modu import sqrt
[...]
x = sqrt(4) # sqrt may be part of modu, if not redefined in between
**Best**
.. code-block:: python
import modu
[...]
x = modu.sqrt(4) # sqrt is visibly part of modu's namespace
As said in the section about style, readability is one of the main features of
Python. Readability means to avoid useless boilerplate text and clutter,
therefore some efforts are spent trying to achieve a certain level of brevity.
But terseness and obscurity are the limits where brevity should stop. Being
able to tell immediately where a class or function comes from, as in the
`modu.func` idiom, greatly improves code readability and understandability in
all but the simplest single file projects.
Packages
--------
Python provides a very straightforward packaging system, which is simply an
extension of the module mechanism to a directory.
Any directory with an __init__.py file is considered a Python package. The
different modules in the package are imported in a similar manner as plain
modules, but with a special behavior for the __init__.py file, which is used to
gather all package-wide definitions.
A file modu.py in the directory pack/ is imported with the statement `import
pack.modu`. This statement will look for an __init__.py file in `pack`, execute
all of its top-level statements. Then it will look for a file `pack/modu.py` and
execute all of its top-level statements. After these operations, any variable,
function, or class defined in modu.py is available in the pack.modu namespace.
A commonly seen issue is to add too much code to __init__.py
files. When the project complexity grows, there may be sub-packages and
sub-sub-packages in a deep directory structure, and then, importing a single item
from a sub-sub-package will require executing all __init__.py files met while
traversing the tree.
Leaving an __init__.py file empty is considered normal and even a good practice,
if the package's modules and sub-packages do not need to share any code.
Lastly, a convenient syntax is available for importing deeply nested packages:
`import very.deep.module as mod`. This allows you to use `mod` in place of the verbose
repetition of `very.deep.module`.
Object-oriented programming
---------------------------
Python is sometimes described as an object-oriented programming language. This
can be somewhat misleading and needs to be clarified.
In Python, everything is an object, and can be handled as such. This is what is
meant when we say that, for example, functions are first-class objects.
Functions, classes, strings, and even types are objects in Python: like any
objects, they have a type, they can be passed as function arguments, they may
have methods and properties. In this understanding, Python is an
object-oriented language.
However, unlike Java, Python does not impose object-oriented programming as the
main programming paradigm. It is perfectly viable for a Python project to not
be object-oriented, i.e. to use no or very few class definitions, class
inheritance, or any other mechanisms that are specific to object-oriented
programming.
Moreover, as seen in the modules_ section, the way Python handles modules and
namespaces gives the developer a natural way to ensure the
encapsulation and separation of abstraction layers, both being the most common
reasons to use object-orientation. Therefore, Python programmers have more
latitude to not use object-orientation, when it is not required by the business
model.
There are some reasons to avoid unnecessary object-orientation. Defining
custom classes is useful when we want to glue together some state and some
functionality. The problem, as pointed out by the discussions about functional
programming, comes from the "state" part of the equation.
In some architectures, typically web applications, multiple instances of Python
processes are spawned to respond to external requests that can
happen at the same time. In this case, holding some state into instantiated
objects, which means keeping some static information about the world, is prone
to concurrency problems or race-conditions. Sometimes, between the initialization of
the state of an object (usually done with the __init__() method) and the actual use
of the object state through one of its methods, the world may have changed, and
the retained state may be outdated. For example, a request may load an item in
memory and mark it as read by a user. If another request requires the deletion
of this item at the same, it may happen that the deletion actually occurs after
the first process loaded the item, and then we have to mark as read a deleted
object.
This and other issues led to the idea that using stateless functions is a
better programming paradigm.
Another way to say the same thing is to suggest using functions and procedures
with as few implicit contexts and side-effects as possible. A function's
implicit context is made up of any of the global variables or items in the persistence layer
that are accessed from within the function. Side-effects are the changes that a function makes
to its implicit context. If a function saves or deletes data in a global variable or
in the persistence layer, it is said to have a side-effect.
Carefully isolating functions with context and side-effects from functions with
logic (called pure functions) allow the following benefits:
- Pure functions are deterministic: given a fixed input,
the output will always be the same.
- Pure functions are much easier to change or replace if they need to
be refactored or optimized.
- Pure functions are easier to test with unit-tests: There is less
need for complex context setup and data cleaning afterwards.
- Pure functions are easier to manipulate, decorate, and pass-around.
In summary, pure functions, without any context or side-effects, are more
efficient building blocks than classes and objects for some architectures.
Obviously, object-orientation is useful and even necessary in many cases, for
example when developing graphical desktop applications or games, where the
things that are manipulated (windows, buttons, avatars, vehicles) have a
relatively long life of their own in the computer's memory.
Decorators
----------
The Python language provides a simple yet powerful syntax called 'decorators'.
A decorator is a function or a class that wraps (or decorates) a function
or a method. The 'decorated' function or method will replace the original
'undecorated' function or method. Because functions are first-class objects
in Python, it can be done 'manually', but using the @decorator syntax is
clearer and thus preferred.
.. code-block:: python
def foo():
# do something
def decorator(func):
# manipulate func
return func
foo = decorator(foo) # Manually decorate
@decorator
def bar():
# Do something
# bar() is decorated
This mechanism is useful for separating concerns and avoiding
external un-related logic 'polluting' the core logic of the function
or method. A good example of a piece of functionality that is better handled
with decoration is memoization or caching: you want to store the results of an
expensive function in a table and use them directly instead of recomputing
them when they have already been computed. This is clearly not part
of the function logic.
Dynamic typing
--------------
Python is said to be dynamically typed, which means that variables
do not have a fixed type. In fact, in Python, variables are very
different from what they are in many other languages, specifically
strongly-typed languages. Variables are not a segment of the computer's
memory where some value is written, they are 'tags' or 'names' pointing
to objects. It is therefore possible for the variable 'a' to be set to
the value 1, then to the value 'a string', then to a function.
The dynamic typing of Python is often considered to be a weakness, and indeed
it can lead to complexities and hard-to-debug code. Something
named 'a' can be set to many different things, and the developer or the
maintainer needs to track this name in the code to make sure it has not
been set to a completely unrelated object.
Some guidelines help to avoid this issue:
- Avoid using the same variable name for different things.
**Bad**
.. code-block:: python
a = 1
a = 'a string'
def a():
pass # Do something
**Good**
.. code-block:: python
count = 1
msg = 'a string'
def func():
pass # Do something
Using short functions or methods helps reduce the risk
of using the same name for two unrelated things.
It is better to use different names even for things that are related,
when they have a different type:
**Bad**
.. code-block:: python
items = 'a b c d' # This is a string...
items = items.split(' ') # ...becoming a list
items = set(items) # ...and then a set
There is no efficiency gain when reusing names: the assignments
will have to create new objects anyway. However, when the complexity
grows and each assignment is separated by other lines of code, including
'if' branches and loops, it becomes harder to ascertain what a given
variable's type is.
Some coding practices, like functional programming, recommend never reassigning a variable.
In Java this is done with the `final` keyword. Python does not have a `final` keyword
and it would be against its philosophy anyway. However, it may be a good
discipline to avoid assigning to a variable more than once, and it helps
in grasping the concept of mutable and immutable types.
Mutable and immutable types
---------------------------
Python has two kinds of built-in or user-defined types.
Mutable types are those that allow in-place modification
of the content. Typical mutables are lists and dictionaries:
All lists have mutating methods, like append() or pop(), and
can be modified in place. The same goes for dictionaries.
Immutable types provide no method for changing their content.
For instance, the variable x set to the integer 6 has no "increment" method. If you
want to compute x + 1, you have to create another integer and give it
a name.
.. code-block:: python
my_list = [1, 2, 3]
my_list[0] = 4
print my_list # [4, 2, 3] <- The same list as changed
x = 6
x = x + 1 # The new x is another object
One consequence of this difference in behavior is that mutable
types are not "stable", and therefore cannot be used as dictionary
keys.
Using properly mutable types for things that are mutable in nature
and immutable types for things that are fixed in nature
helps to clarify the intent of the code.
For example, the immutable equivalent of a list is the tuple, created
with ``(1, 2)``. This tuple is a pair that cannot be changed in-place,
and can be used as a key for a dictionary.
One peculiarity of Python that can surprise beginners is that
strings are immutable. This means that when constructing a string from
its parts, it is much more efficient to accumulate the parts in a list,
which is mutable, and then glue ('join') the parts together when the
full string is needed. One thing to notice, however, is that list
comprehensions are better and faster than constructing a list in a loop
with calls to append().
**Bad**
.. code-block:: python
# create a concatenated string from 0 to 19 (e.g. "012..1819")
nums = ""
for n in range(20):
nums += str(n) # slow and inefficient
print nums
**Good**
.. code-block:: python
# create a concatenated string from 0 to 19 (e.g. "012..1819")
nums = []
for n in range(20):
nums.append(str(n))
print "".join(nums) # much more efficient
**Best**
.. code-block:: python
# create a concatenated string from 0 to 19 (e.g. "012..1819")
nums = [str(n) for n in range(20)]
print "".join(nums)
One final thing to mention about strings is that using join() is not always
best. In the instances where you are creating a new string from a pre-determined
number of strings, using the addition operator is actually faster, but in cases
like above or in cases where you are adding to an existing string, using join()
should be your preferred method.
.. code-block:: python
foo = 'foo'
bar = 'bar'
foobar = foo + bar # This is good
foo += 'ooo' # This is bad, instead you should do:
foo = ''.join([foo, 'ooo'])
Vendorizing Dependencies
------------------------
Runners
-------
Further Reading
---------------
- http://docs.python.org/2/library/
- http://www.diveintopython.net/toc/index.html
Structuring Your Project
========================
By "structure" we mean the decisions you make concerning
how your project best meets its objective. We need to consider how to
best leverage Python's features to create clean, effective code.
In practical terms, "structure" means making clean code whose logic and
dependencies are clear as well as how the files and folders are organized
in the filesystem.
Which functions should go into which modules? How does data flow through
the project? What features and functions can be grouped together and
isolated? By answering questions like these you can begin to plan, in
a broad sense, what your finished product will look like.
In this section we take a closer look at Python's module and import
systems as they are the central element to enforcing structure in your
project. We then discuss various perspectives on how to build code which
can be extended and tested reliably.
Structure is Key
----------------
Thanks to the way imports and modules are handled in Python, it is
relatively easy to structure a Python project. Easy, here, means
that you do not have many constraints and that the module
importing model is easy to grasp. Therefore, you are left with the
pure architectural task of crafting the different parts of your
project and their interactions.
Easy structuring of a project means it is also easy
to do it poorly. Some signs of a poorly structured project
include:
- Multiple and messy circular dependencies: if your classes
Table and Chair in furn.py need to import Carpenter from workers.py
to answer a question such as table.isdoneby(),
and if conversely the class Carpenter needs to import Table and Chair,
to answer the question carpenter.whatdo(), then you
have a circular dependency. In this case you will have to resort to
fragile hacks such as using import statements inside
methods or functions.
- Hidden coupling: each and every change in Table's implementation
breaks 20 tests in unrelated test cases because it breaks Carpenter's code,
which requires very careful surgery to adapt the change. This means
you have too many assumptions about Table in Carpenter's code or the
reverse.
- Heavy usage of global state or context: instead of explicitly
passing ``(height, width, type, wood)`` to each other, Table
and Carpenter rely on global variables that can be modified
and are modified on the fly by different agents. You need to
scrutinize all access to these global variables to understand why
a rectangular table became a square, and discover that remote
template code is also modifying this context, messing with
table dimensions.
- Spaghetti code: multiple pages of nested if clauses and for loops
with a lot of copy-pasted procedural code and no
proper segmentation are known as spaghetti code. Python's
meaningful indentation (one of its most controversial features) make
it very hard to maintain this kind of code. So the good news is that
you might not see too much of it.
- Ravioli code is more likely in Python: it consists of hundreds of
similar little pieces of logic, often classes or objects, without
proper structure. If you never can remember if you have to use
FurnitureTable, AssetTable or Table, or even TableNew for your
task at hand, you might be swimming in ravioli code.
Modules
-------
Python modules are one of the main abstraction layers available and probably the
most natural one. Abstraction layers allow separating code into parts holding
related data and functionality.
For example, a layer of a project can handle interfacing with user actions,
while another would handle low-level manipulation of data. The most natural way
to separate these two layers is to regroup all interfacing functionality
in one file, and all low-level operations in another file. In this case,
the interface file needs to import the low-level file. This is done with the
`import` and `from ... import` statements.
As soon as you use `import` statements you use modules. These can be either built-in
modules such as `os` and `sys`, third-party modules you have installed in your
environment, or your project's internal modules.
To keep in line with the style guide, keep module names short, lowercase, and
be sure to avoid using special symbols like the dot (.) or question mark (?).
So a file name like `my.spam.py` is one you should avoid! Naming this way
will interfere with the way python looks for modules.
In this example python expects to find a "spam.py" file in a folder named "my"
which is not the case. There is an
`example <http://docs.python.org/tutorial/modules.html#packages>`_ of how the
dot notation should be used in the python docs.
If you'd like you could name it as `my_spam.py` but even our friend the
underscore should not be seen often in module names.
Aside for some naming restrictions, nothing special is required for a Python file
to be a module, but the import mechanism needs to be understood in order to use
this concept properly and avoid some issues.
Concretely, the `import modu` statement will look for the proper file, which is
`modu.py` in the same directory as the caller if it exists. If it is not
found, the Python interpreter will search for `modu.py` in the "path"
recursively and raise an ImportError exception if it is not found.
Once `modu.py` is found, the Python interpreter will execute the module in an
isolated scope. Any top-level statement in `modu.py` will be executed,
including other imports if any. Function and class definitions are stored in
the module's dictionary.
Then, the module's variables, functions, and classes will be available to the caller
through the module's namespace, a central concept in programming that is
particularly helpful and powerful in Python.
In many languages, an `include file` directive is used by the preprocessor to
take all code found in the file and 'copy' it into the caller's code. It is
different in Python: the included code is isolated in a module namespace, which
means that you generally don't have to worry that the included code could have
unwanted effects, e.g. override an existing function with the same name.
It is possible to simulate the more standard behavior by using a special syntax
of the import statement: `from modu import *`. This is generally considered bad
practice. **Using `import *` makes code harder to read and makes dependencies less
compartmentalized**.
Using `from modu import func` is a way to pinpoint the function you want to
import and put it in the global namespace. While much less harmful than `import
*` because it shows explicitly what is imported in the global namespace, its
advantage over a simpler `import modu` is only that it will save some typing.
**Very bad**
.. code-block:: python
[...]
from modu import *
[...]
x = sqrt(4) # Is sqrt part of modu? A builtin? Defined above?
**Better**
.. code-block:: python
from modu import sqrt
[...]
x = sqrt(4) # sqrt may be part of modu, if not redefined in between
**Best**
.. code-block:: python
import modu
[...]
x = modu.sqrt(4) # sqrt is visibly part of modu's namespace
As said in the section about style, readability is one of the main features of
Python. Readability means to avoid useless boilerplate text and clutter,
therefore some efforts are spent trying to achieve a certain level of brevity.
But terseness and obscurity are the limits where brevity should stop. Being
able to tell immediately where a class or function comes from, as in the
`modu.func` idiom, greatly improves code readability and understandability in
all but the simplest single file projects.
Packages
--------
Python provides a very straightforward packaging system, which is simply an
extension of the module mechanism to a directory.
Any directory with an __init__.py file is considered a Python package. The
different modules in the package are imported in a similar manner as plain
modules, but with a special behavior for the __init__.py file, which is used to
gather all package-wide definitions.
A file modu.py in the directory pack/ is imported with the statement `import
pack.modu`. This statement will look for an __init__.py file in `pack`, execute
all of its top-level statements. Then it will look for a file `pack/modu.py` and
execute all of its top-level statements. After these operations, any variable,
function, or class defined in modu.py is available in the pack.modu namespace.
A commonly seen issue is to add too much code to __init__.py
files. When the project complexity grows, there may be sub-packages and
sub-sub-packages in a deep directory structure, and then, importing a single item
from a sub-sub-package will require executing all __init__.py files met while
traversing the tree.
Leaving an __init__.py file empty is considered normal and even a good practice,
if the package's modules and sub-packages do not need to share any code.
Lastly, a convenient syntax is available for importing deeply nested packages:
`import very.deep.module as mod`. This allows you to use `mod` in place of the verbose
repetition of `very.deep.module`.
Object-oriented programming
---------------------------
Python is sometimes described as an object-oriented programming language. This
can be somewhat misleading and needs to be clarified.
In Python, everything is an object, and can be handled as such. This is what is
meant when we say that, for example, functions are first-class objects.
Functions, classes, strings, and even types are objects in Python: like any
objects, they have a type, they can be passed as function arguments, they may
have methods and properties. In this understanding, Python is an
object-oriented language.
However, unlike Java, Python does not impose object-oriented programming as the
main programming paradigm. It is perfectly viable for a Python project to not
be object-oriented, i.e. to use no or very few class definitions, class
inheritance, or any other mechanisms that are specific to object-oriented
programming.
Moreover, as seen in the modules_ section, the way Python handles modules and
namespaces gives the developer a natural way to ensure the
encapsulation and separation of abstraction layers, both being the most common
reasons to use object-orientation. Therefore, Python programmers have more
latitude to not use object-orientation, when it is not required by the business
model.
There are some reasons to avoid unnecessary object-orientation. Defining
custom classes is useful when we want to glue together some state and some
functionality. The problem, as pointed out by the discussions about functional
programming, comes from the "state" part of the equation.
In some architectures, typically web applications, multiple instances of Python
processes are spawned to respond to external requests that can
happen at the same time. In this case, holding some state into instantiated
objects, which means keeping some static information about the world, is prone
to concurrency problems or race-conditions. Sometimes, between the initialization of
the state of an object (usually done with the __init__() method) and the actual use
of the object state through one of its methods, the world may have changed, and
the retained state may be outdated. For example, a request may load an item in
memory and mark it as read by a user. If another request requires the deletion
of this item at the same, it may happen that the deletion actually occurs after
the first process loaded the item, and then we have to mark as read a deleted
object.
This and other issues led to the idea that using stateless functions is a
better programming paradigm.
Another way to say the same thing is to suggest using functions and procedures
with as few implicit contexts and side-effects as possible. A function's
implicit context is made up of any of the global variables or items in the persistence layer
that are accessed from within the function. Side-effects are the changes that a function makes
to its implicit context. If a function saves or deletes data in a global variable or
in the persistence layer, it is said to have a side-effect.
Carefully isolating functions with context and side-effects from functions with
logic (called pure functions) allow the following benefits:
- Pure functions are deterministic: given a fixed input,
the output will always be the same.
- Pure functions are much easier to change or replace if they need to
be refactored or optimized.
- Pure functions are easier to test with unit-tests: There is less
need for complex context setup and data cleaning afterwards.
- Pure functions are easier to manipulate, decorate, and pass-around.
In summary, pure functions, without any context or side-effects, are more
efficient building blocks than classes and objects for some architectures.
Obviously, object-orientation is useful and even necessary in many cases, for
example when developing graphical desktop applications or games, where the
things that are manipulated (windows, buttons, avatars, vehicles) have a
relatively long life of their own in the computer's memory.
Decorators
----------
The Python language provides a simple yet powerful syntax called 'decorators'.
A decorator is a function or a class that wraps (or decorates) a function
or a method. The 'decorated' function or method will replace the original
'undecorated' function or method. Because functions are first-class objects
in Python, it can be done 'manually', but using the @decorator syntax is
clearer and thus preferred.
.. code-block:: python
def foo():
# do something
def decorator(func):
# manipulate func
return func
foo = decorator(foo) # Manually decorate
@decorator
def bar():
# Do something
# bar() is decorated
This mechanism is useful for separating concerns and avoiding
external un-related logic 'polluting' the core logic of the function
or method. A good example of a piece of functionality that is better handled
with decoration is memoization or caching: you want to store the results of an
expensive function in a table and use them directly instead of recomputing
them when they have already been computed. This is clearly not part
of the function logic.
Dynamic typing
--------------
Python is said to be dynamically typed, which means that variables
do not have a fixed type. In fact, in Python, variables are very
different from what they are in many other languages, specifically
strongly-typed languages. Variables are not a segment of the computer's
memory where some value is written, they are 'tags' or 'names' pointing
to objects. It is therefore possible for the variable 'a' to be set to
the value 1, then to the value 'a string', then to a function.
The dynamic typing of Python is often considered to be a weakness, and indeed
it can lead to complexities and hard-to-debug code. Something
named 'a' can be set to many different things, and the developer or the
maintainer needs to track this name in the code to make sure it has not
been set to a completely unrelated object.
Some guidelines help to avoid this issue:
- Avoid using the same variable name for different things.
**Bad**
.. code-block:: python
a = 1
a = 'a string'
def a():
pass # Do something
**Good**
.. code-block:: python
count = 1
msg = 'a string'
def func():
pass # Do something
Using short functions or methods helps reduce the risk
of using the same name for two unrelated things.
It is better to use different names even for things that are related,
when they have a different type:
**Bad**
.. code-block:: python
items = 'a b c d' # This is a string...
items = items.split(' ') # ...becoming a list
items = set(items) # ...and then a set
There is no efficiency gain when reusing names: the assignments
will have to create new objects anyway. However, when the complexity
grows and each assignment is separated by other lines of code, including
'if' branches and loops, it becomes harder to ascertain what a given
variable's type is.
Some coding practices, like functional programming, recommend never reassigning a variable.
In Java this is done with the `final` keyword. Python does not have a `final` keyword
and it would be against its philosophy anyway. However, it may be a good
discipline to avoid assigning to a variable more than once, and it helps
in grasping the concept of mutable and immutable types.
Mutable and immutable types
---------------------------
Python has two kinds of built-in or user-defined types.
Mutable types are those that allow in-place modification
of the content. Typical mutables are lists and dictionaries:
All lists have mutating methods, like append() or pop(), and
can be modified in place. The same goes for dictionaries.
Immutable types provide no method for changing their content.
For instance, the variable x set to the integer 6 has no "increment" method. If you
want to compute x + 1, you have to create another integer and give it
a name.
.. code-block:: python
my_list = [1, 2, 3]
my_list[0] = 4
print my_list # [4, 2, 3] <- The same list as changed
x = 6
x = x + 1 # The new x is another object
One consequence of this difference in behavior is that mutable
types are not "stable", and therefore cannot be used as dictionary
keys.
Using properly mutable types for things that are mutable in nature
and immutable types for things that are fixed in nature
helps to clarify the intent of the code.
For example, the immutable equivalent of a list is the tuple, created
with ``(1, 2)``. This tuple is a pair that cannot be changed in-place,
and can be used as a key for a dictionary.
One peculiarity of Python that can surprise beginners is that
strings are immutable. This means that when constructing a string from
its parts, it is much more efficient to accumulate the parts in a list,
which is mutable, and then glue ('join') the parts together when the
full string is needed. One thing to notice, however, is that list
comprehensions are better and faster than constructing a list in a loop
with calls to append().
**Bad**
.. code-block:: python
# create a concatenated string from 0 to 19 (e.g. "012..1819")
nums = ""
for n in range(20):
nums += str(n) # slow and inefficient
print nums
**Good**
.. code-block:: python
# create a concatenated string from 0 to 19 (e.g. "012..1819")
nums = []
for n in range(20):
nums.append(str(n))
print "".join(nums) # much more efficient
**Best**
.. code-block:: python
# create a concatenated string from 0 to 19 (e.g. "012..1819")
nums = [str(n) for n in range(20)]
print "".join(nums)
One final thing to mention about strings is that using join() is not always
best. In the instances where you are creating a new string from a pre-determined
number of strings, using the addition operator is actually faster, but in cases
like above or in cases where you are adding to an existing string, using join()
should be your preferred method.
.. code-block:: python
foo = 'foo'
bar = 'bar'
foobar = foo + bar # This is good
foo += 'ooo' # This is bad, instead you should do:
foo = ''.join([foo, 'ooo'])
Vendorizing Dependencies
------------------------
Runners
-------
Further Reading
---------------
- http://docs.python.org/2/library/
- http://www.diveintopython.net/toc/index.html