Merge pull request #146 from GloryFish/structuring-edits

Structuring edits
This commit is contained in:
Kenneth Reitz
2012-05-11 12:14:42 -07:00
+101 -102
View File
@@ -8,49 +8,49 @@ Structuring your project properly is extremely important.
Structure is Key Structure is Key
---------------- ----------------
Thanks to the way imports and module are handled in Python, it is Thanks to the way imports and modules are handled in Python, it is
relatively easy to structure a python project. Easy, here, means relatively easy to structure a python project. Easy, here, means
actually that you have not many constraints and that the module that you do not have many constraints and that the module
importing model is easy grasp. Therefore, you are left with the importing model is easy to grasp. Therefore, you are left with the
pure architectural task of drawing the different parts of your pure architectural task of crafting the different parts of your
project and their interactions. project and their interactions.
Easy structuration of a project means it is also easy Easy structuring of a project means it is also easy
to do it poorly. Some signs of a poorly structured projects to do it poorly. Some signs of a poorly structured project
include: include:
- Multiple and messy circular dependencies: if your classes - Multiple and messy circular dependencies: If your classes
Table and Chair in furn.py need to import Carpenter from workers.py Table and Chair in furn.py need to import Carpenter from workers.py
to answer to a question such as table.isdoneby(), to answer a question such as table.isdoneby(),
and if convertly the class Carpenter need to import Table and Chair, and if conversely the class Carpenter needs to import Table and Chair,
for example to answer to carpenter.whatdo(), then you to answer the question carpenter.whatdo(), then you
have a circular dependency, and will have to resort to have a circular dependency. In this case you will have to resort to
fragile hacks such has using import statements inside fragile hacks such has using import statements inside
methods or functions. methods or functions.
- Hidden coupling. Each and every change in Table implementation - Hidden coupling: Each and every change in Table's implementation
breaks 20 tests in unrelated test cases because it breaks Carpenter's code, breaks 20 tests in unrelated test cases because it breaks Carpenter's code,
which requires very careful surgery to adapt the change. This means which requires very careful surgery to adapt the change. This means
you have too many assumptions about Table in Carpenter's code or the you have too many assumptions about Table in Carpenter's code or the
reverse. reverse.
- Heavy usage of global state or context: Instead of explicitely - Heavy usage of global state or context: Instead of explicitly
passing ``(height, width, type, wood)`` to each other, Table passing ``(height, width, type, wood)`` to each other, Table
and Carpenter rely on global variables that can be modified and Carpenter rely on global variables that can be modified
and are modified on the fly by different agent. You need to and are modified on the fly by different agents. You need to
scrutinize all access to this global variables to understand why scrutinize all access to these global variables to understand why
a rectangular table became a sqaure, and discover that a remote a rectangular table became a square, and discover that remote
template code is also modifying this context, messing with template code is also modifying this context, messing with
table dimensions. table dimensions.
- Spaghetti code: Multiple pages of nested if clauses and for loops - Spaghetti code: Multiple pages of nested if clauses and for loops
with a lot of copy-pasted procedural code and no with a lot of copy-pasted procedural code and no
proper segmentation are known as spaghetti code. Python's proper segmentation are known as spaghetti code. Python's
meaningful indentation (one of its most controversial feature) make meaningful indentation (one of its most controversial features) make
it very hard to maintain this kind of code. So the good news is that it very hard to maintain this kind of code. So the good news is that
you might not see too much of it. you might not see too much of it.
- Ravioli code is more likely in Python: it consists of hundreds of - Ravioli code is more likely in Python: It consists of hundreds of
similar little pieces of logic, often classes or objects, without similar little pieces of logic, often classes or objects, without
proper structure. If you never can remember if you have to use proper structure. If you never can remember if you have to use
FurnitureTable, AssetTable or Table, or even TableNew for your FurnitureTable, AssetTable or Table, or even TableNew for your
@@ -60,53 +60,53 @@ include:
Modules Modules
------- -------
Python modules are one of the main abstraction layer available and probably the Python modules are one of the main abstraction layers available and probably the
most natural one. Abstraction layers allow separating code into parts holding most natural one. Abstraction layers allow separating code into parts holding
related data and functionalities. related data and functionality.
For example, a layer of a project can handle interfacing with user actions, For example, a layer of a project can handle interfacing with user actions,
while another would handle low-level manipulation of data. The most natural way while another would handle low-level manipulation of data. The most natural way
to separate these two layers is to regroup all interfacing functionalities to separate these two layers is to regroup all interfacing functionality
in one file, and all low-level operations in another file. In this case, in one file, and all low-level operations in another file. In this case,
the interface file need to import the low-level file. This is done with the the interface file needs to import the low-level file. This is done with the
`import` and `from ... import` statements. `import` and `from ... import` statements.
As soon as you use `import` statements you use modules, either builtin modules As soon as you use `import` statements you use modules. These can be either built-in
such as `os` and `sys`, or third-party modules you have installed in your modules such as `os` and `sys`, third-party modules you have installed in your
environment, or project's internal modules. environment, or your project's internal modules.
Nothing special is required for a Python file to be a module, but the import Nothing special is required for a Python file to be a module, but the import
mechanism need to be understood in order to use this concept properly and avoid mechanism needs to be understood in order to use this concept properly and avoid
some issues. some issues.
Concretely, the `import modu` statement will look for the proper file, which is Concretely, the `import modu` statement will look for the proper file, which is
`modu.py` in the same directory as the caller if it exists. If it is not `modu.py` in the same directory as the caller if it exists. If it is not
found, the Python interpreter with search for `modu.py` in the "path" found, the Python interpreter will search for `modu.py` in the "path"
recursively and raise an ImportError exception if it is not found. recursively and raise an ImportError exception if it is not found.
Once `modu.py` is found, the Python interpreter will execute the module in an Once `modu.py` is found, the Python interpreter will execute the module in an
isolated scope. Any top-level statement in `modu.py` will be executed, isolated scope. Any top-level statement in `modu.py` will be executed,
including other imports if any. Function and classes definitions are stored in including other imports if any. Function and class definitions are stored in
the module's dictionary. the module's dictionary.
Then modules variables, functions and classes will be available to the caller Then, the module's variables, functions, and classes will be available to the caller
through the module's namespace, a central concept in programming that is through the module's namespace, a central concept in programming that is
particularly helpful and powerful in Python. particularly helpful and powerful in Python.
In many languages, a `include file` directive is used by the preprocessor to In many languages, an `include file` directive is used by the preprocessor to
take all code found in the file and 'copy' it in the caller's code. It is take all code found in the file and 'copy' it into the caller's code. It is
different in Python: the included code is isolated in a module namespace, which different in Python: the included code is isolated in a module namespace, which
means that you generally don't have to worry that the included code could have means that you generally don't have to worry that the included code could have
unwanted effect, eg override an existing function with the same name. unwanted effects, e.g. override an existing function with the same name.
It is possible to simulate the more standard behavior by using a special syntax It is possible to simulate the more standard behavior by using a special syntax
of the import statement: `from modu import *`. This is generally considered bad of the import statement: `from modu import *`. This is generally considered bad
practice, **using import * makes code harder to read and dependencies less practice. **Using `import *` makes code harder to read and makes dependencies less
compartimented**. compartmentalized**.
Using `from modu import func` is a way to pinpoint the function you want to Using `from modu import func` is a way to pinpoint the function you want to
import and put it is the global namespace. While much less harmful than `import import and put it in the global namespace. While much less harmful than `import
*` because it shows explicitely what is imported in the global namespace, it's *` because it shows explicitly what is imported in the global namespace, its
advantage over a simpler `import modu` is only that it will save some typing. advantage over a simpler `import modu` is only that it will save some typing.
**Very bad** **Very bad**
@@ -134,13 +134,13 @@ advantage over a simpler `import modu` is only that it will save some typing.
[...] [...]
x = modu.sqrt(4) # sqrt is visibly part of modu's namespace x = modu.sqrt(4) # sqrt is visibly part of modu's namespace
As said in the section about style, readability is one of the main feature of As said in the section about style, readability is one of the main features of
Python. Readability means to avoid useless boilerplate text and clutter, Python. Readability means to avoid useless boilerplate text and clutter,
therefore some efforts are spent trying to achieve a certain level of brevity. therefore some efforts are spent trying to achieve a certain level of brevity.
But terseness and obscurity are the limits where brevity should stop: being But terseness and obscurity are the limits where brevity should stop. Being
able to tell immediately from where comes a class or a function, as in the able to tell immediately where a class or function comes from, as in the
`modu.func` idiom, improves greatly code readability and understandability in `modu.func` idiom, greatly improves code readability and understandability in
most cases but the simplest single file projects. all but the simplest single file projects.
Packages Packages
@@ -149,35 +149,35 @@ Packages
Python provides a very straightforward packaging system, which is simply an Python provides a very straightforward packaging system, which is simply an
extension of the module mechanism to a directory. extension of the module mechanism to a directory.
Any directory with a __init__.py file is considered a Python package. The Any directory with an __init__.py file is considered a Python package. The
different modules in the package are imported in a similar manner as plain different modules in the package are imported in a similar manner as plain
modules, will a special behavior for the __init__.py file, that is used to modules, but with a special behavior for the __init__.py file, which is used to
gather all package-wide definitions. gather all package-wide definitions.
A file modu.py in the directory pack/ is imported with the statement `import A file modu.py in the directory pack/ is imported with the statement `import
pack.modu`. This statement will look for a __init__.py file in `pack`, execute pack.modu`. This statement will look for an __init__.py file in `pack`, execute
all its top-level statements. Then it will look for a file `pack/modu.py` and all of its top-level statements. Then it will look for a file `pack/modu.py` and
execute all its top-level statements. After these operations, any variable, execute all of its top-level statements. After these operations, any variable,
function or class defined in modu.py is available in pack.modu namespace. function, or class defined in modu.py is available in the pack.modu namespace.
A commonly seen issue is to add too many code and functions in __init__.py A commonly seen issue is to add too much code to __init__.py
files. When the project complexity grows, there may be sub-packages and files. When the project complexity grows, there may be sub-packages and
sub-sub-packages in a deep directory structure, and then, import a single item sub-sub-packages in a deep directory structure, and then, importing a single item
from a sub-sub-package will require to execute all __init__.py file met while from a sub-sub-package will require executing all __init__.py files met while
descending the tree. traversing the tree.
Leaving a __init__.py file empty is considered normal and even a good pratice, Leaving an __init__.py file empty is considered normal and even a good practice,
if the package's modules and sub-packages do not need to share any code. if the package's modules and sub-packages do not need to share any code.
Lastly, a convenient syntax is available for importing deeply nested packages: Lastly, a convenient syntax is available for importing deeply nested packages:
`import very.deep.module as mod` allow to use `mod` in place of the verbose `import very.deep.module as mod`. This allows you to use `mod` in place of the verbose
repetition of `very.deep.module` in front of each calls to module items. repetition of `very.deep.module`.
Object-oriented programming Object-oriented programming
--------------------------- ---------------------------
Python is sometime described as an object-oriented programming language. This Python is sometimes described as an object-oriented programming language. This
can be somewhat misleading and need to be clarified. can be somewhat misleading and needs to be clarified.
In Python, everything is an object, and can be handled as such. This is what is In Python, everything is an object, and can be handled as such. This is what is
meant when we say that, for example, functions are first-class objects. meant when we say that, for example, functions are first-class objects.
@@ -188,46 +188,46 @@ object-oriented language.
However, unlike Java, Python do not impose object-oriented programming as the However, unlike Java, Python do not impose object-oriented programming as the
main programming paradigm. It is perfectly viable for a Python project to not main programming paradigm. It is perfectly viable for a Python project to not
be object-oriented, ie. to use no or very few class definitions, class be object-oriented, i.e. to use no or very few class definitions, class
inheritance, and any other mechanism that are specific to object-oriented inheritance, or any other mechanisms that are specific to object-oriented
programming. programming.
Moreover, as seen in the modules_ section, the way Python handles modules and Moreover, as seen in the modules_ section, the way Python handles modules and
namespaces gives directly to the developer a natural way to ensure namespaces gives the developer a natural way to ensure
encapsulation and separation of abstraction layers, both being the most common encapsulation and separation of abstraction layers, both being the most common
reasons to use object-orientation. Therefore, Python programmers have more reasons to use object-orientation. Therefore, Python programmers have more
latitude to not use object-orientation, when it is not required by the business latitude to not use object-orientation, when it is not required by the business
model to be constructed. model.
There are some reasons to avoid unnecessary object-orientation. Definining There are some reasons to avoid unnecessary object-orientation. Defining
custom classes is useful when we want to glue together some state and some custom classes is useful when we want to glue together some state and some
functionality. The problem, as pointed out by the discussions about functional functionality. The problem, as pointed out by the discussions about functional
programming, comes from the "state" part of the equation. programming, comes from the "state" part of the equation.
In some architectures, typically web applications, instances of Python In some architectures, typically web applications, multiple instances of Python
processes are spawned simultaneously to answer to external requests that can processes are spawned to respond to external requests that can
happen at the same time. In this case, holding some state into instanciated happen at the same time. In this case, holding some state into instantiated
objects, which means keeping some static information about the world, is prone objects, which means keeping some static information about the world, is prone
to concurrency problems or race-conditions: between the initialization of the to concurrency problems or race-conditions. Sometime between the initialization of the
state of an object, usually done with the __init__() method, and the actual use state of an object, usually done with the __init__() method, and the actual use
of the object state through one of its method, the world may have changed, and of the object state through one of its methods, the world may have changed, and
the retained state may be outdated. For example, a request may load an item in the retained state may be outdated. For example, a request may load an item in
memory and mark it as read by a user. If another request requires the deletion memory and mark it as read by a user. If another request requires the deletion
of this item at the same, it may happen that the deletion actually occur after of this item at the same, it may happen that the deletion actually occurs after
the first process loaded the item, and then we have to mark as read a deleted the first process loaded the item, and then we have to mark as read a deleted
object. object.
This and other issues led to the idea that using stateless functions is a This and other issues led to the idea that using stateless functions is a
better programming paradigm. better programming paradigm.
Another way to say the same thing is to propose to use functions and procedures Another way to say the same thing is to suggest using functions and procedures
with as few implicit context and side-effects as possible. A function's with as few implicit contexts and side-effects as possible. A function's
implicit context is decelable when the function body refers to some global implicit context is made up of any of the global variables or items in the persistence layer
variables or fetches data from the persistence layer. Side-effects are the that are accessed from within the function. Side-effects are the changes that a function makes
opposite: if a function body modifies the global context or save or delete data to it's implicit context. If a function saves or deletes data in a global variable or
on the persistence layer, it is said to have side-effect. in the persistence layer, it is said to have a side-effect.
Isolating carefully functions with context and side-effects from functions with Carefully isolating functions with context and side-effects from functions with
logic (called pure functions) allow the following benefits: logic (called pure functions) allow the following benefits:
- Pure functions are more likely to be deterministic: given a fixed input, - Pure functions are more likely to be deterministic: given a fixed input,
@@ -239,7 +239,7 @@ logic (called pure functions) allow the following benefits:
- Pure functions are easier to test with unit-tests: There is less - Pure functions are easier to test with unit-tests: There is less
need for complex context setup and data cleaning afterwards. need for complex context setup and data cleaning afterwards.
- Pure functions are easier to manipulate, decorate_, pass-around. - Pure functions are easier to manipulate, decorate_, and pass-around.
In summary, pure functions, without any context or side-effects, are more In summary, pure functions, without any context or side-effects, are more
efficient building blocks than classes and objects for some architectures. efficient building blocks than classes and objects for some architectures.
@@ -253,12 +253,12 @@ relatively long life of their own in the computer's memory.
Decorators Decorators
---------- ----------
Python language provides a simple yet powerful syntax called 'decorators'. The Python language provides a simple yet powerful syntax called 'decorators'.
A decorator is a function or a class that wraps (or decorate) a function A decorator is a function or a class that wraps (or decorate) a function
or a method. The 'decorated' function or method will replace the original or a method. The 'decorated' function or method will replace the original
'undecorated' function or method. Because function are first-class objects 'undecorated' function or method. Because functions are first-class objects
in Python it can be done 'manually' but using the @decorator syntax is in Python it can be done 'manually' but using the @decorator syntax is
clearer and thus prefered. clearer and thus preferred.
.. code-block:: Python .. code-block:: Python
@@ -277,8 +277,8 @@ clearer and thus prefered.
# bar() is decorated # bar() is decorated
Using this mechanism is useful for separating concerns and avoiding Using this mechanism is useful for separating concerns and avoiding
external un-related logic to 'pollute' the core logic of the function external un-related logic 'polluting' the core logic of the function
or method. A good example of a functionality that is better handled or method. A good example of a piece of functionality that is better handled
with decoration is memoization or caching: you want to store the results of an with decoration is memoization or caching: you want to store the results of an
expensive function in a table and use them directly instead of recomputing expensive function in a table and use them directly instead of recomputing
them when they have already been computed. This is clearly not part them when they have already been computed. This is clearly not part
@@ -290,18 +290,18 @@ Dynamic typing
Python is said to be dynamically typed, which means that variables Python is said to be dynamically typed, which means that variables
do not have a fixed type. In fact, in Python, variables are very do not have a fixed type. In fact, in Python, variables are very
different from what they are in many other languages, specifically different from what they are in many other languages, specifically
strongly-typed languages: variables are not a segment of the computer's strongly-typed languages. Variables are not a segment of the computer's
memory where some value ir written, they are 'tags' or 'names' pointing memory where some value is written, they are 'tags' or 'names' pointing
to objects. It is therefore possible for the variable 'a' to be set to to objects. It is therefore possible for the variable 'a' to be set to
the value 1, then to the value 'a string', then to a function. the value 1, then to the value 'a string', then to a function.
The dynanic typing of Python is often considered as a weakness, and indeed The dynamic typing of Python is often considered to be a weakness, and indeed
it can lead to complexities and to hard-to-debug code, where something it can lead to complexities and hard-to-debug code. Something
named 'a' can be set to many different things, and the developer or the named 'a' can be set to many different things, and the developer or the
maintainer need to track this name in the code to make sure it has not maintainer needs to track this name in the code to make sure it has not
been set to a completely unrelated object. been set to a completely unrelated object.
Some guidelines allow to avoid this issue: Some guidelines help to avoid this issue:
- Avoid using variables for different things. - Avoid using variables for different things.
@@ -323,9 +323,8 @@ Some guidelines allow to avoid this issue:
def func() def func()
pass # Do something pass # Do something
Using short functions or methods helps writing good code for many Using short functions or methods helps reduce the risk
reasons, one being that their local scope is clearer, and the risk of using the same name for two unrelated things.
of using the same name for two unrelated things is lowered.
It is better to use different names even for things that are related, It is better to use different names even for things that are related,
when they have a different type: when they have a different type:
@@ -340,14 +339,14 @@ when they have a different type:
There is no efficiency gain when reusing names: the assignments There is no efficiency gain when reusing names: the assignments
will have to create new objects anyway. However, when the complexity will have to create new objects anyway. However, when the complexity
grows are each assignment are separated by other lines of code, including grows and each assignment is separated by other lines of code, including
'if' branches and loops, it becomes harder to acertain which type is the 'if' branches and loops, it becomes harder to ascertain what a given
variable at hand. variable's type is.
Some coding practices, like functional programming, even recommend to never re-assign a variable, which Some coding practices, like functional programming, recommend never reassigning a variable.
is done in Java with the keyword final. Python do not have such a keyword, In Java this is done with the `final` keyword. Python does not have a `final` keyword
and it would be against its philosophy anyway, but it may be a good and it would be against its philosophy anyway. However, it may be a good
discipline to avoid setting more than once any variable, and it helps discipline to avoid assigning to a variable more than once, and it helps
in grasping the concept of mutable and immutable types. in grasping the concept of mutable and immutable types.
Mutable and immutable types Mutable and immutable types
@@ -357,12 +356,12 @@ Python has two kinds of built-in or user-defined types.
Mutable types are those that allow in-place modification Mutable types are those that allow in-place modification
of the content. Typical mutables are lists and dictionaries: of the content. Typical mutables are lists and dictionaries:
All lists have muting methods, like append() or pop(), and All lists have mutating methods, like append() or pop(), and
can be modified in place. Same for dictionaries. can be modified in place. The same goes for dictionaries.
Immutable types provide no method for changing their content. Immutable types provide no method for changing their content.
For instance, the variable x set to the integer 6 has no "increment" method. If you For instance, the variable x set to the integer 6 has no "increment" method. If you
want to computed x + 1, you have to create another integer and give it want to compute x + 1, you have to create another integer and give it
a name. a name.
.. code-block:: python .. code-block:: python
@@ -386,8 +385,8 @@ For example, the immutable equivalent of a list is the tuple, created
with ``(1, 2)``. This tuple is a pair that cannot be changed in-place, with ``(1, 2)``. This tuple is a pair that cannot be changed in-place,
and can be used as a key for a dictionary. and can be used as a key for a dictionary.
One particularity of Python that can surprise in the beginning is that One peculiarity of Python that can surprise beginners is that
string are immutable. This means that when constructing a string from strings are immutable. This means that when constructing a string from
its parts, it is much more efficient to accumulate the parts in a list, its parts, it is much more efficient to accumulate the parts in a list,
which is mutable, and then glue ('join') the parts together when the which is mutable, and then glue ('join') the parts together when the
full string is needed. full string is needed.