One up to Python expert (1) – Decorators
Development, Python November 15th, 2007
I have been using Python for almost 4 years. I still remembered how amazed I was by the elegance of code indention, simplicity at the first sight, started from the official tutorial, then dive into it, daily scripts, then couple of side projects, but until today, I still hesitate to claim a Python expert. I would rather share this series of the journey to broaden my vision and deepen my insight on this beautiful language.
Introduction
Dr. David Mertz has a stunning demonstration in Charming Python column show this language feature, decorater. Here is the sample code to add spam, the favorite food for python, to arbitrary function:
def addspam(fn):
def new(*args):
"new method"
print "spam spam spam"
return fn(*args)
return new
@addspam
def add(a,b):
"add method"
print a**2 + b**2
if __name__ == "__main__":
add(3, 4)
The output of is:
25
According to the Python language reference, the @ operation is defined as:
Decorator expressions are evaluated when the function is defined, in the scope that contains the function definition. The result must be a callable, which is invoked with the function object as the only argument. The returned value is bound to the function name instead of the function object. Multiple decorators are applied in nested fashion.
Therefore, our add is defined as:
new is an anonymous function, the name is arbitrary since we would never call it directly. When add is invoked, addspam is evaluated and returns a callable object, new, which accepts the arguments, then is executed. As a well-behaviored decorator, new eventually calls the decoratee after spreads the word, “spam”.
Before we rush to more sophisticated application, let’s take a look at the flaws of the crystal ball. Yes, it is not crystal transparent, new blocks the signature of add:
new method
We can copy the meta data by all means, but is there a smart way to avoid the boilerplate code? Yes, we can use Michele Simionato’s decorator library like this:
from decorator import decorator
@decorator
def addspam(fn, *args, **kw):
"new method"
print "spam spam spam", args
return fn(*args)
@addspam
def add(a,b):
"add method"
print a**2 + b**2
if __name__ == "__main__":
print add.__doc__
add(3, 4)
That is quite dizzying, What is the on the earth under the hood?
Under the hood
First, let us inspect the vanilla version, each decorator would block the signature of the decoratee, illustrated by different colors.
Here is the code snippet of decorator.py
def _decorator(func): # the real meat is here
infodict = getinfo(func)
argnames = infodict[‘argnames’]
assert not (‘_call_’ in argnames or ‘_func_’ in argnames), (
‘You cannot use _call_ or _func_ as argument names!’)
src = "lambda %(signature)s: _call_(_func_, %(signature)s)" % infodict
# import sys; print >> sys.stderr, src # for debugging purposes
dec_func = eval(src, dict(_func_=func, _call_=caller))
return update_wrapper(dec_func, func, infodict)
return update_wrapper(_decorator, caller)
The first difference that caught my eye was the _decorator’s argument, func, instead of new’s *args. Does this matter?
Yes, that is the trick of the magic. decorator decorates the addspam which decorates add, are you still awake? So add is the argument for decorator’s anonymous function, i.e _decorator.
There are two assistants for the magic: getinfo copy the signature of the function; update_wrapper seals the _decorator with caller’s signature. When add is invoked, decorator(addspam) is evaluated, which returns _decortor with addspam’s signature, in another word, decorator is transparent to addspam. _decorator is also the decorator of add, so _decorator(add) is executed:
- Sanity check: make sure keyword is not used in the argument names
- Cook the real meat: build addspam(add) in dec_func. Please check 3.6.2 String Formatting Operations for the syntax of mapping dictionary.
- Seal the can with fun(i.e add)’s signature
Here is the dynamic illustration:
Patterns
I would discuss this topic in detail later, you could take a look at official wiki.
Memorize
This pattern is well-documented in Wikipedia, and here is an effective but obtrusive implementation in DDJ[1], the corresponding python implementation is much more intuitive:
def fib(n):
print "%d is caculated" % n
if n < 2 :
return 1;
else:
return fib(n-1) + fib(n-2)
Programming by Contract
Programming by Contract is an approach for software engineering. Microsoft Visual C++ introduced SAL annotations for precondition and postcondition. Decorators helps to separate the contract and logic[2]:
@precondition("jerry > 1")
@postcondition(lambda x: x > 1)
def foo(tom=1, jerry=2, rose=3, jack=4):
print tom
return jack
Precondition determines the requirement of the arguments, it is more convenient to use names of arguments for evaluation; postcondition specifies the return value, function object is more appropriate to refer the returned value. Here is the full implementation.
Aspect oriented programming
AOP is quite popular in Java community, there is corresponding Python project, PEAK for enterprise environment. For lightweight AOP developer, the decorator could do some help, such as the canonical fund transfer example:
@precondition("fromAccount.amount > amount")
def transfer(fromAccount, toAccount, amount):
# TODO: add transaction here.
fromAccount.withdraw(amount)
toAccount.deposit(amount)
Conclusion
Decorator opens a door to override the default behavior of function. The add-on lets the magic shine and hide the mechanism behind the curtain.
[1] It would be a juicy topic to implement decorator via metaprogramming
[2] Serious DBC users may consider PyDBC
How did you make those graphics? They’re great.
This should be in the standard library in the future..
I needed this for a project, I searched and googled around and I.. didn’t find this module, only the rumor about it and how it worked. So I basically did the same thing myself for my project.
I still wonder why my searches wouldn’t turn up the actual module but only the rumors of it. So thanks for the link.
here was my reimplementation, quite similar:
wrapper.__module__ = func.__module__
wrapper.__name__ = func.__name__
wrapper.__doc__ = func.__doc__
wrapper.__dict__.update(func.__dict__)
return wrapper
def wrap_method(method, func):
"""
Wrap method so that func is called before method;
returns a wrapper that prereserves method’s name
and signature
"""
import inspect
spec = inspect.getargspec(method)
argstring = inspect.formatargspec(*spec)[1:-1] #remove parantheses
def inner_func(*args, **kw):
func(*args, **kw)
return method(*args, **kw)
wrapper_src = "lambda %s: inner_func(%s)" % (argstring, argstring)
context = {"inner_func": inner_func}
wrapper = eval(wrapper_src, context)
_update_wrapper(wrapper, method)
return wrapper
The graphics is made by Microsoft Visio.
Python 2.5 adds wraps() to functools so in your example you can do:
def addspam(fn):
@wraps(fn)
def new(*args):
“new method”
print “spam spam spam”
return fn(*args)
return new
http://docs.python.org/lib/module-functools.html