fredrikholmstrom.com » Programming

Applying programming logic to making friends

Fredrik Holmström — Thu, 06 Aug 2009 10:22:09 +0000

1. Walk out.
2. Go up to a person that looks busy.
3. Talk to the person, and follow him/her.
4. If the person walks through a door that can be locked, make sure you get your foot in.
5. Continue talking to the person.
6. If the police shows up, follow them instead.
7. Talk to the police.
8. Follow the police.
9. If the police walks through a door that can be locked, make sure you get your foot in.
10. If you get locked into a room, wait.
11. Go to 1.

micro DSL for outputting HTML from Python

Fredrik Holmström — Sat, 07 Mar 2009 15:22:14 +0000

Should be self explanatory

def render(tag):
    if not tag:
        return ""
    else:
        if isinstance(tag, str):
            return tag
        else:
            if len(tag) > 1 and isinstance(tag[1], dict):
                return render_tag(tag[0], tag[1], tag[2:])
            else:
                return render_tag(tag[0], None, tag[1:])

def render_tag(tag, attrs, children):
    if not children:
        return "<%s%s/>" % (tag, render_attrs(attrs))
    else:
        return "<%s%s>%s" % (tag, render_attrs(attrs), "\n".join(map(render, children)), tag)

def render_attrs(attrs):
    return "" if not attrs else " " + " ".join(['%s="%s"' % pair for pair in attrs.items()])

if __name__ == "__main__":

    # example

    html =  ("html",
                ("head",
                    ("title", "this is my title"),
                    ("meta", {"name":"keywords", "value":"python, html"}),
                    ("meta", {"name":"description", "value":"outputting html from python dsl"}),
                    ("script", {"src":"static/js/jquery-1.3.2.min.js"}, None) # None forces open
                ),
                ("body",
                    ("div", 

                    )
                )
            )

    print render(html)

Output from the example is something like this:


    
        this is my title

TCO-trampoline in Python, take 2

Fredrik Holmström — Tue, 03 Mar 2009 15:17:28 +0000

My second (and much nicer) version of a tail call optimization trampoline in python

class recur(object):
    def __init__(self, *args):
        self.args = args

    def __call__(self, func):
        return Inner(func)

class Inner(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args):
        result = recur(*args)
        call   = self.func

        while True:
            result = call(*result.args)

            if isinstance(result, recur):
                if result.args and isinstance(result.args[0], Inner):
                    call = result.args[0].func
                    result.args = result.args[1:]
            else:
                return result

        return result

if __name__ == "__main__":

    # recursively sums a list of integers
    # calling itself for each item

    @recur()
    def req_sum(lst, n=0):
        return n if not lst else recur(lst[1:], n+lst[0])

    print req_sum([1,2,3,4])

    # You can also recurse
    # with another function
    # here you have ping and pong
    # calling eachother forever

    pings = 0
    pongs = 0

    @recur()
    def ping(n=0):
        global pings
        print "pong ponged %d times, now I'll ping" % n
        pings += 1
        return recur(pong, pings)

    @recur()
    def pong(n):
        global pongs
        print "ping pinged %d times, now I'll pong" % n
        pongs += 1
        return recur(ping, pongs)

    # if you uncomment this next statement and
    # run the code it will loop forever  

    #ping()

Again, self explanatory.

Evolving Django’s Views

Fredrik Holmström — Fri, 23 Jan 2009 15:12:09 +0000

First of all: Last time I used Django was well over a year ago so if this is already possible to do with the current latest code base then shame on me, anyway – after messing around a bit with Django I got annoyed by the simplicity of the views, they’re very simple minded. I wanted some way of calling different functions depending on the request method and an easier way to return different output types, this is what I came up with.

The idea is simple, use a class with __call__ implemented instead of a function, within __call__ you can put custom logic to direct the requests to the appropriate callbacks – here’s my new base view class and a helper decorator (since Python 2.6 you can decorate classes).

def view(cls):
	return cls()

class ClassView(object):

	fallback = "HTML"

	http_accept = {
		"application/xhtml\+xml": "HTML",
		"text/html":"HTML",
		"application/json": "JSON",
		"text/javascript": "JSON",
		"application/xml": "XML"
	}

	request_method = {
		"GET": "READ",
		"POST": "WRITE",
		"PUT": "WRITE",
		"DELETE": "WRITE"
	}

	def __call__(self, request):
		accept = request.META["HTTP_ACCEPT"]
		method = getattr(self, self.request_method[request.META["REQUEST_METHOD"]])

		for regex, call in self.http_accept.items():
			if regex and re.search(regex, accept):
				return getattr(self, call)(request, method(request))

		return getattr(self, self.fallback)(request, method(request))

	def READ(self, request):
		pass

	def WRITE(self, request):
		pass

	def JSON(self, request, method_vars):
		pass

	def HTML(self, request, method_vars):
		pass

	def XML(self, request, method_vars):
		pass

Now when I want to create a new view in django, instead of doing something like this:

def index(request):
     # blah

I do this:

# for python < 2.6 remove/comment out this next line:
@view
class index(ClassView):

	def READ(self, request):
		return {"title": "Hello World!"}

	def HTML(self, request, method_vars):
		return render_to_response("frontend/index.html", method_vars)

	def JSON(self, request, method_vars):
		return HttpResponse('{"title":"%(title)s"}' % method_vars)

# for python < 2.6 uncomment the next line
# index = view(index)

The READ method will be called on all GET requests, while the WRITE method will be called on all POST, PUT and DELETE. After which the appropriate output method will be selected based on the HTTP Accept header. This allows me to group common functionality in READ for all GET requests to one URL and then still be allowed to do output specific (HTML, XML, JSON, etc.) logic and template rendering based on what content types the client accepts (for example I will probably need to load more objects for HTML output since it usually has a rather advanced layout compared to say JSON which probably only wants to return the object collection as JSON string)

Practical usage of python decorators

Fredrik Holmström — Sat, 18 Oct 2008 15:10:03 +0000

This might be a somewhat unorthodox usage of python decorators, but a very practical and nice one – I got a question on IRC from a friend that wanted to store python functions in a dictionary and was complaining that there is no way to use lambdas in python as anonymous closures because of their limitations and the syntax of: def func(): pass; dict["func"] = func; is ugly, and I agree – so here’s a nicer version utilizing an extension of the built in dict-class:

class MyDict(dict):
  def __call__(self, func):
    self[func.__name__] = func

foo = MyDict()

@foo
def bar(arg):
  print "from bar %s" % arg

foo["bar"]("Hello World!");

The idea and usage is simple: Extend the built in dict class with a subclass that is turned into a decorator, this decorator then stores every function it decorates inside it’s own dictionary.

Simple, practical and elegant.

Python decorators explained

Fredrik Holmström — Mon, 22 Sep 2008 15:03:50 +0000

Python decorators is just syntactic sugar to something that has been possible to do in python since version 1.0, for more background/technical information on decorators check out this pep.

The first and most important thing to remember about decorators is this: Whatever that is returned from the decorator function will replace the original function you’re decorating. Let’s start with the simplest of all decorators:

def foo(func):
	print "Decorating %s" % func.__name__
	return func

@foo	# foo is *not* called
def bar():
	print "bar"

bar()

This decorator does very little, it takes the function we’re decorating as it’s first argument, prints the name of it and then returns it, and because it returns the original function *nothing* happens except that the message Decorating bar is printed when we run the code above and when calling bar() at the last line we will get bar printed also.

As the one comment in the code above says note that foo is *not* called, the @ operator means “pass this function as the decorator for the function below” so if we only pass our “foo” function in, the calling of it is handled internally in python.

Lets step it up a notch and replace the original function, here’s the code:

def foo(func):
	def inner():
		print "Bar is gone"
	return inner

@foo # Not called
def bar():
	print "bar"

bar()

Our foo-function is a bit more advanced here since it contains an inner function, so what happens here? @foo still tells python that foo should be invoked as the decorator for bar, which it is – but this time foo returns the function called “inner” instead of the original function (which still is passed as the “func”-argument to foo, we just don’t do anything with it), so when we try to call bar on the last line we will get Bar is gone instead. This is because what I said earlier: Whatever is *returned* from the decorator will take the original functions place, so our bar() function is actually gone now and inner() has taken it’s place.

I guess most of you have seen decorators that take arguments, such as:

@decorator("arg1", "arg2")
def some_function();
	pass

These “decorators” are actually normal functions that you call with arguments, that returns decorators that are then called by python internally, here’s a real example:

def foo(string):
	def decorator(func):
		def replacement(times=1):
			print string*times
		return replacement
	return decorator

@foo("hi ho ") # Called
def bar():
	print "bar"

bar()
bar(5)

So when we call foo(”hi ho “) here, the foo function is *actually* called here and returns a decorator that takes the original decorated function as its only argument, this decorator returns a function called “replacement” that takes one argument with a default value of 1. The replacement function that is returned from the decorator replaces the original bar() function, so when we call it the first time we get
hi ho and when we call it the second time with 5 as the argument, we will get: hi ho hi ho hi ho hi ho hi ho .

The important thing to notice in the last example here is that foo is NOT a decorator here, it’s a function that takes one argument (in this case “hi ho “) and returns a decorator (aptly named “decorator” here).

Setting up mod_wsgi for apache on Windows

Fredrik Holmström — Sat, 20 Sep 2008 14:55:42 +0000

I just installed mod_wsgi on my windows (yeah yeah, I know…) laptop, it turned out to be amazingly easy but I figured I’d put up notes on how to do it here anyways.

Head over to http://adal.chiriliuc.com/mod_wsgi/ and fetch the latest (at the time of this writing it was revision_1018_2.3) apache module for your combination of python/apache
- mod_wsgi_py24_apache20 is for Python 2.4 and Apache 2.0
- mod_wsgi_py24_apache22 is for Python 2.4 and Apache 2.2
- mod_wsgi_py25_apache20 is for Python 2.5 and Apache 2.0
- mod_wsgi_py25_apache22 is for Python 2.5 and Apache 2.2
Save the mod_wsgi.so file into your apache modules directory, for me it was located in C:\Program Files\Apache Software Foundation\Apache2.2\modules
Create a directory somewhere outside of your webroot that will host your mod_wsgi-application, for simplicity’s sake I choose C:\wsgi
Locate your httpd.conf-file and open it up in a text editor, it’s usually located in the conf-directory of your apache installation, for me it was: C:\Program Files\Apache Software Foundation\Apache2.2\conf\httpd.conf

Do a search for “LoadModule” and you’ll get a block of LoadModule-statements, put this on its own line there: LoadModule wsgi_module modules/mod_wsgi.so so it looks something like this:

#LoadModule usertrack_module modules/mod_usertrack.so
LoadModule version_module modules/mod_version.so
#LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule wsgi_module modules/mod_wsgi.so

Now find the
block for your web-root, for me it was: , under it add this block:
```
WSGIScriptAlias /wsgi "C:/wsgi/handler.py"

"C:/wsgi">
        AllowOverride None
        Options None
        Order deny,allow
        Allow from all
```
Notice that the two bold sections point at where I created my application directory so if you didn’t choose C:\wsgi you need to change this to reflect whatever directory you chose, the /wsgi part of the just before the first bold statement is the apache alias we want to use for our wsgi application – which in this case will be /wsgi also. Also note the use of forward instead of backwards slashes even when on windows.

The observant reader might’ve noticed the /handler.py at the end of the first bolded statement in the bullet point above, this is our handler script / entrypoint into our application. So add this code to handler.py and put it in C:\wsgi

def application(environ, start_response):
    status = '200 OK'
    output = 'Hello World!'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]

Now just restart apache and go to http://localhost/wsgi and enjoy the beautiful world of python web development

Managing asynchronous operations with python generators (part3)

Fredrik Holmström — Mon, 01 Sep 2008 14:52:25 +0000

I’m sorry, I lied to you guys – this part will not contain a real world example because I felt there were to many new concepts introduced. In this part we’ll go through something called asynchronous programming and how to create a scheduler to keep track of all the tasks we’re performing.

We’ll start of with the Task-base class, it wraps a generator exposing it’s own .next()-method and a .suspended()-method that is used by our scheduler to decide if we should run the task or not. A task can basically be anything, but more often then not it’s some type of operation that involves a delay we can’t control when it’s complete – such as a network call, disk access or something similar.

The directory structure for all of these examples is the following:

/
	/demo
		__init__.py
		tasks.py
		scheduler.py
	example_N.py

demo/tasks.py:

class Task(object):
	def __init__(self, generator = None):
		self.generator = generator

	def suspended(self):
		return False

	def next(self):
		return self.generator.next()

No doubt a fairly simple class, three methods spanning just one line each. Either download the entire code for these examples or write it yourself and put it in the demo/tasks.py file.

Lets put together our scheduler, the scheduler isn’t a class – it’s just a module with two variables and two functions, it to is fairly simple and spans just about twenty five lines:

demo/scheduler.py:

queue = []

def add(task):
	global queue
	queue.append(task)

def run():
	global queue

	stack = queue
	queue = []

	while len(stack) > 0:
		task = stack.pop(0)

		try:
			if not task.suspended():
				task.next()
		except StopIteration:
			continue

		queue.append(task)

		if len(stack) == 0 and len(queue) > 0:
			stack = queue
			queue = []

One global list named queue, and two functions: add() that just appends an element on the queue and run() that runs through the queue until there are no more generators left in it. Let’s go through the run() function line by line:

stack = queue – We store the queue in a temporary local variable
queue = [] – Set queue to an empty list
task = stack.pop(0) – Pop the first element of the stack
if not task.suspended(): – If the task isn’t suspended continue
task.next() – Call .next() on the task which in turn forwards the call to the generator the task wraps
except StopIteration: – If we get a StopIteration exception from the executing task/generator we should just continue with the next one (this will push the task that was responsible for the exception out of the queue)
queue.append(task) – Add the task in question to the queue again
if len(stack) == 0 and len(queue) > 0: – If the stack is empty and the queue isn’t
stack = queue – Put the queue in the stack
queue = [] – And clear the queue

Let’s put this together in a simple example so we can see that our scheduler works as it should:

example_1.py

from demo import tasks, scheduler

def echo(word):
	while True:
		print word
		yield

scheduler.add(
	tasks.Task(
		echo("Hello")))

scheduler.add(
	tasks.Task(
		echo("World!")))

scheduler.run()

If we run this from the the terminal with python example_1.py we will get Hello and World! looped over our screen for eternity:

...
Hello
World!
Hello
World!
Hello
World!
Hello
World!
Hello
World!
...

Maybe not so exiting, but if you look closely you will see that Hello and World! alternate between each other, meaning that when the generator that prints Hello yields the first time the scheduler will take control passing execution to the generator printing World!, when that has printed out its message the stack is empty so the scheduler fills the stack with the queue again and beings from the top, and it all begins again.

Let’s add a new function called sleeper() to the mix, here you have example_2.py:

from time import sleep
from demo import tasks, scheduler

def echo(word):
	while True:
		print word
		yield

def sleeper(seconds):
	while True:
		sleep(seconds)
		yield

scheduler.add(
	tasks.Task(
		echo("Hello")))

scheduler.add(
	tasks.Task(
		sleeper(1)))

scheduler.add(
	tasks.Task(
		echo("World!")))

scheduler.add(
	tasks.Task(
		sleeper(1)))

scheduler.run()

You should be able to figure out what happens when we run this example from the terminal, Hello will print – and then yield and the scheduler passes execution to the first sleeper, pausing for one second and then passing it to World! printing that which yields and then execution gets passed to our second sleeper generator that pauses execution for one second, this is mainly so you can see that they actually take turn instead of some mindless spaming from an infinite repeating loop without pauses.

Let’s make use of that .suspended()-method on the Task class shall we? Let’s create a generator that gets called every second time the scheduler asks for it instead of every iteration, here’s the code – add it to the end of demo/tasks.py:

class EvenTask(Task):
	def __init__(self, *args, **kwargs):
		super(self.__class__, self).__init__(*args, **kwargs)
		self.counter = 0

	def suspended(self):
		self.counter += 1
		return self.counter % 2 != 0

If you don’t understand the super(self.__class__, self).__init__(*args, **kwargs)-line it’s how you call the parent classes .__init__()-method in Python, .suspended() increases the counter with +1 each time it’s called but only returns false when we’re at an even number, allowing the task to be executed every other time.

Here’s the next example code, example_3.py

from time import sleep
from demo import tasks, scheduler

def echo(word):
	while True:
		print word
		yield

def sleeper(seconds):
	while True:
		sleep(seconds)
		yield

scheduler.add(
	tasks.Task(
		echo("Hello")))

scheduler.add(
	tasks.Task(
		sleeper(1)))

scheduler.add(
	tasks.EvenTask(
		echo("World!")))

scheduler.add(
	tasks.Task(
		sleeper(1)))

scheduler.run()

It’s identical to example_2.py except that the echo(”World!”)-task now is of the type EvenTask instead of Task, if you run this code you will get Hello Hello World! printed, because the World!-task will skip every other time.

...
Hello
Hello
World!
Hello
Hello
World!
Hello
Hello
World!
...

So, I’ll take one last example before calling it for the day – if you remember the line “except StopIteration:” from the scheduler.run()-function we can use that to make a generator drop out of the scheduler’s run()-loop, i present to you the marvelous example_4.py:

from time import sleep
from demo import tasks, scheduler

def echo(word):
	while True:
		print word
		yield

def sleeper(seconds):
	while True:
		sleep(seconds)
		yield

def echo_once(word):
	while True:
		print word
		yield
		raise StopIteration

scheduler.add(
	tasks.Task(
		echo("Hello")))

scheduler.add(
	tasks.Task(
		echo_once("World!")))

scheduler.add(
	tasks.Task(
		sleeper(1)))

scheduler.run()

Running this from the terminal will yield (no pun intended) you something like this, only printing “World!” once:

Hello
World!
Hello
Hello
Hello
...

In the next part, number four I will show you a real world example with asynchronous network i/o, i just wanted to introduce the concept of a scheduler before jumping into a more advanced example.

Indepth generator usage in Python (part2)

Fredrik Holmström — Sat, 30 Aug 2008 14:50:56 +0000

After my last post on generators in Python I realized I missed go through one thing that I wanted to mention in the first part, namely how the return keyword interfaces with yield and generators, take this example function and its usage:

def count_to_3or4():
	counter = 0

	while counter < 3:
		counter += 1
		yield counter

	return counter+1

c = count_to_3or4()
print c.next() # 1
print c.next() # 2
print c.next() # 3
print c.next() # 4, from return - or ?

If you've read the previous post, or have a basic understanding of generators you would probably guess that 1, 2, 3 will print from the thee first .next()-calls - but you would be wrong. If you try to run the above code you will get this thrown back in your face:

  File "generators.py", line 15
    return counter+1
SyntaxError: 'return' with argument inside generator

So you can't use return in generators (functions with yield) - well, yes you can - you just can't use return with a value attached to it. If you call return within a generator function it will exit and any further calls to .next() will throw a StopIteration exception, take this example code:

def count_to_2or3():
	counter = 0

	while counter < 3:
		counter += 1
		yield counter

		if counter is 2:
			return

c = count_to_2or3()
print c.next() # 1
print c.next() # 2
print c.next() # 3, or ?

It will print 1 and 2, when you call .next() a third time it will hit return (since counter == 2, the if-clause evaluates to true) and throw a StopIteration exception. Basically "return" inside a generator-function does what "break" does inside a loop.

Performance

When you're using generators as a type of iterator together with for (or manually, for that matter) working with large datasets you will see a substantial performance increase over list-generating functions. These two functions will generate the exact same output, but one will be significantly faster and use less memory:

def count_to_list(stop):
	_list = []
	counter = 0

	while counter < stop:
		counter += 1
		_list.append(counter)

	return _list

def count_to_generator(stop):
	counter = 0

	while counter < stop:
		counter += 1
		yield counter

The first function will generate a list of numbers (which takes quite some time and memory) and then return that while the second function, our generator will produce one number each time .next() is called on it and only consume as much memory as one integer take up while also being a fair bit faster, running this on my VPS yields (again, no pun intended) these results:

fredrik@holmstrom:~/python/generators$ time python list.py

real    0m0.611s
user    0m0.540s
sys     0m0.040s

fredrik@holmstrom:~/python/generators$ time python generator.py

real    0m0.385s
user    0m0.380s
sys     0m0.000s

I didn't measure memory usage here, but trust me - generator.py will consume a lot less memory, this technique is also called "lazy evaluation" in proper CS terms - there's a lot more information on this topic alone, but this will do for now.

Advanced usage

As I mentioned in the history introduction in my previous post about generators, Python 2.5 gave generators a substantial usability boost allowing us to pass information back into the function through the yield statement and the .send() and .throw() methods on the generator-object. send() works exactly like next() except that you can pass a value back into the function as it's first argument, but there are a few caveats you should look out for - take this snippet of code:

def echo():
	while True:
		print yield

Will give you the following SyntaxError:

  File "explained.py", line 3
    print yield
              ^

SyntaxError: invalid syntax

Changing the print yield to this:

def echo():
	while True:
		val = yield
		print val

Will make the code execute properly, however this seems pretty non-pythonic having to store the variable we want to store the result in a temporary variable - so instead we can do this:

def echo():
	while True:
		print (yield)

Wrapping the yield in parenthesizes will allow you to use the result of it directly instead of storing it in a temporary variable, so let's put our echo() generator to use:

def echo():
	while True:
		print (yield)

e = echo()
e.send("Hello")
e.send("World!")

But, running this code will show you the second caveat of trying to pass values back into the generator function, this TypeError will be thrown in your face if you run this code:

Traceback (most recent call last):
  File "generator.py", line 6, in 
    e.send("Hello")
TypeError: can't send non-None value to a just-started generator

Remember how I said that yield paused the execution of the generator function and that when you call the generator function (in this case e = echo()) no code is yet to be executed until you call .next() on your generator-object? So if .send() can be used to pass data back into a yield statement while the generator is paused, we can't call .send() when no code has been executed and no yield statement has paused the generator, right?

What this means in practice is that you either have to call .next() or .send(None) the first time you call a generator, and when the generator reaches its first yield statement it will pause execution waiting for another call to .send() (or .next() if you don't want to pass any data back) that will pass data back into it at the yield statement, confusing? So changing the code above to this:

def echo():
	while True:
		print (yield)

e = echo()
e.send(None) # or e.next()
e.send("Hello")
e.send("World!")

Will make it run, printing:

Hello
World!

To illustrate exactly what's happening here, I'll take another example - slightly more advanced but still achieving the same result as above:

def echo():
	counter = 0

	while True:
		counter += 1
		print (yield counter)

e = echo()
print "Yeild nr %s" % e.send(None) 	# Sending nothing in (since we havn't paused
					# anything with yield yet) and yielding nr 1
					# back to the print statement

print "Yeild nr %s" % e.send("Hello")	# the pause from nr 1 gets resumed, passing "Hello"
					# back in and printing it, then doing another loop
					# and yielding nr 2 back and pausing execution

print "Yeild nr %s" % e.send("World!")	# the pause from nr 2 gets resumed, passing "World!"
					# back in and printing it, then doing another loop
					# and yielding nr 3 back to and pausing execution

					# If we would call the same e.send("Blah"), etc.
					# here we could go on forever since the yield
					# statement is stuck in a "while True"-loop

Make sure to read the comments in the above code since I figured it would be a lot easier to explain if the comments where attached to the correct line, running the above code will yield (again, no pun intended ;p) the following results:

Yeild nr 1
Hello
Yeild nr 2
World!
Yeild nr 3

Quite simple, and yet so powerful. There is one last thing I want to demonstrate in this, second part, of the tutorial - the method .throw() those of you familiar with other languages then python might recognize the word throw and figure it would have something to do with exceptions, and you'd be correct - it does.

As I've demonstrated, .send() sends in data to the paused yield statement, and .throw() does something similar: it sends in an exception that gets thrown and the paused yield statements line, let's demonstrate:

def exceptional():
	while True:
		yield

e = exceptional()
e.next()
e.throw(Exception)

Will give you this output:

Traceback (most recent call last):
  File "generator.py", line 7, in 
    e.throw(Exception)
  File "generator.py", line 3, in exceptional
    yield
Exception

Which is correct, because you sent an Exception in. It is possible to call .throw() as the first method on a new generator object, before any call to .next() or .send(), however that will throw an exception before any code is executed in the method and you will not have a chance to handle it.

In the stack trace above you also see that the exception is actually thrown at the "yield" line when it's resumed after being paused by .next() the first time.

Let's do a more advanced example, with a custom exception class:

def exceptional():
	counter = 0
	while True:
		try:
			counter += 1
			yield counter
		except DemoException, exc:
			print "Caught exception with message: %s" % exc

class DemoException(Exception):
	pass

e = exceptional()
print e.next()
print e.throw(DemoException("Hello World"))

The above code will print this:

1
Caught exception with message: Hello World
2

And here's the magic - if you handle the exception that gets thrown in at the line yield was called at (by wrapping it in a try/except/finally-block) the code will continue executing like it should and .throw() will return the result of the next invocation of yield. All in all .send() and .throw() work exactly the same way except that .throw() raises whatever you feed it with as an exception.

The ability to pass errors (exceptions) *into* generators allows you to do some really neat error handling that doesn't require your wrapping code to have any information about the generator resulting in a very clean and loosely coupled code.

In the next, and last, part I will go through a real world example using asynchronous i/o and network calls utilizing all the techniques explained in these two posts.

Introduction to generators in Python

Fredrik Holmström — Fri, 29 Aug 2008 14:32:18 +0000

First, some history…

Generators is a concept that was introduced in Python at version 2.2, back then they were unidirectional that only allowed information to be passed out of the generator and not back into it, which limited their use to simple iterators and not much else. This was changed / enhanced in Python 2.5 when both data and exceptions now can be passed back into to generator. The changes made in 2.5 allowed for generators to be used as coroutines enabling them to function in complex event-driven programming such as asynchronous I/o, games, etc.

So how does one define a generator in Python? It’s actually very simple, you just define a normal subroutine (or function, if you will) that has the keyword yield somewhere inside of its body, here’s a quick example:

def foo():
    yield

What does yield do to a subroutine then? When a subroutine encounters the yield expression it suspends execution so that it can be resumed at a later time, as chosen by the programmer. You basically tell the routine “I don’t want to continue executing you now, but at a later stage I might want to and you should resume from the point where the yield statement was and not start over”, it’s also important to note that when yield is called the subroutine’s state (variable values, etc.) are all saved, so when you continue executing it everything will be the way you left it.

When you call a generator-function (a subroutine/function with the yield-keyword in its body) you don’t get a result back, instead you get a generator-object back that is used to control the execution of the subroutine, take the foo()-routine we defined above, if we do this:

gen = foo()
print gen

This is what python will print about the “result” of foo(): , so when we call a generator function we get a generator object back, not the result of the function call. Note that none of the code inside foo() has yet been executed, as I’ve said the generators execution is controlled through the generator-object, primarily by it’s next()-method which will start/resume execution until a yield statement is found, and then return. So if we do this instead:

gen = foo()
print gen.next()

We get back this: None, not very useful at all, if we try calling gen.next() again we will get something like this:

Traceback (most recent call last):
  File "generators.py", line 7, in 
    print gen.next()
StopIteration

Because the yield statement only gets executed once in our foo-generator and the generator then reaches its end, we can only “resume” execution with next() once. So what if add two yields to foo() instead, making the code look like this:

def foo():
	yield
	yield

gen = foo()
print gen.next()
print gen.next()

This works, giving us back:

None
None

calling gen.next() a third time will, again, raise a StopIteration-exception. We’re still only getting back a lot of nothing (None) from our generators, how about passing something back out from our yield statements, modifying foo again making it look like this:

def foo():
	yield "Hello"
	yield "World!"

gen = foo()
print gen.next()
print gen.next()

Will yield (no pun intended) this result:

Hello
World!

Kind of what you were expecting, huh? So let’s do something a bit more interesting, or well – something that shows what generators are useful for:

def counter(count_to):
	counter = 0;

	while counter < count_to:
		counter = counter+1
		yield counter

As you see this generator named counter takes one argument, an integer which decides how far we should count, remember when we call counter(3) to count to three no code inside the generator gets executed until we call the generator-objects next() method, it then executes normally until it hits a yield statement and then suspends returning (through next()) whatever we fed to yield, let's see it in action:

c = counter(3)
print c.next()
print c.next()
print c.next()

This will, maybe not to our surprise now, print:

1
2
3

When the three gets "yielded" to us, we can not call next() again without raising an StopIteration-exception because the while-loops condition would return false skipping the yield statement within it and counter() would end, without yielding anything back to us through next().

What happens if we call counter() several times? We will get several generator-objects each representing one invocation of counter() with its own internal state, you can almost think of it like creating an two object instances of a class:

c1 = counter(3)
c2 = counter(4)
print c1.next()
print c2.next()
print c1.next()
print c2.next()
print c1.next()
print c2.next()

print c2.next() # We can run c2 once more c1 since its counting to four and c1 to three

The above code will print,

, demonstrating that each invocation of a generator function creates its own generator-object and scope. Generators are used everywhere in python, in most cases they are used as iterators together with the for statement but they have other uses to. Using a generator together with a for statement is very straightforward, take the above counter()-function, we can use it the same way range() is used in python:

for i in counter(5):
	print i

The for-language construct in python has a built in way of handling generators, when it gets fed a generator-object (the result of calling counter(5) in this case) it will call .next() on it putting the value returned in the iteration-variable, i in this case. When for gets an StopIteration exception thrown from the generator for calling .next() one to many times it will silently kill the exception and stop the loop, neat huh?

Lets write a, again useless, generator-function that iterates through every letter in a word and call it with for:

def letters(word):
	for i in range(len(word)):
		yield word[i]

for letter in letters("Hello World"):
	print letter

I think you can guess what this will print, yeah. While the above function is practically useless in python - it's a good example on how generators and the for statement work together. I hope this run-through gave you a quick look into what generators are, if you're interested in learning more about them make sure to check out part two of this article series.