fredrikholmstrom.com

First, some history…

Generators is a concept that was introduced in Python at version 2.2, back then they were unidirectional that only allowed information to be passed out of the generator and not back into it, which limited their use to simple iterators and not much else. This was changed / enhanced in Python 2.5 when both data and exceptions now can be passed back into to generator. The changes made in 2.5 allowed for generators to be used as coroutines enabling them to function in complex event-driven programming such as asynchronous I/o, games, etc.

So how does one define a generator in Python? It’s actually very simple, you just define a normal subroutine (or function, if you will) that has the keyword yield somewhere inside of its body, here’s a quick example:

def foo():
    yield

What does yield do to a subroutine then? When a subroutine encounters the yield expression it suspends execution so that it can be resumed at a later time, as chosen by the programmer. You basically tell the routine “I don’t want to continue executing you now, but at a later stage I might want to and you should resume from the point where the yield statement was and not start over”, it’s also important to note that when yield is called the subroutine’s state (variable values, etc.) are all saved, so when you continue executing it everything will be the way you left it.

When you call a generator-function (a subroutine/function with the yield-keyword in its body) you don’t get a result back, instead you get a generator-object back that is used to control the execution of the subroutine, take the foo()-routine we defined above, if we do this:

gen = foo()
print gen

This is what python will print about the “result” of foo(): <generator object at 0x2b8cbb061098>, so when we call a generator function we get a generator object back, not the result of the function call. Note that none of the code inside foo() has yet been executed, as I’ve said the generators execution is controlled through the generator-object, primarily by it’s next()-method which will start/resume execution until a yield statement is found, and then return. So if we do this instead:

gen = foo()
print gen.next()

We get back this: None, not very useful at all, if we try calling gen.next() again we will get something like this:

Traceback (most recent call last):
  File "generators.py", line 7, in 
    print gen.next()
StopIteration

Because the yield statement only gets executed once in our foo-generator and the generator then reaches its end, we can only “resume” execution with next() once. So what if add two yields to foo() instead, making the code look like this:

def foo():
	yield
	yield

gen = foo()
print gen.next()
print gen.next()

This works, giving us back:

None
None

calling gen.next() a third time will, again, raise a StopIteration-exception. We’re still only getting back a lot of nothing (None) from our generators, how about passing something back out from our yield statements, modifying foo again making it look like this:

def foo():
	yield "Hello"
	yield "World!"

gen = foo()
print gen.next()
print gen.next()

Will yield (no pun intended) this result:

Hello
World!

Kind of what you were expecting, huh? So let’s do something a bit more interesting, or well – something that shows what generators are useful for:

def counter(count_to):
	counter = 0;

	while counter < count_to:
		counter = counter+1
		yield counter

As you see this generator named counter takes one argument, an integer which decides how far we should count, remember when we call counter(3) to count to three no code inside the generator gets executed until we call the generator-objects next() method, it then executes normally until it hits a yield statement and then suspends returning (through next()) whatever we fed to yield, let's see it in action:

c = counter(3)
print c.next()
print c.next()
print c.next()

This will, maybe not to our surprise now, print:

1
2
3

When the three gets "yielded" to us, we can not call next() again without raising an StopIteration-exception because the while-loops condition would return false skipping the yield statement within it and counter() would end, without yielding anything back to us through next().

What happens if we call counter() several times? We will get several generator-objects each representing one invocation of counter() with its own internal state, you can almost think of it like creating an two object instances of a class:

c1 = counter(3)
c2 = counter(4)
print c1.next()
print c2.next()
print c1.next()
print c2.next()
print c1.next()
print c2.next()

print c2.next() # We can run c2 once more c1 since its counting to four and c1 to three

The above code will print,

1
1
2
2
3
3
4

, demonstrating that each invocation of a generator function creates its own generator-object and scope. Generators are used everywhere in python, in most cases they are used as iterators together with the for statement but they have other uses to. Using a generator together with a for statement is very straightforward, take the above counter()-function, we can use it the same way range() is used in python:

for i in counter(5):
	print i

The for-language construct in python has a built in way of handling generators, when it gets fed a generator-object (the result of calling counter(5) in this case) it will call .next() on it putting the value returned in the iteration-variable, i in this case. When for gets an StopIteration exception thrown from the generator for calling .next() one to many times it will silently kill the exception and stop the loop, neat huh?

Lets write a, again useless, generator-function that iterates through every letter in a word and call it with for:

def letters(word):
	for i in range(len(word)):
		yield word[i]

for letter in letters("Hello World"):
	print letter

I think you can guess what this will print, yeah. While the above function is practically useless in python - it's a good example on how generators and the for statement work together. I hope this run-through gave you a quick look into what generators are, if you're interested in learning more about them make sure to check out part two of this article series.