2 Part 2 Advanced Python
2.3 Iterator Objects
while 1:
target = raw_input('Enter a target line ("q" to quit): ') if target == 'q':
break
repl = raw_input('Enter a replacement: ') result = pat.sub(repl, target)
print 'result: %s' % result
Here is another example of the use of a function to insert calculated replacements.
import sys, re, string pat = re.compile('[am]+') def replacer(mo):
return string.upper(mo.group(0)) print 'Uppercasing am.'
while 1:
target = raw_input('Enter a target line ("q" to quit): ') if target == 'q':
break
result = pat.sub(replacer, target) print 'result: %s' % result
Notes:
● If the replacement argument to sub is a function, that function must take one argument, a match object, and must return the modified (or replacement) value.
The matched substring will be replaced by the value returned by this function.
● In our case, the function replacer converts the matched value to upper case.
This is also a convenient use for a lambda instead of a named function, for example:
import sys, re, string pat = re.compile('[am]+') print 'Uppercasing am.' while 1:
target = raw_input('Enter a target line ("q" to quit): ') if target == 'q':
break
result = pat.sub(
lambda mo: string.upper(mo.group(0)), target)
print 'result: %s' % result
Note 2: The iterator protocol has changed slightly in Python version 3.0.
Goals for this section:
● Learn how to implement a generator function, that is, a function which, when called, returns an iterator.
● Learn how to implement a class containing a generator method, that is, a method which, when called, returns an iterator.
● Learn the iterator protocol, specifically what methods an iterator must support and what those methods must do.
● Learn how to implement an iterator class, that is, a class whose instances are iterator objects.
● Learn how to implement recursive iterator generators, that is, an iterator generator which recursively produces iterator generators.
● Learn that your implementation of an iterator object (an iterator class) can
"refresh" itself and learn at least one way to do this.
Definitions:
● Iterator And iterator is an object that satisfies (implements) the iterator protocol.
● Iterator protocol An object implements the iterator protocol if it implements both a next() and an __iter__() method which satisfy these rules: (1) the
__iter__() method must return the iterator; (2) the next() method should return the next item to be iterated over and when finished (there are no more items) should raise the StopIteration exception. The iterator protocol is described at Iterator Types
http://docs.python.org/library/stdtypes.html#iteratortypes.
● Iterator class A class that implements (satisfies) the iterator protocol. In particular, the class implements next() and __iter__() methods as described above and in Iterator Types
http://docs.python.org/library/stdtypes.html#iteratortypes.
● (Iterator) generator function A function (or method) which, when called, returns an iterator object, that is, an object that satisfies the iterator protocol. A function containing a yield statement automatically becomes a generator.
● Generator expression An expression which produces an iterator object.
Generator expressions have a form similar to a list comprehension, but are enclosed in parentheses rather than square brackets. See example below.
A few additional basic points:
● A function that contains a yield statement is a generator function. When called, it returns an iterator, that is, an object that provides next() and __iter__() methods.
● The iterator protocol is described here: Python Standard Library: Iterator Types
http://docs.python.org/library/stdtypes.html#iteratortypes.
● A class that defines both a next() method and a __iter__() method satisfies the iterator protocol. So, instances of such a class will be iterators.
● Python provides a variety of ways to produce (implement) iterators. This section describes a few of those ways. You should also look at the iter() builtin function, which is described in The Python Standard Library: Builtin Functions:
iter() http://docs.python.org/library/functions.html#iter.
● An iterator can be used in an iterator context, for example in a for statement, in a list comprehension, and in a generator expression. When an iterator is used in an iterator context, the iterator produces its values.
This section attempts to provide examples that illustrate the generator/iterator pattern.
Why is this important?
● Once mastered, it is a simple, convenient, and powerful programming pattern.
● It has many and pervasive uses.
● It helps to lexically separate the producer code from the consumer code. Doing so makes it easier to locate problems and to modify or fix code in a way that is localized and does not have unwanted sideeffects.
● Implementing your own iterators (and generators) enables you to define your own abstract sequences, that is, sequences whose composition are defined by your computations rather than by their presence in a container. In fact, your iterator can calculate or retrieve values as each one is requested.
Examples The remainder of this section provides a set of examples which implement and use iterators.
2.3.1 Example A generator function
This function contains a yield statement. Therefore, when we call it, it produces an iterator:
def generateItems(seq):
for item in seq:
yield 'item: %s' % item anIter = generateItems([])
print 'dir(anIter):', dir(anIter) anIter = generateItems([111,222,333]) for x in anIter:
print x
anIter = generateItems(['aaa', 'bbb', 'ccc']) print anIter.next()
print anIter.next() print anIter.next() print anIter.next()
Running this example produces the following output:
dir(anIter): ['__class__', '__delattr__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__iter__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', 'gi_frame', 'gi_running', 'next']
item: 111 item: 222 item: 333 item: aaa item: bbb item: ccc
Traceback (most recent call last):
File "iterator_generator.py", line 14, in ? print anIter.next()
StopIteration
Notes and explanation:
● The value returned by the call to the generator (function) is an iterator. It obeys the iterator protocol. That is, dir(anIter) shows that it has both
__iter__() and next() methods.
● Because this object is an iterator, we can use a for statement to iterate over the values returned by the generator.
● We can also get its values by repeatedly calling the next() method, until it raises the StopIteration exception. This ability to call the next method enables us to pass the iterator object around and get values at different locations in our code.
● Once we have obtained all the values from an iterator, it is, in effect, "empty" or
"exhausted". The iterator protocol, in fact, specifies that once an iterator raises the StopIteration exception, it should continue to do so. Another way to say this is that there is no "rewind" operation. But, you can call the the generator function again to get a "fresh" iterator.
An alternative and perhaps simpler way to create an interator is to use a generator expression. This can be useful when you already have a collection or iterator to work with.
Then following example implements a function that returns a generator object. The effect is to generate the objects in a collection which excluding items in a separte collection:
DATA = [ 'lemon', 'lime', 'grape', 'apple', 'pear',
'watermelon', 'canteloupe', 'honeydew', 'orange',
'grapefruit', ]
def make_producer(collection, excludes):
gen = (item for item in collection if item not in excludes) return gen
def test():
iter1 = make_producer(DATA, ('apple', 'orange', 'honeydew', )) print '%s' % iter1
for fruit in iter1:
print fruit test()
When run, this example produces the following:
$ python workbook063.py
<generator object <genexpr> at 0x7fb3d0f1bc80>
lemon lime grape pear
watermelon canteloupe grapefruit
Notes:
● A generator expression looks almost like a list comprehension, but is surrounded by parentheses rather than square brackets. For more on list comprehensions see section Example A list comprehension.
● The make_producer function returns the object produced by the generator expression.
2.3.2 Example A class containing a generator method
Each time this method is called, it produces a (new) iterator object. This method is analogous to the iterkeys and itervalues methods in the dictionary builtin object:
#
# A class that provides an iterator generator method.
#
class Node:
def __init__(self, name='<noname>', value='<novalue>', children=None):
self.name = name self.value = value
self.children = children if children is None:
self.children = []
else:
self.children = children
def set_name(self, name): self.name = name def get_name(self): return self.name
def set_value(self, value): self.value = value def get_value(self): return self.value
def iterchildren(self):
for child in self.children:
yield child #
# Print information on this node and walk over all children and # grandchildren ...
def walk(self, level=0):
print '%sname: %s value: %s' % (
get_filler(level), self.get_name(), self.get_value(), ) for child in self.iterchildren():
child.walk(level + 1)
#
# An function that is the equivalent of the walk() method in
# class Node.
#
def walk(node, level=0):
print '%sname: %s value: %s' % (
get_filler(level), node.get_name(), node.get_value(), ) for child in node.iterchildren():
walk(child, level + 1) def get_filler(level):
return ' ' * level def test():
a7 = Node('gilbert', '777') a6 = Node('fred', '666') a5 = Node('ellie', '555') a4 = Node('daniel', '444')
a3 = Node('carl', '333', [a4, a5]) a2 = Node('bill', '222', [a6, a7]) a1 = Node('alice', '111', [a2, a3])
# Use the walk method to walk the entire tree.
print 'Using the method:' a1.walk()
print '=' * 30
# Use the walk function to walk the entire tree.
print 'Using the function:' walk(a1)
test()
Running this example produces the following output:
Using the method:
name: alice value: 111
name: bill value: 222 name: fred value: 666 name: gilbert value: 777 name: carl value: 333
name: daniel value: 444 name: ellie value: 555
==============================
Using the function:
name: alice value: 111 name: bill value: 222 name: fred value: 666 name: gilbert value: 777 name: carl value: 333
name: daniel value: 444 name: ellie value: 555
Notes and explanation:
● This class contains a method iterchildren which, when called, returns an iterator.
● The yield statement in the method iterchildren makes it into a generator.
● The yield statement returns one item each time it is reached. The next time the iterator object is "called" it resumes immediately after the yield statement.
● A function may have any number of yield statements.
● A for statement will iterate over all the items produced by an iterator object.
● This example shows two ways to use the generator, specifically: (1) the walk method in the class Node and (2) the walk function. Both call the generator iterchildren and both do pretty much the same thing.
2.3.3 Example An iterator class
This class implements the iterator protocol. Therefore, instances of this class are iterators.
The presence of the next() and __iter__() methods means that this class implements the iterator protocol and makes instances of this class iterators.
Note that when an iterator is "exhausted" it, normally, cannot be reused to iterate over the sequence. However, in this example, we provide a refresh method which enables us to
"rewind" and reuse the iterator instance:
#
# An iterator class that does *not* use ``yield``.
# This iterator produces every other item in a sequence.
#
class IteratorExample:
def __init__(self, seq):
self.seq = seq self.idx = 0 def next(self):
self.idx += 1
if self.idx >= len(self.seq):
raise StopIteration value = self.seq[self.idx]
self.idx += 1 return value def __iter__(self):
return self def refresh(self):
self.idx = 0
def test_iteratorexample():
a = IteratorExample('edcba') for x in a:
print x
print '' a.refresh()
for x in a:
print x print '=' * 30
a = IteratorExample('abcde') try:
print a.next() print a.next() print a.next() print a.next() print a.next() print a.next()
except StopIteration, e:
print 'stopping', e test_iteratorexample()
Running this example produces the following output:
d b
d b
==============================
b d
stopping
Notes and explanation:
● The next method must keep track of where it is and what item it should produce next.
● Alert: The iterator protocol has changed slightly in Python 3.0. In particular, the next() method has been renamed to __next__(). See: Python Standard Library: Iterator Types
http://docs.python.org/3.0/library/stdtypes.html#iteratortypes.
2.3.4 Example An iterator class that uses yield
There may be times when the next method is easier and more straightforward to implement using yield. If so, then this class might serve as an model. If you do not feel the need to do this, then you should ignore this example:
#
# An iterator class that uses ``yield``.
# This iterator produces every other item in a sequence.
#
class YieldIteratorExample:
def __init__(self, seq):
self.seq = seq
self.iterator = self._next() self.next = self.iterator.next def _next(self):
flag = 0
for x in self.seq:
if flag:
flag = 0 yield x else:
flag = 1 def __iter__(self):
return self.iterator def refresh(self):
self.iterator = self._next() self.next = self.iterator.next def test_yielditeratorexample():
a = YieldIteratorExample('edcba') for x in a:
print x
print '' a.refresh()
for x in a:
print x print '=' * 30
a = YieldIteratorExample('abcde') try:
print a.next() print a.next() print a.next() print a.next() print a.next() print a.next()
except StopIteration, e:
print 'stopping', e test_yielditeratorexample()
Running this example produces the following output:
d b
d b
==============================
b d
stopping
Notes and explanation:
● Because the _next method uses yield, calling it (actually, calling the iterator object it produces) in an iterator context causes it to be "resumed" immediately after the yield statement. This reduces bookkeeping a bit.
● However, with this style, we must explicitly produce an iterator. We do this by calling the _next method, which contains a yield statement, and is therefore a generator. The following code in our constructor (__init__) completes the setup of our class as an iterator class:
self.iterator = self._next() self.next = self.iterator.next
Remember that we need both __iter__() and next() methods in
YieldIteratorExample to satisfy the iterator protocol. The __iter__() method is already there and the above code in the constructor creates the next() method.
2.3.5 Example A list comprehension
A list comprehension looks a bit like an iterator, but it produces a list. See: The Python Language Reference: List displays
http://docs.python.org/reference/expressions.html#listdisplays for more on list comprehensions.
Here is an example:
In [4]: def f(x):
...: return x * 3 ...:
In [5]: list1 = [11, 22, 33]
In [6]: list2 = [f(x) for x in list1]
In [7]: print list2 [33, 66, 99]
2.3.6 Example A generator expression
A generator expression looks quite similar to a list comprehension, but is enclosed in
parentheses rather than square brackets. Unlike a list comprehension, a generator
expression does not produce a list; it produces an generator object. A generator object is an iterator.
For more on generator expressions, see The Python Language Reference: Generator expressions http://docs.python.org/reference/expressions.html#generatorexpressions.
The following example uses a generator expression to produce an iterator:
mylist = range(10) def f(x):
return x*3
genexpr = (f(x) for x in mylist) for x in genexpr:
print x
Notes and explanation:
● The generator expression (f(x) for x in mylist) produces an iterator object.
● Notice that we can use the iterator object later in our code, can save it in a data structure, and can pass it to a function.