Free lesson · Python Basic to Advance

The Iterator Protocol

Medium7 min readpython

The Iterator Protocol

What an Iterator Is

An iterator is an object that produces a sequence of values one at a time. Iterators are the foundation of Python's looping mechanism - for loops, comprehensions, sum, max, zip, and many other tools all consume iterators.

Two methods define the iterator protocol:

__iter__(self) - returns the iterator object itself
__next__(self) - returns the next value, or raises StopIteration when exhausted

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for n in Countdown(3):
    print(n)
# 3
# 2
# 1

This code defines a simple iterator. Iter returns self - the object is its own iterator. Next returns the next value or raises stop iteration when done. The for loop calls next repeatedly until stop iteration is raised. Countdown produces three then two then one and stops.

This is the manual form. Most iterators in practice are written as generators (Topic 19.3) - much less code.

Iterables vs Iterators

Python distinguishes two related concepts:

Iterable - an object that can produce an iterator (has __iter__)
Iterator - the actual stepping object (has __next__ and __iter__)

# Lists are iterables, not iterators
nums = [1, 2, 3]

# Calling iter() on a list creates an iterator
it = iter(nums)
print(next(it))    # 1
print(next(it))    # 2

# A list itself isn't an iterator
next(nums)
# TypeError: 'list' object is not an iterator

This code shows the distinction. Lists are iterables but not iterators. Calling iter on a list produces an iterator over its elements. Calling next on a list directly raises type error. The iterator is the stepping object; the iterable is what produces it. For loops handle the iter call automatically.

The for loop, comprehensions, and most consumers handle this automatically - they call iter() on the iterable to get an iterator.

How `for` Loops Work

A for loop is shorthand for the manual iterator protocol:

nums = [1, 2, 3]

# for loop
for n in nums:
    print(n)

# Equivalent manual form
it = iter(nums)
while True:
    try:
        n = next(it)
    except StopIteration:
        break
    print(n)

This code shows the for loop's underlying mechanism. The for statement calls iter to get an iterator, then calls next in a loop until stop iteration is raised. The manual form is what for compiles to. Knowing this helps debug iterator-related issues and design custom iterables.

This is exactly what Python does. Knowing the underlying protocol helps when implementing custom iterables.

Iterators Are Single-Use

An iterator is exhausted after one full pass:

nums = [1, 2, 3]
it = iter(nums)

list(it)    # [1, 2, 3] - first pass

list(it)    # [] - already exhausted

This code shows iterator exhaustion. After consuming the first three elements, the iterator is empty. Subsequent calls to next raise stop iteration. Useful for streaming data once. Surprising when you didn't realize it - comprehensions and other consumers all exhaust the iterator.

To iterate multiple times, either keep the underlying iterable (the list) and create new iterators, or save the values into a list.

`StopIteration` and Sentinel

StopIteration is the protocol signal for "no more values":

nums = [1, 2]
it = iter(nums)

print(next(it))    # 1
print(next(it))    # 2
print(next(it))    # raises StopIteration

# With a default - return value instead of raising
print(next(it, "done"))    # "done" (we'd get this instead of raise)

This code shows stop iteration. After the last value, next raises stop iteration by default. With a second argument, next returns that default instead of raising. Useful when you want to know "is there more" without exception handling.

The default-second-argument form is convenient when reading iterators that might be empty.

Built-In Iterators

Python's standard library provides many iterators:

# range
for i in range(5):
    print(i)

# enumerate - pairs (index, value)
for i, item in enumerate(["a", "b", "c"]):
    print(i, item)

# zip - combine multiple iterables
for a, b in zip([1, 2, 3], ["x", "y", "z"]):
    print(a, b)

# reversed
for x in reversed([1, 2, 3]):
    print(x)

# filter - produce items matching a predicate
for x in filter(lambda x: x > 2, [1, 2, 3, 4]):
    print(x)

# map - apply a function to each item
for x in map(str.upper, ["hello", "world"]):
    print(x)

This code shows common built-in iterators. Range, enumerate, zip, reversed, filter, map. Each is lazy - it produces values one at a time as iterated. Combining them gives concise data pipelines. The itertools module has many more - covered in module 10.

These are all iterators - lazy, single-use, memory-efficient.

The `iter()` Function

iter() has two forms:

# Single-argument: convert iterable to iterator
it = iter([1, 2, 3])

# Two-argument: call a function until it returns a sentinel value
import random
def random_byte():
    return random.randint(0, 255)

# Iterate until we get 0
for byte in iter(random_byte, 0):
    print(byte)

This code shows the two forms of iter. Single arg converts an iterable. Two arg calls a function until it returns the sentinel. Useful for reading until end of file or other "until some condition" patterns.

The two-argument form is less common but useful for callback-style sources.

Implementing an Iterable

For a class to be iterable, define __iter__ returning an iterator. The class itself doesn't need to be the iterator - separating them allows multiple concurrent iterations:

class Doubles:
    def __init__(self, items):
        self.items = items

    def __iter__(self):
        return _DoublesIterator(self.items)

class _DoublesIterator:
    def __init__(self, items):
        self.items = items
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index >= len(self.items):
            raise StopIteration
        value = self.items[self.index] * 2
        self.index += 1
        return value

d = Doubles([1, 2, 3])

# Each for-loop creates its own iterator - independent state
for x in d:
    print(x)    # 2, 4, 6

for x in d:
    print(x)    # 2, 4, 6 again - fresh iterator

This code separates iterable from iterator. The Doubles class is iterable; each iter call creates a fresh DoublesIterator. This lets the same Doubles be iterated multiple times without exhausting. For most cases, generators - Topic 19.3 - handle this more cleanly with much less code.

The separation lets the same iterable produce multiple concurrent iterators.

Iterator vs Generator

For most cases, generators (Topic 19.3) are the better choice - they're iterators implemented via the yield statement, requiring far less code:

# Iterator class - verbose
class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# Generator function - same behavior, less code
def countdown(start):
    while start > 0:
        yield start
        start -= 1

This code compares the two approaches. The class form is explicit but verbose. The generator form does the same thing in three lines. For most "produce a sequence of values" needs, generators are the right tool. Topic 19.3 covers them.

The class form is appropriate when iteration needs to interleave with other state, or when the class has methods beyond iteration.

Composing Iterators

Iterators chain efficiently - each step is lazy, so no intermediate lists are built:

import itertools

# Multiply each, filter to even, take the first 5
nums = range(100)
doubled = map(lambda x: x * 2, nums)
even_doubled = filter(lambda x: x % 4 == 0, doubled)
first_five = itertools.islice(even_doubled, 5)

print(list(first_five))
# [0, 4, 8, 12, 16]

This code chains iterators. Each step is lazy. Map doesn't run until consumed. Filter doesn't run until consumed. Islice takes only the first 5 - at which point only 5 elements have been produced. The chain processes a streaming sequence without building intermediate lists.

This composability is what makes iterators efficient for data pipelines.

`itertools` Module

itertools provides many specialized iterators:

Function	What it does
`count(start, step)`	Infinite counter
`cycle(iterable)`	Repeat forever
`chain(*iterables)`	Concatenate
`islice(iterable, ...)`	Slice an iterator
`groupby(iterable, key)`	Group consecutive equal items
`permutations(iterable)`	All orderings
`combinations(iterable, r)`	All r-length combinations
`product(*iterables)`	Cartesian product

import itertools

# Take first 5 from an infinite counter
first_five = list(itertools.islice(itertools.count(), 5))
print(first_five)
# [0, 1, 2, 3, 4]

# Chain
combined = list(itertools.chain([1, 2], [3, 4]))
# [1, 2, 3, 4]

This code uses itertools functions. They produce iterators, often combining or transforming other iterators. Useful for many common data manipulations. The full reference is in the itertools documentation.

itertools is one of the most-used standard library modules.

Performance

Iterators are lazy and memory-efficient - produce one value at a time. For large datasets, iteration consumes constant memory.

The overhead per iteration is small but real. For tight numeric loops on large arrays, libraries like numpy are far faster because they use vectorized operations bypassing Python's iteration overhead.

Interview Relevance

Interview: The iterator protocol is fundamental Python:

The two methods: __iter__ and __next__

The distinction between iterables and iterators

That for loops use the protocol

StopIteration as the end signal

Iterators are single-use

How generators relate to manual iterators

Common built-ins: range, zip, map, filter, enumerate

A common interview moment: implementing a custom iterator. The right shape involves __iter__ returning self or a helper, and __next__ raising StopIteration when done.

Another moment: code that iterates the same iterator twice and gets nothing the second time. The diagnosis is exhaustion.

Module 30 covers Python interview preparation.

Common Pitfalls

Warning: Confusing iterables with iterators. Lists are iterables; iter(list) creates an iterator.

Iterating an exhausted iterator. After one pass, it produces nothing. Create a new iterator.

Forgetting __iter__ returns self for an iterator. Without it, the iterator can't be used in for loops.

Using __next__ without StopIteration. The loop runs forever. Always have a termination condition.

Writing iterator classes when generators would do. Generators are usually shorter and clearer.

Calling next() on a list directly. Lists are not iterators. Wrap with iter() first.

Storing iterator state in the iterable. Prevents multiple concurrent iterations. Use a separate iterator object or a generator.

Try It

Exercise: Practice with iterators:

Write a Countdown iterator class with __iter__ and __next__. Use it in a for loop.

Convert a list to an iterator with iter() and step through with next().

Try iterating an exhausted iterator. Observe that it produces nothing.

Use next(it, default) to safely read from an iterator that may be empty.

Chain map, filter, and itertools.islice to process the first 10 even squares.

Implement an iterable class that separates iteration state into a helper class - supporting multiple concurrent iterations.

Back to Python Basic to Advance

The Iterator Protocol

The Iterator Protocol

What an Iterator Is

Iterables vs Iterators

How for Loops Work

Iterators Are Single-Use

StopIteration and Sentinel

Built-In Iterators

The iter() Function

Implementing an Iterable

Iterator vs Generator

Composing Iterators

itertools Module

Performance

Interview Relevance

Common Pitfalls

Try It

How `for` Loops Work

`StopIteration` and Sentinel

The `iter()` Function

`itertools` Module