Python Generators - Yield, Lazy Evaluation, and Efficiency

Introduction to Python Generators

When working with large datasets or continuous data streams, we often seek smarter ways to process information without gobbling up memory. That’s where Python Generators come in — they allow you to iterate lazily, computing values only as needed.

What is a Generator?

A generator is a special type of iterator in Python. It lets you generate values on the fly instead of storing them in memory like lists. You define a generator using a regular function but with a twist — the yield keyword.

Why Use Generators?

• Efficient memory usage
• Faster for large or infinite sequences
• Cleaner code for custom iterators

Creating Your First Generator

Let’s take a simple example to see generators in action.

def count_up_to(n):
    count = 1
    while count <= n:
        yield count
        count += 1

gen = count_up_to(3)
print(gen)  # Just a generator object
for num in gen:
    print(num)

<generator object count_up_to at 0x...>
1
2
3

Explanation

Unlike regular functions, calling a generator function doesn't run it. Instead, it returns a generator object. When used in a for loop, it executes until the next yield is hit — pausing there and resuming from the same spot in the next iteration.

The Power of Lazy Evaluation

Generators don’t compute values until you ask for them. This approach is known as lazy evaluation.

def infinite_numbers():
    num = 0
    while True:
        yield num
        num += 1

gen = infinite_numbers()
for _ in range(5):
    print(next(gen))

0
1
2
3
4

This example shows how generators can theoretically run forever — only producing the next value when needed.

Generator vs List: A Memory Comparison

import sys

def gen_func():
    for i in range(1000):
        yield i

gen_obj = gen_func()
list_obj = [i for i in range(1000)]

print(sys.getsizeof(gen_obj))   # Lightweight
print(sys.getsizeof(list_obj))  # Much heavier

112
9024

Notice how the generator object is tiny in memory compared to the full list — even though they represent the same range of values.

Using Generators with next()

You can manually control a generator using the next() function.

def greeting():
    yield "Hi"
    yield "Hello"
    yield "Hey"

g = greeting()
print(next(g))
print(next(g))
print(next(g))

Hi
Hello
Hey

If you call next() again after it's exhausted, it will raise a StopIteration exception.

Generator Expressions

Much like list comprehensions, Python allows you to create generators in a compact form using generator expressions.

gen_exp = (x * x for x in range(5))
for val in gen_exp:
    print(val)

0
1
4
9
16

They’re shorter to write and just as memory-efficient.

Common Pitfalls and Checks

• Don't reuse a generator — once exhausted, it's done.
• Be cautious with infinite generators — always add a break or limit when testing.
• If you're expecting a return value, note that yield doesn’t return data like return. Use return only when you want to terminate the generator early.

How to Check if an Object is a Generator?

from types import GeneratorType

def sample():
    yield 1

print(isinstance(sample(), GeneratorType))  # True

True

Next Steps

• Try converting a regular function that builds a list into a generator.
• Use generators in file reading (line by line) for large files.
• Explore itertools — many of its tools work seamlessly with generators.