Back to the blog

Tony CantaveHow I Use Python Generators to Process Big Files Without Running Out of Memory

By Tony Cantave ·

When I started writing Python scripts for real work, I did what most beginners do: I read the whole file into memory, turned it into a list, and then looped over the list. That works fine for a 200 row CSV. It stops working when a log export or a data dump grows to a few gigabytes and the script either crawls or crashes. The fix that changed how I write file scripts is the generator.

What a generator actually is

A generator is a function that uses yield instead of return. Each time you ask it for the next value, it runs until the next yield, hands you one item, and pauses. Nothing else is computed until you ask again. That means memory use stays flat no matter how big the input is.

def read_lines(path):
    with open(path, encoding="utf8") as f:
        for line in f:
            yield line.rstrip("\n")

A file object in Python is already lazy when you loop over it, so this function is mostly a wrapper. The real power shows up when you chain several small generators together.

Building a pipeline out of small steps

Here is the pattern I use for cleaning a large export. Each step does one job and passes items along to the next.

import csv

def read_rows(path):
    with open(path, newline="", encoding="utf8") as f:
        yield from csv.DictReader(f)

def skip_blank(rows):
    for row in rows:
        if any(value.strip() for value in row.values() if value):
            yield row

def normalize(rows):
    for row in rows:
        row["email"] = (row.get("email") or "").strip().lower()
        yield row

def only_valid(rows):
    for row in rows:
        if "@" in row["email"]:
            yield row

Then I wire the steps together:

rows = read_rows("export.csv")
rows = skip_blank(rows)
rows = normalize(rows)
rows = only_valid(rows)

with open("clean.csv", "w", newline="", encoding="utf8") as out:
    writer = None
    for row in rows:
        if writer is None:
            writer = csv.DictWriter(out, fieldnames=row.keys())
            writer.writeheader()
        writer.writerow(row)

At no point does the whole file sit in memory. One row travels through all four steps, gets written, and then the next row starts. I can process a file far bigger than my laptop's RAM and the memory use barely moves.

Why I like this for debugging

Each step is a plain function that takes an iterable and returns an iterable, so I can test any of them alone with a short list:

sample = [{"email": "  Someone@Example.com "}, {"email": "not an email"}]
print(list(only_valid(normalize(sample))))

That prints one row with a clean lowercase email. No fixtures, no temp files. When something breaks in a big run, I can feed the bad rows into one step at a time and see exactly where it goes wrong.

Counting and summarizing without lists

Generator expressions are the one line version of the same idea. If I only need a total, I never build a list:

total = sum(1 for _ in only_valid(normalize(read_rows("export.csv"))))

The same works for adding up a numeric column:

amount = sum(float(r["amount"] or 0) for r in read_rows("orders.csv"))

Notice the parentheses instead of square brackets. Square brackets build the full list first. Parentheses stream it one value at a time.

Mistakes I made along the way

Using a generator twice. A generator can only be consumed once. If you loop over it and then try to loop again, the second loop gets nothing, and Python does not raise an error to warn you. I lost an afternoon to this. If you need two passes, call the generator function again, or, if the data is small, convert it to a list on purpose.

Closing the file too early. If you open a file in one function and return a generator that reads from it after the with block ends, you get "I/O operation on closed file". Keeping the with inside the generator, like read_rows above, keeps the file open exactly as long as it is needed.

Losing the bad row in a long chain. When a step fails deep in a pipeline, the traceback points at the right line, but the row that caused it is gone. Now I wrap risky conversions in a small step that reports the row number and the bad value, then skips it:

def to_float(rows, field):
    for n, row in enumerate(rows, start=1):
        try:
            row[field] = float(row[field])
        except (TypeError, ValueError):
            print(f"row {n}: bad {field} value {row[field]!r}")
            continue
        yield row

Why this pattern stuck with me

A lot of my background is in logistics and manufacturing systems, where exports from dispatch tools and ERP systems were large and messy. In my AI evaluation work I also handle batches of samples and scores. In both places the same rule applies: move one record at a time through small steps you can test on their own. Generators make that the default instead of something you have to fight for.

If you write Python scripts that touch files, try rewriting one of them as a chain of three or four generators. The code usually gets shorter and easier to test, and it keeps working when the file gets ten times bigger.