Python 腳本讀取大型檔案崩潰？用 10 行 mmap 避免整檔載入記憶體

You open a 6 GB log file, call .read(), and the process dies with MemoryError (or your laptop starts swapping and the fan sounds like a jet).

The fix is not a bigger machine. It is not loading the file at all. Python ships a module for this: mmap.

The problem

with open("big.log", "rb") as f:
    data = f.read()          # tries to allocate the whole file in RAM
    print(data.count(b"ERROR"))

This works on a 5 MB file and falls over on a 5 GB one, because read() copies the entire file into a Python bytes object.

The fix: memory-map the file

import mmap

with open("big.log", "rb") as f:
    with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
        count = 0
        pos = mm.find(b"ERROR")
        while pos != -1:
            count += 1
            pos = mm.find(b"ERROR", pos + 1)
        print(count)

What changed: mmap asks the operating system to map the file into your address space. Nothing is copied up front. The OS loads pages from disk only when you touch them, and it can drop them again when memory is tight. Your script behaves as if the whole file were a bytes object, but your resident memory stays small.

Three things you can do with the mapped object

mm[0:100]            # slice bytes without reading the rest
mm.readline()        # read the next line, like a normal file
mm.seek(5_000_000)   # jump anywhere instantly

mm.find() and mm.rfind() search without building intermediate copies, and slicing only materializes the bytes you ask for.

Gotchas I would have liked to know earlier

• Empty files raise ValueError. You cannot map a zero-length file, so check the size first.
• Use ACCESS_READ if you only read. It prevents accidental writes to the underlying file.
• It is not magic for random access on slow disks. The OS still has to fetch pages, so a spinning disk with random jumps will be slow.
• Text vs bytes: mmap gives you bytes. Decode only the slice you need: mm[a:b].decode("utf-8", errors="replace").
• Your results are still bounded by address space. On a 64-bit Python that is not a practical limit.

Why this matters right now

The same idea is behind a lot of the current "run a model bigger than your RAM" work. Local LLM runtimes such as llama.cpp memory-map model weights so the OS pages them in on demand instead of loading many gigabytes at startup. If you have been following the experiments with streaming parts of a model from SSD, they are building on the same OS feature you just used for a log file.

Full reference for the module: Python docs: mmap.

Try it

Take the biggest file you have on disk, run the 10-line version above, and watch your process memory while it runs.

What is the largest file you have ever tried to process in Python, and what finally made it work for you: chunked reads, mmap, or a different tool entirely?