Dagster 與 Airflow：小型數據團隊應如何選擇

"Dagster vs Airflow" sounds like a feature contest. It mostly isn't. Both schedule Python, both retry failures, both have a UI, both are free to self-host. The real difference is what each one thinks it is scheduling. Airflow schedules tasks. Dagster schedules the things those tasks produce.

For a big team with an Airflow platform already running, that difference rarely justifies a migration. For a small data team starting from nothing, it decides how much you can hold in your head. I run a production Dagster pipeline alone: 112 assets, 93 asset checks, 57 schedules. This post is about why that model fit, what still hurt, and when Airflow is the better answer. I have not run Airflow in production, so the Airflow side comes from its documentation, not from my scars.

Tasks or assets: the one difference that matters

In Airflow you write a DAG: task A, then task B, then task C. Airflow knows whether each task succeeded. It does not, by default, know what the task wrote.

In Dagster you write an asset: a function that produces a table, a file or a model, and names the assets it depends on. The graph comes from those dependencies. Dagster records each materialization, so it knows which partitions of which table exist, when they were built, and whether the checks on them passed.

from dagster import asset, AssetIn

@asset
def daily_prices(): ...

@asset(ins={"prices": AssetIn("daily_prices")})
def sector_statistics(prices): ...

That is the whole pitch. If your job is "keep these 100 tables correct and current", the asset model matches the job. If your job is "call these 40 systems in this order", Airflow's task model matches it at least as well.

Why the asset model fit a 112-asset pipeline run by one person

My pipeline ingests market data, builds features, trains models and writes to ClickHouse. The counts today: 112 assets, 36 of them partitioned (34 daily, 2 static), 100 jobs, 57 schedules, 3 sensors, 93 asset checks. It has 913 commits since November 2025 and about 3,000 tests. The teardown of the whole platform gives the architecture; it has grown since that was written.

Three things made the asset model worth it for one person:

1. Partitions are data, not a loop. Dagster tracks the status of every daily partition. When a scrape fails for one date, I see one red square and rerun one square. I don't write backfill scripts that remember which dates they have done.
2. Checks sit next to the thing they check. 93 asset checks run against the assets they guard and show up on the asset's page. When something is wrong, the check and the data are in one place.
3. Granularity is a decision, not a default. I could choose to drop partitions where they cost more than they saved. One sector-statistics asset rebuilt 1,882 dates, 109,152 rows, in about 45 seconds as a single unpartitioned asset. The per-partition version would have meant over 2,000 partitions and 5,000 to 20,000 queries. The first phase of that change materialised 9 of 10 such "full" assets in about six minutes.

None of this is impossible in Airflow. Airflow 3 now has assets too (more below). But in Dagster the asset is the centre of the tool, and with one maintainer, the centre of the tool is what you get for free.

This is the first part. The full post — including the rest of the working details — is on my site: Dagster vs Airflow: which to choose for a small data team