Dagster 與 dbt：兩者都需要嗎？
Dagster and dbt: do you need both?
Dagster 與 dbt 功能有重疊，但前者負責流程編排，後者是 SQL 轉換框架；是否同用，取決於轉換工作和團隊需要。大部分轉換都在同一個數據倉庫以 SQL 執行，或由分析師編寫模型時，dbt 的功能和可移植性較有用。若轉換工作以 Python 為主，或由一名工程師負責並願意在 Python 資產中處理 SQL，則可不使用 dbt。
"Why use dbt if I have Dagster?" is the number-two Google result for this topic, and it's a Reddit thread. That tells you the question is real. People see two tools that both draw a lineage graph, both run on a schedule, and both run tests, and they wonder if they are paying twice.
They aren't, quite. dbt is a SQL transformation framework. Dagster is an orchestrator. They overlap in the middle, and the overlap is where the confusion lives. My own production pipeline runs on Dagster with no dbt at all, so I'll also say when skipping dbt is the right call.
What each tool actually does
dbt takes a folder of SELECT statements with Jinja templating. It works out their order from ref() calls, compiles them for your warehouse, runs them as tables or views, and runs tests (not_null, unique, custom SQL) against the results. It does not schedule itself. Something has to call dbt build.
Dagster decides when things run, records what each run produced, retries failures, tracks partitions, and runs arbitrary Python. It has its own data-quality checks (asset checks) and its own lineage graph, built from the dependencies you declare.
The overlap: both have a lineage graph and both have tests. The difference: dbt's graph covers SQL models in one warehouse. Dagster's graph covers everything, including the Python that loads data before dbt runs and the models you train after it.
Why use dbt if you have Dagster
You still want dbt when:
-
Most of your transformations are SQL in one warehouse. dbt's templating,
ref(), incremental materializations and the docs site are a decade of work you would otherwise rebuild badly. - Analysts write the models. dbt is the tool analysts already know. Asking them to write Dagster assets in Python is a bigger change than it sounds.
- You want the project to be portable. A dbt project runs under Dagster, Airflow, Prefect, cron or dbt Cloud. Your SQL is not locked to the orchestrator.
You can skip dbt when the transformations are mostly Python, or when one engineer writes everything and is happy to keep the SQL inside Python assets. That is my situation, below.
This is the first part. The full post — including the rest of the working details — is on my site: Dagster and dbt: do you need both?
來源：DEV Community · dev.to