技術堆疊的選擇,以及哪些設計經受住了實際運作考驗
Choosing the Stack, and What Survived Production
團隊以 Airbyte、Airflow、Athena 上的 dbt、S3 上的 Iceberg、AWS Glue Data Catalog、Lake Formation、OpenMetadata 和 Metabase 作為起始技術堆疊,並為每項決策訂下驗證準則。
TL;DR
- The plan: Airbyte, Airflow, dbt on Athena, Iceberg on S3, the Glue catalog, Lake Formation, a metadata catalog and Metabase.
- Five things changed: the ingestion tool, where tables live, the write mode, the number of layers and where metadata lives.
- Each decision said what would prove it right. That made it cheap to notice when it wasn't.
The first decision record
Last week I listed the symptoms of our old platform. We answered them with a decision record, one short document per choice, each with a status of recommended direction and a list of validation criteria. The first one picked the whole stack:
| Responsibility | First choice |
|---|---|
| Ingestion | Airbyte |
| Orchestration | Airflow |
| Transformation | dbt on Athena |
| Storage | S3 + Apache Iceberg |
| Technical catalog | AWS Glue Data Catalog |
| Access control | Lake Formation |
| Metadata catalog | OpenMetadata |
| BI and ad hoc SQL | Metabase |
The reasoning was short:
- A serverless engine. Athena bills for data processed, so nothing is billed while nobody queries. That answered the idle-cluster symptom directly.
- An open table format. Iceberg tables on S3 outlive the engine that writes them.
- dbt and Airflow in one repository. Astronomer Cosmos turns a dbt project into Airflow DAGs, so we wouldn't hand-write a task per model, and models and DAGs would ship together.
- One BI tool that reads Athena and can receive descriptions from dbt.
The validation criteria were just as concrete: critical dashboards running in the new BI tool, costs monitored by layer, and the old warehouse shut down only after a stability period.
What changed
1. The ingestion tool. We started with Airbyte and moved to OLake, which replicates databases into Iceberg with change data capture. The move had a side effect I didn't see coming, and it gets its own post in two weeks.
2. Where the tables live. The plan said to treat Iceberg maintenance as part of the product from day zero: compaction, snapshot expiration and orphan-file cleanup, orchestrated by Airflow. Then we moved the tables to Amazon S3 Tables, whose table buckets store Iceberg tables and run that maintenance themselves. The maintenance DAG was never built. Our docs now say the opposite of the plan: never run OPTIMIZE or VACUUM yourself. The move came with constraints of its own, and those are a whole arc of this series.
3. The write mode. The plan said merge-on-read in the raw layer and copy-on-write in the curated ones, so reads stay fast. Then we read the Athena docs closely: UPDATE, MERGE INTO and DELETE always use merge-on-read with positional deletes, and copy-on-write isn't supported. A merge that rewrites rows that didn't change leaves delete files behind for nothing. So we learned to design merges that touch only the rows that changed.
4. The number of layers. The plan had five: raw, staging, trusted, datamarts and reports. Within a few months, staging and trusted were doing the same job twice: cast, rename, deduplicate. We collapsed staging into trusted, with one model per source entity and no joins across sources. Four layers were enough.
5. Where metadata lives. The plan put a metadata catalog server in the middle, collecting descriptions and lineage from dbt, the catalog, Airflow and the BI tool. We postponed it. Descriptions are written once in dbt and pushed into the BI tool every day, because that is where people actually read them. Later, the BI tool's AI assistant read them there too. Lineage to dashboards lives in dbt itself.
What didn't change
Athena, dbt, Airflow, Iceberg and Metabase are all still there, doing the jobs the first record gave them. So the core held up. Most of what we changed sat around the core: how data gets in, how tables are kept healthy, and where people read descriptions.
The lesson
The first record wasn't wrong. It was a starting point that said what would prove it. Because every decision had validation criteria, "this isn't working" was a measurement, not an opinion, and changing course didn't need a debate about who had been right.
Two habits made that work:
- Status is part of the decision. "Recommended direction" told everyone the decision was open to evidence. Some decisions moved to "done", others got replaced, and the old text stayed readable.
- Read the engine's docs before designing for it. The write-mode plan was sound Iceberg advice in general. It just wasn't something our engine supported.
Takeaways
- Write each architecture decision with a status and the evidence that would confirm it.
- Before you design maintenance jobs, check whether the storage layer already does that work for you.
- Check which write modes your engine actually supports before choosing one per layer.
- Collapse layers that do the same job twice.
References
- Amazon Athena: Update Iceberg table data, merge-on-read only.
- Amazon S3: Working with S3 Tables and table buckets and Maintenance for tables.
- OLake documentation: database replication into Iceberg with CDC.
- Astronomer Cosmos: dbt projects as Airflow DAGs.
Next week: the whole architecture on one page, from the source databases to the dashboards.
來源:DEV Community · dev.to