數據湖、數據倉庫與數據湖倉：簡化數據工程

I have seen a lot of engineers think about data one way: how do I store it, and how do I query it?

Data engineers ask something else: where does all this data land, and what can I build from it?

That gap gets wider as your system grows.

5 services means 5 data shapes, 5 formats, 5 update patterns. Reading and writing rows is the least interesting thing you can do with them.

Clean them, join them, and you get product features, stakeholder dashboards, and an honest view of what’s actually happening inside your system.

There are 3 places that data can live:

Data Lake
Where everything lands first. Raw, in any format: JSON events, logs, CSVs, Parquet, images.
No schema forced on write. You describe it later (schema-on-read).
On AWS that’s S3, plus the Glue Data Catalog to define tables, plus Athena to query them.
Weakness: no transactions, and updates or deletes are painful. Without governance it quietly turns into a data swamp nobody trusts.
Data Warehouse
Clean, modeled, schema-on-write. Built for heavy joins and aggregations, so it’s what your BI dashboards sit on (Redshift, Snowflake, BigQuery).
Weakness: you pay to ETL and store a second copy. It wants structured data only, and ML teams usually end up exporting data back out of it.

So most teams ran both.

A lake for raw data and ML. A warehouse for BI. Pipelines copying between them. Two copies, two bills, and one of them is always stale.

Data Lakehouse Keep the data in cheap object storage, in open formats, and put a table layer on top: Apache Iceberg, Delta Lake, or Hudi. That layer gives you ACID transactions, schema enforcement, time travel, and row-level updates. One copy of the data. BI and ML read the same tables.

It’s not a new storage system. It’s metadata on top of files you already have.

Weakness: you own more of the operations, like compaction, the small-files problem, and picking a catalog.

Rule of thumb:
→ Raw data, any shape, cheapest storage: lake
→ Fast, trusted SQL for the business: warehouse
→ Both, without duplicating data: lakehouse

I walk through all three in the full video:
https://youtu.be/dRDaQ7wZHTg

Over to you: is your team still running a lake and a warehouse side by side, or have you moved to Iceberg or Delta?