Databricks LTAP Direct Writes 如何讓零售與電商的 OLTP、OLAP 協同運作而不拖慢應用程式

Every retailer runs two businesses on two kinds of databases.

The first is the store itself: carts, checkouts, order status, loyalty balances, store-level inventory, and the price a shopper sees on a shelf tag or product page. That’s OLTP, transactional work that has to answer in milliseconds because a customer is waiting.

The second is the intelligence behind the store: demand forecasts by SKU, store, and day; personalization scores for every shopper; optimized prices; assortment plans; and fraud models. That’s OLAP, analytics over billions of rows and years of history.

Retail has gotten very good at the second business. The trouble has always been getting its output into the first one.

Two engines, one platform

Databricks handles these two kinds of work with two engines that sit side by side on the same platform.

The lakehouse is where analytics and AI happen. It combines the low-cost, open storage of a data lake with the reliability and performance of a data warehouse. Data is stored in open Delta Lake tables on cloud object storage, and it’s processed by engines such as Apache Spark and Databricks SQL, which can scan billions of rows in parallel. This is where a retailer’s sales history, loyalty data, supply chain feeds, and clickstream come together, and where forecasting, pricing, and personalization models get built and run. It’s the OLAP side.

Lakebase is where applications live. It’s a fully managed Postgres database built into the Databricks Platform, designed for the fast, high-concurrency reads and writes that apps need: looking up one order, one shopper’s recommendations, or one item’s price, in milliseconds, thousands of times a second. Because it’s standard Postgres, app teams use the tools, drivers, and SQL they already know. Unlike a separate operational database bolted on from outside, Lakebase is governed by the same Unity Catalog as the lakehouse, so permissions, lineage, and the definition of “customer” or “item” stay consistent on both sides. It’s the OLTP side.

Having both on one platform is the starting point. The hard part, and the subject of this article, is moving large volumes of data from the lakehouse into Lakebase without slowing down the applications Lakebase is serving.

The nightly load problem

Here’s a scenario most retail data teams will recognize.

Every night the lakehouse produces something big. It might be a fresh price file for every item in every store, a new set of recommendation scores for millions of shoppers, or a demand forecast covering tens of thousands of SKUs across a couple of thousand locations. That last one alone runs to hundreds of millions of rows and is rebuilt from scratch every cycle.

All of it has to land in the operational Postgres database the apps actually read from, and that’s where it hurts.

In standard Postgres, every write goes through a single primary instance, the same one serving your storefront, store systems, and mobile app. A bulk load competes directly with live traffic for CPU and memory. Teams end up with familiar workarounds: maintenance windows, loads squeezed into the overnight gap before stores open, overprovisioned databases sized for the load instead of the shoppers, and a separate reverse-ETL tool just to manage the pain.

One such customer, loading about 1 billion rows a day through synced tables, needed more than 8 hours per load cycle and had to overprovision infrastructure just to keep their application from degrading. In retail, 8 hours is the whole window between the last online order of the night and the first store opening in the morning.

The fix: Build the data somewhere else, then hand it over

Databricks announced LTAP Direct Writes in Beta on October 8, 2026. It’s a new way to load data into Lakebase Postgres. The headline numbers are 1 TB in under 5 minutes and up to 147x faster heap loading, without consuming the resources your live application depends on.

With Direct Writes, the heavy construction work moves out of the Postgres primary and into Spark, running in parallel:

1. Parallel page building in sandboxes. Each Spark executor runs its own isolated Postgres instance and builds native Postgres data pages for its slice of the table. The driver assigns non-overlapping ID ranges so the slices never collide.
2. Frozen binary COPY. Inside each sandbox, data is loaded with binary COPY and FREEZE. Rows are treated as already committed, so there’s no transaction history to maintain.
3. Checksums on every page. Workers checksum the pages they build, and Lakebase’s pageserver validates them on ingest. A corrupted page can never become the source of truth.
4. Index building without re-downloading the table. Workers export key columns and row locations to object storage. Postgres’s standard B-tree builder then builds indexes from those, without pulling the full table back down.
5. One atomic handoff. The Spark driver writes a manifest and makes a single SQL call. The primary records only a small log entry describing the import, not the terabytes of data. The pageserver claims the files, warms them on the local SSD, and atomically swaps them into the visible table at commit.

The result is that the primary barely notices. Your storefront’s Postgres writes a tiny log entry instead of absorbing a terabyte of inserts.

This is what LTAP (Lake Transactional/Analytical Processing) means in practice. OLAP computes the bulk work it’s built for; OLTP keeps serving customers; and both share a single governed copy of the data under Unity Catalog.

What this changes for retail and e-commerce

Faster loads are nice. For retailers, the bigger change is that bulk refreshes no longer constitute an operational event.

Price and promotion changes stop waiting for a window. A full refresh of store-item pricing can land during the business day without slowing down point-of-sale lookups or the product page. Merchandising teams get to act when they decide, not when the loading schedule allows.

Personalization goes from nightly to whenever. If scoring millions of shoppers produces a big table, reloading it in minutes means you can rescore after a campaign launches, after a weather event, or ahead of a holiday weekend, and the app picks up the new scores right away.

Forecasts reach the systems that act on them. SKU-store-day demand forecasts can be served from Lakebase to replenishment apps, store ordering tools and delivery-promise logic, without a separate serving database and a pipeline to feed it.

Reference data rebuilds stop being scary. Item master, store hierarchies, assortment and planogram tables are rebuilt wholesale all the time. Snapshot syncs with Direct Writes make a full rebuild routine instead of a weekend project.

You can retire some reverse-ETL infrastructure. The write-up puts it directly: large bulk-refreshed datasets like nightly scoring tables, fresh feature sets and rebuilt reference data can stay in Lakebase. They no longer need a separate reverse-ETL tool just to avoid the load penalty.

Peak season gets safer. During Black Friday or holiday weeks, the worst thing a data team can do is compete with shoppers for database capacity. Direct Writes means that the bulk load and the checkout surge aren’t competing for the same CPU.

How it fits with synced tables

Direct Writes is an option on synced tables, which keep a Unity Catalog Delta table available in Lakebase as a read-only Postgres table. Synced tables have three modes, and it matters which one you pick:

• Snapshot does a full refresh on every cycle. Direct Writes speeds up every sync in this mode, so it’s the natural fit for price files, forecasts, scores and reference data that get rebuilt wholesale.
• Triggered and Continuous use Change Data Feed, so updates after the first load are already incremental. Direct Writes speeds up the initial load, which is often the biggest load you’ll ever do.

That suggests a simple retail split. Use Snapshot with Direct Writes for big tables rebuilt nightly or on demand. Use Continuous for data that changes transaction by transaction, like recent sales for returns lookups or live inventory positions.

Getting started

https://docs.databricks.com/aws/en/oltp/projects/sync-tables#accelerated-sync-with-ltap-direct-writes

The bottom line

Retail has always had a gap between what the business knows and what the storefront can act on. Moving analytics results into operational systems meant risking the customer experience, so it happened less often, more slowly, and with more infrastructure than anyone wanted.

Direct Writes closes much of that gap for the biggest, most disruptive loads. The lakehouse builds the data in parallel; Postgres accepts it in a single atomic handoff; and shoppers at the shelf, in the app, or at checkout never notice.

That’s the shift: OLTP and OLAP working together, on one platform, without the analytics side overwhelming the application side.

Databricks #Lakebase #Retail #Ecommerce #OLTP #OLAP #Postgres #DataEngineering

Originally published on Medium.