DuckLess：在自己的 Google Cloud 專案中執行無伺服器 DuckDB

This article was originally published on Medium.

I like DuckDB a lot. On a laptop it chews through gigabytes of Parquet faster than most of the platforms I use at work, and it asks for nothing: no cluster, no warehouse, no service to keep alive.

Then the data moves to a GCS bucket and grows to 100 GB, and the laptop stops being an option. On Google Cloud, the usual answers are BigQuery, or a VM that someone has to size, start, watch and remember to stop. For heavier pipelines, Spark or Dataflow.

In my client work, I also see a lot of batch jobs on Cloud Run that process files in chunks, in parallel, to stay under the container's memory limit. It works and response times stay reasonable, but the code drifts toward something custom, and it could be much faster. And reaching for Spark or Dataflow means distributed computing for data that often fits on one large machine, where DuckDB is very fast on its own.

What I actually wanted was simple to say: run this SQL file next to the data, on a machine big enough for it, and give me back the result and the bill for the minutes it took.

So I built that. It's called DuckLess, it's open source (Apache 2), and this article is about what it does, what I measured, and what I learned along the way.

What it looks like

You write a job. Most of the time it's a plain SQL file:

CREATE VIEW orders AS
SELECT * FROM read_parquet('gs://my-data/raw/orders/*.parquet');

COPY (
    SELECT customer_id, date_trunc('month', ordered_at) AS month, sum(amount) AS revenue
    FROM orders
    GROUP BY ALL
) TO 'gs://my-data/marts/revenue_by_month'
  (FORMAT parquet, PER_THREAD_OUTPUT, FILE_SIZE_BYTES '256MB', OVERWRITE);

SELECT count(*) FROM orders;

And you run it:

duckless run revenue.sql -m n2-highmem-32 --spot

DuckLess creates a VM of that size in your project, starts DuckDB on it, runs the file, writes the metrics and the last result next to the job, and deletes the VM. You follow along in the terminal, and duckless result <job-id> gives you the rows of the last SELECT afterwards.

Python works the same way, for the jobs where SQL alone gets awkward:

from duckless_runtime import connect, copy_to_parquet

con = connect()  # GCS auth, memory, threads and spill directory already set for this VM
copy_to_parquet(con, "SELECT ...", "gs://my-data/out/")

And if you already have a tool, dbt-duckdb for instance, duckless exec runs any image with any command.

Parquet is the fast path, but jobs read CSV, JSON, Excel, Avro and Iceberg from GCS the same way, and BigQuery tables directly. More on that one below.

Your project, not mine

This was the first design decision and the one I'm least willing to bend: everything runs in your GCP project. There is no DuckLess service between you and your data. No data leaves your project, nothing reads your buckets except a service account you own, and the bill comes from Google, not from me.

What it needs is small: a work bucket, a runner service account with the least privileges it needs, and an Artifact Registry remote repository that proxies the runner image published on GitHub (ghcr.io). The job VMs have no external IP, so they can't pull from GitHub directly: they reach GCS and that repository through Private Google Access, and the image is cached in the region.

If your team manages infrastructure as code, and most teams I work with do, that's a Terraform module you call from your own code and CI/CD:

module "duckless" {
  source           = "git::https://github.com/tosun-si/duckless.git//duckless/terraform?ref=v0.4.2"
  project_id       = "my-project"
  region           = "europe-west1"
  runner_image_tag = "0.4.2"
}

If you just want to try it, duckless init --project my-project deploys the same module through Infrastructure Manager, after checking your organization policies (a project that forbids service account creation should fail in five seconds, not ten minutes into an apply). The account it uses holds its admin roles only while init or destroy runs, and can only grant the runner's roles: nothing powerful stays behind.

GCS access goes through the VM's own credentials, with DuckDB's community gcs extension. No HMAC keys to create, rotate or leak.

The numbers

I didn't want to build this on vibes, so the first thing I did was a spike against BigQuery on the same data: TPC-H at scale factor 100, 35.6 GB of Parquet on GCS, 600 million rows in lineitem. The workload was a full scan, four TPC-H queries (1, 9, 18 and 21) and an aggregate written back to GCS.

BigQuery wins on latency: no startup, and native tables are faster. If you need an answer in ten seconds, use BigQuery. But for transformations that run every day and take minutes, paying for the VM's minutes instead of the bytes scanned changes the bill by one or two orders of magnitude. Part of the gap is also how BigQuery counts: it billed about 105 GB of logical bytes for a scan of 35 GB of Parquet.

Three other results surprised me more than the headline:

• Memory pressure doesn't break jobs, it slows them down. The same queries with DuckDB limited to 7 GB instead of 205 GB took 133 s instead of 47 s, spilling to local SSD. DuckLess attaches local NVMe SSDs to every Batch VM for exactly this reason.
• The default way of writing Parquet to GCS is slow. A single writer tops out around 110 MB/s, whatever the machine. With PER_THREAD_OUTPUT, the same write ran at 900 MB/s on 32 vCPUs. The bottleneck was the sequential writer, not GCS.
• Reading one big file from GCS has the same problem. A 12 GB CSV loaded at about 90 MB/s, because DuckDB's gcs extension fetches only five ranges of a file at once. Copying it to local SSD with 64 parallel reads first took the load from over two minutes to 15 seconds. That one deserves its own article.

Small jobs: Cloud Run Jobs

Cloud Batch is great for big machines, Spot and local SSD, but a VM takes about a minute to start. For a job on four cores that runs for twenty seconds, that's most of the time.

So since v0.2.0, DuckLess has a second runner. With the default --on auto, a job goes to Cloud Run Jobs when the machine fits there (up to 8 vCPUs and 32 GiB) and you didn't ask for Spot or local SSD; everything else stays on Batch. On a small TPC-H job, that took the total time from about 1 min 50 to under a minute.

Cloud Run had its own surprises: DuckDB saw the host's CPUs instead of the four I was paying for, and the first GCS call over gRPC hung for 37 seconds. DuckLess now handles both. Cloud Run has no local disk either, so spill stays in memory, which is why the routing only picks it when nothing changes for the job.

Tables, not just files: DuckLake

Parquet files in a bucket are great until you need to update a row, load data incrementally or ask what the table looked like yesterday. That's what DuckLake is for: the data stays Parquet in your bucket, and a small Postgres database keeps track of which files make which version of which table.

v0.3.0 adds it with one flag:

duckless init --project my-project --ducklake

That creates a Cloud SQL Postgres instance, private IP only, with backups and point-in-time recovery. Every job then starts with the catalog attached as lake:

CREATE TABLE lake.orders AS SELECT * FROM read_parquet('gs://my-data/orders/*.parquet');
INSERT INTO lake.orders SELECT * FROM read_parquet('gs://my-data/orders_today/*.parquet');

SELECT count(*) FROM lake.orders AT (VERSION => 3);  -- time travel

There is no password anywhere: the runner reaches the catalog through the Cloud SQL Auth Proxy with IAM authentication. Getting this right took two spikes, a token that expired after an hour, and a network peering that destroy left in a strange state. That story deserves its own article, and it will get one.

BigQuery, in and out

Plenty of teams don't have Parquet in a bucket: their data is in BigQuery, and what they want is to run Python on it without paying a query every time. Since v0.4.0, a job reads a BigQuery table directly:

SELECT customer_id, sum(amount) AS revenue
FROM bigquery_scan('my-project.sales.orders', filter = 'order_date >= DATE ''2026-01-01''')
GROUP BY ALL;

from duckless_runtime import connect, read_bigquery

con = connect()
orders = read_bigquery(con, "my-project.sales.orders", ["order_id", "amount"], "order_date >= DATE '2026-01-01'")
for batch in orders.fetch_record_batch():   # Arrow batches, never the whole table in memory
    ...

Under the hood it's DuckDB's bigquery extension and the BigQuery Storage Read API: the table streams as Arrow, one stream per thread, and only the columns the job uses and the rows the filter keeps leave BigQuery. That read is billed at Storage Read API prices, about six times less than an on-demand query over the same data.

I tried a hand-written Python client first. The extension was four to six times faster, so the Python helper is now a thin wrapper around it. On 60 million rows, copying the whole table to GCS took 16 seconds on an n2-highmem-16.

Writing back is where BigQuery bills get scary, so v0.4.2 makes it one call that picks the cheap path:

from duckless_runtime import connect, to_bigquery

con = connect()
# ... transformations in DuckDB ...
to_bigquery(con, "SELECT * FROM accepted", "my-project.finance.journal_lines", mode="replace_partitions")

The result goes to GCS as Parquet and comes in with a load job, which is free and atomic. Then it depends on the mode. On a 100-million-row table with a batch of three days, a plain MERGE billed 4.81 GB, the whole table. Deleting the three partitions and inserting the new rows billed 0.05 GB: deleting whole partitions is free, so you pay for the data you write, not for the size of the table. When a batch only holds some rows of a partition, a MERGE restricted to the partitions it touches billed 0.09 GB. There's a trick to that last one, and the next article is about it.

Checking raw data before trusting it

The other job I kept seeing is validation: a raw CSV arrives from a source system, and someone has to check it before anything uses it. Valid emails, known country codes, unique keys, dates that aren't in the future, lines that parse at all.

The usual way is Python with a Pydantic model per row, reading in chunks to keep memory under control. It works, and it's slow: Python validating rows one by one does thousands per second. With DuckDB, every rule becomes one SQL predicate over the whole file, and lines that don't fit the expected schema are caught by the CSV reader itself instead of crashing the read. Pydantic is still there, but where it's cheap: it validates the rules file, so a typo in a rule stops the job before a single byte of data is read.

On 100 million lines (7.8 GB of CSV), checking six rules and collecting every error took 63 seconds on an n2-highmem-32 Spot VM. The job writes each error to GCS and fails when the error rate crosses a threshold, so a pipeline can stop right there.

If your team speaks Soda, the same checks work as a Soda v4 data contract: Soda's DuckDB connector runs them through the job's DuckDB connection. Once the file was loaded, Soda's nine checks on those 100 million lines took under two seconds. Both versions are in the repository's examples.

Skills for the agent next to you

A lot of the knowledge above isn't something a CLI can decide for you. Which machine for this data? Why did this job sit in SCHEDULED for ten minutes? Why does gs:// suddenly fail on credentials? (Someone installed httpfs, which takes over gs://. Ask me how I know.)

So DuckLess ships five Agent Skills (setup, writing jobs, sizing, troubleshooting and DuckLake) for Claude Code and the other agents that read .agents/skills:

duckless skills install          # into this project
duckless skills install --user   # into your home directory

They are versioned with the CLI, so the advice matches the version you run, and committing them in a repository gives the whole team the same ones. They're also available as a Claude Code plugin with /plugin marketplace add tosun-si/duckless.

The split I try to keep: if a decision can be made from facts (the runner, the smallest valid SSD count, whether the quota is there), the CLI makes it. If it takes judgment, it goes in a skill.

What it is not

DuckLess runs one job on one machine. It's not a distributed engine, it won't replace a warehouse that dozens of analysts query all day, and it doesn't do interactive queries: there is a minute of startup on Batch and about thirty seconds on Cloud Run before anything happens.

What it is good at is the large middle ground of data work: daily transformations, backfills, exports, validating raw files, Python logic over a BigQuery table, a job that has outgrown Cloud Run, a dbt project on DuckDB. Jobs where a single large machine is enough, and where you'd rather pay for minutes than for bytes or for a cluster that idles.

Try it

uv tool install duckless          # or: pip install duckless
duckless preflight -m n2-highmem-16 --project my-project   # checks quota, creates nothing
duckless init --project my-project      # or the Terraform module, from your own code
duckless run job.sql -m n2-highmem-16 --spot

With the Terraform module, terraform destroy removes it; with init, duckless destroy does. The examples are a good next step: daily marts, late corrections with DuckLake, raw data validation (plain and with Soda), BigQuery to Python. The first one generates the data the others use, so they run as they are.

DuckLess is at v0.4.2. It works, it's tested on real projects, and it's young. I'm especially interested in real use cases: if you have a job that's too big for a laptop and too small or too expensive for what you run today, I'd love to hear about it, in an issue or in the comments.

• Code: github.com/tosun-si/duckless
• Docs: tosun-si.github.io/duckless

If DuckLess looks useful to you, a ⭐ on GitHub helps other people find it.

If you enjoyed this article, follow me for more content on Google Cloud, data engineering, AI agents, Software and DevOps:

• LinkedIn
• YouTube
• Medium
• dev.to
• X