Jikwang Moon 如何用 OpenResty 和一段 Valkey 腳本為 Corvus 建立公平的虛擬排隊系統

A waiting room (or "virtual queue") is what you see when a ticket sale opens: "You are number 1,342 in line." Behind it is a simple promise: let in only as many people as the site can handle, and let the rest in, in the order they arrived.

I built one into Corvus, a self-hosted gateway on OpenResty. This post walks through how it works, the parts that were harder than they looked, and the trade-offs I made.

What "working" means

Before writing code I wrote down what the queue must guarantee:

1. Capacity: never more than N visitors inside at once.
2. Order: first come, first served. A new visitor must not skip people who are already waiting.
3. No races: several nginx workers (or several gateways) must never hand out the same free slot twice.
4. Cheap for people inside: visitors who are already admitted should not cost a database round trip on every request.
5. No trust in the browser: closing a tab sends nothing reliable, so slots must free themselves.
6. Survive bad days: a Valkey outage or a bot flood must not take the site down or empty the queue.

The data model: five Valkey keys per room

All the state of one room lives in Valkey, the open-source fork of Redis that ships inside the Corvus container. It speaks the same protocol and runs the same Lua scripts, so everything below works unchanged on Redis too, and Corvus can point at an external Redis or Valkey when several gateways share one queue.

The key names share a hash tag, corvus:wr:{room}:active, corvus:wr:{room}:queue…, so in a Valkey (or Redis) Cluster they land in the same slot and one script can touch all of them.

Sorted sets do a lot of the work here:

• "Free expired slots" is one ZREMRANGEBYSCORE active -inf now.
• "Who is next" is ZPOPMIN queue.
• "What is my position" is ZRANK queue sid.

One Lua script, run atomically inside Valkey

Requirement 3 is the classic trap. "Read the number of active sessions, and if there is room, add one" is a race if two workers do it at the same time. Valkey runs a Lua script (EVALSHA) atomically, so the whole state machine is a single script and nothing else can run in between.

The script supports a handful of operations: enter, poll, heartbeat, release, stats. Every call starts the same way:

local t = redis.call('TIME')
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

-- Expired sessions free their slot (TTL/heartbeat based release; browser close is never trusted).
redis.call('ZREMRANGEBYSCORE', K_ACTIVE, '-inf', now)

The time comes from Valkey's TIME, not from the gateway. With several gateways there is exactly one clock, so "expired" means the same thing everywhere.

No queue jumping

A new visitor gets in directly only if nobody is waiting and there is a free slot:

-- New arrival: admit directly only when nobody is waiting (no queue jumping).
if redis.call('ZCARD', K_QUEUE) == 0 and budget() > 0 then
  admit(sid, active_ttl)
  return finish(ACTIVE, sid)
end

local seq = redis.call('INCR', K_SEQ)
redis.call('ZADD', K_QUEUE, seq, sid)
redis.call('ZADD', K_SEEN, now, sid)
return finish(WAITING, sid)

Without the ZCARD == 0 check, a visitor who happens to arrive the moment a slot frees up would take it, ahead of someone who has been waiting ten minutes.

Promotion: who gets the free slot

Before answering, the script moves people from the head of the line into free slots:

local function promote()
  local scanned = 0
  while scanned < promote_max and budget() > 0 do
    local head = redis.call('ZPOPMIN', K_QUEUE)
    if #head == 0 then break end
    scanned = scanned + 1
    local m = head[1]
    local seen = tonumber(redis.call('ZSCORE', K_SEEN, m))
    redis.call('ZREM', K_SEEN, m)
    if seen and (now - seen) <= queue_ttl then
      admit(m, grant_ttl)
    end
  end
end

Two details matter:

• Abandoned waiters are skipped. Someone who closed the tab an hour ago is still in the sorted set. If they stopped checking in for longer than queue_ttl, they are dropped instead of being handed a slot nobody will use.
• A promoted visitor gets a short slot first. The slot lasts only grant_ttl (30 s by default). If they don't come back to claim it, it frees up quickly. When they do come back, it is extended to the full active_ttl.

budget() is the smaller of "free slots" and "admissions left this second", so the optional admission rate ("let in at most 50 people per second") is enforced in the same atomic step.

Slots free themselves

An admitted session's score in active is its expiry time. Every request from that visitor pushes it forward (ZADD ... XX now + active_ttl). When they stop making requests, the score falls behind now and the next script call removes it. There is also a leave endpoint for the "Leave queue" button, but nothing depends on it. You can't count on a closing browser to tell you anything.

Keeping Valkey off the hot path

If every request of every admitted visitor ran the script, Valkey would become the bottleneck for the whole site. So the gateway hands out a signed admission token in a cookie:

v2.<key id>.a.<room>.<session id>.<issued at>.<expires>.<HMAC-SHA256>

On each request, OpenResty checks it like this:

local tok = admission_from_cookie(room)
if tok and ngx.time() - tok.iat < room.refresh then
  return -- fast path: no Redis
end

• Token younger than refresh (15 s): the HMAC check is all that happens. No network call.
• Older but valid: one heartbeat call extends the slot in Valkey, and a fresh token is issued.
• No or invalid token: an enter call admits or queues the visitor.

So an admitted visitor costs Valkey at most one tiny call every 15 seconds, however many requests their browser makes. The trade-off is a revocation lag: an "emergency release" or a capacity change can take up to refresh seconds to reach everyone. For a waiting room, that's fine.

The key id in the token allows rotating the signing secret: the gateway accepts tokens signed with the current and the previous secret and signs new ones with the current one.

In the project's benchmark (Corvus pinned to 2 vCPU, tiny responses), a host behind an access queue served about 58k req/s to admitted visitors, compared to about 75k req/s for a plain host. On that path Valkey is barely involved; the cost is the extra Lua work per request.

Surviving floods

A queue is a juicy target. If every request without a cookie created a queue entry, a simple curl loop would fill the line with millions of fake visitors.

Step 1: a cookie check that creates no state. A first visit doesn't touch Valkey at all. The visitor gets a signed queue cookie and a redirect to the same URL:

if not sid then
  -- Cookie check: a first visit creates no server-side state. The client gets a
  -- signed queue cookie and must come back with it before Redis is touched, so
  -- cookie-less floods can neither fill the queue nor take admission slots.
  issue_queue(room, new_sid())
  return ngx.redirect(ngx.var.request_uri, 302)
end

A bot that ignores cookies loops on this redirect forever and never reaches Valkey.

Step 2: limits for clients that do keep cookies.

• At most 60 new queue entries per IP per minute (configurable). This is kept in an nginx shared dict, so it costs no Valkey work.
• At most 3 status checks per second per session.
• A maximum queue length (100,000 by default). Beyond it, visitors get a "come back later" page instead of growing Valkey forever.
• If Valkey reports it is out of memory, visitors get the same "busy" answer. A full Valkey must not be treated as "Valkey is down, let everyone in".

When Valkey is actually down

Each room has a policy:

• fail-open lets everyone in and favours availability.
• fail-closed shows a "try again" page and protects capacity.

Either way, visitors who already hold a valid admission token always get through, because their check doesn't need Valkey.

To avoid every request waiting for a connection timeout during an outage, there is a small circuit breaker. After one transport failure, all nginx workers skip Valkey for 3 seconds. The flag lives in a shared dict, so one worker's failure protects all of them.

The waiting page and the API

Browsers get a waiting page that polls /__corvus/wr/<room>/status. The poll interval grows with your position: 3 s near the front, up to 30 s far back. So 50,000 waiting people don't hammer the gateway every second. On errors the page backs off exponentially, up to 60 s. Several tabs of the same browser share one session, so opening ten tabs doesn't give you ten places in line.

Requests that can't show a page (fetch, XHR, mobile apps) get 429 Too Many Requests with a Retry-After header and a small JSON body (rank, waiting, retry_after). I deliberately avoided 503. The site isn't broken, and a waiting room shouldn't light up your 5xx alerts.

Trade-offs I accepted

• Capacity counts sessions, not requests. "500 visitors" means 500 browsers inside, whatever they do there. That is what people expect from a waiting room, but it isn't a rate limiter. (Corvus has a separate one.)
• Cross-site API calls stay queued. The cookie is SameSite=Lax, so a web app on another site calling a queued API won't send it. Only put the queue on what people open in a browser.
• Shared IPs and the per-IP limit. A big office behind one NAT can hit the 60-entries-per-minute limit. It's configurable for that reason.
• Optional network binding. Admission tokens can be bound to the visitor's /24 (IPv6 /48), so a token can't be passed around. Visitors who switch from Wi-Fi to mobile data then have to queue again, which is why it's off by default.

Wrapping up

The core idea fits in one sentence: put the whole state machine in one atomic Valkey script, and use signed tokens so admitted visitors almost never need it. Most of the remaining work went into the unhappy paths: abandoned tabs, floods, a full Valkey, a dead Valkey.

The full code is on GitHub:

• the Valkey script: gateway/redis/waitroom.lua
• the OpenResty side: gateway/lua/corvus/waitroom.lua

In Corvus you turn it on from the console in a couple of clicks: create an access queue, then attach it to a proxy host.

Questions, or a case I didn't think of? I'd love to hear it in the comments.

Corvus is built with heavy help from an AI coding assistant, and this post was drafted with one too. The design decisions, verification and testing are my own.