Rekognition 的 ProvisionedThroughputExceededException：TPS 配額與穩健退避方法

Editor note for Ashish (delete before publishing): this draft was written with AI. Rewrite it in your own voice and check every claim. If any AI-written text remains, add a line saying the post was written with AI assistance (DEV.to and Medium both require this).

TL;DR Rekognition's default quota for IndexFaces, SearchFacesByImage and SearchFaces is 5 transactions per second per account in most regions, and 50 in us-east-1, us-west-2 and eu-west-1. A naive parallel backfill blows through that in under a second and gets ProvisionedThroughputExceededException (or ThrottlingException). Retries alone don't fix it. Rate-limit on the client to just under the quota, share one limiter across every worker in the account and region, set the SDK to adaptive retry mode, and request a quota increase before a big job, not during it.

The usual story: a face-search feature works fine in development, then someone kicks off a backfill of 200,000 existing photos with a thread pool of 32 workers. Within seconds the logs fill with this:

botocore.errorfactory.ProvisionedThroughputExceededException: An error occurred
(ProvisionedThroughputExceededException) when calling the IndexFaces operation (reached max retries: 2):
Provisioned rate exceeded

"reached max retries: 2" is the SDK telling you it already retried twice with backoff and gave up. The problem isn't the retry policy. The job is asking for more throughput than the account has.

The default quotas

From the Rekognition service quotas page. These are per account, per region, per operation, so IndexFaces and SearchFacesByImage each have their own budget.

Default TPS quotas as listed by AWS on 2026-10-31. Mumbai, Frankfurt, Tokyo, Sydney, Singapore, London, Ohio and others are in the right-hand column.

Do the arithmetic for that backfill. 200,000 photos at 5 TPS is 40,000 seconds, about 11 hours, and that's at a perfect 5 per second with zero throttles. At 50 TPS it's a bit over an hour. Search has its own quota, so live searches aren't affected by the backfill. Live uploads are: IndexFaces from your upload path and from the backfill share one 5 TPS budget, so new uploads start failing while the backfill runs.

Note: Throttling is per account. Two services, two environments or two developers' scripts in the same AWS account and region all draw on the same quota. A staging backfill can throttle production.

Why more retries make it worse

Each worker that gets throttled backs off and tries again. With 32 workers retrying on similar schedules, the retries arrive in waves that are themselves over quota. Raising max_attempts to 10 just means each photo fails more slowly. In the SDK's default standard mode, failed retries also drain a client-side retry quota (a token bucket), so a sustained overload ends with the SDK refusing to retry at all.

The AWS SDK retry docs say the default is standard mode with max_attempts of 3 (one request plus two retries). adaptive mode adds a client-side rate limiter that slows the initial requests too once it sees throttling, which is what a bulk job needs. AWS's own guidance is that adaptive is for "single-resource, throttling-heavy, latency-tolerant" work. A backfill fits that description. A user-facing search request doesn't.

A backfill that stays under quota

Three pieces: a token bucket set slightly below the quota, shared by all workers; adaptive retries as a safety net; and a dead-letter list for photos that still fail, so the job finishes and you retry the remainder later.

One process, many threads, one shared limiter. For several processes, put the bucket in Redis.

import threading, time
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
from concurrent.futures import ThreadPoolExecutor

REGION = "ap-south-1"
QUOTA_TPS = 5          # check Service Quotas for your account; it may have been raised
TARGET_TPS = QUOTA_TPS * 0.8   # headroom for live traffic and clock jitter

rek = boto3.client(
    "rekognition",
    region_name=REGION,
    config=Config(retries={"mode": "adaptive", "max_attempts": 6}),
)

class TokenBucket:
    def __init__(self, rate: float, burst: int = 1):
        self.rate, self.capacity = rate, burst
        self.tokens, self.updated = burst, time.monotonic()
        self.lock = threading.Lock()

    def take(self) -> None:
        while True:
            with self.lock:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                wait = (1 - self.tokens) / self.rate
            time.sleep(wait)

bucket = TokenBucket(TARGET_TPS)
failed: list[tuple[str, str]] = []

def index_one(key: str) -> None:
    bucket.take()
    try:
        rek.index_faces(
            CollectionId="gallery-2026",
            Image={"S3Object": {"Bucket": "my-photos", "Name": key}},
            ExternalImageId=key.replace("/", "_"),  # ExternalImageId allows [a-zA-Z0-9_.\-:]+
            MaxFaces=100,
        )
    except ClientError as e:
        failed.append((key, e.response["Error"]["Code"]))

keys = [...]  # your S3 keys
with ThreadPoolExecutor(max_workers=8) as pool:   # more threads than TPS buys nothing
    list(pool.map(index_one, keys))
print(f"done, {len(failed)} to retry later")

A few notes on the numbers. The pool size only needs to cover latency: if an IndexFaces call takes around 400 ms, four threads already sustain 10 calls a second, so eight is plenty for a 5 TPS budget. Burst is 1 on purpose, so the job never sends a spike on startup. And 80% of quota leaves room for whatever else in the account calls the same operation.

Live traffic: different rules

• Keep retries short. A user waiting for search results shouldn't sit through six attempts with growing backoff. Use standard mode with the default 3 attempts, and return a "busy, try again" response after that.
• Separate the budgets in your head. Live SearchFacesByImage has its own quota, but live IndexFaces (new uploads) shares one with any backfill. Pause or slow the backfill during peak upload hours.
• Alarm on throttles, not just errors. CloudWatch publishes a ThrottledCount metric for Rekognition. An alarm on it tells you you're near the ceiling before users notice.

Requesting a quota increase

TPS quotas are adjustable through Service Quotas or the AWS Support Center, per operation and per region. Ask with a specific number and a reason ("IndexFaces to 50 TPS in ap-south-1 for a one-time backfill of 2M photos"), and ask days before you need it, since approval isn't instant. Ask for each operation separately: a raise for IndexFaces does nothing for SearchFacesByImage.

Or send bulk work through a queue

We build SightRadar, a face search API that uses Rekognition's collection model, and we went the other way for bulk work. POST /v1/batches takes up to 1,000 photo URLs in one call, queues them, and sends one signed webhook per photo as results arrive. Your code never meets a per-second ceiling during a backfill, and batch photos are billed at $0.00062 each instead of the $0.00093 real-time rate. See batch and webhooks.

import os, requests

photos = [{"url": f"https://cdn.example.com/{k}", "external_id": k} for k in keys[:1000]]
r = requests.post(
    "https://api.sightradar.com/v1/batches",
    headers={"Authorization": f"Bearer {os.environ['SIGHTRADAR_API_KEY']}"},
    json={"collection_id": "gallery-2026", "photos": photos},
    timeout=30,
)
print(r.json()["batch_id"])  # results arrive per photo via webhook or GET /v1/batches/{id}

Real-time calls do have limits: 50 requests per second per API key and 10 in-flight requests per account by default, returning 429. So the token-bucket advice above still applies to a synchronous loop. The difference is that bulk jobs don't need one. If your existing code is boto3, the Rekognition shim keeps your index_faces calls as they are.

Estimate a backfill at batch rates before you start one. Open the pricing calculator

Frequently asked questions

What is the default TPS limit for AWS Rekognition IndexFaces?

5 transactions per second per account in most regions, and 50 in US East (N. Virginia), US West (Oregon) and Europe (Ireland). SearchFaces and SearchFacesByImage have the same defaults, each as a separate quota. The limits are adjustable through Service Quotas.

What is the difference between ProvisionedThroughputExceededException and ThrottlingException in Rekognition?

Both mean you sent requests faster than the account's quota allows. Handle them the same way: slow down on the client, retry with backoff and jitter, and request a quota increase if the sustained rate you need is above the default.

Should I use adaptive retry mode for Rekognition?

For bulk, latency-tolerant jobs such as backfills, yes. Adaptive mode adds a client-side rate limiter that slows new requests after throttling. For user-facing requests, standard mode with few attempts is better, so a user is not kept waiting.

Do Rekognition TPS limits apply per collection?

No. They apply per AWS account, per region, per operation. Every collection, service and script in the same account and region shares the same quota for a given operation.

• the face search API used in the examples. Originally published on the SightRadar blog.*