Socialprofiler 如何設計按次收費的長任務 API

Disclosure: I'm the founder of Socialprofiler, a people-search tool that resolves a US phone number, username or name into someone's public social accounts. This post is about one API design problem we had to solve, with the warts left in. The spec and docs I quote are public and MIT-licensed; no product pitch beyond this line.

TL;DR

• When one API call spends the customer's money and kicks off a 90-second job, a single POST /do-the-thing is the wrong shape.
• We split it into three phases: discover (free, idempotent), confirm (one call, one credit), read (free forever).
• Progress goes over Server-Sent Events, with polling as the fallback and /check as the source of truth.
• Partial failures are data, not HTTP errors. A private Instagram account must not 500 the whole report.
• Five things I'd do differently are at the bottom. One of them is a bug in our own OpenAPI file.

The problem

Our job looks like this: a user gives us a handle (nasa on Instagram) or a name plus a US state. We search several networks, show candidate profiles, the user picks the right ones, and then we pull the public follow graph for each, map it to interests, and write an AI summary. The pull takes anywhere from 30 seconds to five minutes, and it costs us real money per profile, so it costs the customer one report credit.

The naive design is one endpoint:

POST /report
{ "source": "ig", "username": "nasa" }

It fails in three ways at once:

1. Ambiguity. nasa matches twelve Instagram accounts. Which one did you mean? If we guess and charge, you're angry. If we ask, the single call is already not single.
2. Latency. A request that holds a connection open for 90 seconds is a request that gets killed by a proxy at 60.
3. Billing. The customer needs a moment where they can look at what they're about to pay for and say yes. Charging on the first byte of input is how you get chargebacks.

The shape we landed on

 ┌───────────┐   ┌──────────────┐   ┌───────────┐   ┌──────────┐   ┌────────┐
 │  create   │──▶│  candidates  │──▶│    add    │──▶│ confirm  │──▶│  read  │
 └───────────┘   └──────────────┘   └───────────┘   └──────────┘   └────────┘
  POST /create    GET /create/{id}   POST .../add    POST .../confirm  GET /report/{id}
      free              free             free        ⚠ 1 credit          free

Every box before confirm is free and repeatable. confirm is the only call that spends anything, and the body of that call is an explicit list of what you're paying for.

Phase 1: discover

curl -s -X POST $BASE/create \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"source":"ig","username":"nasa"}'
# → { "id": "665f1c9a2b4e8d0011aa77bc" }

The server starts searching asynchronously and hands back a report id immediately. You then poll GET /create/{id} until the sources you care about report loaded: true:

{
  "request": { "source": "ig", "username": "nasa" },
  "profiles": [
    {
      "source": "ig",
      "loaded": true,
      "matches": 12,
      "items": [
        { "id": "528817151", "username": "nasa", "fullname": "NASA", "link": "https://instagram.com/nasa" }
      ],
      "hasMore": false
    }
  ]
}

Wrong handle? Search again inside the same report with POST /create/{id}/{source}/search instead of creating a new one. The report id is a workspace, not a transaction.

Phase 2: pick and confirm

# one call per profile, no bulk endpoint (more on that below)
curl -s -X POST $BASE/create/$ID/ig/add -d '{"id":"528817151"}' ...
curl -s -X POST $BASE/create/$ID/tw/add -d '{"id":"11348282"}' ...

# the only call that costs anything
curl -s -X POST $BASE/create/$ID/confirm \
  -d '[{"source":"ig","id":"528817151"},{"source":"tw","id":"11348282"}]' ...
# → { "estimatedTime": 90, "loadTimeout": 300 }

Two deliberate rules here:

• The confirm body must match exactly what you added. You cannot sneak a new profile in at the paying step. This makes the confirm call auditable: the array is the invoice line.
• The response tells you how long to wait and when the server will give up. loadTimeout is the server's own ceiling, so the client doesn't have to invent one.

If estimatedTime comes back as 0, the report was already generated. Re-confirming an existing report is a no-op, not a second charge.

Phase 3: wait, then read

This is where most long-job APIs get lazy and say "poll every few seconds". We offer that, but the recommended path is a Server-Sent Events stream:

GET /sse/{reportId}
Authorization: Bearer <jwt>
Accept: text/event-stream

Every message is a small JSON object with type and status:

{ "type": "follows", "status": "inprogress", "source": "ig", "id": "528817151" }
{ "type": "follows", "status": "done",       "source": "ig", "id": "528817151" }
{ "type": "follows", "status": "error",      "source": "tw", "id": "11348282", "error": "private" }
{ "type": "summary", "status": "done" }
{ "type": "ping",    "status": "inited" }

Progress is derived client-side: build a map of source:id → status from the follows events and count how many have settled. Here's the whole consumer in Node 18 with no dependencies:

const res = await fetch(`${BASE}/sse/${reportId}`, {
  headers: { Authorization: `Bearer ${token}`, Accept: 'text/event-stream' },
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const frames = buffer.split('\n\n');
  buffer = frames.pop();                       // keep the incomplete tail
  for (const frame of frames) {
    const payload = frame.split('\n')
      .filter(l => l.startsWith('data:'))
      .map(l => l.slice(5).trim()).join('');
    if (!payload) continue;
    const msg = JSON.parse(payload);
    if (msg.type === 'ping') continue;
    console.log(msg.type, msg.status, msg.source ?? '', msg.error ?? '');
  }
}

The ping every ~30 seconds exists purely so that proxies don't close an idle connection. The useful inversion: if you haven't seen a ping in 60 seconds, the connection is dead even if your socket says otherwise. Reconnect. Reconnecting is safe because the stream sends you the current state, not a replay.

The decision I'd defend hardest: failures are data

Once a report is confirmed, individual profiles can fail to load. The account is private. The account was renamed. The person opted out. None of those are HTTP errors, and none of them stop the report.

// GET /check/{reportId}
{
  "generated": true,
  "ready": true,
  "loadError": "private",   // present only on failure
  "errorSource": "ig"
}

The reason codes are a closed enum: private, notFound, zeroFollows, optOut, unloaded, unmapped, unexpected. The first four are deterministic and the client should not retry them. optOut in particular is one we treat as sacred: if someone asked to be removed, the answer is no, and it stays no.

If one of three confirmed profiles is private, the report still generates from the other two. The alternative, failing the whole job because one third of the input was unavailable, would mean charging a credit and returning nothing. We tried that version early on. Support tickets taught us the current one.

The same thinking shows up in the error table. 402 means out of credits and retrying will not help; 408 and 503 mean try again with backoff; 401 means refresh the token exactly once. The docs say which is which so a client can be written as a lookup table instead of a guess:

RETRYABLE = {408, 429, 500, 502, 503, 504}

def call(method, url, **kw):
    for attempt in range(5):
        r = requests.request(method, url, timeout=30, **kw)
        if r.status_code not in RETRYABLE:
            return r
        time.sleep(min(2 ** attempt, 30) + random.random())
    return r

Small things that paid off

• Report ids never expire. GET /report/{id} works forever with a valid token and never charges again. Idempotent reads make retries boring, which is the goal.
• ETag on the report page. Send If-None-Match, get 304, skip the download. Costs nothing to implement, saves real bandwidth for dashboards that refresh.
• The AI summary is its own status. The report can be ready while the summary is still pending. Decoupling them meant the slow LLM step stopped blocking the fast graph step.
• One rate limit, stated plainly. 60 requests per minute per account. Clients that cache the JWT and use SSE instead of polling never get near it.

Five things I'd do differently

Honesty section. All of these are visible in the public spec today.

1. There is no bulk add. One request per profile. For a report with six profiles that's six round trips before you can confirm. It was a shortcut in the first version and it never got fixed because it never broke anything. It should be an array.
2. The OpenAPI file stamps bearerAuth on every operation, including POST /auth/signin. Generated clients will try to send a token to the endpoint that issues tokens. The README warns about it; the spec should just be right.
3. Form-style validation errors on the auth endpoints return 200 with {"status":"error"}. This is a leftover from the web app's form handler. Every other endpoint uses real status codes. Mixed conventions are worse than either convention alone.
4. EventSource in the browser can't send an Authorization header. So browser clients either proxy the stream or rely on a same-origin cookie. A short-lived signed URL for the SSE endpoint would have been the cleaner answer.
5. The rate limiter's rejection code isn't in the spec. We document "treat unknown 4xx as non-retryable", which is a workaround for not writing down one number.

If you're building something similar

The pattern generalises to any API where a call is expensive, slow, or both: image generation, batch enrichment, anything that hits a third-party meter.

• Make everything before the paying step free and repeatable, and make the paying step's body the complete, explicit list of what's being bought.
• Return the server's own timeout expectations with the acknowledgement so the client doesn't have to guess.
• Stream progress, keep polling as a fallback, and name one endpoint as the truth when the two disagree.
• Decide up front which failures are HTTP errors and which are data. Then write the enum down.

The full spec (36 endpoints, OpenAPI 3.0.3), the six doc pages this post quotes from, and runnable curl, Python and Node examples live in github.com/twicethinker/socialprofiler-api-docs. The interactive version is at dashboard.socialprofiler.com/docs. The consumer product that sits on top of all this, the one that lets you find someone's public social accounts from a phone number, only works for US subjects and is geo-blocked for EU, UK and Russian visitors; that constraint is legal, not technical, and it shaped the optOut reason code more than anything else did.

If you've solved the "one call costs money" problem a different way, I'd genuinely like to hear it in the comments. Especially if you found a good answer to the browser SSE auth problem.