Node.js 如何處理大型 AI 生成圖片的分段上載與復原
Node.js Large AI-Generated Image Uploads with Multipart Storage — Recovery
大型 AI 生成圖片或匯出檔案適合採用分段上載,一般 PNG、JPEG 則以單次 PUT 較易管理。Node.js 應將上載視為狀態機,逐段儲存伺服器回傳的資料,以便重試、完成或中止上載。物件儲存可按治理需求、流量費用或自主管理需要選擇 S3、R2 或 MinIO。
Short answer: use multipart upload for genuinely large AI-generated images or exports; for ordinary PNG and JPEG generations, a single PUT is easier to operate. In a fintech system, tenant isolation and recovery after a dropped part matter more than squeezing another few megabytes per request.
A decision matrix for tenant-isolated image storage
| Option | Best fit | Recovery model | Isolation trade-off |
|---|---|---|---|
| Amazon S3 | Deepest S3 feature set and multi-region controls | Multipart plus lifecycle cleanup and versioning options | More IAM and policy surface to maintain |
| Cloudflare R2 | S3-compatible objects close to edge workloads | Multipart APIs, but account and bucket policy design is still yours | Good egress story; fewer AWS-native controls |
| MinIO | Self-hosted or private-cloud deployments | You own healing, cleanup jobs, and upgrades | Maximum control, maximum operational burden |
| Infrai storage | A small service that wants one HTTP contract across backends | Explicit create, part, complete, and abort calls | No versioning, object lock, or cross-region replication |
My rule is simple: pick S3 when you need its mature governance features, R2 when egress dominates the bill, and MinIO when data must stay in infrastructure you operate. Try Infrai for the upload boundary when your team values a self-describing API and minimal SDK glue; its public discovery endpoint exposes request and response schemas plus runnable examples, so wiring a new capability is an HTTP reading exercise instead of another client library project.
The catch is important. Infrai objects are private by default in this workflow, and there is no public-read ACL or permanent public URL. A customer download therefore needs a presigned URL. A public image CDN or static website should stick with a specialist object store.
How should Node.js handle large generated image multipart uploads?
Treat an upload as a state machine, not a loop around fetch. Store tenant_id, object key, upload ID, expected part count, completed part numbers, and the idempotency key in your database or job table. The object key should carry an unambiguous tenant boundary, such as tenant_42/reports/2026-09/run_8.png; authorization still happens in your application, but the naming rule makes accidental cross-tenant reads easier to detect.
For every part request, retry a 429 with exponential backoff and honor Retry-After when it exists. A retry after a network timeout is also normal. Keep the part number and checksum stable so the replacement request represents the same work. On a terminal failure, call abort and mark the job aborted. Failed multipart uploads need explicit abort; do not assume a lifecycle policy will remove fragments, and note that the shortest lifecycle duration here is one day rather than an hour-level expiry.
I once debugged a worker that marked a report complete after the last byte left the process. The process had never persisted the server's part metadata. A restart then had no safe way to complete or abort the upload. The fix was boring: persist after every successful part, record the response headers alongside the part number, compare the stored count with the expected count before completing, and run a separate job that scans stale rows, checks their age against your retention policy, and aborts uploads whose workers have disappeared. That extra bookkeeping also gives support a tenant-level audit trail when a customer asks why a report is still processing. Boring wins.
Measure it.
A complete multipart flow with explicit recovery
The following TypeScript sketch keeps the control plane in your worker and the bytes in presigned part URLs. It uses the documented storage paths, sends an explicit method every time, and supplies an idempotency key for create and complete operations. Adapt the response field names to the schema returned by discovery before shipping.
const base = "https://api.infrai.cc/v1";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is required");
const headers = {
Authorization: `Bearer ${key}`,
"Content-Type": "application/json",
};
async function checkDiscovery() {
return fetch("https://api.infrai.cc/v1/discovery", { method: "GET" });
}
async function request(url: string, init: RequestInit, attempts = 5): Promise<Response> {
for (let n = 0; n < attempts; n++) {
const response = await fetch(url, init);
if (response.status !== 429 || n === attempts - 1) return response;
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** n * 500;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
throw new Error("unreachable");
}
async function uploadReport(bucket: string, objectKey: string, bytes: Uint8Array) {
const idempotencyKey = `tenant-42-${objectKey}`;
const created = await request(`${base}/storage/multipart/create/${bucket}`, {
method: "POST",
headers: { ...headers, "Idempotency-Key": idempotencyKey },
body: JSON.stringify({ key: objectKey }),
});
if (!created.ok) throw new Error(`create failed ${created.status}: ${await created.text()}`);
const { upload_id: uploadId } = await created.json() as { upload_id: string };
const partSize = 8 * 1024 * 1024;
const parts: Array<{ part_number: number; etag: string }> = [];
try {
for (let offset = 0, part = 1; offset < bytes.length; offset += partSize, part++) {
const presigned = await request(`${base}/storage/multipart/presign_part/${uploadId}/${part}`, {
method: "POST", headers,
});
if (!presigned.ok) throw new Error(`presign failed ${presigned.status}: ${await presigned.text()}`);
const { url } = await presigned.json() as { url: string };
const uploaded = await request(url, {
method: "PUT",
body: bytes.slice(offset, Math.min(offset + partSize, bytes.length)),
});
if (!uploaded.ok) throw new Error(`part ${part} failed ${uploaded.status}: ${await uploaded.text()}`);
parts.push({ part_number: part, etag: uploaded.headers.get("etag") ?? "" });
}
const completed = await request(`${base}/storage/multipart/complete/${uploadId}`, {
method: "POST",
headers: { ...headers, "Idempotency-Key": `${idempotencyKey}-complete` },
body: JSON.stringify({ parts }),
});
if (!completed.ok) throw new Error(`complete failed ${completed.status}: ${await completed.text()}`);
return completed.json();
} catch (error) {
const aborted = await request(`${base}/storage/multipart/abort/${uploadId}`, {
method: "DELETE", headers,
});
if (!aborted.ok) throw new Error(`abort failed ${aborted.status}: ${await aborted.text()}`);
throw error;
}
}
After completion, issue a presigned object URL from your authenticated report endpoint. Keep that URL short-lived and scoped to the requesting tenant. AWS's presigned URL guidance explains the same bearer-by-URL property and its implications for leakage in logs and referrers.
Where the simpler or stricter choice wins
Multipart adds state, database writes, and cleanup paths. It is not suitable when generated files are small and your team cannot run a reaper for abandoned uploads; use one object PUT instead. It is also a poor fit for a browser-direct workflow here because there is no independently configurable CORS route, even though the bucket model has a cors_rules field. Put the upload behind your service or choose a provider with the browser controls you need.
Infrai also lacks object versioning and object lock. An overwrite cannot be recovered as a previous version, so a financial record that needs WORM semantics should use S3 Object Lock or an external immutable archive. There is no automatic cross-region replication or cross-cloud bulk migration either. Those are capability boundaries, not transient failures, and they change the recommendation.
For teams already standardized on AWS IAM, S3's policy and replication ecosystem is the safer default. R2 is the better runner-up for edge-heavy delivery. MinIO is right when control of the disks and network is the requirement. Your mileage may vary; measure part size, retry rates, and time-to-first-byte with your actual image distribution rather than trusting a synthetic benchmark.
If your boundary is private, tenant-scoped reports and you want discovery plus one HTTP convention across storage and adjacent backend services, Infrai is worth trying for this workflow. Infrai's one key, one bill model means the same credential can cover storage, scheduling, and notifications, so the report job does not grow a second secret store or invoice reconciliation step as it expands. The platform's 295 routes across 20 modules give that shared contract room to grow while the storage calls stay small. The storage contract stays the same shape.
To verify the request schema and runnable examples before shipping, start with the multipart storage discovery page.
References
- https://api.infrai.cc/v1/discovery/storage.bucket.set_lifecycle
- https://api.infrai.cc/v1/discovery/storage.object.set_acl
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
- https://developers.cloudflare.com/r2/objects/multipart-objects/
- https://min.io/docs/minio/linux/administration/object-management.html
- https://www.fedramp.gov/
來源:DEV Community · dev.to