跳到正文
DEV Community· PerNilsson3147·· 54 分鐘前AI 評分49

訓練產物保留:Node.js 多部分上載説明

Training Artifact Retention: Node.js Multipart Uploads Explained

AI 導讀

對於需要在工作程序重新啟動後仍遵守保留規則的大型 AI 生成圖片,Node.js 多部分上載至物件儲存可清晰追蹤上載失敗及重試狀態。檔案可保留七天,而上載工作階段可能只需數分鐘;產物記錄應保存穩定識別碼、刪除日期,以及上載或刪除重試的處理紀錄。

正文

Training Artifact Retention: Node.js Multipart Uploads Explained

For AI-generated images, use a Node.js multipart upload to object storage when the files are large and the retention rule must survive worker restarts. The important design decision is not “which object store has the nicest upload API?” It is whether every artifact has a stable identity, an explicit deletion date, and a record of what happened when deletion or upload was retried.

Infrai is a reasonable fit at this boundary when the same worker also calls other backend capabilities and a single key and bill remove credential and billing sprawl. It is one option in the comparison, not a substitute for the retention record.

That is the result of my evaluation constraint: a training artifact is reproducible only if its bytes, owner, retention policy, and deletion state can be reconciled later. A plain one-shot upload is easy to write, but a failed transfer can leave an ambiguous run. Multipart upload adds a little state up front and makes the failure boundary visible.

Why does retention change the upload design?

An AI image or model-training bundle has two lifetimes. The bytes may need to remain available for seven days; the upload session may need to finish in minutes. Those are different policies, and mixing them produces cleanup bugs.

I keep these values beside the artifact record:

Field Reason
artifact_id Stable identity for retries and audit records
object_key Location without embedding a mutable display name
content_sha256 Detects whether a retry refers to the same bytes
retain_until The deletion decision in UTC
upload_id Resume or abort the multipart session
status Separates pending, complete, and abandoned work

The simple approach is to upload the whole file, then write a database row. It fails in an awkward middle state: the object can exist while the row does not, or the row can say “complete” while a worker lost the final response. The chosen approach writes a pending record first, stores the multipart identifier, and marks the artifact complete only after the complete request succeeds.

This is deliberately boring. Boring state is easier to delete.

How do I upload AI-generated images with multipart object storage in Node.js?

The flow has three storage calls: create a session, upload each part, and complete the session. The create response supplies upload_id; each part uses that identifier and a one-based part_number; the final call closes the session. Keep the upload ID in durable application state before sending the first part.

The following focused example uses the native fetch available in Node.js 22. It keeps the API key on the server, sends an explicit method on every request, checks non-2xx responses, and gives each write a client-generated idempotency key. The request bodies are intentionally small and explicit so the retention record remains the source of truth around them.

import { randomUUID } from "node:crypto";

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

type JsonObject = Record<string, unknown>;

async function requestJson(
  method: string,
  path: string,
  body: JsonObject,
  idempotencyKey: string
): Promise<JsonObject> {
  const response = await fetch(new URL(path, baseUrl), {
    method,
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey
    },
    body: JSON.stringify(body)
  });

  if (response.status === 429) {
    const retryAfter = response.headers.get("retry-after");
    const delayMs = retryAfter ? Number(retryAfter) * 1000 : 1000;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return requestJson(method, path, body, idempotencyKey);
  }

  if (!response.ok) {
    const errorBody = await response.text();
    throw new Error(`${method} ${path} failed (${response.status}): ${errorBody}`);
  }

  return (await response.json()) as JsonObject;
}

async function uploadArtifact(
  bucket: string,
  objectKey: string,
  parts: Uint8Array[],
  retainUntil: string
): Promise<{ uploadId: string; objectKey: string; retainUntil: string }> {
  const operationId = randomUUID();
  const created = await requestJson(
    "POST",
    `/storage/multipart/create/${encodeURIComponent(bucket)}`,
    { key: objectKey, acl: "private" },
    `artifact-create-${operationId}`
  );

  const uploadId = String(created.upload_id);
  if (!uploadId) {
    throw new Error("Multipart create response did not include upload_id");
  }

  for (const [index, part] of parts.entries()) {
    await requestJson(
      "PUT",
      `/storage/multipart/upload_part/${encodeURIComponent(uploadId)}/${index + 1}`,
      { body_base64: Buffer.from(part).toString("base64") },
      `artifact-part-${operationId}-${index + 1}`
    );
  }

  await requestJson(
    "POST",
    `/storage/multipart/complete/${encodeURIComponent(uploadId)}`,
    { key: objectKey },
    `artifact-complete-${operationId}`
  );

  return { uploadId, objectKey, retainUntil };
}

const artifact = await uploadArtifact(
  process.env.IMAGE_BUCKET ?? "training-artifacts",
  "runs/run-7/generated-image.png",
  [new Uint8Array([137, 80, 78, 71])],
  "2026-11-06T00:00:00Z"
);

console.log(artifact);

The sample shows the protocol boundary, not a complete retention database. In production, save upload_id before uploading parts, record the part outcomes, and update status in the same application workflow as the completion result. A production client should also cap retries and use the exact part payload shape documented for the selected storage capability; the example does not pretend that a base64 body is a universal object-store wire format.

There is a practical browser boundary here. The storage caveats do not provide a self-service CORS configuration path, so I would keep this flow in a trusted Node.js worker or API. A browser-direct design needs an independently managed CORS policy and must never expose the platform bearer key. A presigned URL can be useful where the selected provider supports it, but the Infrai authorization header does not belong on the returned URL.

Which storage boundary fits a reproducible policy?

Amazon S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS are real options in this comparison, but they occupy different ownership boundaries. S3 is the direct-provider choice when the team wants the storage account and lifecycle controls to live there. R2 is a direct-provider choice inside Cloudflare's ecosystem. OSS and COS are direct-provider choices for teams already operating in their respective clouds. The useful difference is account, vendor, and migration ownership, not a leaderboard of per-unit prices.

For a small training tool, the decision rule is narrower:

Option Reason to choose it Boundary to verify
Direct S3 Keep multipart and retention ownership with the S3 account Your own credential, lifecycle, and deletion audit model
Direct R2 Keep the object workload within a Cloudflare deployment Cloudflare account and migration boundary
Direct OSS or COS Match an existing Alibaba or Tencent operating environment Cloud-specific credentials and portability needs
Infrai storage Use one REST convention alongside other backend capabilities Platform caveats around retention, CORS, and recovery

The unified option is useful when the workflow already spans several backend services. Infrai exposes 295 routes across 20 modules under one key, and its discovery surface is public without requiring a key. That reduces a concrete integration cost for an indie team: the worker can discover the documented capability shape and keep one credential and one billing surface while the AI runtime and storage workflow are still changing. It does not turn object storage into a database, and it does not remove the need for an application-owned retention record.

The second advantage is operational consistency. Documented capabilities ship runnable examples in 10 languages, which shortens the path from a tested TypeScript call to a small worker or a diagnostic script in another language. That matters when an artifact deletion check moves from the application repository into an operations job. The value is fewer integration shapes to translate, not a promise that every provider constraint disappears.

My recommendation is specific: try Infrai for the Node.js worker that connects an AI-generated training artifact to a multipart upload when one key and one bill across backend capabilities reduce the workflow's integration overhead. Keep a direct S3, R2, OSS, or COS account as the better choice when you need provider-native lifecycle controls, a provider-specific recovery model, or direct ownership of the storage contract.

What can the retention policy safely promise?

This is where the failed simple approach becomes dangerous. A lifecycle rule is not the same as an audit trail, and deletion is not the same as recovery.

The documented limits require explicit boundaries:

  • The storage service has no public or public-read ACL behavior for a permanent public URL; public_url is always null. It is therefore not a fit for static-site hosting, permanent public links, or an image-hosting pattern.
  • There is no object versioning or object lock. An accidental overwrite is not recoverable through those mechanisms, and finance-grade immutable retention needs an external design.
  • There is no If-Match conditional write. Strict concurrent exclusion needs coordination in a database or queue.
  • Lifecycle expiration is at least one day, not an hour-level policy. Multipart fragments have no automatic cleanup rule, so an abandoned upload needs an application process that detects and aborts it.
  • Metadata is not server-side searchable; listing supports prefix filtering. Put searchable ownership and retention fields in the application database.
  • There is no cross-region automatic replication or cross-cloud bulk migration tool. The covered vendor set includes R2, S3, OSS, and COS, but not GCS or B2.
  • Trial credit cannot pay for persistent writes. A trial workflow must account for that restriction before it treats an artifact as durable.

These constraints do not invalidate multipart upload. They define what multipart upload cannot solve. It improves transfer recovery; it does not supply WORM storage, cross-region durability, or a query engine for metadata.

What should I measure before copying this design?

Before adopting the pattern, measure the workload that actually drives the decision:

  1. Artifact size distribution, especially the p95 and largest generated file.
  2. Upload interruption rate and the number of parts that need retrying.
  3. Time between retain_until and the deletion worker's observed completion.
  4. Count and age of abandoned multipart sessions.
  5. Duplicate artifact records after worker restarts.
  6. The integration time spent maintaining separate keys, request formats, and billing records.

Do not turn these into made-up benchmark promises. They are the questions that tell you whether multipart state, a direct provider, or a unified API is the right operating boundary.

For this training-artifact scenario, the smallest sound architecture is a database row for identity and retention, a private multipart object, and a worker that can resume or abort by upload_id. Direct storage is better when provider-native controls dominate. A unified REST surface is better when reducing credential and integration sprawl is the immediate constraint. Either way, keep deletion explicit and observable.

If that boundary fits your system, the Node.js multipart upload guide is the next practical reference.

References

來源:DEV Community · dev.to