AI 數據儲存：如何構建模型背後的數據管道

AI data storage is the layer that holds every file a model touches: raw training media, preprocessed shards, checkpoints, and logs. Most teams run it on S3-compatible object storage because the S3 API is the one interface every ML framework already speaks.

What counts as AI data in storage terms

In storage terms, AI data splits into four buckets that get treated very differently. Raw inputs are the source files: images, audio, video, PDFs, or sensor captures. Preprocessed shards are those inputs packed into training-ready chunks, usually tar for media or parquet for tabular, often already tokenized. Checkpoints are periodic snapshots of model weights plus optimizer state, written while training runs. Run logs and evaluation artifacts are the small, high-value files you regret not keeping. Each bucket has its own access pattern. Raw and prepared data is read at high concurrency during training. Checkpoints are write-heavy and bursty. Logs are tiny but you want them durable and queryable. Packing all four into one flat bucket is the most common early mistake, because it forces the storage layer to serve a streaming read workload and a bursty write workload from the same prefix.

Why did object storage win the ML dataset layer

Object storage won the ML dataset layer because it removed the filesystem from the conversation. A training cluster might span dozens of nodes, and a POSIX filesystem that survives that fan-out is expensive to run and easy to corrupt. S3-compatible object storage gives every worker the same URL for the same object, scales the read path horizontally, and costs a fraction of block storage per gigabyte. The kicker is the API itself: boto3, fsspec, PyTorch, and WebDataset all speak S3, so the same code that reads a cloud bucket reads a self-hosted one by changing one endpoint. That portability is why teams keep object storage even after they move off the public cloud. The trade is that object storage is not a filesystem. You do not get POSIX semantics, random in-place edits, or cheap directory renames. For training data, that is usually fine, and the next section shows why.

How does a training-data pipeline actually flow

A training-data pipeline moves through five stages, and most storage decisions happen before a single GPU wakes up. Collection lands raw inputs in an object-store bucket, usually with versioned prefixes so you can reproduce a dataset later. Preprocessing reads those inputs and writes packed shards back to a different prefix, often tar for images or parquet for tabular. Storage then holds both the raw and the prepared layers, which is why capacity planning matters early. Streaming is the hot path: each training worker pulls shards on demand instead of copying the whole set locally. Checkpointing writes weight snapshots on a schedule so a crashed job restarts from minutes, not days, behind. The pattern that works is to keep the raw layer immutable and let the prepared layer be rebuildable. If preprocessing is deterministic and the raw bucket is versioned, you can always regenerate shards after a schema change without re-collecting data.

How do models read data during training

Models read data through three layers, and the bottom one is almost always an S3 client. At the framework level, PyTorch feeds samples through a DataLoader that spins up worker processes to fetch and decode in parallel. Below that, the data is usually read with boto3 or fsspec pointed at an S3 endpoint, which is where a self-hosted store like RustFS slots in by accepting the same API. For large image or multimodal sets, WebDataset streams tar shards so each worker reads one shard at a time instead of listing a million objects. The bottleneck is rarely the model. It is the read path keeping N GPUs fed, which means parallel workers, prefetching, and shards sized so no single object becomes a hot key. A practical RustFS read looks like a standard boto3 client with the endpoint pointed at port 9000:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:9000",
    aws_access_key_id="rustfsadmin",
    aws_secret_access_key="rustfsadmin",
)

resp = s3.list_objects_v2(Bucket="training-data", Prefix="shards/")
for obj in resp.get("Contents", []):
    print(obj["Key"], obj["Size"])

The same client reads a cloud bucket by swapping the endpoint and credentials.

Checkpoints and artifacts are the write-heavy half nobody plans for

Checkpoints are where AI storage stops being a read problem and becomes a write problem. A single checkpoint holds the full model weights plus optimizer state, and at fp16 a 7-billion-parameter model is about 14 GB, while a 70-billion model is roughly 140 GB. Teams save them every few hundred steps, so during a long run the storage layer absorbs repeated multi-gigabyte writes while training reads the dataset in parallel. That bursty pattern is what breaks naive single-disk setups. Two habits make it survivable. First, write checkpoints to a bucket with versioning on, so a half-written file from a killed job does not overwrite the last good one. Second, use Object Lock (WORM) on the checkpoint bucket if you need to prove a snapshot is unaltered for audit or regulatory reasons. RustFS supports both versioning and Object Lock, and keeps the metadata for those operations inside the store itself.

Where does RustFS fit as a self-hosted S3 layer

RustFS fits where a team wants the S3 API without sending every training byte to a metered cloud account. It is Apache 2.0, S3-compatible, and built in Rust, with core S3 plus Object Lock, Versioning, Lifecycle, and Replication available today. Setup is one container on ports 9000 and 9001. Because the container runs as a non-root user (UID/GID 10001:10001), make the mounted data and log directories writable by that user before start:

# RustFS runs as non-root UID/GID 10001:10001; make mounts writable first
mkdir -p data logs
chown -R 10001:10001 data logs

docker run -d -p 9000:9000 -p 9001:9001 \
  -v $(pwd)/data:/data \
  -v $(pwd)/logs:/logs \
  rustfs/rustfs:latest

The README documents default credentials rustfsadmin/rustfsadmin; change them before any real deployment. One honest trade: RustFS has no FUSE or POSIX mount, so you use the S3 API, not a local filesystem. That suits pipelines built on boto3 or WebDataset and frustrates tools that demand a mount. The metadata for versioning, lifecycle, and locking lives inside RustFS. That is one fewer service to run, but it also means RustFS reliability is your durability, and backup and recovery are your responsibility, not a vendor's.

The lakehouse edge: Iceberg and the catalog question

If your AI work touches tabular features or training labels at scale, you will meet Apache Iceberg. Iceberg is a table format on top of object storage that tracks snapshots, schema, and partitioning, turning a bucket of parquet files into a queryable table. The catalog is the piece teams underestimate. RustFS ships S3 Tables as an Iceberg REST Catalog, but it is Preview, so I would not bet a production lakehouse on it yet. For that, run a standalone catalog such as Apache Polaris or Project Nessie in front of RustFS: both are Apache-2.0, independent of any storage engine, and mature enough that the Iceberg community builds against them daily. The choice is embed versus separate. Embedding the catalog in your storage keeps moving parts down; running it separately gives you one catalog that can track multiple buckets or storage backends. Choose separate once you have more than one store to manage.

What we would tell a team starting fresh

Start with object storage as the single source of truth and treat the local filesystem as a cache, not the system of record. Keep raw data immutable and versioned, pack it into shards before training, and size shards so no object becomes a hot key. Put checkpoints in their own versioned bucket, switch on Object Lock for auditability, and rehearse a restore before you trust the setup. If you self-host with RustFS, accept the S3 API as the interface and stop looking for a POSIX mount; build on boto3 or WebDataset from day one. Plan around two real gaps: S3 Tables on RustFS is Preview, so use Polaris or Nessie for a production lakehouse, and RustFS holds your metadata, so your backup discipline is the only thing between you and data loss. None of this is exotic. It is the same S3 pattern every cloud ML stack uses, pointed at infrastructure you control.

FAQ

What is the difference between a data lake and object storage for AI?

A data lake is an organizing pattern; object storage is the substrate. You build a data lake by storing raw and prepared data in object storage with consistent prefixes, versioning, and lifecycle rules, then layering query engines or training pipelines on top. Object storage alone is just the bucket.

Do I need a POSIX filesystem mount for training data?

Most modern training pipelines do not. Frameworks read through boto3, fsspec, or WebDataset against an S3 endpoint, which avoids the cost and fragility of a distributed POSIX filesystem. RustFS has no FUSE or POSIX mount, so plan to use the S3 API directly.

How big are model checkpoints, really?

Checkpoint size is the parameter count times the bytes per parameter. At fp16, a 7-billion-parameter model is about 14 GB and a 70-billion model about 140 GB. Optimizer state can add roughly the same again, so budget for double those numbers on long runs.

Can RustFS replace cloud object storage for training?

For the data and checkpoint layers, yes, if you want to control cost and data residency. RustFS is Apache 2.0 and S3-compatible, so existing boto3 and WebDataset code runs against it by changing the endpoint. You take on durability and backup responsibility, since the metadata lives inside the store.

What is S3 Tables and do I need it for AI?

S3 Tables is RustFS's Iceberg REST Catalog for queryable tables on object storage. It is currently Preview, so for a production lakehouse use a standalone catalog like Apache Polaris or Project Nessie in front of RustFS. Most training pipelines do not need it; they read shards directly through the S3 API.

A note on how this was verified

Article type: this is a documentation-driven explainer, not a production deployment report. I pulled the RustFS commands, ports, license, and Feature & Status table verbatim from the project README (checked 2026-10-30); the docker run command is quoted exactly, not paraphrased, and the default credentials are the ones the README documents. The ML framework claims (PyTorch DataLoader, WebDataset, boto3) come from each project's official documentation. I did not stand up a live multi-GPU training cluster or run a throughput benchmark in this environment, and the sandbox here has no Docker daemon, so the RustFS command was not executed locally. The checkpoint size figures are derived from published parameter counts and standard dtype sizes (2 bytes per param at fp16), not measured on RustFS. Treat the architecture guidance as a starting point: validate against your own data sizes and network before committing, and keep versioned backups, because durability for a self-hosted store is a discipline you own.

Sources

• RustFS GitHub README (Feature & Status, Docker Quick Start, Accessing RustFS) — https://github.com/rustfs/rustfs (checked 2026-10-30)
• RustFS documentation, installation — https://docs.rustfs.com/en/installation
• boto3 documentation — https://boto3.amazonaws.com/v1/documentation/api/latest/
• WebDataset documentation — https://webdataset.github.io/webdataset/
• PyTorch DataLoader — https://pytorch.org/docs/stable/data.html
• Apache Iceberg — https://iceberg.apache.org/
• Apache Polaris — https://polaris.apache.org/
• Project Nessie — https://projectnessie.org/