真的需要從零打造 LLMOps 平台嗎？

Building a production-ready platform for open-weight models by extending an existing internal developer platform

The LLMOps platform trap

Open-weight models are becoming an increasingly attractive choice for enterprises because they offer cost efficiency, regulatory compliance and digital sovereignty, and the gap with proprietary models narrows with every new open model. For many enterprises, next phase of AI adoption will run largely on open-weight models.

But adopting open-weight models comes with a trade-off: enterprises take on more responsibility for operating them.

It starts with one model, then grows to include fine-tuned variants, cheaper alternatives, and newer versions. Soon, an enterprise is managing dozens of models across teams, environments, and GPU clusters.

Getting one model to answer a prompt is easy. Operating many of them in production is not.

Every new model adds management complexity. Who can deploy it? Where can it run, and on which GPUs? How does it move from development to production? How is inference traffic routed?

At this point, it is tempting to build a dedicated platform for all of this: an LLMOps platform.

But look closely and most of these challenges are not new. Internal developer platforms (IDPs) already solve them for every other workload: identity, access control, environments, deployment workflows, networking, observability and developer self-service.

What is new is a set of model-specific concerns: versioning, GPU allocation, evaluation, token usage and rollouts. This is where an existing IDP falls short, and where the real engineering effort belongs.

So, rather than building an LLMOps platform from scratch, let's identify what we can reuse, what we need to extend, and what we need to build.

Which problems are actually new?

I suggest introducing a simple framework to decide which capabilities we can reuse from an existing IDP, which need to be extended, and which need to be built specifically for AI workloads.

Adopt

Capabilities a model workload needs, but that are not model-specific:

• identity and access control
• projects and tenancy
• environments
• network isolation
• deployment and GitOps
• gateways
• baseline observability
• multi-cluster management
• developer self-service

Rebuilding these just because the workload contains a large model adds little value.

Extend

Capabilities where the platform's abstractions still fit, but need a model-shaped version.

Model serving is the clearest example. Instead of every team assembling Kubernetes resources, networking, scaling, health checks and an inference runtime, the platform offers one model-serving abstraction. The developer only specifies:

model: llama-3-70b
quantization: awq
gpu: 4
scaling: autoscale

The platform turns that into the required infrastructure. The same approach works for evaluation, fine-tuning and other AI workflows.

Build

Capabilities that are genuinely model-native. For example, request count is a poor measure of inference cost when one request uses a hundred tokens and the next uses a hundred thousand.

Which gives us a simple rule:

Adopt the foundation. Extend the platform. Build what makes models different.

Why an IDP is the right foundation

An IDP exists to solve one problem:

How do you hide infrastructure complexity without taking away the flexibility developers need?

Developers describe what they want; the platform handles the complexity. That is exactly the problem an LLMOps platform has to solve, which makes an IDP its natural foundation.

So the next practical question is:

Is there a platform that already works this way, and is open enough to build on?

One such project is OpenChoreo, an open-source IDP for Kubernetes and a CNCF Sandbox project. It is built on the same principles,

• Abstract: Hide infrastructure complexity behind simple, developer-facing interfaces.
• Extensible: Allow platform engineers to add model-specific capabilities without rebuilding the underlying platform.

Applying adopt, extend, build framework on OpenChoreo

Here is how a few of the harder LLMOps problems map onto OpenChoreo, and whether we adopt each one as-is, extend it, or build on top.

Organizational structure and isolation

Adopt.

Model workloads must respect the same boundaries as everything else. A finance team's fine-tuned model should not be reachable from a customer-support project, and neither should call an external model API unless explicitly allowed.

OpenChoreo already models this:

Namespace           → organization or business unit
 └── Project         → team or application domain (a Cell at runtime)
      └── Component  → a service, job, or model deployment

Each Project runs as a Cell: an isolated runtime boundary with default-deny network policies and resource quotas. Traffic across the boundary must go through well-defined gateways. Access control follows the same hierarchy, so who can deploy or promote a model is governed by boundaries that already exist.

Nothing here needs to be model-aware. A model deployment is just another Component in a Project.

Serving

Extend. OpenChoreo provides ComponentTypes and Traits. Platform engineers author a model-serving ComponentType on top.

An enterprise may run vLLM today and SGLang tomorrow. Developers should not need to know how either is deployed. The runtime lives inside the ComponentType, so it can change without changing the developer's configuration.

That is an extension of the platform, not a reason to build a separate deployment system.

Placement

Build. OpenChoreo provides DataPlanes and Environments. GPU inventory and model placement are built on top.

GPU capacity is usually spread across clusters and providers, and models differ in the GPU memory, topology, region and cost they need. OpenChoreo's DataPlanes already describe where workloads run. The model layer enriches them with GPU type, memory, available capacity, topology, region and cost.

The IDP answers: where **can* workloads run?*
The model layer adds: where **should* this model run?*

Evaluation and promotion

Extend. OpenChoreo provides DeploymentPipelines and Workflows. Evaluation runs are authored as Workflows, and their results become a promotion signal.

For applications, a passing build and test suite is usually enough to promote. Models need more: a candidate must be evaluated for quality, latency, cost and safety first.

That does not require a separate deployment system. Evaluation becomes one more signal in the existing promotion workflow. The promotion mechanism stays a platform capability; only the evidence behind the decision changes.

Lineage

Build. OpenChoreo and Kubernetes provide the resource storage. The model lineage graph is built on top.

Models introduce relationships that application platforms never track. When a production model regresses, the platform must answer what produced it: which base model, dataset, fine-tuning run, adapter, quantization and evaluation.

That means recording the links between models, weights, datasets, training runs, evaluations and deployments.

Conclusion

Open models give enterprises more control over models, weights, data and infrastructure. That control also means operating the whole system.

It does not mean building a developer platform, deployment system, networking layer, observability stack and multi-cluster control plane first. That foundation already exists.

The interesting engineering is the layer above it:

• How should models be represented?
• How should GPU capacity be managed?
• How should models be evaluated before promotion?
• How should token usage become a platform-level cost signal?
• How should model lineage be maintained?
• How should model versions be rolled out?

These are the problems that make an LLMOps platform different. OpenChoreo provides the foundation and the extension points; the model-specific layer evolves on top of it.

Adopt the foundation. Extend the platform. Build what makes models different.

Models will keep changing quickly. The platform underneath them does not need to be reinvented.