跳到正文
DEV Community· 3deep5me·· 4 小時前AI 評分58

如何透過 oauth2-proxy 讓 Kubernetes ServiceAccount 以自身身份存取 Forgejo

Giving Kubernetes ServiceAccounts an identity in Forgejo with oauth2-proxy

AI 導讀

作者示範以 Kubernetes ServiceAccount 投射的 JWT，經 oauth2-proxy 驗證後，讓編碼 Agent 以各自身份存取 Forgejo。

正文

In my little agent platform every coding agent is a Kubernetes Job: it gets a repository, a task and a deadline, works, pushes a branch and opens a pull request. Which means every one of those Jobs has to authenticate against Forgejo, our Git host - and my first version of that was exactly the thing you'd expect: a Personal Access Token per agent, sitting in a SealedSecret next to the agent definition.

It worked. I still didn't like it, and the reasons were never "it's a secret in Git", because it wasn't a secret in Git - it was sealed. The reasons were:

  • The credential outlives the work. An agent run lasts two hours. The token lasts until somebody remembers to revoke it.
  • It is a shared identity, not a per-run one. Anybody who can create a Job can mount the SealedSecret. That includes a fellow agent, and it means "who pushed this?" has exactly one answer for the whole platform.
  • It is a value somewhere. In a pod spec, in env, in a crash dump, in a diff of a debug session. A credential that never exists as a value cannot leak.

So instead of managing a token per agent, I made the agents log in as themselves. Kubernetes already knows who a pod is - a ServiceAccount - and it will happily mint a signed token for it. That token is a JWT with a sub claim. Forgejo accepts an HTTP header as an authenticated user if you ask it to. And oauth2-proxy is a small piece of software that verifies JWTs.

All three pieces exist. The interesting part is the glue - and the diagram below is that glue: an agent pod whose only credential is a projected token, a gate with two doors, and a Forgejo that nothing but the gate may reach.

The gate: the pod's projected token (aud=forgejo) is verified by oauth2-proxy, the verified Kubernetes subject is mapped to a Forgejo account, and only the gate pod may open a connection to the instance

The pods: an identity, not a secret

The pod gets one volume. That's the entire credential configuration of an agent:

volumes:
  - name: forgejo-identity
    projected:
      sources:
        - serviceAccountToken:
            path: token              # /var/run/secrets/forgejo/token
            audience: forgejo        # what makes it acceptable to the gate
            expirationSeconds: 86400 # longer than any run may be, and no longer

Two things matter here.

audience is the entire point. A normal ServiceAccount token is minted for the API server. A token with audience: forgejo is useless against the API server and only usable by whoever accepts that audience - which is the thing in front of Forgejo. It is a scoping mechanism, and it costs one line.

projected is the other half: the kubelet writes this file and rewrites it in place while the token approaches expiry. So nothing in the container may freeze its value. The entrypoint refuses to start without it:

if [ ! -s "${PI_FORGEJO_SA_TOKEN_FILE:-}" ]; then
  log "fatal: no projected ServiceAccount token - refusing to run unauthenticated"
  exit 1
fi

I'd rather have a run that does not happen than a run that silently talks to Forgejo unauthenticated.

The pod doesn't write the token anywhere either - git is authenticated by a wrapper that is put on PATH and reads the file on every call:

REAL_GIT=$(command -v git)
cat > /agent-bin/git <<WRAPPER
#!/bin/sh
exec "$REAL_GIT" -c "http.${PI_FORGEJO_URL}/.extraHeader=Authorization: Bearer \$(cat "$PI_FORGEJO_SA_TOKEN_FILE")" "\$@"
WRAPPER

Because the header is scoped to our Forgejo URL, the token is never offered to any other host; because it is read per call, a rotation in the middle of a two-hour run is invisible. Nothing lands in .git/config and no credential helper is involved - which also means a git command that fails cannot print a credential in its error message.

The cluster as an identity provider

This is the part that surprised me the most: the Kubernetes API server is a working OIDC issuer, and you can check the claims yourself without installing anything:

kubectl create token hermes-pke --audience=forgejo --duration=10m \
  | cut -d. -f2 | base64 -d 2>/dev/null; echo

You get iss (the API server), aud: ["forgejo"], an expiry, and

"sub": "system:serviceaccount:hermes-pke:hermes-pke"

That subject is the identity: namespace and ServiceAccount name, unforgeable, signed with the cluster's own keys, expiring on its own.

The catch is that the API server is a verification-only issuer. Its discovery document advertises no authorization and no token endpoint, because there is nothing to authorize - you don't log into Kubernetes, you present a token Kubernetes gave you. Every OIDC client library that insists on doing a full discovery will therefore fail against it, and you have to tell it to skip discovery and fetch the signing keys directly.

oauth2-proxy as the verifier

oauth2-proxy normally does "log in with GitHub/Google" for a web app. It also understands bearer tokens, and that mode is what we use: it takes the Authorization: Bearer <token> header, checks signature, issuer, audience and expiry against the API server's own keys, and forwards the verified username to its upstream.

args:
  - --provider=oidc
  - --oidc-issuer-url=https://kubernetes.default.svc.cluster.local
  - --skip-oidc-discovery=true
  - --oidc-jwks-url=https://kubernetes.default.svc.cluster.local/openid/v1/jwks
  - --provider-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  - --client-id=forgejo            # = the audience the token must carry
  - --user-id-claim=sub            # a ServiceAccount has no e-mail address
  - --email-domain=*
  - --skip-jwt-bearer-tokens=true  # accept the bearer token, no browser session
  - --bearer-token-login-fallback=false  # invalid token => 403, not a redirect
  - --pass-user-headers=true       # hand the subject on as X-Forwarded-User
  - --upstream=http://127.0.0.1:8081/
  - --http-address=0.0.0.0:4180

Four of those lines are the ones I'd keep if I could only keep four: --client-id (a token for another audience is rejected), --user-id-claim=sub (there is no e-mail to claim), --skip-jwt-bearer-tokens (this is what makes a Kubernetes token a login at all) and --bearer-token-login-fallback=false - with this issuer there is no interactive flow to redirect to, so an invalid token must be a hard 403.

Why oauth2-proxy and not Forgejo itself verifying the token? Because Forgejo, like most applications, does not grow custom token verification for my homelab. It has one standard mechanism for "something in front of me authenticated this request", and I'd rather use the standard mechanism with a dedicated verifier than patch the application. Which brings us to the interesting part.

Forgejo as the relying party

Forgejo supports reverse-proxy authentication: if a request arrives with a certain header, that header value is the username. Turned on in app.ini:

[service]
ENABLE_REVERSE_PROXY_AUTHENTICATION = true      # the web UI
ENABLE_REVERSE_PROXY_AUTHENTICATION_API = true  # and the API - agents use this
ENABLE_REVERSE_PROXY_AUTO_REGISTRATION = false  # never invent an account
ENABLE_REVERSE_PROXY_EMAIL = false
ENABLE_REVERSE_PROXY_FULL_NAME = false

[security]
REVERSE_PROXY_AUTHENTICATION_USER = X-Forwarded-User

Auto-registration off is deliberate: an account that the gate can present must already exist, with its rights declared in the repository that manages them. An unknown subject stays an unauthenticated request.

Now the part I measured rather than assumed, and the reason this post is worth writing: in Forgejo 15.0.10 there is no trusted-proxy check for this method. The reverse-proxy login code takes the header on faith. REVERSE_PROXY_TRUSTED_PROXIES exists, but in this version it only influences the source address used for logging - it does not gate the login.

Which means the header is a login credential, and reachability is part of the authentication: whoever can open a TCP connection to Forgejo's HTTP port can be whoever they like. Two consequences:

  1. Forgejo's Service is a ClusterIP, and a Cilium network policy allows exactly one pod to speak to it - the gate. Re-opening that port for anything else is a hole, so the "extra allowed CIDRs" list in my values file is empty and documented as such.
  2. My own gate must not be reachable in a way that lets someone skip the verifier.

Hence the gate is a pod with two doors:

:8080   the classic door   - browsers, the CI runner, Forgejo tokens
                            BLANKS every identity header a client sends
:8081   the identity door  - only this pod's own oauth2-proxy may reach it
                            (enforced by the network policy)

Both doors proxy to the same Forgejo. The classic door deletes the headers, always:

proxy_set_header X-Forwarded-User "";
proxy_set_header X-WEBAUTH-USER "";
proxy_set_header X-Auth-Request-User "";
proxy_set_header X-Forwarded-Email "";

Not "overwrites if trusted" - blanks. A client sending those headers is not lying more convincingly, it is simply ignored. Without this line, the whole design collapses back into "become anyone by sending a header".

The identity door is the one oauth2-proxy forwards to, and it strips the token on the way through, because the token has done its job by then:

proxy_set_header X-Forwarded-User $identity_user;
proxy_set_header Authorization "";

One header, many subjects

Now the cosmetic problem: the subject arriving from Kubernetes is system:serviceaccount:agent-reviewer:reviewer. Forgejo usernames cannot contain colons, and I don't want a name pattern to create accounts - an account must exist, and it must be the right account.

So the identity door maps the subject to a login, with a default that fails closed:

map $http_x_forwarded_user $identity_user {
  default "";
  "system:serviceaccount:agent-tofu:tofu"        "agent-tofu";
  "system:serviceaccount:hermes-pke:hermes-pke"  "agent-hermes-pke";
  "system:serviceaccount:agent-reviewer:reviewer" "agent-reviewer";
}

Anything not listed - including an empty header, including something that merely looks like a ServiceAccount - becomes an empty string, i.e. an unauthenticated request. No account is ever invented from a name pattern.

Adding an agent is therefore two reviewed lines in two repositories: one line in this table (which Kubernetes subject may act as which Forgejo account) and one row in the access repository (what that account may do in which repository). Revoking an agent is deleting the line. That asymmetry - the identity table is small, explicit and reviewable - is what I actually wanted from all of this, much more than "no secret".

What bit me

Because a post that only shows the working end state is useless, here are the failures, with the errors:

  • cookie_secret must be 16, 24 or 32 bytes, but is 44. Even in bearer-token mode oauth2-proxy validates the cookie config. My SealedSecret held a base64 string (44 chars) instead of the 32 bytes it decodes to.
  • 502 from the gate. oauth2-proxy's upstream pointed at the Forgejo pod's port (:3000) instead of the Service (:80). Use the Service - it is what a chart should depend on, and it survives rescheduling.
  • host not found in upstream, but only on the node where the new pod landed. The network policy's DNS egress rule didn't match: toEndpoints.matchLabels matches inside the policy's own namespace, so k8s-app: kube-dns needs Cilium's own io.kubernetes.pod.namespace: kube-system label next to it.
  • nginx died silently in a read-only root filesystem. Every temp path it would create under /var/cache/nginx has to be redirected onto an emptyDir mounted at /tmp.
  • The web UI's live updates died at the gate until I forwarded Upgrade/Connection for websockets - and client_max_body_size 0, because a git push is not a small request and must not be buffered onto a small disk.
  • My first cut had the identity door open to the whole cluster. Anyone could have sent the header straight to :8081 and skipped oauth2-proxy entirely. The gate's own network policy now allows that port from its own pod only - which is the same rule as before, just applied to the other door.

Why not the obvious alternatives

  • A token per agent (PAT). Works, and it is how I started. Long-lived, shared by whoever can create a Job, present as a value, and revocation is a human task. The Git identity of an agent should not be something a human has to remember to take away.
  • Deploy keys. Nice for a push-only robot, but they are per repository and they are not a user: no attribution, no pull-request authorship, no reviews. With n agents and m repositories you also get n×m keys.
  • Username + password for the agent account. The same as a PAT, with a worse rotation story and a real login session.
  • Trusting the wire. That's the trap above, not an alternative.

What it costs, what it buys

Costs: one additional hop in front of Git (the gate), an identity table to maintain, and the discipline that only the gate may reach the instance. Both gates are ordinary Kubernetes objects, so all of it is in Git and reconciled.

Buys: no credential exists as a value anywhere - not in Git, not in a pod, not in a backup; each agent run is itself; a runaway agent is revoked by deleting one line; every commit, branch and pull request is attributable to the agent that made it; and the identity of a Job is the same identity the cluster already uses for its RBAC, which means there is exactly one story about "who is this" in the platform instead of two.

Next on my list: an agent that starts the runs of another agent (a reviewer that reviews every pull request), and what that costs in admission control. That's a post of its own. 🙂

Tested against Forgejo 15.0.10 (chart 17.3.1), a Cilium-based cluster and oauth2-proxy in bearer-token mode. The reverse-proxy behaviour without a trusted-proxy check is a measured fact about this version - if you rely on it, measure it on yours.

Thanks for reading! If you're doing something similar - or you know a Forgejo version that gates the reverse-proxy header properly - I'd love to hear it in the comments.

This post was completely written by LLM also the architecture and wiring behind this. It's only validated by a human if it works. (It works!)
But tbh I don't know if it's secure - so before you use it in production you should verify it by yourself - because rbac is nothing to play with.

來源：DEV Community · dev.to

#教學/實踐#Agent#部署/工程