自然梯度：為何最速下降取決於所選幾何

Gradient descent looks like a purely mechanical rule: follow the negative gradient downhill. But hidden inside that rule is a geometric assumption so familiar that it is easy to miss. When you take a step of a fixed size in parameter space, you are implicitly deciding what "a small step" means. Change the parameterisation, and the same algorithm can behave like a completely different optimiser — not because the model changed, but because the ruler changed.

Natural gradient descent makes that ruler explicit. Instead of measuring distance between parameter vectors, it measures distance between the probability distributions those parameters define. That one change explains why it is invariant to reparameterisation, why it can be dramatically faster on ill-conditioned problems, and why it is so expensive that practitioners usually have to approximate it.

The hidden metric in ordinary gradient descent

Consider minimising a loss L(θ). The gradient ∇L(θ) points in the direction of steepest increase — but steepest with respect to what? Steepest only has meaning once you fix how distances are measured.

Ordinary gradient descent solves, at each step, a local problem of the form: minimise the linear approximation to the loss, subject to the step staying small in Euclidean distance:

min⁡δ  L(θ)+∇L(θ)⊤δsubject to∣δ∣2≤ϵ2

The solution is the familiar update δ∝−∇L(θ) . The Euclidean norm ∣δ∣2 is doing real work here. It says that moving θ₁ by 0.01 is exactly as costly as moving θ₂ by 0.01, regardless of what those parameters mean.

That is rarely true. In a neural network, a small change to an early-layer weight can completely change the function, while a large change to a later bias may barely matter. In a probabilistic model, a small change in a probability near 0 or 1 is far more consequential than the same change near 0.5. Euclidean distance treats all of these as equal, so ordinary gradient descent can crawl along flat directions and overshoot along sensitive ones — the classic picture of a long, narrow valley.

The natural question is: what distance should replace Euclidean distance? If the parameters define a distribution pθ , a principled answer is a divergence between distributions, not a distance between vectors.

From KL divergence to the Fisher information matrix

Take a small perturbation δ and ask how much the distribution changes, measured by Kullback–Leibler (KL) divergence:

DKL(pθ ∣ pθ+δ)

For small δ , the first-order term vanishes — the KL divergence is minimised at δ=0 — and the second-order Taylor expansion gives a quadratic form:

DKL(pθ ∣ pθ+δ)≈12 δ⊤F(θ) δ

where F(θ) is the Fisher information matrix:

F(θ)=Ex∼pθ[∇θlog⁡pθ(x) ∇θlog⁡pθ(x)⊤]

Equivalently, under regularity conditions, it is the negative expected Hessian of the log-likelihood:

F(θ)=−Ex∼pθ[∇θ2log⁡pθ(x)]

This is a beautiful object. It is positive semi-definite, it transforms as a metric tensor under reparameterisation, and it turns the space of distributions into a Riemannian manifold: at each point θ, the local geometry is given by F(θ) . Two parameter vectors that are far apart in Euclidean terms may define nearly identical distributions, and F knows that.

Now repeat the steepest-descent derivation, but constrain the step by KL divergence instead of Euclidean distance:

min⁡δ  L(θ)+∇L(θ)⊤δsubject to12 δ⊤F(θ) δ≤ϵ2

Using a Lagrange multiplier, the solution is the natural gradient update:

δ∝−F(θ)−1∇L(θ)

That is the whole idea. Ordinary gradient descent preconditions by the identity matrix; natural gradient descent preconditions by the inverse Fisher information. Directions in which the distribution changes rapidly get small steps, and directions in which it barely changes get large steps — exactly the correction a narrow valley needs.

Why reparameterisation invariance matters

Suppose you reparameterise with ϕ=g(θ) , where g is invertible and smooth. The model itself is unchanged: every ϕ corresponds to the same distribution as some θ . An optimiser that only cares about the model should therefore trace out the same path through distribution space, whatever coordinates you use.

Ordinary gradient descent fails this test. Its steps depend on the coordinates, because the Euclidean metric does not transform correctly. The natural gradient passes it: F transforms as a covariant tensor, its inverse as a contravariant tensor, and the two transformation laws cancel. In distribution space, the natural gradient direction is the same object no matter how you parameterise it.

A concrete case makes this vivid. Take a Bernoulli distribution with mean μ in (0, 1), and the same distribution written in logit coordinates, η=log⁡(μ/(1−μ)) . The Fisher information in μ-coordinates is:

F(μ)=1μ(1−μ)

It blows up near 0 and 1, correctly reflecting that a small change in μ there is a large change in the distribution. In η-coordinates, the Fisher information is F(η)=μ(1−μ) , the variance of the Bernoulli. Ordinary gradient descent in μ-coordinates will take timid steps near the boundaries and bold steps in the middle, for no reason other than the choice of coordinates. The natural gradient corrects for this in both coordinates, and — crucially — gives the same trajectory in distribution space either way.

This is not a cosmetic property. In deep learning, different layers, activation scales, and normalisation choices amount to reparameterisations of the same function class. An optimiser that is sensitive to those choices forces you to retune learning rates every time you change the architecture. Invariance removes a whole class of tuning artefacts.

A worked example: fitting a Bernoulli by natural gradient

The Bernoulli case is simple enough to compute by hand and instructive enough to generalise. Suppose we observe data and want to fit μ by minimising the negative log-likelihood. The gradient with respect to μ and the Fisher information give a natural gradient step that, after simplification, moves μ directly toward the empirical mean at a rate independent of where μ currently sits — the geometry has removed the distortion that made steps near the boundaries so awkward.

In code, the contrast is easy to see for a logistic-style parameterisation. The snippet below fits a single probability with ordinary gradient descent in logit coordinates and with the natural gradient, using the Fisher information F=μ(1−μ) :

import numpy as np

# Observed fraction of successes
y_bar = 0.8

def sigmoid(eta):
    return 1.0 / (1.0 + np.exp(-eta))

# Ordinary gradient descent in logit coordinates
eta = 0.0
for _ in range(200):
    mu = sigmoid(eta)
    # Gradient of negative log-likelihood w.r.t. eta
    grad = mu - y_bar
    eta -= 0.1 * grad
ordinary_mu = sigmoid(eta)

# Natural gradient descent: precondition by 1 / F, where F = mu * (1 - mu)
eta = 0.0
for _ in range(200):
    mu = sigmoid(eta)
    grad = mu - y_bar
    fisher = mu * (1.0 - mu)
    eta -= 0.1 * grad / max(fisher, 1e-6)
natural_mu = sigmoid(eta)

print("ordinary GD mu:", ordinary_mu)
print("natural GD mu:", natural_mu)

For well-behaved, one-dimensional problems the difference is modest. The point of the example is the mechanism: dividing by the Fisher information rescales the step by exactly the local sensitivity of the distribution. When that sensitivity varies by orders of magnitude across parameters — as it does in deep networks — the same mechanism is what rescues optimisation from pathological curvature.

Where natural gradients show up in practice

Exact natural gradient descent is rarely used directly, because forming and inverting F(θ) for millions of parameters is infeasible. But its approximations are everywhere, once you know what to look for:

• K-FAC (Kronecker-factored approximate curvature) approximates the Fisher information for each layer as a Kronecker product of two much smaller matrices — one from activations, one from backpropagated gradients — making inversion tractable. It has been used to train large models in far fewer steps than first-order methods, at the cost of more computation per step.
• Adaptive methods such as Adam and RMSProp can be viewed, loosely, as diagonal approximations to natural gradient ideas: they rescale each coordinate by a running estimate of gradient magnitude, which is related to (but not the same as) the diagonal of the Fisher information. This connection explains part of their robustness to rescaling, and also why they are not truly invariant.
• Variational inference uses natural gradients for exponential-family variational distributions, where the Fisher information has closed form and the natural gradient update often simplifies beautifully — in some cases to a simple averaging of sufficient statistics. This is one of the cleanest success stories of the method.
• Trust-region and second-order methods such as TRPO in reinforcement learning constrain policy updates by KL divergence, which is the same geometric idea in a different algorithmic costume: do not let the distribution move too far in one step, where "far" is measured distributionally.

The pattern is consistent: whenever an algorithm measures progress or step size in distribution space rather than parameter space, the Fisher geometry is lurking underneath.

Limitations and common pitfalls

Natural gradients are not a free lunch, and several misunderstandings recur:

The Fisher information is not the Hessian. For a loss that is exactly a negative log-likelihood, the Fisher information equals the expected Hessian at the optimum under the true model, but in general they differ. The Hessian describes the curvature of a particular loss on particular data; the Fisher describes the local geometry of the model family itself. Conflating them leads to wrong claims about what natural gradient descent is optimising.

The empirical Fisher is not the Fisher. A common shortcut replaces the expectation over pθ with an average over training data, using squared gradients of the loss. This "empirical Fisher" is easy to compute but is a different matrix with different properties — notably, it need not be positive semi-definite in the same way, and it changes with the loss rather than depending only on the model. Results proved for the true Fisher do not automatically transfer.

Inversion needs damping. In practice F is singular or nearly singular — overparameterised models have directions that change the distribution hardly at all. Implementations add damping, solving (F+λI)−1∇L instead. Too little damping and the update explodes along flat directions; too much and you are back to ordinary gradient descent. Damping is not a minor detail; it is often the difference between a method that works and one that diverges.

Per-step cost versus number of steps. A natural gradient step can reduce the loss far more than a first-order step, but if it costs a hundred times more to compute, it may still lose on wall-clock time. The practical question is never "is the natural direction better?" but "is it better per unit of compute, on this problem, at this scale?" For small-to-medium probabilistic models, often yes. For very large networks, only structured approximations like K-FAC are competitive, and even then the engineering is substantial.

Minibatch noise interacts with preconditioning. Preconditioning amplifies noise along directions where F is small, precisely because those directions get large steps. With noisy stochastic gradients, an undamped natural gradient can be much noisier than ordinary stochastic gradient descent. Good implementations damp, average curvature estimates over many batches, and update the preconditioner less frequently than the parameters.

The takeaway

Ordinary gradient descent is not geometry-free; it silently assumes Euclidean geometry in parameter space. That assumption is convenient and often harmless, but it is an assumption, and on problems where parameters differ wildly in their effect on the model, it is the wrong one.

The natural gradient replaces that assumption with one grounded in the model itself: measure steps by how much the distribution changes, using the Fisher information as the metric. The result is reparameterisation-invariant, well-suited to ill-conditioned problems, and expensive — which is why the field lives on a spectrum from exact natural gradients through K-FAC to diagonal adaptive methods, trading geometric fidelity for computational tractability.

Next time an optimiser behaves strangely after a seemingly innocent reparameterisation — a rescaled input, a different activation, a new normalisation layer — ask what metric it is implicitly using. The answer is usually the whole story.

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial