Adjoints and the Discrete-Time Pontryagin Principle

Finite-horizon optimal control produces a structured nonlinear program and the KKT conditions characterize its local solutions. How do those multipliers organize themselves when the equality constraints are a forward dynamical recursion?

If we take the Bolza formulation of the DOCP and apply the KKT conditions directly, we obtain an optimization system with many multipliers and constraints. Written in raw form, it looks like any other nonlinear program. But in control, this structure has a long history and a name of its own: the Pontryagin principle. In fact, the discrete-time version can be seen as the structured KKT system that results from introducing multipliers for the dynamics and collecting terms stage by stage.

We work with the Bolza program

Introduce costates $\boldsymbol{\lambda}_{t+1}\in\mathbb{R}^n$ for the dynamics, multipliers $\boldsymbol{\mu}_t\ge \mathbf{0}$ for path inequalities, and $\boldsymbol{\nu}$ for terminal equalities. The Lagrangian is

It is convenient to package the stagewise terms in a Hamiltonian

Then

Necessary conditions¶

How do state feasibility, costate recursion, control stationarity, and complementarity emerge from the stagewise Lagrangian?

Taking first-order variations and collecting terms gives the discrete-time adjoint system, control stationarity, and complementarity. At a local minimum $\{\mathbf{x}_t^\star,\mathbf{u}_t^\star\}$ with multipliers $\{\boldsymbol{\lambda}_t^\star,\boldsymbol{\mu}_t^\star,\boldsymbol{\nu}^\star\}$:

State dynamics (primal feasibility)

Costate recursion (backward “adjoint” equation)

with the terminal condition

Control stationarity (first-order optimality in $\mathbf{u}_t$) If $\mathcal{U}_t=\mathbb{R}^m$ (no explicit set constraint), then

If $\mathcal{U}_t$ imposes bounds or a convex set, the condition becomes the variational inequality

where $N_{\mathcal{U}_t}(\cdot)$ is the normal cone to $\mathcal{U}_t$. For simple box bounds, this reduces to standard KKT sign and complementarity conditions on the components of $\mathbf{u}_t^\star$.

Path-constraint multipliers (primal/dual feasibility and complementarity)

Terminal equalities (if present)

The triplet “forward state, backward costate, control stationarity” is the discrete-time Euler–Lagrange system tailored to control with dynamics. It is the same KKT logic as before, but organized stagewise through the Hamiltonian.

Recap. The discrete-time Pontryagin principle is the KKT system for trajectory optimization, organized to exploit temporal structure. It has a forward-backward decomposition: states propagate forward through the dynamics, while costates propagate backward through the adjoint equation. The Hamiltonian $H_t$ packages together the stage cost, the dynamics (weighted by the next costate), and any path constraints (weighted by their multipliers). Control stationarity says that optimal controls minimize the Hamiltonian at each stage. Complementarity ensures that only binding constraints carry nonzero multipliers. This structure underlies both analytical solution methods (such as LQR) and numerical algorithms (such as the adjoint method for gradient computation).

The adjoint equation as reverse accumulation¶

The costate equations characterize stationarity, but can the same backward recursion compute every control derivative with one reverse sweep?

Optimization needs sensitivities. In trajectory problems we adjust decisions (controls or parameters) to reduce an objective while respecting dynamics and constraints. First‑order methods in the unconstrained case (e.g., gradient descent, L‑BFGS, Adam) require the gradient of the objective with respect to all controls, and constrained methods (SQP, interior‑point) require gradients of the Lagrangian, i.e., of costs and constraints. The discrete‑time adjoint equations provide these derivatives in a way that scales to long horizons and many decision variables.

Consider

A single forward rollout computes and stores the trajectory $\mathbf{x}_{1:T}$. A single backward sweep then applies the reverse‑mode chain rule stage by stage.

Defining the costate by

yields exactly the discrete‑time adjoint (PMP) recursion.

The gradient with respect to each control follows from the same reverse pass:

This reverse accumulation produces every control gradient with one forward rollout and one backward adjoint pass. The costate $\boldsymbol{\lambda}_t$ measures the marginal effect of perturbing the state at time $t$ on the total objective. Each control gradient combines a direct contribution from $c_t$ with an indirect contribution through the next state. Backpropagation through an unrolled dynamical system performs the same calculation.

Finite differences instead perturb one decision at a time and rerun the system. They require on the order of $p$ rollouts for $p=(T-1)m$ control variables and introduce a finite-difference step size. Forward-mode sensitivities propagate a separate Jacobian-vector product for each parameter direction, so their work also scales with $p$. Reverse mode propagates one costate vector backward and reads all partial derivatives from that sweep. For a scalar objective, this replaces one rollout per parameter by one forward-backward pass, at the cost of storing or checkpointing the state trajectory.

The adjoint recursion is therefore the reverse-mode derivative of the trajectory objective. States carry the nominal trajectory forward, costates carry its sensitivity backward, and the local control derivatives combine the two at each stage.

Summary and Outlook¶

The KKT conditions organize local optimality into primal feasibility, stationarity, dual feasibility, and complementarity. Applied along a trajectory, they give the discrete-time Pontryagin principle. States propagate forward, costates propagate backward, and the Hamiltonian supplies the local control stationarity condition. The same backward recursion computes all control gradients with one reverse pass.

The necessary conditions do not choose how states and actions should be exposed to a numerical solver. Should the states remain decision variables, be eliminated by forward simulation, or appear only at segment boundaries? Numerical trajectory optimization compares those formulations.

Exercises¶