Policy Gradients and Actor-Critic

The previous chapters optimized policies through Q-functions, soft Bellman relations, or path-consistency equations. Can expected return itself supply the policy objective without differentiating the environment dynamics? The score function estimator applies directly to trajectories because the policy terms are the only factors in their probability law that depend on the policy parameters.

The estimator requires only the ability to evaluate and differentiate $\log \pi_{\boldsymbol{w}}(a|s)$, so it also works with discrete actions where reparameterization is unavailable.

Let $G(\tau) \equiv \sum_{t=0}^T r(s_t, a_t)$ be the sum of undiscounted rewards in a trajectory $\tau$. The stochastic optimization problem we face is to maximize:

where $\tau = (s_0,a_0,s_1,a_1,...)$ is a trajectory and $G(\tau)$ is the total return. Applying the score function estimator, we get:

We have eliminated the need to know the transition probabilities in this estimator since the probability of a trajectory factorizes as:

Therefore, only the policy depends on $\boldsymbol{w}$. When taking the logarithm of this product, we get a sum where all the $\boldsymbol{w}$-independent terms vanish. The final estimator samples trajectories under the distribution $p(\tau; \boldsymbol{w})$ and computes:

This is a direct application of the score function estimator. However, we rarely use this form in practice and instead make several improvements to further reduce the variance.

Leveraging Conditional Independence¶

Given the Markov property of the MDP, rewards $r_k$ for $k < t$ are conditionally independent of action $a_t$ given the history $h_t = (s_0,a_0,...,s_{t-1},a_{t-1},s_t)$. This allows us to only need to consider future rewards when computing policy gradients.

The conditional independence assumption means that the term $\mathbb{E}_{\tau}\left[\sum_{t=0}^T\nabla_{\boldsymbol{w}}\log \pi_{\boldsymbol{w}}(a_t|s_t)\sum_{k=0}^{t-1} r_k \right]$ vanishes. To see this, factor the trajectory distribution as:

We can now re-write a single term of this summation as:

The inner expectation is zero because

The Monte Carlo estimator becomes:

This gives us the REINFORCE algorithm:

The benefit of this estimator compared to the naive one (which would weight each score function by the full trajectory return $G(\tau)$) is that it generally has less variance. This variance reduction arises from the conditional independence structure we exploited: past rewards do not depend on future actions. More formally, this estimator is an instance of a variance reduction technique known as the Extended Conditional Monte Carlo Method.

The Surrogate Loss Perspective¶

The algorithm above computes a gradient estimate $\hat{g}$ explicitly. In practice, implementations using automatic differentiation frameworks take a different approach: they define a surrogate loss whose gradient matches the REINFORCE estimator. For a single trajectory, consider:

where the returns $G_t$ and actions $a_t$ are treated as fixed constants (detached from the computation graph). Taking the gradient with respect to $\boldsymbol{w}$:

Minimizing this surrogate loss via gradient descent yields the same update as maximizing expected return via REINFORCE. The negative sign converts our maximization problem into a minimization suitable for standard optimizers.

This surrogate loss is not the expected return $J(\boldsymbol{w})$ we are trying to maximize. It is a computational device that produces the correct gradient at the current parameter values. Several properties distinguish it from a true loss function:

1. It changes each iteration. The returns $G_t$ come from trajectories sampled under the current policy. After updating $\boldsymbol{w}$, we must collect new trajectories and construct a new surrogate loss.

2. Its value is not meaningful. Unlike supervised learning where the loss measures prediction error, the numerical value of $L_{\text{surrogate}}$ has no direct interpretation. Only its gradient matters.

3. It is valid only locally. The surrogate loss provides the correct gradient only at the parameters used to collect the data. Moving far from those parameters invalidates the gradient estimate.

This perspective explains why policy gradient code often looks different from the pseudocode above. Instead of computing $\hat{g}$ explicitly, implementations define the surrogate loss and call loss.backward():

# Surrogate loss implementation (single trajectory)
log_probs = [policy.log_prob(a_t, s_t) for s_t, a_t in trajectory]
returns = compute_returns(rewards)
surrogate_loss = -sum(lp * G for lp, G in zip(log_probs, returns))
surrogate_loss.backward()  # computes REINFORCE gradient
optimizer.step()

Variance Reduction via Control Variates¶

Recall that the REINFORCE gradient estimator, after leveraging conditional independence, takes the form:

This is a sum over trajectories and timesteps. The gradient contribution at timestep $t$ of trajectory $i$ is:

While unbiased, this estimator suffers from high variance because the return $\sum_{k=t}^T r_k$ can vary significantly across trajectories even for the same state-action pair. The control variate method provides a principled way to reduce this variance.

General Control Variate Theory¶

For a general estimator $Z$ of some quantity $\mu = \mathbb{E}[Z]$, and a control variate $C$ with known expectation $\mathbb{E}[C]=0$, we can construct:

This remains unbiased since $\mathbb{E}[Z_{\text{cv}}] = \mathbb{E}[Z] - \alpha\mathbb{E}[C] = \mathbb{E}[Z]$. The variance is:

The $-2\alpha\text{Cov}(Z,C)$ term is what enables variance reduction. If $Z$ and $C$ are positively correlated, we can choose $\alpha > 0$ to make this term negative and large in magnitude, reducing the overall variance. However, the $\alpha^2\text{Var}(C)$ term grows quadratically with $\alpha$, so if we make $\alpha$ too large, this quadratic term will eventually dominate and the variance will increase rather than decrease. The variance as a function of $\alpha$ is a parabola opening upward, with a unique minimum. Setting $\frac{d}{d\alpha}\text{Var}(Z_{\text{cv}}) = 0$ gives:

This is the coefficient from ordinary least squares regression: we predict the estimator $Z$ using the control variate $C$ as the predictor. Since $\mathbb{E}[C] = 0$, the linear model is $Z \approx \mathbb{E}[Z] + \alpha^* C$, where $\alpha^*$ is the OLS slope coefficient. The control variate estimator $Z_{\text{cv}} = Z - \alpha^* C$ computes the residual: the part of $Z$ that cannot be explained by $C$.

Substituting $\alpha^*$ into the variance formula yields:

where $R^2 = \frac{[\text{Cov}(Z,C)]^2}{\text{Var}(Z)\text{Var}(C)}$ is the coefficient of determination from regressing $Z$ on $C$. The variance reduction is $R^2 \text{Var}(Z)$: the better $C$ predicts $Z$, the more variance we eliminate.

Application to REINFORCE¶

In the reinforcement learning setting, our REINFORCE gradient estimator is a sum over timesteps: $\sum_{t=0}^T Z_t$ where each $Z_t$ represents the gradient contribution at timestep $t$. We apply control variates separately to each term. Since $\text{Var}(\sum_t Z_t) = \sum_t \text{Var}(Z_t) + \sum_{t \neq s} \text{Cov}(Z_t, Z_s)$, reducing the variance of each $Z_t$ reduces the total variance, though we do not explicitly address the cross-timestep covariance terms.

For a given trajectory at state $s_t$, the gradient contribution at time $t$ is:

This is the product of the score function $\nabla_{\boldsymbol{w}}\log \pi_{\boldsymbol{w}}(a_t|s_t)$ and the return-to-go $\sum_{k=t}^T r_k$. We can subtract any state-dependent function $b(s_t)$ from the return without introducing bias, as long as $b(s_t)$ does not depend on $a_t$. This is because:

where the last equality follows from the score function identity (8).

We can now define our control variate as:

where $b(s_t)$ is a baseline function that depends only on the state. This satisfies $\mathbb{E}[C_t|s_t] = 0$. Our control variate estimator becomes:

The optimal baseline $b^*(s_t)$ minimizes the variance. To find it, consider the scalar parameter case for simplicity. Write $g(a_t) \equiv \nabla_w \log \pi_w(a_t|s_t)$ and $G_t = \sum_{k=t}^T r_k$. We want to minimize:

Since the mean does not depend on $b$, minimizing the variance is equivalent to minimizing the second moment $\mathbb{E}[g(a_t)^2(G_t - b)^2|s_t]$. Expanding and taking the derivative with respect to $b$ gives:

For vector-valued parameters $\boldsymbol{w}$, we minimize a scalar proxy such as the trace of the covariance matrix, which yields the same formula with $\|\nabla_{\boldsymbol{w}}\log \pi_{\boldsymbol{w}}(a_t|s_t)\|^2$ in place of $g(a_t)^2$:

This is the exact optimal baseline: a weighted average of returns where the weights are the squared norms of the score function. In practice, we treat the squared norm as roughly constant across actions at a given state, which leads to the simpler and widely used choice:

With this approximation, the variance-reduced gradient contribution at timestep $t$ becomes:

The term in parentheses is exactly the advantage function: $A^{\pi_{\boldsymbol{w}}}(s_t, a_t) = q^{\pi_{\boldsymbol{w}}}(s_t, a_t) - v^{\pi_{\boldsymbol{w}}}(s_t)$, where the Q-function is approximated by the Monte Carlo return $\sum_{k=t}^T r_k$. The full gradient estimate for a trajectory is then the sum over all timesteps:

In practice, we do not have access to the true value function and must learn it. Unlike the methods in the amortization chapter, where we learned value functions to approximate the optimal Q-function, here our goal is policy evaluation: estimating the value of the current policy $\pi_{\boldsymbol{w}}$. The same function approximation techniques apply, but we target $v^{\pi_{\boldsymbol{w}}}$ rather than $v^*$. The simplest approach is to regress from states to Monte Carlo returns, learning what Williams (1992) called a “baseline”:

When implementing this algorithm nowadays, we always use mini-batching to make full use of our GPUs. Therefore, a more representative variant for this algorithm would be:

The value function is trained by regressing states directly to their sampled Monte Carlo returns $G$. Advantage normalization (step 2.5) is not part of the optimal baseline derivation but improves optimization in practice and is standard in modern implementations.

Generalized Advantage Estimation¶

The baseline construction gave us a gradient estimator of the form:

where $G_t = \sum_{k=t}^T r_k$ is the Monte Carlo return from time $t$. For each visited state-action pair $(s_t, a_t)$, the term in parentheses

is a Monte Carlo estimate of the advantage $A^{\pi}(s_t, a_t) = q^{\pi}(s_t, a_t) - v^{\pi}(s_t)$. If the baseline equals the true value function, $v = v^{\pi}$, then $\mathbb{E}[\widehat{A}_t^{\text{MC}} | s_t, a_t] = A^{\pi}(s_t, a_t)$, so this estimator is unbiased.

However, as an estimator it has two limitations. First, it has high variance because $G_t$ depends on all future rewards. Second, it uses the value function only as a baseline, not as a predictor of long-term returns. We essentially discard the information in $v(s_{t+1}), v(s_{t+2}), \ldots$

GAE addresses these issues by constructing a family of estimators that interpolate between pure Monte Carlo and pure bootstrapping. A parameter $\lambda$ controls the bias-variance tradeoff.

Decomposing the Monte Carlo Advantage¶

Fix a value function $v(s)$ (not necessarily equal to $v^{\pi}$) and define the one-step residual:

Start from the Monte Carlo advantage and add and subtract $\gamma v(s_{t+1})$:

Applying this decomposition recursively yields:

The Monte Carlo advantage is exactly the discounted sum of future residuals. This is an algebraic identity, not an approximation.

The sequence $\{\delta_{t+l}\}_{l \geq 0}$ provides incremental corrections to the value function as we move forward in time. The term $\delta_t$ depends only on $(s_t, a_t, s_{t+1})$; $\delta_{t+1}$ depends on $(s_{t+1}, a_{t+1}, s_{t+2})$, and so on. As $l$ increases, the corrections become more noisy (they depend on more random outcomes) and more sensitive to errors in the value function at later states. Although the full sum is unbiased when $v = v^{\pi}$, it can have high variance and can be badly affected by approximation error in $v$.

GAE as a Shrinkage Estimator¶

The decomposition above suggests a family of estimators that downweight residuals farther in the future. Let $\lambda \in [0,1]$ and define:

This is the generalized advantage estimator $A_t^{\text{GAE}(\gamma,\lambda)}$.

Two special cases illustrate the extremes. When $\lambda = 1$, we recover the Monte Carlo advantage:

When $\lambda = 0$, we keep only the immediate residual:

Intermediate values $0 < \lambda < 1$ interpolate between these extremes. The influence of $\delta_{t+l}$ decays geometrically as $(\gamma\lambda)^l$. The parameter $\lambda$ acts as a shrinkage parameter: small $\lambda$ shrinks the estimator toward the one-step residual; large $\lambda$ allows the estimator to behave more like the Monte Carlo advantage.

If $v = v^{\pi}$ is the true value function, then $\mathbb{E}[\delta_t | s_t, a_t] = A^{\pi}(s_t, a_t)$ and $\mathbb{E}[\delta_{t+l} | s_t, a_t] = 0$ for $l \geq 1$. In this case:

for all $\lambda \in [0,1]$. When the value function is exact, GAE is unbiased regardless of $\lambda$; changing $\lambda$ only affects variance.

In practice, we approximate $v^{\pi}$ with a function approximator, and the residuals $\delta_{t+l}$ inherit approximation error. Distant residuals involve multiple applications of the approximate value function and are more contaminated by modeling error. Downweighting them (choosing $\lambda < 1$) introduces bias but can reduce variance and limit the impact of those errors.

Mixture of Multi-Step Estimators¶

Another perspective on GAE comes from multi-step returns. Define the $k$-step return from time $t$:

and the corresponding $k$-step advantage estimator $A_t^{(k)} = G_t^{(k)} - v(s_t)$. Each $A_t^{(k)}$ uses $k$ rewards before bootstrapping; larger $k$ means more variance but less bootstrapping error.

The GAE estimator can be written as a geometric mixture:

GAE is a weighted average of the $k$-step advantage estimators, with shorter horizons weighted more heavily when $\lambda$ is small.

Using GAE in the Policy Gradient¶

Once we choose $\lambda$, we plug $A_t^{\lambda}$ in place of $G_t - v(s_t)$ in the policy gradient estimator:

We still use a control variate to reduce variance (the baseline $v$), but now we construct the advantage target by smoothing the sequence of residuals $\{\delta_t\}$ with a geometrically decaying kernel.

For the value function, it is convenient to define the $\lambda$-return:

When $\lambda = 1$, $G_t^{\lambda}$ reduces to the Monte Carlo return; when $\lambda = 0$, it becomes the one-step bootstrapped target $r_t + \gamma v(s_{t+1})$.

When $\lambda = 1$, this reduces (up to advantage normalization) to the Monte Carlo baseline algorithm earlier in the chapter. When $\lambda = 0$, advantages become the one-step residuals $\delta_t$, and the $\lambda$-returns reduce to standard one-step bootstrapped targets.

Actor-Critic as the $\lambda = 0$ Limit¶

The case $\lambda = 0$ is particularly simple. The advantage becomes:

and the policy update reduces to:

while the value update becomes a standard one-step regression toward $r_t + \gamma v(s_{t+1})$. This gives the online actor-critic algorithm:

This algorithm was derived by Sutton in his 1984 thesis as an “adaptive heuristic” for temporal credit assignment. In the language of this chapter, it is the $\lambda = 0$ member of the GAE family: it uses the most local residual $\delta_t$ as both the target for the value function and the advantage estimate for the policy gradient.

Likelihood Ratio Methods in Reinforcement Learning¶

How do importance ratios and constrained or clipped surrogates reuse trajectories collected under an earlier policy?

The score function estimator from the previous section is a special case of the likelihood ratio method where the proposal distribution equals the target distribution. We now consider the general case where they differ.

Recall the likelihood ratio gradient estimator from the beginning of this chapter. For objective $J(\theta) = \mathbb{E}_{x \sim p(x;\theta)}[f(x)]$ and any proposal distribution $q(x)$:

where $\rho(x; q, \theta) = \frac{p(x;\theta)}{q(x)}$ is the likelihood ratio. The partial derivative $\frac{\partial \rho}{\partial \theta} = \rho \nabla_\theta \log p$ holds because $x$ is treated as fixed, having been sampled from $q$, which does not depend on $\theta$.

In reinforcement learning, let $x = \tau$ be a trajectory, $f(\tau) = G(\tau)$ the return, $p(\tau;\boldsymbol{w})$ the trajectory distribution under policy $\pi_{\boldsymbol{w}}$, and $q(\tau)$ the trajectory distribution under some other policy $\pi_q$. The gradient becomes:

where the trajectory likelihood ratio simplifies because transition probabilities cancel:

This product of $T+1$ ratios can become extremely large or small as $T$ grows, leading to high variance. The temporal structure provides some relief: since $\mathbb{E}_{a \sim \pi_q}[\rho] = 1$, future ratios $\rho_{k}$ for $k > t$ that do not affect the reward $r_t$ can be marginalized out. However, past ratios $\rho_{0:t-1}$ are still needed to correctly weight the probability of reaching state $s_t$.

In practice, algorithms like PPO and TRPO make an additional approximation: they use only the per-step ratio $\rho_t$ rather than the cumulative product $\rho_{0:t}$. This ignores the mismatch between the state distributions induced by the two policies. Combined with a baseline $b(s_t)$, the approximate estimator is:

This approximation corresponds to maximizing the importance-weighted surrogate objective:

where $A_t = G_t - b(s_t)$. Taking the gradient with respect to $\boldsymbol{w}$, only $\rho_t$ depends on $\boldsymbol{w}$ (since trajectories are sampled from $\pi_q$):

The gradient of the ratio is:

Substituting back:

This matches equation (46). When $\pi_q = \pi_{\boldsymbol{w}}$, the ratios $\rho_t = 1$ and we recover the score function estimator. The approximation error grows as the policies diverge, which motivates the trust region and clipping mechanisms discussed below.

Variance and the Dominance Condition¶

The ratio $\rho_t = \pi_{\boldsymbol{w}}(a_t|s_t)/\pi_q(a_t|s_t)$ is well-behaved only when the two policies are similar. If $\pi_{\boldsymbol{w}}$ assigns high probability to an action where $\pi_q$ assigns low probability, the ratio explodes. For example, if $\pi_q(a|s) = 0.01$ and $\pi_{\boldsymbol{w}}(a|s) = 0.5$, then $\rho = 50$, amplifying any noise in the advantage estimate.

Importance sampling also requires the dominance condition: the support of $\pi_{\boldsymbol{w}}$ must be contained in the support of $\pi_q$. If $\pi_{\boldsymbol{w}}(a|s) > 0$ but $\pi_q(a|s) = 0$, the ratio is undefined. Stochastic policies typically have full support, but the ratio can still become arbitrarily large as $\pi_q(a|s) \to 0$.

A common use case is to set $\pi_q = \pi_{\boldsymbol{w}_{\text{old}}}$, a previous version of the policy. This allows reusing data across multiple gradient steps: collect trajectories once, then update $\boldsymbol{w}$ several times. But each update moves $\boldsymbol{w}$ further from $\boldsymbol{w}_{\text{old}}$, making the ratios more extreme. Eventually, the gradient signal is dominated by a few samples with large weights.

Proximal Policy Optimization¶

The variance issues suggest a natural solution: keep the ratio $\rho_t$ close to 1 by ensuring the new policy stays close to the behavior policy. This keeps the importance-weighted surrogate $L^{\text{IS}}(\boldsymbol{w})$ from (47) well-behaved.

Trust Region Policy Optimization (TRPO) formalizes this by adding a constraint on the KL divergence between the old and new policies:

The KL constraint ensures that the two distributions remain similar, which bounds how extreme the importance weights can become. This is a constrained optimization problem, and one could in principle apply standard methods such as projected gradient descent or augmented Lagrangian approaches (as discussed in the trajectory optimization chapter). TRPO takes a different approach: it uses a second-order Taylor approximation of the KL constraint around the current parameters and solves the resulting trust region subproblem using conjugate gradient methods. This involves computing the Fisher information matrix (the Hessian of the KL divergence), which adds computational overhead.

Proximal Policy Optimization (PPO) achieves similar behavior through a simpler mechanism: rather than constraining the distributions to be similar, it directly clips the ratio $\rho_t$ to prevent it from moving too far from 1. This is a construction-level guarantee rather than an optimization-level constraint.

From Trajectory Expectations to State-Action Averages¶

Before defining the PPO objective, we need to clarify the relationship between the trajectory-level surrogate (47) and the state-action level objective that PPO actually optimizes. The importance-weighted surrogate is defined as an expectation over trajectories:

We can rewrite this as an expectation over state-action pairs by introducing a sampling distribution. For a finite horizon $T$, define the averaged time-marginal distribution:

where $d_t^{\pi_q}(s)$ is the probability of being in state $s$ at time $t$ when following policy $\pi_q$ from the initial distribution. This is the uniform mixture over the time-indexed state-action distributions: we pick a timestep $t$ uniformly at random from $\{0, 1, \ldots, T\}$, then sample $(s, a)$ from the joint distribution at that timestep.

With this definition, the trajectory expectation becomes:

The factor $(T+1)$ is just a constant that does not affect the optimization. This reformulation shows that the importance-weighted surrogate is equivalent to an expectation over state-action pairs drawn from the averaged time-marginal distribution. This is not a stationary distribution or a discounted visitation distribution, but the empirical mixture induced by the finite-horizon rollout procedure.

The Clipped Surrogate Objective¶

PPO replaces the linear importance-weighted term $\rho A$ with a clipped version. For a state-action pair $(s, a)$ with advantage $A$ and importance ratio $\rho(\boldsymbol{w}) = \pi_{\boldsymbol{w}}(a|s) / \pi_{\boldsymbol{w}_{\text{old}}}(a|s)$, define the per-sample clipped objective:

where $\epsilon$ is a hyperparameter (typically 0.1 or 0.2) and $\text{clip}(x, a, b) = \max(a, \min(x, b))$ restricts $x$ to the interval $[a, b]$.

The population-level PPO objective is then:

where the expectation is taken over the averaged time-marginal distribution (53) induced by $\pi_{\boldsymbol{w}_{\text{old}}}$.

In practice, we never compute this expectation exactly. Instead, we collect a batch of transitions $\mathcal{D} = \{(s_t^{(i)}, a_t^{(i)}, A_t^{(i)})\}$ by running $\pi_{\boldsymbol{w}_{\text{old}}}$ and approximate the expectation with an empirical average:

This is the same plug-in approximation used in fitted Q-iteration: replace the unknown population distribution with the empirical distribution $\hat{P}_{\mathcal{D}}$ induced by the collected batch, then compute the sample average. The empirical surrogate $\hat{L}^{\text{CLIP}}$ is simply an expectation under $\hat{P}_{\mathcal{D}}$. No assumptions about stationarity or discounted visitation are needed. We just average over the transitions we collected.

Intuition for the Clipping Mechanism¶

The $\min$ operator in (55) selects the more pessimistic estimate. Consider the two cases:

• Positive advantage ($A > 0$): The action is better than average, so we want to increase $\pi_{\boldsymbol{w}}(a|s)$. The unclipped term $\rho A$ increases with $\rho$. The clipped term stops increasing once $\rho > 1 + \epsilon$. Taking the minimum means we get the benefit of increasing $\rho$ only up to $1 + \epsilon$.

• Negative advantage ($A < 0$): The action is worse than average, so we want to decrease $\pi_{\boldsymbol{w}}(a|s)$. The unclipped term $\rho A$ becomes less negative (improves) as $\rho$ decreases. The clipped term stops improving once $\rho < 1 - \epsilon$. Taking the minimum means we get the benefit of decreasing $\rho$ only down to $1 - \epsilon$.

In both cases, the clipping removes the incentive to move the probability ratio beyond the interval $[1-\epsilon, 1+\epsilon]$. This keeps the new policy close to the old policy without explicitly computing or constraining the KL divergence.

The algorithm collects a batch of trajectories, then performs $K$ epochs of mini-batch updates on the same data. The empirical surrogate $\hat{L}^{\text{CLIP}}$ approximates the population objective (56) using samples from the averaged time-marginal distribution. The ratio $\rho$ is computed in log-space for numerical stability. Clipping removes the local incentive to move a sampled probability ratio beyond the chosen interval. It does not guarantee that the sampled updates will find a successful policy, and it says nothing about physical constraints absent from the reward or environment.

Experiment: PPO on the SwingRL Plant¶

The SwingRL model from the modeling chapter provides a matched comparison between supplied structure and sampled policy optimization. Both controllers act on the same articulated standing rider, use the same two bounded commands, receive the same observations, and count one full unwrapped rotation as success. The structured controller receives a phase-locked pumping rule whose phases were selected by a model sweep. PPO receives trajectories and the environment reward.

The experiment asks whether the clipped policy-gradient update discovers a full rotation under one fixed, reproducible protocol. The protocol was chosen before inspecting the five final runs.

Each complete rollout batch contains 2,048 transitions, so the last update is recorded at 1,001,472 interactions. A checkpoint is the first complete update at or beyond its nominal 50,000-interaction target. Evaluation angles are uniform between $-5^\circ$ and $5^\circ$, angular velocities are uniform between -0.1 and 0.1 rad/s, and the deterministic policy uses the tanh of the Gaussian mean. The environment and the structured baseline use the same 100 states.

import sys
from pathlib import Path
from IPython.display import HTML, display

sys.path.insert(0, "code")
from swing_ppo import replay_player_html

display(HTML(replay_player_html(Path("_static/swing_ppo"))))

The replay is an illustration from one prespecified run, not the evidential comparison. Future observations remain hidden until the movie reaches their checkpoint. The five-seed result below carries the comparison across runs.

Download the recorded seed-0 replay

The final policies settle on low-motion behavior. Across the five showcase rollouts, the largest angle lies between $5.45^\circ$ and $5.62^\circ$. Return improves because the policy changes the rider while keeping effort and time penalties modest, but the behavior never approaches the $360^\circ$ success condition.

The comparison does not establish a general ranking between control and RL. The structured controller receives the oscillation phase and a controller family adapted to the mechanism, while PPO must infer useful coordination from sampled returns. Conversely, the structured controller’s nominal success requires negative suspension tension during 7.37 percent of its active steps. A rigid rod can supply that outward force; a playground chain cannot. PPO avoids the violation by barely moving, which is feasible but does not solve the task.

The two failures answer different audit questions. Optimization has not found a successful sampled policy under this protocol. The successful structured policy exposes an inadequate rigid-link model. More interactions or a revised learning objective might address the first failure. A unilateral chain model or an explicit tension constraint is required for the second.

The complete Python implementation includes policy sampling, tanh-corrected log probabilities, GAE, PPO updates, batched SwingRL evaluation, checkpoint serialization, and replay rendering. The experiment record specifies the artifact layout and reproduction command. Policy training is deliberately absent from the MyST build; the page reads the recorded checkpoints, tables, figures, and movie.

The Policy Gradient Theorem¶

How can the trajectory-level score estimator be rewritten as an expectation over discounted state visitation and action values?

The algorithms developed so far (REINFORCE, actor-critic, GAE, and PPO) all estimate policy gradients from sampled trajectories. We now establish the theoretical foundation for these estimators by deriving the policy gradient theorem in the discounted infinite-horizon setting.

Sutton et al. (1999) provided the original derivation. Here we present an alternative approach using the Implicit Function Theorem, which frames policy optimization as a bilevel problem:

subject to:

The Implicit Function Theorem states that if there is a solution to the problem $F(\mathbf{v}, \mathbf{w}) = 0$, then we can “reparameterize” our problem as $F(\mathbf{v}(\mathbf{w}), \mathbf{w})$ where $\mathbf{v}(\mathbf{w})$ is an implicit function of $\mathbf{w}$. If the Jacobian $\frac{\partial F}{\partial \mathbf{v}}$ is invertible, then:

Here we made it clear in our notation that the derivative must be evaluated at root $(\mathbf{v}(\mathbf{w}), \mathbf{w})$ of $F$. For the remaining of this derivation, we will drop this dependence to make notation more compact.

Applying this to our case with $F(\mathbf{v}, \mathbf{w}) = (\mathbf{I} - \gamma \mathbf{P}_{\pi_{\boldsymbol{w}}})\mathbf{v} - \mathbf{r}_{\pi_{\boldsymbol{w}}}$:

Then:

where we have defined the discounted state visitation distribution:

Recall the vector notation for MDPs from the infinite-horizon MDP chapter:

Taking derivatives with respect to $\mathbf{w}$ gives: