PPO — Proximal Policy Optimization Algorithms

01 — Every symbol, explained

The vocabulary first

Nothing later on this page uses a symbol that is not defined here. Click any of them — each one gets a plain-English explanation, a precise one, and a small drawing of what it does.

\pi_\theta(a \mid s)

the policy

in plain words

The thing being trained. Show it a situation and it gives you a probability for every action it could take — not one answer, a spread of preferences.

precisely

A parameterised conditional distribution over actions given a state. In the continuous-control experiments it outputs the mean of a Gaussian, with a separate learnable standard deviation.

why it is there

Everything in policy-gradient methods is built on this being a distribution you can differentiate, rather than a lookup table or an argmax.

where it appears

1 of 16 — use the arrows, or click a symbol above.

If you read only one of these, read the advantage. Almost every difficulty in policy-gradient methods comes from the fact that Â is a noisy estimate of something you cannot observe, computed from a policy you have already changed.

02 — The problem

Experience is expensive, and reusing it is dangerous

A vanilla policy gradient does one update per sample and then throws it away. Reusing the batch would be far cheaper — but the moment the policy moves, the data no longer describes the policy that generated it.

vanilla policy gradient

Collect experience, take exactly one step, discard everything.

naively reusing the batch

Ten steps on the same data with the plain objective. Each step makes the data a worse description of the current policy.

the estimator everyone starts from

\hat g = \hat{\mathbb{E}}_t\big[\nabla_\theta \log \pi_\theta(a_t \mid s_t)\,\hat A_t\big]

Read it right to left. Â says whether the action beat expectations. ∇ log π says which way to move the weights to make that action more likely. Multiply, average over the batch, step uphill.

In practice you do not code the gradient — you construct an objective whose derivative is this, and let autodiff do the rest:

L^{PG}(\theta) = \hat{\mathbb{E}}_t\big[\log \pi_\theta(a_t \mid s_t)\,\hat A_t\big]

Why reuse breaks it

The objective above scores actions using probabilities from the current policy against advantages measured under the old one. After one step that mismatch is small. After ten it is not, and nothing in the objective notices or cares — so it happily keeps pushing a direction that stopped being valid several steps ago.

what TRPO did about it

\max_\theta\ \hat{\mathbb{E}}_t\!\left[\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}\hat A_t\right] \ \text{ s.t. } \ \hat{\mathbb{E}}_t\big[\mathrm{KL}\big] \le \delta

A hard constraint on how far the policy may move, solved with conjugate gradients after a linear approximation to the objective and a quadratic one to the constraint. It works — but it is complicated, and incompatible with dropout or with sharing parameters between the policy and the value function.

03 — The idea

Remove the reward for overshooting

PPO does not forbid a large step. It makes one pointless. Past a band around the old policy the objective goes flat, so gradient ascent simply stops pulling in that direction.

step one

Score the new policy with old data

r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)}

At the start of the update every ratio is exactly 1, because θ = θ_old. As you optimise, actions the new policy likes more drift above 1 and actions it likes less drift below.

step two

The obvious objective, and its problem

L^{\mathrm{CPI}}(\theta) = \hat{\mathbb{E}}_t\big[r_t(\theta)\hat A_t\big]

Perfectly sensible for a small step. But nothing bounds r, so maximising this drives good actions' probabilities toward 1 and bad ones toward 0 in a single update — on the strength of a noisy advantage estimate.

step three

Clip it, then take the minimum

\min\big(r\hat A,\; \mathrm{clip}(r, 1-\epsilon, 1+\epsilon)\hat A\big)

Clipping flattens the objective outside the band. The minimum makes the result a lower bound on the unclipped version — so the change in ratio is ignored only when ignoring it would flatter the update, and included whenever it makes the update look worse.

Figure 1, made playable — drag the ratio, flip the sign of the advantage

r = 1.00

r·Â — unclippedclip(r)·ÂL^CLIP = min of the two

probability ratio r1.000

advantage Â1.0

r · Â1.000

clip(r) · Â1.000

L^CLIP = min1.000

gradient here

alive

You are inside the band, or on the side where the min keeps the unclipped term. The gradient points somewhere, so the update does something.

note the asymmetry

For a good action the curve flattens on the right; for a bad one it flattens on the left. The min is what puts the flat part on the side where flattening protects you, and leaves the slope alive on the side where the update is a correction rather than a runaway.

L^{\mathrm{CLIP}}(\theta) = \hat{\mathbb{E}}_t\Big[\min\big(r_t(\theta)\hat A_t,\; \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\,\hat A_t\big)\Big]

The part that is easy to miss. Clipping alone is not enough. If you clipped without the minimum, an update that moved the ratio the wrong way could still register as an improvement, because the clip would hide how bad it had become. The min is what makes L^CLIP a pessimistic bound rather than merely a bounded one.

first-order equivalence

L^CLIP equals L^CPI to first order around θ_old — at r = 1 they have the same value and the same slope. They only diverge as the policy actually moves, which is exactly when you want them to. That is why PPO can behave like an unconstrained method locally while behaving like a trust region globally.

04 — The other half

Where the advantage comes from

The clipped objective decides how far to move. The advantage decides which way. Getting it from a finite slice of experience is its own problem, and the answer has one dial.

what an advantage is

A(s, a) = Q(s, a) - V(s)

Q(s,a) is how well things go if you take this action here. V(s) is how well they go on average from here. The difference is the only thing that should move a policy: not “was this good”, but “was this better than what I usually do”.

Subtracting V is what makes the estimate low-variance. Without it, every action taken in a promising state looks good, and the policy learns the state rather than the choice.

the trouble with a fixed-length slice

\hat A_t = -V(s_t) + r_t + \gamma r_{t+1} + \cdots + \gamma^{T-t+1}r_{T-1} + \gamma^{T-t}V(s_T)

The run stops after T steps, so you have real rewards up to the cut and then nothing. The value function is spliced in to stand for everything after it. That works, but it inherits every mistake the critic makes about the tail.

Generalised advantage estimation generalises this, and recovers it exactly at λ = 1.

Move λ and watch the estimate change character

λ = 0.95

per-step TD error δ_tresulting advantage Â_t

spread of Â across the segment

0.261

higher means a noisier learning signal

what you are choosing

The middle. Enough real reward to correct the critic, enough critic to damp the noise. The paper uses 0.95 everywhere.

try this

Turn the critic noisy, then slide λ down. At low λ the advantage follows the critic's mistakes closely; at high λ the real rewards wash them out.

the estimator

\hat A_t = \delta_t + (\gamma\lambda)\delta_{t+1} + \cdots + (\gamma\lambda)^{T-t+1}\delta_{T-1}

\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)

what δ actually is

One step of surprise. You expected V(s_t); you got the reward r_t plus whatever the next state is worth. The gap is how much better or worse that step went than the critic predicted. Generalised advantage estimation is nothing more than an exponentially weighted sum of those surprises, with λ setting how far the sum reaches.

The bias–variance trade, in one sentence

Trusting the critic gives you a smooth but possibly wrong signal; trusting the observed rewards gives you an honest but noisy one. λ says how much of each, and 0.95 is the answer PPO uses in every experiment in the paper.

05 — Putting it together

Three terms, one loss

When the policy and value function share a network you cannot optimise them separately, so everything is folded into a single expression. Select a term.

L^{\mathrm{CLIP}+VF+S}_t(\theta) = \hat{\mathbb{E}}_t\Big[\;\underbrace{L^{\mathrm{CLIP}}_t(\theta)}_{\text{policy}}\; - \;c_1\underbrace{L^{VF}_t(\theta)}_{\text{critic}}\; + \;c_2\underbrace{S[\pi_\theta](s_t)}_{\text{exploration}}\Big]

what it does

Push up the probability of actions that beat expectations, push down the ones that did not — but stop caring once the ratio leaves its band.

worth knowing

This is the term the whole paper is about. Everything else here is standard practice bolted on around it.

Note the signs. The expression is maximised. The policy term and the entropy bonus are things you want more of, so they carry a plus. The value loss is an error you want less of, so it is subtracted. Every deep-RL implementation flips this into a minimised loss by negating the whole thing, which is why the same equation appears in code with all the signs reversed.

06 — The algorithm

Eight lines, actor-critic style

This is Algorithm 1 in full. The paper is explicit that it is only a few lines different from a vanilla policy gradient implementation — the change is in the objective, not the loop.

Algorithm 1 · PPO, actor-critic style

line 1

The outer loop. Each iteration is: collect a batch, then squeeze several epochs of learning out of it.

what is active

MuJoCo · 1M timesteps

Table 3. Policy is an MLP, two hidden layers of 64 units, tanh. No parameter sharing with the value function, so c₁ is irrelevant; no entropy bonus.

Horizon T	2048
Adam stepsize	3 × 10⁻⁴
Num. epochs	10
Minibatch size	64
Discount γ	0.99
GAE λ	0.95

07 — The ablation

Every way of constraining the step, compared

Seven MuJoCo environments, three seeds each, one million timesteps, scores normalised so a random policy is 0 and the best result is 1 — then averaged over all 21 runs into a single number per setting.

No clipping or penalty

worse than a random policy on half-cheetah

-0.39

Clipping, ε = 0.1

0.76

Clipping, ε = 0.2

0.82

Clipping, ε = 0.3

0.70

Adaptive KL, d_targ = 0.003

0.68

Adaptive KL, d_targ = 0.01

0.74

Adaptive KL, d_targ = 0.03

0.71

Fixed KL, β = 0.3

0.62

Fixed KL, β = 1

0.71

Fixed KL, β = 3

0.72

Fixed KL, β = 10

0.69

average normalised score · Table 1 · β initialised at 1

No constraint at all

Nothing stops the ratio, the policy takes a destructive step, and on one environment it ends up worse than where it started. This row is why the rest of the paper exists.

Clipped surrogate

The winner, and not very sensitive: 0.1 and 0.3 both work, 0.2 works best. Clipping is a single hyperparameter with an intuitive meaning — how far the action probability may move.

Adaptive KL penalty

The coefficient β is adjusted after each update to hit a target divergence — halved if the policy moved too little, doubled if too much. Respectable, but consistently behind clipping.

Fixed KL penalty

A constant β. The paper's argument against it is that no single value works across problems, or even across the course of one problem as the characteristics change.

Read the first row again. With nothing constraining the update the average score is −0.39 — worse than not training at all, because on half-cheetah the policy is driven somewhere catastrophic. Every other row is an attempt to prevent that, and the cheapest attempt is also the best one.

08 — Results

Where it lands

Two domains, two different stories. On continuous control PPO simply wins. On Atari it learns faster than the alternatives, and finishes roughly level with a much more complicated method.

Atari · 49 games, three trials each

Number of games each algorithm wins under two different scoring rules. The rules disagree, and the paper reports both rather than picking the flattering one.

average reward over all of trainingfavours fast learning

PPO 30

ACER 18

A2C 1

average reward over the last 100 episodesfavours final performance

PPO 19

ACER 28

A2C 1

tie 1

PPO dominates on learning speed and ACER edges ahead on final score. ACER is a substantially more involved algorithm — off-policy corrections, a replay buffer, a trust region in the policy's output space. PPO gets close to it with an objective you can write in one line.

MuJoCo · one million timesteps

Compared against tuned implementations of five methods, across seven environments. PPO comes out ahead on almost all of them.

HalfCheetahHopperInvertedDoublePendulumInvertedPendulumReacherSwimmerWalker2d

versus

A2CA2C + trust regionCEMVanilla PG (adaptive)TRPO

the humanoid showcase

Three Roboschool tasks with a 3D humanoid: run forward; run toward a target that jumps to a new position every 200 steps; and the same again while being pelted with cubes, so the robot has to get up off the ground. Trained with the adaptive-stepsize variant across 32 or 128 parallel actors.

why this paper stuck

Not because it is the most accurate method on any single benchmark, but because it is the one you can implement correctly from memory, it tolerates shared networks and dropout, and it does not need a second-order solver. It became the default for most of what followed — including the reinforcement learning used to fine-tune large language models.

The biggest stepyou can takewithout falling off