The Reinforcement Ladder
From dynamic programming to autonomous driving
07
Year — 2025
Categories
Five exercises, each removing an assumption of the last: exact planning with a known model, tabular learning without one, deep function approximation for continuous states, actor-critic control for continuous actions — and finally, performance engineering. The ladder ends with an enhanced DDPG agent steering a multi-lane simulated racetrack, tuned across a sweep of more than sixty trained models.
Approach
The progression is the pedagogy. Value and policy iteration solve a known Markov decision process exactly. Q-Learning and Monte Carlo abandon the model on FrozenLake’s slippery grid. DQN replaces the table with a network on CartPole and MountainCar, with a systematic comparison of exploration-decay strategies. DDPG steps into continuous action space — steering and throttle on a simulated racetrack.
The final rung is engineering: prioritised experience replay with importance-sampling correction, adaptive noise with periodic exploration bursts, orthogonal initialisation, and gradient clipping — swept across architectures and target-update rates, three seeds each.
Each exercise removes an assumption of the previous one — until the agent is driving.
Sweep
- Algorithms implemented
- 8
- Environments mastered
- 5
- Models in final sweep
- 60+
- Best mean return
- 374.86 ± 52.78
Findings
The sweep’s best configuration paired a larger critic with a compact actor and a faster target-update rate — larger critics improve value estimation, compact actors keep training stable, and higher τ propagates learned values sooner. Along the way, the comparative studies articulated why: why Monte Carlo is more discount-sensitive than Q-Learning, why DQN’s loss is non-monotonic, why linear exploration decay generalises across environments.
- University of Edinburgh, School of Informatics
Next case study
08 / 11