Open the maritime collision-avoidance literature and you'll find a familiar fight: discrete-action methods (DQN and descendants) vs continuous-action methods (PPO, SAC, and cousins). The papers treat the algorithm choice as the central variable. The honest reading of the results is that it isn't — the reward function is.
This is a useful article precisely because the literature is noisy: understanding why the algorithm debate is secondary clarifies what actually determines whether an RL agent navigates safely.
Key findings
- DQN-family agents dominate early maritime RL because discrete rudder/heading actions map cleanly to COLREGS manoeuvre categories. Established
- PPO/SAC handle continuous control better but add instability and tuning burden; neither produces a decisive safety advantage. Established
- Reward shaping — how collision proximity, rule-compliance and smoothness are weighted — is the dominant performance determinant, not the algorithm. Research demonstration
- Both families produce COLREGS-plausible behaviour in simulation; neither produces demonstrably compliant behaviour in the field. Established
- The real frontier is not algorithm choice but robustness: partial observability, sensor noise, multi-ship interaction — the dimensions the benchmarks underserve. Maritime-GPT analysis
Why the algorithms compare the way they do
DQN's discrete actions — port, starboard, steady, slow — match COLREGS' discrete manoeuvre vocabulary, which is why it was the first and remains the most-cited approach. PPO's continuous control is more expressive (any rudder angle, any speed) but needs reward shaping that maps continuous action to rule-compliant behaviour — which is where the papers diverge into reward engineering. The algorithm chooses the action space; the reward chooses the behaviour; and the behaviour is what COLREGS judges. Established
What the comparisons actually show
| Axis | DQN family | PPO family | Verdict |
|---|---|---|---|
| Action space | Discrete — clean COLREGS mapping | Continuous — smoother control | Situational |
| Training stability | Q-learning instability known | On-policy, more stable | PPO edge |
| Sample efficiency | Higher — discrete is simpler | Lower — more samples needed | DQN edge |
| COLREGS compliance | Encodable via discrete rewards | Needs shaped continuous reward | Depends on reward, not algo |
| Multi-ship scaling | Struggles — fixed encounters | Better — but still hard | Neither solves it |
The reward is the real research object
Every credible comparison finds the same thing: tune the reward wrong and either algorithm produces a ship that violates COLREGS; tune it right and both produce broadly rule-compliant paths — with residual edge cases the reward didn't cover. The reward function is where the rules are actually encoded: collision proximity penalties, stand-on/give-way asymmetry, smoothness, safety margins. The algorithm is the delivery mechanism for a policy the reward defined. That's the honest centre of the literature. Maritime-GPT analysis
What actually matters more than the choice
- <strong>Partial observability.</strong> Real ships see noisy, incomplete pictures — an algorithm evaluated on full-state sims is a different problem than one seeing the real sensor feed. Established
- <strong>Multi-ship interaction.</strong> Two-ship evaluation misses the cases where the decision matters most. Established
- <strong>Non-compliant targets.</strong> Fishing boats, leisure craft, vessels that don't follow the rules — the distribution the training assumes is the one the sea violates. Maritime-GPT analysis
- <strong>Verification.</strong> A policy that scores well in simulation is not a policy that's provably safe — the assurance gap is the same regardless of algorithm. Established
What we know
- DQN's discrete actions fit COLREGS manoeuvre categories cleanly.
- PPO's continuous control is more expressive but reward-dependent.
- Reward design is the determinant of rule-compliance, not the algorithm.
What we don't
- A published comparison on a shared, realistic, restricted-visibility benchmark.
- Whether either family scales to real multi-ship, noisy, non-compliant traffic.
- The reward structure that produces certifiably compliant behaviour.
Being tested
- Reward learning / inverse RL from real ship tracks.
- Partially-observable (POMDP) formulations.
- Hybrid RL with rule-constraint layers; how to verify compliance remains open.
Maritime-GPT analysis
The DQN-vs-PPO question is the wrong question dressed as the right one. The literature's real finding — which the papers state but the summaries don't — is that the reward function is where the safety lives, and the algorithm is interchangeable plumbing. A lab that argues about the algorithm is optimising the wrong axis; the one that can state its reward's failure modes is the one worth listening to. Maritime-GPT analysis
This is a research feature, not operational guidance. Simulation results are reported as simulation; demonstrations are not deployments; company claims are attributed as such. Verify safety-critical decisions against official nautical publications and flag-state requirements.
Sources & evidence
The recent survey cataloguing the DQN/PPO/hybrid landscape.
The DQN paper — the discrete-action family the maritime work descends from.
The wider RL landscape this comparison sits in.