Reinforcement learning learns by reward. Give a ship-shaped agent a reward for reaching its destination, a penalty for nearing another vessel, and a penalty for breaking the rules, run a million simulated encounters, and a policy emerges that steers like a navigator — without ever being told how. That's the promise. The whole research problem lives in the second sentence: a penalty for breaking the rules. Maritime-GPT analysis
COLREGS isn't a cost function. "Safe speed," "early and substantial action," "the ordinary practice of seamen" — the rules deliberately leave room for judgement, because the drafters knew the sea invents cases no rulebook enumerates. Compressing that into reward terms is where every DRL paper makes its real contribution — and its compromises. Maritime-GPT analysis
Key findings
- Multiple 2024–25 studies report COLREGS-consistent avoidance in simulation across DQN, PPO and hybrid variants — the approach is real, not theoretical. Research demonstration
- An enhanced DQN reports ~31% lower collision rates than multi-agent DRL baselines and only ~12.6% degradation in dense traffic — high-density robustness is the active frontier. Research demonstration
- Compliance is engineered into the reward, not learned — "COLREGS-compliant RL" means the designers encoded the rules as reward terms, which is exactly where interpretation risk hides. Maritime-GPT analysis
- Evaluation environments differ wildly — gridworlds, Imazu-type scenarios, realistic ship dynamics with wind/current — so cross-paper numbers aren't comparable. Established
- Everything is simulation. No published DRL collision-avoidance policy has been validated in real traffic. Established
How the reward encodes the rules
Every credible DRL collision-avoidance paper is, at heart, a reward-engineering paper. The Frontiers PPO work is representative: encounter-situation awareness feeds a multi-objective reward balancing collision risk, navigational efficiency, path smoothness and destination progress — with COLREGS shaping the risk terms so rule-violating manoeuvres cost the agent. Research demonstration The Scientific Reports enhanced-DQN variant adds dueling/double-Q architecture and reports dense-traffic robustness as its headline. Research demonstration
The subtlety: "COLREGS-compliant" here means the agent satisfies the reward's encoding of COLREGS — typically stand-on/give-way classification, safe-distance and starboard-turn terms. Rule 2's judgement calls, not-under-command situations, and the obligation to depart from the rules when collision is imminent don't fit cleanly into scalar reward and are usually out of scope. A policy can be perfectly compliant with its reward function and still wrong at sea. Maritime-GPT analysis
DQN, PPO, hybrids — the trade space
| Approach | Evidence | Strength | Weakness |
|---|---|---|---|
| DQN / enhanced DQN | Research demonstration | Discrete actions, stable training, strong dense-traffic results | Coarse steering; discrete action set limits manoeuvre |
| PPO (continuous) | Research demonstration | Smooth continuous control, cooperative multi-agent training | Sample-hungry; reward hacking on edge cases |
| Hybrids (DEDRL, BC + DRL) | Research demonstration | Optimizer refinement on top of learned policy; real wind/current in Imazu tests | More components to verify; still simulation-bound |
The continuous-vs-discrete choice matters at sea: a real helm takes any course and speed, so discrete-action DQN is a simplification; continuous PPO is realistic but harder to train stably. The hybrid line — a learned policy refined by classical optimisation, or behaviour cloning seeded from human tracks — points at the more interesting question: not "can RL do it" but "what's the best split of learned and engineered." Maritime-GPT analysis
Where sim-to-sea breaks
The gap between the papers and the ocean is consistent and specific: Maritime-GPT analysis
- Idealised targets. Simulated ships follow rules and constant dynamics; real targets err, drift, misread, and fish. Policies trained on compliant targets fail on non-compliant ones.
- Clean sensors. Simulators give perfect CPA/TCPA; the sea gives radar noise, AIS gaps and latency — a policy trained on truth performs worse on estimates.
- Single-ship focus. Multi-vessel encounters — where most real risk lives — are the least-tested setting, though dense-traffic work is closing it.
- The Imazu problem. Many studies evaluate on the same canonical scenario sets; beating the benchmark is not the same as being safe.
The most honest studies — the Imazu-environment DRL work including wind and current — explicitly push against this, but even they are validation in a better simulator, not at sea. The field's own reviews repeatedly flag the missing step: no replay against real tracked encounters, no at-sea trials. Established
What we know / don't / being tested
What we know
- DRL can learn rule-shaped avoidance in simulation across several architectures.
- Dense-traffic robustness is improving — sub-13% degradation in the best studies.
- Reward design is the load-bearing part of the result.
What we don't know
- Whether reward-encoded COLREGS matches human/legal interpretation in edge cases.
- How policies behave against non-compliant, non-standard, or deceptive targets.
- How to certify a learned policy — it can't be inspected like code.
Being tested
- Richer dynamics — wind, current, under-actuation in training environments.
- Multi-agent cooperative policies — ships that reason about each other's policies.
- Shielded RL — a deterministic safety layer over the learned policy.
Maritime-GPT analysis
RL will reach the bridge as a shield, not a pilot. The defensible deployment is a learned policy inside hard constraints — proposing manoeuvres that a deterministic safety gate and a human can veto — exactly the layered pattern the LLM work converged on. The open question that decides the timeline isn't learning quality, it's verification: until someone can certify what a trained policy won't do, no flag State will let it steer. Maritime-GPT analysis
Open research problems
- Policy verification — formal bounds on a trained network's behaviour across the input space.
- Non-compliant-target robustness — encounters with targets that break rules or sensor assumptions.
- Real-encounter replay benchmarks — evaluating policies against actual AIS-tracked traffic, not canonical scenarios.
- Reward vs rule alignment — encoding "depart from the rules when required" without teaching the policy to cheat.
Evidence scorecard
| Dimension | Rating | Basis |
|---|---|---|
| Evidence strength | Moderate | Consistent simulation results across independent teams; no sea trials. |
| Research maturity | Experimental | Policies in scenario suites; deployment unproven. |
| Maritime deployment | None verified | No published operational RL avoidance system aboard. |
| Confidence in conclusions | High | Results quoted per-paper; sim-vs-sea gap is itself sourced. |
Related tools & reading
- Multi-agent collision avoidance
- Can an LLM be trusted with COLREGS?
- Can AI explain its navigation decisions?
- The MASS Code
Sources
- COLREGs-compliant ship collision avoidance strategy based on proximal policy optimization — Frontiers in Marine Science (2025)Peer-reviewed
PPO with encounter-awareness mechanisms and a multi-objective reward balancing risk, efficiency, smoothness and destination — multi-ship compliance in simulation. - Deep reinforcement learning model for multi-ship collision avoidance decision making — Scientific Reports (2025), s41598-025-05636-3Peer-reviewed
Enhanced DQN (dueling + double Q): reports ~31% lower collision rate than multi-agent DRL baselines, ~12.6% degradation in dense traffic vs 18–45% for baselines. - Optimizing multi-vessel collision avoidance decision-making for ASVs: a COLREGs-compliant DRL approach — J. Marine Science and Engineering 12(3):372Peer-reviewed
COLREGS-shaped reward in continuous action space; validated in the enhanced Imazu environment including wind and current — closer to real dynamics than gridworld tests. - Differential evolution deep reinforcement learning for dynamic multiship collision avoidance with COLREGs compliance — J. Marine Science and Engineering 13(3):596Peer-reviewed
DEDRL — DQN for planning plus differential evolution refinement and a quaternion ship domain for risk; shows the hybrid-optimizer direction.
All figures are the cited papers' simulation results under their stated test conditions. Accessed 5 October 2026. Editorial analysis, not navigation advice.