COLREGS Rule 5 is blunt: every vessel shall maintain a proper lookout 'by sight and hearing as well as by all available means'. For two hundred years that meant a human on the wing with binoculars. For autonomous shipping it means a sensor — and the cheapest, densest sensor is the camera.
Cameras are also the sensor the ocean most enjoys breaking. Water moves. Glare saturates. Fog erases. Spray coats lenses. The datasets that maritime computer vision is trained on are a fraction the size of the ones that taught cars to drive, and the environment is less forgiving: a missed pedestrian appears once; a missed kayak at 10 knots closes at hundreds of metres per minute of combined speed.
This article reviews what the published maritime vision research actually demonstrates — the datasets, the detection results, the known failure modes — and where the honest architecture places the camera: inside a fusion stack, supervised by other sensors and ultimately by a human.
Key findings
- Maritime-specific benchmarks — SeaShips (31,455 images, six ship classes), the Singapore Maritime Dataset, MODD/MODD2 — are real and usable, but orders of magnitude smaller and narrower than terrestrial-automotive datasets. Established
- Detection quality collapses under precisely the conditions COLREGS cares about: fog, rain, darkness, glare, and small objects at long range. Research demonstration
- Background-subtraction methods that pass standard benchmarks produce 'poor recall and extremely poor precision' on real maritime video, because the water itself is a moving background. Research demonstration
- No published evidence supports a camera-only lookout meeting Rule 5 in unrestricted waters; every credible architecture fuses vision with radar and AIS. Established
- A cross-modal gap persists: a model trained on shore-surveillance video degrades sharply aboard a pitching, yawing vessel. Onboard vs onshore footage is effectively a different domain. Research demonstration
The sensor the sea fights
Terrestrial autonomous driving had one advantage shipping never will: a road is a flat, marked, structured surface. The sea is none of those. The camera's background is a moving, specular, wave-textured plane that generates phantom edges, real wakes, and glints that look like targets. The 2019 benchmark of 23 classical background-subtraction methods on the Singapore Maritime Dataset — methods that succeed on terrestrial benchmarks — produced poor recall and extremely poor precision across the board, with spurious water dynamics, wakes, and ghost effects listed as the structural causes. Research demonstration
Deep learning papered over the classical methods' weaknesses — but inherited a deeper one: data scarcity. The canonical maritime ship dataset, SeaShips, contains 31,455 annotated frames from a coastline surveillance network. Useful, and honestly collected — but shore-fixed, Chinese-coastal, daylight-weighted, and limited to six vessel classes. For comparison, the automotive benchmark nuScenes contains roughly 1.4 million annotated object instances in 3D across radar, lidar and camera. Established
| Dataset | Source | Frames | Perspective | Limitation |
|---|---|---|---|---|
| SeaShips (2018) | Shao et al., IEEE T-MM | ~31,455 images | Shore surveillance | Single coastline; 6 classes; daylight-heavy |
| Singapore Maritime Dataset (2015–) | Prasad et al., IEEE T-ITS | Multi-hour visible + NIR video | Onshore + onboard + near-infrared | Tropical Singapore waters only; limited class labels |
| MODD / MODD2 | Kristan, Bovcon (Ljubljana) | USV-mounted video | USV onboard | Small craft; limited geography |
| SeaDronesSee (2022) | Varga et al., WACV | UAV imagery | Aerial | Altitude view ≠ bridge view |
Maritime-GPT synthesis of the four most-cited maritime vision corpora. None covers a commercial-ship bridge perspective at scale. Maritime-GPT analysis
Where vision actually earns its keep
- <strong>Lookout augmentation.</strong> A camera never blinks; it tracks a target the human glanced away from. As a second pair of eyes feeding an alert layer, vision demonstrably adds coverage. Research demonstration
- <strong>Collision-course support.</strong> In clear conditions, on bridge-mounted gear, CNN detectors reliably flag approaching vessels inside a few kilometres — the traffic-density scenario COLREGS spends most pages on. Research demonstration
- <strong>Sensor-fusion truth-check.</strong> A camera gives the fusion engine something radar and AIS cannot: appearance. Two contacts, same range and bearing — radar sees a blob, the camera sees a fishing boat vs a bulk carrier. That distinction decides COLREGS manoeuvre class. Maritime-GPT analysis
- <strong>IR fill-in.</strong> Thermal and near-IR extend vision into partial darkness, but maritime IR is its own niche dataset problem. Research demonstration
The failure catalogue
A list every autonomy pitch should print in bold: Research demonstration
- <strong>Fog.</strong> Visible-band cameras degrade roughly with human vision — a sensor that sees like the human it replaces cannot satisfy Rule 5 where the human already fails.
- <strong>Sun glare and backscatter.</strong> Low sun over water saturates the image precisely where a crossing target's silhouette sits.
- <strong>Sea state.</strong> A pitching vessel rotates the horizon in the frame; horizon-detection errors cascade into false positives above the waterline.
- <strong>Small objects.</strong> A jet-ski, kayak, or floating container is a handful of pixels at collision-relevant range — the regime where detector recall collapses.
- <strong>Confusers.</strong> Birds, spray, whitecaps, moon-glint, harbour lights — the sea generates decoys faster than datasets can catalog them.
- <strong>Domain shift.</strong> A detector trained on shore video fails aboard; one trained on Singapore's waters fails in the North Sea. Ships move through domains continuously.
The Rule 5 reality check
The legal question underneath is whether a camera satisfies 'sight'. The honest answer is <em>partially, and never alone</em>. The IMO's regulatory-scoping exercise and the MASS Code both treat the lookout function as a composite: radar, AIS, vision, and IR together. No instrument published in the verified literature certifies a camera-only lookout for unrestricted waters. Established
Which is the correct engineering answer. Radar sees range and bearing through fog; AIS sees identity — when honest; vision sees class and aspect. Remove any one and the encounter classification degrades. The MASS-era sensor suite is not 'camera plus extras'; it is a fusion in which the camera supplies what it alone can — and is not asked to carry the lookout alone. Maritime-GPT analysis
What we know
- Dataset progress is real: maritime-specific corpora exist, are public, and grow.
- Visible-band detection of medium+ targets in good conditions is reliable enough to be an assistant layer.
- Fusion architectures (camera+radar+AIS) demonstrably outperform any single modality.
What we don't
- Whether any vision stack meets Rule 5 in fog/night/glare at commercial-ship speeds in unrestricted waters.
- Published false-positive rates on real bridge footage under real sea states, at scale.
- Whether the domain gap can be closed by more data or must be engineered around (augmentation, IR, fusion).
Being tested
- Onboard camera trials feeding COLREGS-classification pipelines.
- Thermal-IR fusion for night and fog.
- Synthetic-to-real transfer to expand scarce maritime data.
Maritime-GPT analysis
The takeaway is not 'vision fails' — it is 'vision alone fails'. The research supports cameras as an always-on assistant layer and a fusion component; it does not support a camera replacing the lookout. Engineers building autonomy stacks should budget the camera's share of the lookout role accordingly — high in clear traffic, near-zero in fog and glare — and spend the difference on radar, IR, and the human the code still requires. Maritime-GPT analysis
Open problems
- A large, public, bridge-height, multi-geography, all-weather dataset with ground truth. It does not exist.
- Quantified recall under fog/night at COLREGS decision ranges, not at benchmark-average ranges.
- Standardised metrics for 'what a lookout must detect' — currently no regulation-derived vision benchmark exists.
- Whether self-supervised pretraining on unlimited unlabelled sea footage closes the data gap.
This is a research feature, not operational guidance. Simulation results are reported as simulation; demonstrations are not deployments; company claims are attributed as such. Verify safety-critical decisions against official nautical publications and flag-state requirements.
Sources & evidence
The canonical survey of maritime EO vision — establishes the field's failure modes and the sensor-fusion requirement.
The largest classic maritime ship-detection benchmark — 31,455 shore-surveillance frames; the field's baseline dataset and its limits.
23 classical methods all produce poor recall/precision on real maritime video — the benchmark that made maritime its own subfield.
The main public onboard (not shore) detection dataset — the perspective a real ship needs.
The small-object-recognition frontier — person-sized targets in open water is the detection regime COLREGS actually penalises.
Establishes that the lookout function under the MASS framework is composite — no single-sensor certification path.