No single maritime sensor sees everything — radar reaches far and works in the dark and fog but can't read a hull's detail; the camera resolves the detail but fails in the dark and weather. The fusion — combining the two into a single track — is the perception architecture that covers each sensor's blindness with the other's strength.
Radar-camera fusion is the workhorse of the multimodal maritime stack — the pairing that powers the situational awareness both autonomous ships and surveillance systems depend on.
Key findings
- Radar provides range, all-weather operation, and tracking; camera provides classification, identification, and detail. Established
- The fusion levels — early (raw data), mid (features), late (tracks) — trade off information richness vs engineering simplicity. Established
- The result is a single track with both range and identity — the contact that's 'a vessel at 3nm' <em>and</em> 'a fishing vessel heading north'. Research demonstration
- The fusion handles the failures — radar loses the small target; the camera is blind in fog; each covers the other. Established
- The calibration problem is the engineering core — aligning the sensors' frames so the radar contact and the camera image refer to the same object. Established
The complementary senses
Radar and camera are complementary in every dimension that matters: radar works at range and in darkness but returns only a position; the camera reads the detail — the hull colour, the light pattern, the type — but only when it can see. Fused, the system produces a contact that's both located and identified: the radar track carries the range and motion, the camera adds the classification and the visual confirmation. The pair doesn't just add capabilities; it covers each other's failure modes. Established
The fusion levels
- <strong>Early fusion.</strong> Raw radar returns overlaid on the image — rich information, hardest to engineer. Established
- <strong>Mid fusion.</strong> Radar detections matched to camera detections — the track-to-object association. Established
- <strong>Late fusion.</strong> Each sensor produces a track; a fusion layer combines them — simplest, most robust to sensor failure. Research demonstration
The calibration is the hard part
The fusion's engineering problem is association — the radar's polar coordinates must map to the camera's image coordinates so both refer to the same object, and the mapping shifts with sea state, mounting, and motion. A mis-calibrated fusion produces 'the radar contact and the camera object are different things' — a false split — or 'the radar track matched the wrong detection' — a false merge. The alignment isn't a footnote; it's the fusion's correctness. Established
What the fusion enables
The fused track powers the downstream decisions — collision avoidance that knows both where the target is and what it is, surveillance that flags the unidentified contact, the COLREGS reasoning that needs the vessel's type. The camera-only system sees the hull but not the range; the radar-only system sees the range but not the hull; the fusion is the complete picture the decision needs. Established
What we know
- Radar gives range/all-weather; camera gives class/detail — the pair is complementary.
- Fusion level is a design trade-off between richness and engineering simplicity.
- Calibration (radar-to-camera alignment) is the correctness-critical component.
What we don't
- Optimal fusion level for different vessel types and environments.
- How fusion holds up under adversarial inputs to either sensor.
- The certification standard for fused-sensor perception.
Being tested
- Radar-camera fusion on live bridge systems.
- Fusion robustness under sensor degradation.
- Track-association accuracy in dense traffic.
Maritime-GPT analysis
Radar-camera fusion is the maritime stack's proof that the multimodal answer is the right one — not because fusion is fashionable, but because the sensors' blindness is complementary. Radar fails on the small vessel, the camera fails in the fog, and the fusion is the coverage neither has alone. The engineering core is the calibration — the alignment that makes the two sensors' outputs refer to the same world — and when it drifts, the fusion produces a confident wrong answer. Maritime-GPT analysis
This is a research feature, not operational guidance. Simulation results are reported as simulation; demonstrations are not deployments; company claims are attributed as such. Verify safety-critical decisions against official nautical publications and flag-state requirements.