Every autonomy system is a trust relationship: the human who relies on it too much becomes the automation-bias casualty; the human who relies on it too little wastes the tool and misses its real help. Between those poles is calibration — reliance proportional to reliability — and it's not a property of the system, it's a property of the human's model of the system.
The maritime version is sharpened by stakes: a misjudged trust on a 200m vessel is not a spreadsheet error. This article is about the calibration problem and why it's the human-factors core of the whole autonomy question.
Key findings
- Trust calibration — matching reliance to the system's actual reliability — is the operative human-factors goal, not simply 'more trust'. Established
- Over-trust produces automation bias — deferring to the machine even when wrong; under-trust produces disuse — ignoring a system that would have helped. Established
- Explainability is the mechanism the literature points to: a system that shows its reasoning is one whose reliability the human can estimate. Research demonstration
- Trust is earned through calibrated failure — a system that admits its uncertainty builds more appropriate trust than one that's always confident. Established
- The failure mode is asymmetric — the first confident error after a streak of correct outputs is the dangerous one. Maritime-GPT analysis
The two failure modes
Over-trust is the documented failure: the operator who defers to the autopilot, the officer who accepts the ETA the system produced without checking. Under-trust is the quieter failure: the alert that's ignored because the last three were false alarms — the system that was right this time, disbelieved. The research literature names both; the maritime safety case needs both bounded. Established
Calibration is a property of the interface
A system that always says 'answer: X' teaches blind trust; a system that says 'answer: X, confidence: low because Y' teaches calibrated trust. The explainability work is precisely this — the display that shows its reasoning is the display that lets the human estimate when it's wrong. A Voyage Intel Pro that shows the weather-analysis uncertainty teaches trust calibrated to the evidence; one that shows only the answer teaches over-reliance. Maritime-GPT analysis
The learning curve problem
Trust calibrates through experience — but the operator's first experiences with a new system set the calibration baseline, and a system that's right 95% of the time produces a user who's calibrated for 95% reliability. The dangerous case is the 5% that arrives after the trust has been built: the confident wrong answer after a streak of right ones. The design answer is to make the system's uncertainty visible <em>before</em> the trust is earned — to show the error bars from day one, not after the incident. Maritime-GPT analysis
What calibrated trust looks like
- <strong>Uncertainty shown by default.</strong> The display includes the confidence interval, the data age, the assumption — the cues the trust is calibrated on. Research demonstration
- <strong>Failure modes trained.</strong> The operator has seen the system be wrong in simulation — the calibration happens in training, not the first casualty. Maritime-GPT analysis
- <strong>The override is cheap.</strong> The human can countermand without friction — a system that can't be questioned teaches compliance, not trust. Established
- <strong>Uncalibrated trust is the alert.</strong> A watchkeeper who hasn't checked the system in an hour isn't trusting it — they've stopped supervising it. Maritime-GPT analysis
What we know
- Trust calibration, not trust maximisation, is the correct human-factors goal.
- Explainability is the mechanism that lets the human estimate reliability.
- The first confident error after a reliable streak is the dangerous one.
What we don't
- How to measure trust calibration in a live operation, not a lab.
- Whether displayed uncertainty actually changes operator reliance in practice.
- The training that produces calibrated trust fastest.
Being tested
- Uncertainty-display interfaces for decision support.
- Simulator-based trust-calibration training.
- Reliance-behaviour metrics aboard autonomous-assist vessels.
Maritime-GPT analysis
Trust is not the AI's property — it's the interface's. A system that hides its uncertainty earns the wrong kind; one that shows it earns the right amount. The design that survives the safety review is the one that helps the human know <em>when</em> not to trust it — which means showing the confidence, the failure modes, and the edge cases from the start, not after the first confident error. Maritime-GPT analysis
This is a research feature, not operational guidance. Simulation results are reported as simulation; demonstrations are not deployments; company claims are attributed as such. Verify safety-critical decisions against official nautical publications and flag-state requirements.
Sources & evidence
The canonical automation-misuse/disuse paper — the trust literature's foundation.
The supervision context trust calibration sits in.
The explainability mechanism that enables calibrated trust.