Designing Human-Centric Interfaces for Autonomous Systems
The best autonomous system is not the one that needs people least. It is the one that keeps people ready.
Four moments that explain the problem
Autonomous systems already fly our aircraft, guard our airspace and steer our cars. Most of the time they do it well. The moments that matter are the ones where a human and a machine have to share a decision under pressure. Here are four real cases, three that went wrong and one that went right.
Boeing 737 MAX, 2018 and 2019
Two airliners crashed within five months and 346 people died. A flight control function called MCAS repeatedly pushed the nose down after a faulty sensor reading. The US National Transportation Safety Board found that Boeing had assumed pilots would recognise and respond to such a failure quickly, and that those assumptions did not hold when crews faced several alerts and warnings at once.
Patriot air defence, Iraq 2003
Patriot batteries shot down two coalition aircraft, and three aircrew were lost. The US Defense Science Board found that the operating protocol was largely automatic and that operators were trained to trust the system's software, in a sky where friendly aircraft outnumbered enemy missiles by about four thousand to one. Its remedy was more operator involvement, which meant changes to software, displays and training.
Tesla Autopilot, Mountain View 2018
A driver was killed when his car, under Autopilot, steered into a motorway barrier. The NTSB concluded that the driver was likely distracted by a game on his phone and over reliant on the automation, and that the car's ineffective monitoring of driver engagement contributed to the crash.
US Airways 1549, the Hudson 2009
After a bird strike stopped both engines, the captain started the auxiliary power unit immediately, without waiting to reach that item on the engine failure checklist. The NTSB found that this kept the Airbus in its normal flight control mode with its envelope protections active, including protection against a stall. Everyone on board survived. Human judgement and automated protection worked as one team.
The pattern is the same in every case. The technology was capable. What decided the outcome was whether the human understood what the machine was doing, trusted it the right amount and could act at the right moment. That is the job of the interface, and it is why human centric design for autonomy is a safety discipline, not decoration.
Lisanne Bainbridge saw this coming in 1983 in a paper she called Ironies of Automation. The more we automate, the less practice people get, and yet we still expect them to step in during the rarest and hardest moments. With AI agents, drones and decision support tools arriving in every sector, that irony has become a design brief.
Our point of view
At Niki Shu Designs we start from a simple position. An interface for an autonomous system should be measured by how ready it keeps the human, not by how little it asks of them.
Readiness means three things. The person knows what the system is doing and why. They have a fair sense of how far to trust it right now. And they can take control smoothly when they need to. Everything else in this article follows from that idea.
"Design for the handover, not the happy path."
Trust should be calibrated, not maximised
Product teams often talk about building trust in AI as though more trust were always better. The research says otherwise. John Lee and Katrina See, in their widely cited 2004 review, argued that the goal is appropriate reliance. Too little trust and people ignore a system that would have helped them. Too much and they stop checking it when it is wrong.
This changes what a good interface looks like. A dashboard that always looks confident is quietly encouraging overtrust. A better design shows the system's confidence and its limits, and it changes visibly when conditions move outside the situations the system was built for. Uncertainty is not a weakness to hide. It is information the operator needs.
Show your interface to a new user during a scenario where the automation is wrong. If they cannot tell within a few seconds, the interface is calibrating trust upwards when it should be pulling it down.
Transparency in three layers
How much should a system explain? Jessie Chen and colleagues at the US Army Research Laboratory offer a useful answer in their Situation Awareness based Agent Transparency model, known as SAT. It builds on Mica Endsley's well known model of situational awareness, which moves from perceiving what is happening, to understanding it, to anticipating what comes next.
SAT asks an agent to share three matching layers of information. The first is what it is doing and what it is trying to achieve. The second is why, meaning the reasoning and constraints behind its choice. The third is what it expects to happen, including how uncertain it is. Studies by Chen's group and others have generally found that adding the reasoning and projection layers helps people calibrate their trust, though more detail also has a cost in attention.
That trade off is the real design work. We treat SAT as a layered disclosure pattern. Level one is always visible. Level two is one glance or one tap away. Level three appears when it matters, for example when confidence drops or a decision is irreversible.
Diagram 1. Transparency in three layers, after Chen et al. (SAT)
Make modes impossible to miss
Many automation accidents are really mode accidents. Nadine Sarter and David Woods studied pilots on highly automated flight decks and found that they were regularly surprised by what the automation did. Their 1995 paper took its title from a pilot's question, "How in the world did we ever get into that mode?". The system had changed its behaviour, and the change was technically on the display, but nobody had noticed.
The crew of Air France 447 faced a version of this in 2009. When ice blocked the airspeed probes, the autopilot disconnected and the flight controls changed mode. The French investigators at the BEA cited the lack of a clear cockpit display of the airspeed inconsistencies the computers had detected, and the crew never diagnosed the stall that followed.
Modern autonomous products repeat this mistake. A vehicle shifts from one driving mode to another. An AI agent moves from suggesting actions to taking them. A drone switches from operator guidance to its own route. If that change is shown with a small icon in the corner, people will miss it.
We design mode as a first class element. It should be shown persistently, in plain words, in a place the eye returns to. Changes in mode should be announced, and a change the human did not request should be treated as an event that needs acknowledgement.
Levels of automation are a design dial
Raja Parasuraman, Thomas Sheridan and Christopher Wickens proposed in 2000 that automation is not one setting but many. A system can automate the gathering of information, the analysis of it, the choice of a decision and the carrying out of an action, and each of these can be set to a different level. The SAE J3016 standard for driving automation, with its familiar levels from zero to five, is a product of the same thinking.
For designers this is liberating. You do not have to choose between a manual tool and a black box. You can let the machine do the heavy lifting on sensing and analysis while the human keeps the final decision. You can let it act on its own for low risk tasks and ask permission for high risk ones. The interface then becomes the place where that split is made visible and, where appropriate, adjustable.
Diagram 2. Levels of automation as a dial per stage, after Parasuraman, Sheridan and Wickens
Handover is the hardest moment
Taking back control from automation is where Bainbridge's irony bites. Driving simulator studies, including work by Christian Gold and colleagues, found that people given only a few seconds of warning before a takeover reacted faster but often less safely than those given longer. The lesson for interface design is that a handover is a process, not a button.
A good takeover sequence warns early, explains the reason in plain language, shows the current state and the most urgent hazard, and confirms clearly who is in control once the switch is complete. It also works in the other direction. When a person hands a task back to the machine, the system should confirm what it has understood the goal to be.
Diagram 3. Handover as a process
Fewer alarms, better alarms
When everything beeps, nothing does. In hospitals, the US Joint Commission issued a Sentinel Event Alert in 2013 on alarm fatigue, warning that the sheer number of device alarms was leading staff to tune them out, with harmful consequences for patients. It estimated that between 85 and 99 percent of alarm signals do not require clinical intervention. The same pattern appears in control rooms and cockpits.
Autonomous systems make this worse because they can generate an endless stream of notifications about their own state. Our rule is that every alert must earn its place. It should be prioritised, it should say what the person needs to do, and it should go away when the condition clears. Anything that does not need a human decision belongs in a log, not an alarm.
"Every alert must earn its place."
Meaningful human control is a design property
In defence the debate about autonomy is often framed in policy language. The US Department of Defense Directive 3000.09 requires that autonomous and semi autonomous weapon systems be designed to allow commanders and operators to exercise appropriate levels of human judgement over the use of force. Filippo Santoni de Sio and Jeroen van den Hoven have argued that meaningful human control depends on the system tracking human reasons and on there being a clear line of responsibility back to people who understand it.
Those are not only legal or ethical ideas. They are interface requirements. If an operator cannot see why a system recommended an action, cannot easily stop it, or cannot reconstruct what happened afterwards, then human control exists on paper only. Explanations, interruptible actions and audit trails are where principle becomes practice.
The NSD readiness checklist
Use these seven questions when you review any interface for an autonomous system.
- Mode. Can a new user say, in one glance, what the system is doing and in which mode?
- Reasoning. Can they find out why it chose that action within one step?
- Confidence. Does the interface show uncertainty, and does it change when conditions leave the system's design envelope?
- Allocation. Is it clear which tasks the machine owns and which the human owns?
- Handover. Is there a rehearsed, early and explained takeover sequence in both directions?
- Alerts. Does every alert carry a priority and a required action, with everything else sent to a log?
- Accountability. Can a person stop the system easily and reconstruct afterwards what it did and why?
Closing thought
Autonomy is getting better quickly. That makes the human more important, not less, because the moments when people are needed will be rarer, faster and harder. Teams that design only for the happy path will build systems that work beautifully until the one day they really matter. Teams that design for readiness will build systems people can trust for the right reasons.
If you are building an autonomous product and want to put these ideas into practice, we would be glad to talk.
Sources
- NTSB (2019). Assumptions used in the safety assessment process and the effects of multiple alerts and indications on pilot performance (737 MAX). ASR-19-01. source
- Defense Science Board (2005). Report of the Task Force on Patriot System Performance, report summary. source
- NTSB (2020). Collision between a sport utility vehicle operating with partial driving automation and a crash attenuator, Mountain View, California. HAR-20/01. source
- NTSB (2010). Loss of thrust in both engines after encountering a flock of birds and subsequent ditching on the Hudson River, US Airways Flight 1549. AAR-10/03. source
- BEA (2012). Final report on the accident on 1 June 2009 to the Airbus A330-203 F-GZCP, flight AF 447. source
- The Joint Commission (2013). Sentinel Event Alert 50, medical device alarm safety in hospitals. source
- Bainbridge, L. (1983). Ironies of automation. Automatica 19(6). source
- Lee, J. D. and See, K. A. (2004). Trust in automation, designing for appropriate reliance. Human Factors 46(1). source
- Chen, J. Y. C. et al. (2014). Situation awareness based agent transparency. ARL-TR-6905. source
- Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors 37(1). source
- Sarter, N. B. and Woods, D. D. (1995). How in the world did we ever get into that mode? Human Factors 37(1). source
- Parasuraman, R., Sheridan, T. B. and Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Trans. SMC Part A 30(3). source
- SAE International (2021). J3016 Taxonomy and definitions for driving automation. source
- Gold, C. et al. (2013). Take over! How long does it take to get the driver back into the loop? Proc. HFES Annual Meeting 57(1). source
- US Department of Defense (2023). Directive 3000.09, Autonomy in weapon systems. source
- Santoni de Sio, F. and van den Hoven, J. (2018). Meaningful human control over autonomous systems. Frontiers in Robotics and AI 5. source
Working on something similar? Tell us what you’re working on.