For years we designed assistants that could only listen and speak, or only read and write. Today a single agent can hear your voice, read what you type, look at a photo you show it through the camera and display information on a screen. That’s what we call a multimodal agent: one that combines several channels, such as voice, text, images and screens. The technology already makes this possible. The design question is which channel to use at each moment.
Models that see, hear and speak
In December 2023, Google introduced Gemini, a family of models designed from the start to work with text, images, audio and video together (Gemini Team, 2023). In May 2024, OpenAI introduced GPT-4o, which can reason across voice, images and text in real time (OpenAI, 2024). And in October 2024, Anthropic gave Claude the ability to use a computer: look at the screen, move the cursor, click and type (Anthropic, 2024).
With that, an agent can take in information through any channel and respond through any other. That opens up a lot of possibilities, and a lot of ways to get it wrong.
What we already knew about multimodal design
Multimodal interaction isn’t new in research. In 1999, Sharon Oviatt published a classic paper with ten myths about it that are still useful (Oviatt, 1999). Three of them in particular:
- A multimodal system doesn’t mean people will combine channels. Often they use just one, whichever is most convenient at that moment.
- Channels tend to complement each other more than repeat each other. A person says “put this here” while pointing: the voice provides the action, and the gesture, the place.
- The main advantage isn’t speed. Having several channels mostly helps avoid and correct errors, because what one channel doesn’t catch, the other can resolve.
Then there’s the context of use. As we explain in what to ask yourself before building an assistant, every place frees up or ties up your hands, eyes, ears and voice. Good multimodal design lets people use whichever channel they have free.
Which channel for what
There are no fixed rules, but a few patterns keep coming up in our projects:
- Voice works well for quick requests, especially when your hands or eyes are busy, and for short answers. It works poorly for long lists, comparisons or details you need to remember.
- A screen works well for showing options, comparing, reviewing a summary before confirming and showing what you can do. And for things you’d rather not say out loud in front of other people.
- Images or the camera work well for showing instead of describing: a broken part, a label, a document, an error on another screen.
- Written text works well for exact details that are hard to catch by ear, like an email address, a code or a street address.
We saw this clearly in our project with Sanitas. Patients control the lights, the air conditioning and the TV by voice from their bed, but when the device isn’t in use, its screen shows example phrases that teach people what they can ask for (Sanitas case study). Voice is for acting, and the screen is for discovering.
The new challenges
Privacy
A camera, or an agent that looks at the screen, sees far more than it needs to: other people, documents, notifications. You have to decide what gets captured, when, and what gets stored, and tell people clearly.
Consistency across channels
If the agent says one thing and the screen shows another, people don’t know which to believe. Voice and screen have to tell the same story, each in its own way. Nor should the agent read aloud everything it shows, or show everything it says.
Vision errors
A model can misread an image with the same confidence it misreads a text. If the agent is going to act on what it sees, it should confirm what it understood before doing anything important.
Accessibility
Multiple channels help more people: someone who can’t see the screen can use their voice, and someone who can’t speak can tap or type. But only if every important task can be completed through more than one channel. A design that forces you to look at the screen at a key step leaves someone out.
How to get started
Map out the person’s journey step by step and, at each moment, ask yourself where they are, what they have free and what they need: to act, to choose, to review or to show something. Pick the main channel for that moment and a backup. And test it in the real place, with real noise, light and hurry, because in the office everything seems to work.
Designing experiences that combine voice, text, images and screens is part of our work in conversation and agent design and in applied research and prototypes.
References
