How Do Humanoid Robots Talk and Follow Spoken Instructions?

The short answer

A talking robot chains several separate systems. Microphones capture sound. Speech recognition turns the sound into a best-guess text. A language component works out what was asked and writes a reply. Text-to-speech turns the reply into audio for a loudspeaker. None of those steps moves a joint.

Following a spoken instruction needs a second path. The request has to be turned into something the robot’s perception, planning and control software can execute, and then the result has to be checked in the physical world. A robot that says “I will bring the cup” has produced a sentence. Bringing the cup is a different, separately testable outcome. [1]

A simple picture: the waiter and the kitchen

Think of a restaurant. The waiter listens to your order, writes it on a ticket and tells you it is on its way. The kitchen reads the ticket and cooks. Listening, writing and speaking are the robot’s speech system. Cooking is its body, controlled by other software.

The comparison has a limit, and the limit is the point. In a restaurant, the same person who promised the dish can usually check the kitchen. In a robot, the reply and the action can come from different programs that do not automatically agree. The waiter can say “coming right up” while the kitchen has no ingredients. A well-designed robot reports the outcome only after a check has confirmed it.

Hearing: microphones and the noise around a robot

Many robots carry several microphones rather than one. Pollen Robotics lists four PDM microphones on a Seeed reSpeaker XMOS XVF3800 board in Reachy Mini, sampling at up to 16 kHz, together with a 5 W speaker. Unitree lists a four-microphone array and a speaker for the standard R1. These are manufacturer hardware descriptions, not measurements of how well either robot hears. [2] [3]

An array helps because sound reaches each microphone at a slightly different time. Software can use those differences to estimate where a voice comes from and to favour sound from that direction, a family of methods called beamforming. A robot is also a noisy place to listen: motors, gearboxes and cooling fans run while it moves, and its own loudspeaker can feed back into its microphones. Echo cancellation subtracts the robot’s own voice from what the microphones record.

Before recognition, software has to decide when someone is speaking at all. This is voice activity detection. The Whisper speech model, for example, predicts a no-speech probability for each segment, and its authors report that combining a no-speech threshold with a confidence threshold made that detection more reliable. Getting this wrong cuts off the start of a request or sends background noise to the recognizer. [4]

Speech recognition: from sound to a text guess

Automatic speech recognition, or ASR, converts audio into text. Whisper is a well-documented example. Its authors resample audio to 16,000 samples per second, convert it into an 80-channel log-Mel spectrogram using 25 ms windows every 10 ms, and process it in 30-second segments with an encoder-decoder Transformer. The models were trained on 680,000 hours of audio paired with transcripts. [4]

Two terms need defining. A spectrogram is a picture of sound: time runs along one axis, pitch bands along the other, and brightness shows how much energy each band holds at each moment. The Mel scale spaces those bands roughly the way human hearing separates pitches. An encoder-decoder model first turns the whole spectrogram into an internal representation, then writes the text one token at a time. [4]

The output is the model’s most likely text, not a certain record of what was said. The Whisper paper reports that all tested models degrade as background noise becomes more intense, with Whisper doing better than models trained only on clean audiobook speech under pub noise. It also lists errors that a person would rarely make: getting stuck repeating a phrase, skipping the first or last words of a segment, and occasionally writing a transcript unrelated to the audio. [4]

Robots use different recognizers, and some do not run on the robot at all. The UCLA PhoneBot paper describes speech-to-text, language-model and text-to-speech features as backend calls over Wi-Fi, separate from the walking loop that runs on the phone. Where recognition happens decides what fails when the network drops. [5]

Interpretation: from text to a request the robot can act on

A transcript such as “put the red cup in the sink” still has to be turned into parts the robot can use: an action (put), an object (the red cup), a destination (the sink) and any conditions. Words like “that one” or “over there” cannot be resolved from text alone. They need the camera view, and sometimes where the person is pointing or looking. When two objects match, a careful system asks instead of guessing. [1]

Language models connect to robot actions in a few documented ways. They differ in what the model outputs, and therefore in what still has to check that output. [6] [7] [8]

[6] [7] [8]

Each approach is limited by what lies underneath it. The Code as Policies authors write that their method is restricted by what the perception functions can describe and by a handful of control primitives, that it struggles with long or complex commands, and that it assumes every instruction is feasible. A model cannot pick up an object that the vision system cannot find. [7]

The full chain from an instruction to a placed object is traced in how humanoid robots make decisions. That article follows a bottle through perception, planning, control and verification.

Speaking back: text-to-speech

Text-to-speech, or TTS, runs the recognition problem in reverse. Tacotron 2 is a documented two-stage design: a sequence-to-sequence network predicts a mel spectrogram from the input characters, and a modified WaveNet model, called a vocoder, turns that spectrogram into an audio waveform. In the authors’ listening test, people rated its speech 4.53 on a five-point scale, against 4.58 for professional recordings. [9]

Newer systems use different architectures, but the lesson holds: a natural-sounding voice measures the speech synthesizer. It says nothing about whether the robot understood the request or can do the task it is describing. [9]

Talking is not doing

Generating a response and executing a physical action are different operations with different evidence. A reply is finished when the audio has played. An action is finished when the object is where it should be, which the robot can only know by looking or measuring again.

SayCan shows the gap with numbers. Across 101 instructions in a real office kitchen, PaLM-SayCan produced a correct plan 81% of the time, but the robot executed the task successfully 60% of the time. Human raters judged the plan and the execution separately. A system that announced success whenever it had a plan would have overstated its results. [6]

This is also why a spoken claim in a demonstration video needs checking. A robot can talk on its own while a person moves its arms through teleoperation. The voice and the motion are separate systems, and the voice does not reveal who chose the motion.

The operator’s side of that arrangement is described in how humanoid robot teleoperation works.

Follow one spoken request

[1] [4]

Every step adds delay. The total time from the end of the sentence to the first motion depends on where recognition runs, how large the language model is and how the robot plans. Measure it on the robot rather than adding up figures from different papers.

Can a humanoid robot hold a conversation?

If a large language model is connected, a robot can produce fluent conversation. Holding a useful conversation in a room is harder. People interrupt each other, talk over the robot, refer back to things said minutes ago and point at objects. The robot has to decide who is speaking, when its turn begins, what “it” or “that” refers to, and whether a remark is a request at all.

Fluent text also hides recognition errors. In a 2023 study of Whisper’s API on recordings from AphasiaBank, about 1% of transcriptions contained whole phrases or sentences that the speaker never said, and 38% of those hallucinations contained content the authors classed as harmful. Hallucinations were associated with longer pauses, which were more common among speakers with aphasia. The result describes that service version and dataset, but it shows why a robot should not treat every transcript as fact. [10]

Talking about the scene is a further step. A language model can describe a kitchen fluently from its training data alone. Whether the robot actually perceives the kitchen in front of it depends on its cameras and perception software, not on how well it speaks.

Where voice control fails

[4] [10] [5]

The safety point deserves emphasis. Voice is a convenient way to ask for a task. It should not be the only way to stop one, because the same noise and recognition errors that garble a request can garble the word “stop”.

Questions to ask about a talking robot

• Does speech recognition run on the robot or in the cloud, and what happens offline?
• Was the spoken request scripted for the demonstration or chosen freely?
• Who chose the arm and leg motions: the robot’s software or a remote operator?
• How does the robot respond when it mishears or when two objects match?
• Does it report success after a check, or as soon as it has a plan?
• How many attempts, retries and human interventions were there?

For the wider question of which decisions a robot makes by itself, read are humanoid robots autonomous?.

Vision-language-action models are explained separately in vision-language-action models explained.