How Robotics Teams Collect Training Data: Teleoperation, Demonstrations and Datasets

The short answer

A common way to get robot learning data is to have people show the robot what to do. An operator controls the robot through a teleoperation device while software records what the robot saw and felt, and what it was told to do, at every moment. Those recordings, called demonstrations or episodes, are checked, labelled and stored, then used to train a policy: a program that maps new observations to actions. [1] [2]

Collecting demonstrations is not the same as getting a robot that works. The trained policy still has to be tested, on its own, in conditions it was not trained on. Many policies that look good on their training data fail when an object, a camera angle or a room changes. [3] [4]

A simple picture: teaching by guiding a hand

Think of teaching a child to write a letter by holding their hand and moving the pen together. After many repetitions the child can do it alone. Robot demonstration learning works in a similar way: a person guides the robot through a task many times, and a model learns to repeat the pattern.

The recording works like a flight recorder that logs every camera frame, every joint angle and every command with a precise time. Later, the training program looks at pairs of “what the robot observed” and “what the operator did next”.

The analogy has an important limit. A child understands what a letter is for. A policy learns statistical links between what it saw and what was done. If the cup in front of it looks different from every cup in the recordings, it has no understanding to fall back on. [3]

What one demonstration contains

[2] [5] [6]

DROID is a well-documented example. Each episode includes three stereo RGB streams at 1280 × 720 recorded at 15 Hz, joint and end-effector state, gripper state, actions, camera calibration, one to three crowdsourced language instructions, the building name and the collector’s ID. [2]

The difference between observations and actions is central. Observations are what the robot measures. Actions are what it was commanded to do. A policy learns to predict actions from observations, so both must be recorded, and they must be matched to the right moment. [1]

Hardware for collecting demonstrations

[1] [5] [2] [7] [4] [8]

Each setup records different things. A leader-follower arm records robot joint angles directly. A hand-held gripper like UMI records the tool’s motion and camera view, and the robot has to be able to reproduce that motion later. The UMI authors note that the robot’s joint limits are unknown while the data is collected, so infeasible demonstrations have to be filtered out afterwards. [4] [1]

Timestamps and synchronization

Every stream runs at its own rate. In TWIST, motion capture ran at 120 Hz, retargeting and policy commands at 50 Hz, and the G1’s joint controller at 1,000 Hz. DROID recorded its cameras at 15 Hz. A recorder has to give every sample a timestamp from a common clock so that training can match an image with the joint state and the command from the same instant. [7] [2]

There is also delay. A command sent now shows up in the camera image a little later, after the motors move and the camera captures and transmits a frame. UMI handles this with inference-time latency matching, aligning observation streams and sending actions early to offset execution delay. LeRobot lets a training program ask for frames at offsets in seconds, such as 0.2 s and 0.1 s before the current frame. [4] [5]

Measuring command-to-response delay on a real robot is described in real-to-sim-to-real robotics.

How demonstrations are stored

Datasets need a format that keeps streams, rates and episode boundaries together. The Open X-Embodiment collection stores data in RLDS, a format built on serialized TFRecord files that supports different sensors and action spaces. LeRobotDataset v3 stores frame-by-frame states, actions and timestamps in Parquet files, camera video as MP4 files per camera, and the feature list, shapes and frames per second in a metadata file. Episode boundaries live in metadata rather than in file names. [9] [5]

Whatever the format, record the units, coordinate frames and camera calibration with the data. A joint angle without its unit or an image without its camera pose cannot be reused reliably by someone else. [10] [2]

Labelling and success criteria

Each episode needs at least a task description and an outcome. In DROID, collectors marked each episode as successful or not. About 16,000 unsuccessful trajectories are included in the release but are not counted in the 76,000 successful ones. Language instructions were added by crowdworkers, one to three per episode. [2]

Define success before recording, in physical terms. “Picked up the cup” is ambiguous. “Cup lifted at least 5 cm off the table and held for 2 seconds without slipping” can be checked by a person or by software. The same definition should be used later when the trained policy is evaluated, so that the two numbers mean the same thing.

Failed and interrupted attempts are worth keeping, with a clear label. They show what goes wrong, and some training and evaluation methods use them. Mixing them silently with successful demonstrations teaches the policy the mistakes. [2]

Quality checks before training

• Every stream present for the whole episode, with no dropped camera frames beyond a set limit.
• Timestamps increasing and on a common clock; stream offsets measured.
• Joint values inside their limits; no sudden jumps that indicate a sensor fault.
• Camera calibration recorded and checked; DROID found initial calibration could be inaccurate and refined it afterwards.
• Success label and task text present and consistent with the video.
• Episodes that a human interrupted or reset marked as such.
• Demonstrations that the target robot cannot reproduce removed or flagged.

[2] [4] [5]

Diversity: objects, scenes and viewpoints

A policy can only cope with variation it has seen. DROID was collected across 564 scenes in 52 buildings by 50 collectors at 13 institutions over 12 months, covering 86 tasks counted as unique verbs. Open X-Embodiment pooled more than a million trajectories from 22 robot types, and its authors report that the RT-X model trained on it improved several robots by using experience from other platforms. [2] [9]

Variation that matters includes different instances of the object, positions on the table, lighting, backgrounds, camera placement, and grasp attempts that slip or miss. Recording the same scene a thousand times usually adds less than recording a hundred scenes ten times each. [2]

Collecting, training and evaluating are three separate jobs

[1] [3]

UMI shows why evaluation conditions matter. Its cup-arrangement policy succeeded in 20 of 20 trials in the lab, and about 72% of the time with new environments and cups. Open-TeleVision reported 92% picking and 88% placement success for a trained policy on an H1, measured as separate stages over five episodes. These are autonomous results after training, not the operator’s teleoperation score. [4] [11]

[3] [12] [1]

Worked example: teaching a robot arm to pick up a cup

[1] [5] [2]

The final number is successes divided by attempts under stated conditions. Report the failures by type, such as missed grasp, slip, collision or wrong placement, because each one points to a different fix: more varied data, better calibration, or a different gripper.

Limits of teleoperation-based training

• Distribution shift: once the policy makes a small error, it sees situations no demonstration showed, and errors can compound.
• Operator style: different people solve the task differently, and inconsistent demonstrations are harder to learn from.
• Embodiment mismatch: human hands, VR controllers or hand-held tools do not move exactly like the robot.
• Missing senses: if the operator relied on touch or sound that was not recorded, the policy cannot learn it.
• Delay: lag in the teleoperation link changes how operators move, and that style ends up in the data.
• Cost: every demonstration needs a person’s time; UMI’s authors note that their tool is slower to use than bare-hand demonstration.

[13] [3] [4] [14]

How the operator’s interface affects what is recorded is covered in how humanoid robot teleoperation works.

Privacy and consent

Robot cameras record whatever is in view: the operator, other people, rooms, screens and documents. Motion capture records how a specific person moves. In many places, recordings of identifiable people are personal data under privacy law.

• Tell operators and bystanders what is recorded and why, and get consent where it is required.
• Avoid recording faces, screens and documents that the task does not need, or blur them before sharing.
• Keep a record of where each episode was collected and under what agreement.
• Check the dataset licence and the consent terms before publishing or reusing data.