Humanoid Locomotion as Next Token Prediction: What the Berkeley Paper Measured

The claim: control as next-token prediction

The paper casts real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Its authors are Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath and Jitendra Malik, all at UC Berkeley. It was submitted to arXiv on 29 February 2024 and published in the NeurIPS 2024 main track as a spotlight poster. [1] [2]

The robot is Agility Robotics' Digit: 1.6 m tall, 45 kg, 30 degrees of freedom of which 20 are actuated. The policy outputs joint positions. A causal transformer is trained autoregressively on sequences of sensory observations and actions, with each input token predicting the next token of its own modality, which the paper calls modality-aligned prediction. [3]

Four data sources, not all with actions

The training set mixes four kinds of trajectories. Two come from controllers in Agility's simulator; two come from human motion. Only the first contains actions that match the policy's joint-position action space. Where actions are absent, the paper substitutes a learned mask token, trained end to end, and applies no loss at those positions. This is what lets observation-only data enter the same sequence model. [3]

The paper reports that including action-free trajectories consistently leads to better performance. That is the practical result of the mask token: the motion-capture and video data improve a controller even though they carry no robot actions. [3] [4]

A small transformer

The deployed architecture has a hidden size of 192, four transformer blocks, four attention heads and a context of 16 steps, with batch normalisation on the input, layer normalisation inside, ReLU activations and a mean-squared-error loss. The scaling study trains models of 1, 2 and 8 million parameters and contexts of 16, 32 and 48 steps. The paper does not state the parameter count of the deployed model as a single number. [3]

Three trends are reported: tracking error falls with more training data, with longer context and with larger models, the last described as decreasing monotonically with model size. Across 14 trained models, the correlation between next-token prediction error and tracking error is a Pearson coefficient of 0.87. That correlation is the paper's argument that a language-model-style loss is a usable proxy for control quality. [3] [4]

What the robot did in San Francisco

The policy was deployed zero-shot on the physical Digit and walked in San Francisco over the course of one week, on walkways, concrete, asphalt, tiled plazas and dirt roads. The project page carries the video. [3] [5]

Against the authors' reinforcement-learning baseline, the paper reports that the transformer consistently outperforms it on tracking error over 245 trials; the values are plotted, not tabulated. The policy also walked backward at up to 0.5 m/s, a command absent from training. [3] [4]

What is not in the paper

• No control frequency in hertz.
• No mapping between the abstract's 27 hours and a data subset.
• No count or duration for the YouTube data.
• No released code or data; the NeurIPS checklist answers No.
• No tabulated baseline numbers; the comparison is a figure.

A reader reproducing the idea therefore has the architecture, the data recipe and the mask-token mechanism, but not the dataset, the simulator access or the training schedule. The paper is a demonstration that the formulation works on one robot, not a package to run. [4]