Causal Data Tiers for World-Action Models

A Survey of Egocentric Data, Model Design, and Evaluation

Abstract

World action models predict how environments change under action, but their accuracy depends on training data that records agent-environment causality. Third-person datasets capture state; egocentric data captures the action that causes it. This survey maps egocentric data sources for world model training. We organise existing data by causal completeness: uncurated video records outcomes without actions; laboratory telemetry links physical actions to states but at limited scale; synthetic environments generate full counterfactual trajectories. We then trace how this determines how models are constructed, and how they are designed to take advantage of the various strengths of the different data types. This is followed by an analysis on the state of the egocentric data economy, outlining the saturation of data collection businesses and highlighting the need to own dynamic data infrastructure and processing pipelines as opposed to just a raw data supplier. We conclude with a discussion on the ongoing problem of accurately evaluating the effectiveness of egocentric datasets in improving model performance.