Most physical AI teams know they need data. Few know they need a stack of it. The capabilities a deployed humanoid, AV, or warehouse robot needs — perception, action, instruction following, multi-step workflow execution — each map to a different layer of training data, with different collection methods, annotation depth, and quality controls. The physical AI dataset stack is a way to think about those layers as one integrated system rather than four disconnected procurement decisions.
Key Takeaways
- The physical AI dataset stack has four layers tied to four real-world capabilities.
- Layer 1 covers human activity and demonstration data for perception and understanding.
- Layer 2 captures robot manipulation data for repeatable task execution.
- Layer 3 aligns vision, language, and action for instruction following at scale.
- Layer 4 supports long-horizon, multi-step task completion in real environments.
- Each layer feeds the next; weaknesses below propagate up the stack.
Why think about physical AI data as a stack?
Physical AI data behaves as a stack because each capability layer depends on the layers beneath it. Perception data without action data produces a model that sees but cannot move. Action data without language alignment produces a model that moves but cannot follow instructions. Long-horizon workflow data without robust instruction following collapses on the first multi-step task.
NVIDIA’s open physical AI dataset, released to the developer community, comprises thousands of hours of multicamera video at unprecedented diversity (NVIDIA, 2025), and even at that scale, downstream teams still need their own task-specific layers above it. Pre-training data is necessary, not sufficient.
Layer 1: What does human understanding data cover?
Human understanding data is human activity and demonstration data — first-person and third-person footage of humans doing tasks in real environments. It teaches the model what the world looks like and how humans move through it.
Human demonstration data: Video and sensor recordings of humans performing tasks, with annotations that align observations to actions, intents, or outcomes.
This layer feeds perception, scene understanding, and intent inference. Quality questions to ask:
- Does the data cover the environments your robot will operate in?
- Are demonstrations annotated at the atomic-action level, or just per-clip?
- Is participant consent documented and traceable?
Shaip’s L1 data collection layer captures real-world activity across kitchens, factories, warehouses, healthcare facilities, and roads — environments that match deployment contexts rather than lab conditions.
Layer 2: What does task execution data cover?
Task execution data is robot manipulation data — trajectories, joint states, object interactions, and contact dynamics for repeatable physical tasks. It teaches the model how to act, not just what to perceive.
Robot manipulation data: Time-stamped sequences of robot states, end-effector poses, and object interactions, captured during teleoperation, scripted execution, or demonstration replay.
This is where embodiment-specific structure shows up. Joint configurations, gripper geometries, and action spaces vary across robots, so manipulation data is rarely portable across embodiments without retargeting. Cross-embodiment efforts — such as datasets unifying 22 robot embodiments under one action schema (DeepMind/Stanford et al., 2024) — have made this slightly easier, but task-specific manipulation data remains a hands-on collection program.
Layer 3: What does VLA data add?
VLA data adds language alignment to vision and action — every episode carries a natural-language instruction tied to the trajectory that fulfills it.
Vision-Language-Action (VLA) data: Episode-level training data containing synchronized visual observations, natural-language instructions, and action trajectories with success labels.
This layer is what enables instruction following. Without it, a manipulation model can execute one trained task; with it, the same backbone can generalize across hundreds of instructions. The catch: language descriptions need to be atomic, specific, and aligned with actual action boundaries — not vague summaries. Annotation precision at this layer determines whether a fine-tuned VLA generalizes to new prompts or memorizes the training set.
Layer 4: What does long-horizon task data cover?
Long-horizon task data covers multi-step workflows — sequences where the robot must complete one sub-task to start the next. Cooking a meal, sorting a warehouse pallet, and assembling a kit are long-horizon tasks. Each requires the model to track state, recover from sub-task failure, and chain skills.
A research dataset focused on long-horizon tabletop manipulation comprised 200 episodes across 20 multi-step tasks with cluttered scenes (LHManip authors, arXiv, 2024) — small in scale but tightly structured. Production teams typically build evaluation sets with hundreds to thousands of long-horizon episodes, plus exception-handling traces for failure recovery.