Overview of the ESRP task. Given a top-down target layout, an embodied agent needs to rearrange furniture in a 3D scene from an initial configuration (red) to a target state (green). The agent operates under realistic constraints, relying only on egocentric observations without access to global state information or ground-truth localization.
This paper introduces Embodied Scene Rearrangement Planning (ESRP), a task in which an embodied agent rearranges furniture to match a target configuration using egocentric observations and a top-down layout image. The agent has no access to the global state, and furniture can occlude other objects. It must therefore connect partial local views to a global target over a long sequence of actions. We built ESRP-Bench on OmniGibson with more than 5,400 scene pairs and 8,100 objects. The benchmark includes three evaluation metrics and four baselines: a task-and-motion planner, a vision-language-model-based method, and imitation-learning and reinforcement-learning agents. All four methods have difficulty completing the task efficiently, particularly in scenes with several objects. The results point to scene understanding and long-horizon planning as the main open problems.
The policy must align partial, first-person views with the target layout and infer spatial information solely from local perception.
The agent must generate plausible action sequences while reasoning about inter-object dependencies to avoid physical deadlocks and mutual occlusions.
Every moved object changes the scene, so the agent must update its perception, localization, and navigation plan as it works.
Initial-target scene layout pairs. We group layouts into three difficulty levels: easy (1 object to rearrange), medium (2 to 3 objects), and hard (4 to 6 objects). ESRP-Bench uses 3D-FRONT scenes simulated in OmniGibson. It contains more than 5,400 scenes and 8,100 movable objects from several room and furniture categories.
Pipeline overview. ESRP-Bench contains the dataset and simulation environment. We evaluate four baselines: ESRP-BC, a diffusion policy trained on expert demonstrations; ESRP-PPO, a reinforcement-learning agent; ESRP-VLM, which prompts a pretrained vision-language model in a ReAct loop; and ESRP-PLAN, a hierarchical task-and-motion planner with access to the global state.
| Baseline | SR ↑ | OSR ↑ | RDR ↓ |
|---|---|---|---|
| ESRP-PPO | 20.18% | 20.35% | 0.9845 |
| ESRP-BC | 19.09% | 20.31% | 1.0170 |
| ESRP-VLM | 17.64% | 18.97% | 0.9830 |
| ESRP-PLAN | 30.20% | 36.59% | 0.6916 |
The strongest baseline has access to the global state but reaches only a 30.20% success rate. Long-horizon planning remains difficult when the learning-based agents receive only egocentric observations.
Successful rearrangements. The red curve shows the agent’s trajectory; the blue curve shows the moved object’s trajectory.
Failure cases. As the number of rearrangeable objects increases, long-horizon complexity makes the task substantially harder; the agent can only approximate the target layout.
@article{chen2026esrp,
title = {Embodied Scene Rearrangement Planning},
author = {Chen, Canzhi and Wang, Zan and Zhu, Siqi and Wu, Qi and Li, Yixuan and Liang, Wei},
journal = {IEEE Robotics and Automation Letters},
year = {2026},
publisher = {IEEE}
}