OmniRe:

Omni Urban Scene Reconstruction

Ziyu Chen*,¶, Jiawei Yang, Jiahui Huang§, Riccardo de Lutio§,
Janick Martinez Esturo§, Boris Ivanovic§, Or Litany†,§, Zan Gojcic§,
Sanja Fidler‡,§, Marco Pavone$,§, Li Song*, Yue Wang§,¶
* Shanghai Jiao Tong University † Technion ‡ University of Toronto
$ Stanford University § NVIDIA Research University of Southern California
arXiv DriveStudio

System-level 3DGS Framework for Urban Scenes Recon&Sim!





Abstract

We introduce OmniRe, a holistic approach for efficiently reconstructing high-fidelity dynamic urban scenes from on-device logs. Recent methods for modeling driving sequences using neural radiance fields or Gaussian Splatting have demonstrated the potential of reconstructing challenging dynamic scenes, but often overlook pedestrians and other non-vehicle dynamic actors, hindering a complete pipeline for dynamic urban scene reconstruction. To that end, we propose a comprehensive 3DGS framework for driving scenes, named OmniRe, that allows for accurate, full-length reconstruction of diverse dynamic objects in a driving log. OmniRe builds dynamic neural scene graphs based on Gaussian representations and constructs multiple local canonical spaces that model various dynamic actors, including vehicles, pedestrians, and cyclists, among many others. This capability is unmatched by existing methods. OmniRe allows us to holistically reconstruct different objects present in the scene, subsequently enabling the simulation of reconstructed scenarios with all actors participating in real-time (~60Hz). Extensive evaluations on the Waymo dataset show that our approach outperforms prior state-of-the-art methods quantitatively and qualitatively by a large margin. We believe our work fills a critical gap in driving reconstruction.

method overview

We build a dynamic Gaussian scene graph leveraging the collective strengths of different representations:

  1. Vehicles are modeled as static Gaussians, transformed using rigid body transformations to simulate their motion.
  2. Close-range walking pedestrians are fitted with a template-based SMPL model, enabling joint-level control.
  3. Far-range and other template-less dynamic actors are reconstructed using self-supervised deformation fields.



Comparison with Baselines

More Applications

Let People Dance!

Super Slow Motion

10x slow motion by interpolating 10 frames between adjacent frames.

Pedestrians Dreamer

Replacing our reconstructed pedestrians with any generated characters (e.g. GART).

Scene Simulation

OmniRe enables diverse simulation including scene editing, traffic and human behavior simulation.

BibTeX

@article{chen2024omnire,
  author    = {Ziyu Chen and Jiawei Yang and Jiahui Huang and Riccardo de Lutio and Janick Martinez Esturo and Boris Ivanovic and Or Litany and Zan Gojcic and Sanja Fidler and Marco Pavone and Li Song and Yue Wang},
  title     = {OmniRe: Omni Urban Scene Reconstruction},
  journal   = {arXiv preprint arXiv:2408.16760},
  year      = {2024}
}

Acknowledgements

We appreciate Jiageng Mao, Junjie Ye, and Ziyi Yang for their helpful discussions.