RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

◆Huang Huang, ◆Wensi Ai, ◆Ziyu Chen, ◆Youhui Wang, Zijian Du, Yang Liu, Jiaolong Yang, Li Fei-Fei, Jiajun Wu

RoboRender turns simulated robot trajectories into photorealistic videos while preserving geometry, robot motion, and action labels. Policies trained on these videos transfer to real-world robots without real-world fine-tuning.

A Language-Informed Loss for Visual Generation

Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen, Kyle Sargent, Juan Carlos Niebles, Li Fei-Fei, Jiajun Wu

LILIE learns an invertible image encoder through caption prediction from noisy features. The resulting language-informed loss improves text-to-image generation over pixel-only supervision.

RealWonder: Real-Time Physical Action-Conditioned Video Generation

◆Wei Liu, ◆Ziyu Chen, Zizhang Li, Yue Wang, ◇Hong-Xing Yu, ◇Jiajun Wu

RealWonder turns a single image into an interactive scene using a controllable video model. By incorporating physics, it enables physically grounded 3D interactions.

The Neverwhere Visual Parkour Benchmark Suite

Ziyu Chen, Henghui Bao, Haoran Chang, Alan Yu, Ran Choi, Kai McClennen, Gio Huh, Kevin Yang, Ri-Zhao Qiu, Yajvan Ravan, John J. Leonard, Xiaolong Wang, Phillip Isola, ◇Ge Yang, ◇Yue Wang

IROS 2026

A toolchain and benchmark suite for hyper-realistic visual locomotion, building high-fidelity digital twins of real-world environments for closed-loop evaluation.

OmniRe: Omni Urban Scene Reconstruction

Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojcic,
Sanja Fidler, Marco Pavone, Li Song, Yue Wang

ICLR 2025 (Spotlight)

OmniRe is a holistic framework for dynamic scene reconstruction that provides comprehensive coverage, including static backgrounds, driving vehicles, and non-rigidly moving dynamic actors.

360-Degree Panorama Generation from Few Unregistered NFoV Images

◆Jionghao Wang, ◆Ziyu Chen, Jun Ling, Rong Xie, Li Song

ACM Multimedia 2023

We proposed a diffusion-based pipeline that generates complete 360 panoramas using one or more unregistered NFoV images captured from arbitrary angles. This pipeline not only delivers superior quality panoramas but also offers a broader range of outcomes due to its text-conditioned generation capability.

L-Tracing: Fast Light Visibility Estimation on Neural Surfaces by Sphere Tracing

Ziyu Chen, Chenjing Ding, Jianfei Guo, Dongliang Wang, Yikang Li, Xuan Xiao, Wei Wu, Li Song

ECCV 2022

Our L-Tracing based reflectance factorization framework produces photo-realistic novel view images with nearly 10x speedup, compared with the same framework applying volumetric integration for light visibility estimation.