StyleLight: HDR Panorama Generation for Lighting Estimation and Editing

S-Lab, Nanyang Technological University


Abstract

We present a new lighting estimation and editing framework to generate high-dynamic-range (HDR) indoor panorama lighting from a single limited field-of-view (FOV) image captured by low-dynamic-range (LDR) cameras. Existing lighting estimation methods either directly regress lighting representation parameters or decompose this problem into FOV-to-panorama and LDR-to-HDR lighting generation sub-tasks. However, due to the partial observation, the high-dynamic-range lighting, and the intrinsic ambiguity of a scene, lighting estimation remains a challenging task. To tackle this problem, we propose a coupled dual-StyleGAN panorama synthesis network (StyleLight) that integrates LDR and HDR panorama synthesis into a unified framework. The LDR and HDR panorama synthesis share a similar generator but have separate discriminators. During inference, given an LDR FOV image, we propose a focal-masked GAN inversion method to find its latent code by the LDR panorama synthesis branch and then synthesize the HDR panorama by the HDR panorama synthesis branch. StyleLight takes FOV-to-panorama and LDR-to-HDR lighting generation into a unified framework and thus greatly improves lighting estimation. Extensive experiments demonstrate that our framework achieves superior performance over state-of-the-art methods on indoor lighting estimation. Notably, StyleLight also enables intuitive lighting editing on indoor HDR panoramas, which is suitable for real-world applications.

Pipeline

HDR Panorama Generation

Relighting

Lighting Editing




Overview Video



Results of lighting estimation




Results of lighting editing




Related Links

Text2Light: Zero-Shot Text-Driven HDR Panorama Generation, TOG 2022.

CaG: Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs, Technical Report, 2022.

Relighting4D: Neural Relightable Human from Videos, ECCV 2022.

Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis, ECCV 2022.

Gardner et al. Learning to Predict Indoor Illumination from a Single Image, SIGGRAPH Asia, 2017.

Gardner et al. Deep Parametric Indoor Lighting Estimation, ICCV 2019.

Zhan et al. EMlight:Lighting Estimation via Spherical Distribution Approximation, AAAI 2021.

BibTeX

@inproceedings{wang2022stylelight,
   author    = {Wang, Guangcong and Yang, Yinuo and Loy, Chen Change and Liu, Ziwei},
   title     = {StyleLight: HDR Panorama Generation for Lighting Estimation and Editing},
   booktitle = {European Conference on Computer Vision (ECCV)},   
   year      = {2022},
  }