A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

EGSR 2024 (Computer Graphics Forum)

Yohan Poirier-Ginter1,2      Alban Gauthier1      Julien Philip3      Jean-François Lalonde2      George Drettakis1     
1Inria, Université Côte d'Azur      2Université Laval      3Adobe Research     
Paper Video Results Viewer Code Datasets Group Publ. Page

Abstract

Relighting radiance fields is severely underconstrained for multi-view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single-illumination data by exploiting priors extracted from 2D image diffusion models. We first fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by light direction, allowing us to augment a single-illumination capture into a realistic – but possibly inconsistent – multi-illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low-frequency lighting, we represent appearance with a multi-layer perceptron parameterized on light direction. To enforce multi-view consistency and overcome inaccuracies we optimize a per-image auxiliary feature vector. We show results on synthetic and real multi-view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes.

Method

Our method produces relightable radiance fields directly from single-illumination multi-view dataset, by using priors from generative data in the place of an actual multi-illumination capture. It is composed of three main parts. First, we create a 2D relighting neural network with direct control of lighting direction. Second, we use this network to transform a multi-view capture with single lighting into a virtual multi-lighting capture. Finally, we create a relightable radiance field that accounts for inaccuracies in the synthesized relit input images and provides a multi-view consistent lighting solution.

Results

Since it does not rely on accurate geometry and surface normals, as compared to many prior works our method is better at handling cluttered scenes with complex geometry and reflective BRDFs. We compare to Outcast, Relightable 3D Gaussians, and TensoIR.

Ground Truth

Ours

Outcast

R3DG

TensoIR

View more results

Ablations

Radiance fields like 3DGS rely on multi-view consistency, and breaking it introduces additional floaters and holes in surfaces.

To allow the neural network to account for this inconsistency and correct accordingly, we optimize a per-image auxiliary latent vector:

Without latents

Final method


Additionally, during training all gaussian primitives that project within the view frustum of a camera but are located in front of its znear plane are culled. This pruning process, inspired by Floaters No More, removes most of the floaters:

Without pruning

Final method

Concurrent Work

The following concurrent works also propose diffusion-based relighting:

BibTeX

@article{
      10.1111:cgf.15147,
      journal = {Computer Graphics Forum},
      title = {{A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis}},
      author = {Poirier-Ginter, Yohan and Gauthier, Alban and Philip, Julien and Lalonde, Jean-François and Drettakis, George},
      year = {2024},
      publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
      ISSN = {1467-8659},
      DOI = {10.1111/cgf.15147}
    }

Acknowledgments and Funding

This research was funded by the ERC Advanced grant FUNGRAPH No 788065, supported by NSERC grant DGPIN 2020-04799 and the Digital Research Alliance Canada. The authors are grateful to Adobe and NVIDIA for generous donations, and the OPAL infrastructure from Université Côte d’Azur. Thanks to Georgios Kopanas and Frédéric Fortier-Chouinard for helpful advice.