Improving NeRF Quality by Progressive Camera Placement for Free-Viewpoint Navigation

Vision, Modeling, and Visualization (VMV) 2023


Georgios Kopanas1,2      George Drettakis1,2
1Inria      2Université Côte d'Azur     
1 2
Paper Code Results (40GB)

We present a new method that proposes the next best camera placement for NeRF capture (left). We introduce two metrics that can be easily computed, observation frequency and angular uniformity (middle). On the right, we show that our approach outperforms two baseline camera placement strategies, "Hemisphere" which is the typical approach used in most NeRF methods and "Random", as well as ActiveNeRF a recent related work.

Abstract

Neural Radiance Fields, or NeRFs, have drastically improved novel view synthesis and 3D reconstruction for rendering. NeRFs achieve impressive results on object-centric reconstructions, but the quality of novel view synthesis with free-viewpoint navigation in complex environments (rooms, houses, etc) is often problematic. While algorithmic improvements play an important role in the resulting quality of novel view synthesis, in this work, we show that because optimizing a NeRF is inherently a data-driven process, good quality data play a fundamental role in the final quality of the reconstruction. As a consequence, it is critical to choose the data samples – in this case the cameras – in a way that will eventually allow the optimization to converge to a solution that allows free-viewpoint navigation with good quality. Our main contribution is an algorithm that efficiently proposes new camera placements that improve visual quality with minimal assumptions. Our solution can be used with any NeRF model and outperforms baselines and similar work.

Evaluation

We used 5 synthetic scenes modeled by professional artists to represent realistic indoor environments. For each scene we construct a training set corresponding to each one of the algorithms we want to evaluate and multiple test sets that provide a good overview of the total quality throughout the scene. Our test-sets contain a total 150 views that are distinct from the training views. The test-sets are split in 3 sub-sets: 1) 50 random views using the "Hemisphere" capture style 2) 50 views using "Random" caputre style and 3) 50 views using our sampling process. The purpose of the multiple test sets is to evaluate each algorithm fairly throughout different camera distributions such that the quantitative metric evaluate the total quality throughout the scene. This avoids bias towards one of the aforementioned distributions, and allows a more comprehensive overall evaluation of our algorithm

Visual Comparisons

Ours
Random
Ours
ActiveNeRF
Ours
Hemisphere
Ours
Ground Truth

BibTeX

@inproceedings {10.2312:vmv.20231222,
		booktitle = {Vision, Modeling, and Visualization},
		editor = {Guthe, Michael and Grosch, Thorsten},
		title = {{Improving NeRF Quality by Progressive Camera Placement for Free-Viewpoint Navigation}},
		author = {Kopanas, Georgios and Drettakis, George},
		year = {2023},
		publisher = {The Eurographics Association},
		ISBN = {978-3-03868-232-5},
		DOI = {10.2312/vmv.20231222}
}

Acknowledgments and Funding

This research was funded by the ERC Advanced grant FUNGRAPH No 788065. The authors are grateful to Adobe for generous donations, the OPAL infrastructure from Université Côte d’Azur and for the HPC resources from GENCI–IDRIS (Grant 2022-AD011013409). The authors also thank the anonymous reviewers for their valuable feedback.