Point-Based Neural Rendering with Per-View Optimization

Georgios Kopanas1,2 Julien Philip1,2,3 Thomas Leimkühler1,2 George Drettakis1,2
1Inria 2Université Côte d'Azur 3Adobe Research
1 2 3
Paper Supplemental Presentation Code Group Publication Page
Comparisons Ablations Other

Our differentiable point-based pipeline allows optimization of attributes such as reprojected features or depth in each input view.
We achieve state-of-the-art image-based rendering, multi-view stylization and multi-view harmonization.

Abstract

There has recently been great interest in neural rendering methods. Some approaches use 3D geometry reconstructed with Multi-View Stereo (MVS) but cannot recover from the errors of this process, while others directly learn a volumetric neural representation, but suffer from expensive training and inference.

We introduce a general approach that is initialized with MVS, but allows further optimization of scene properties in the space of input views, including depth and reprojected features, resulting in improved novel-view synthesis. A key element of our approach is our new differentiable point-based pipeline, based on bi-directional Elliptical Weighted Average splatting, a probabilistic depth test and effective camera selection. We use these elements together in our neural renderer, that outperforms all previous methods both in quality and speed in almost all scenes we tested. Our pipeline can be applied to multi-view harmonization and stylization in addition to novel-view synthesis.

Video

Comparisons

We compare our IBR method with previous IBR and recent neural rendering methods; the most meaningful comparisons are qualitative visual inspections of the videos. We also provide quantitative comparisons in the paper and an ablation study.

Ours
Textured Mesh
Ours
Deep Blending [Hedman 2018]
Ours
Stable View Synthesis [Riegler 2021]
Ours
ULR [Buehler 2001]
Ours
Free View Synthesis [Riegler 2020]
Ours
Nerf++ [Zhang 2020]

Multi-View Style Transfer


For multi-view style-transfer we leverage the differentiability of our point-based reprojection to jointly optimize each input image to match a given style, while maintaining photo-consistency. We base our stylization method on the approach of [Mechrez 2018].

Harmonization

Rendered with ULR [Buehler 2001]

We demonstrate a multi-view harmonization technique that works well for one of the most common problems in real-world multi-view captured content: Exposure and other camera parameters can fluctuate between images, creating multi-view inconsistencies breaking basic IBR algorithm assumptions. We multiply the color of each image with an optimized brightness coefficient that controls it’s total brightness and allows our pipeline to optimize for it during training.

BibTeX

@Article{KPLD21,
      author       = "Kopanas, Georgios and Philip, Julien and Leimkühler, Thomas and Drettakis, George",
      title        = "Point-Based Neural Rendering with Per-View Optimization",
      journal      = "Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering)",
      number       = "4",
      volume       = "40",
      month        = "June",
      year         = "2021",
      url          = "http://www-sop.inria.fr/reves/Basilic/2021/KPLD21"}

Acknowledgments and Funding

This research was funded by the ERC Advanced grant FUNGRAPH No 788065. The authors are grateful to the OPAL infrastructure from Université Côte d'Azur for providing resources and support. The authors thank G. Riegler for help with comparisons. Thanks to A. Bousseau for proofreading earlier drafts, E. Yu for help with the figures and S.Diolatzis for thoughtful discussions throughout. Finally, the authors thank the anonymous reviewers for their valuable feedback.

References

[Buehler 2001] Buehler, Chris, et al. "Unstructured lumigraph rendering." Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 2001.

[Hedman 2018] Hedman, Peter, et al. "Deep blending for free-viewpoint image-based rendering." ACM Transactions on Graphics (TOG) 37.6 (2018): 1-15.

[Mechrez 2018] Mechrez, Roey, Itamar Talmi, and Lihi Zelnik-Manor. "The contextual loss for image transformation with non-aligned data." Proceedings of the European conference on computer vision (ECCV). 2018.

[Riegler 2020] Riegler, Gernot, and Vladlen Koltun. "Free view synthesis." European Conference on Computer Vision. Springer, Cham, 2020.

[Zhang 2020] Zhang, Kai, et al. "Nerf++: Analyzing and improving neural radiance fields." arXiv preprint arXiv:2010.07492 (2020).

[Riegler 2021] Riegler, Gernot, and Vladlen Koltun. "Stable view synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.