Our method is divided into two main steps. First, we sample a random vector \(\mathbf{z}_\textrm{obj}\in \mathbb{R}^{512}\) and generate a volumetric representation of triplane features. During training, we also sample a camera from the estimated poses of the training data, and apply our camera adjustment with latent \(\mathbf{z}_\textrm{cam}\in \mathbb{R}^{512}\). At inference, the camera is set by the user and the adjustment is not applied. To render from a specific camera viewpoint, we cast a ray for each pixel and sample positions along the ray to compute their corresponding triplane features, which are decoded into base color and density using a lightweight decoder MLP. The base color and density are volume-rendered. We also compute the surface normals by backpropagating the density through the decoder w.r.t to the position of the sample. Second, we sample an environment map from our dataset and shade our car representation using the efficient physically-based approximation of a microfacet BRDF. We render the images at \(128\times 128\) for efficiency (and memory constraints during training) and then apply a super-resolution network to reach \(256\times 256\). Green denotes trainable neural networks and blue fixed functions.

Our method enables interactive physically-based rendering (PBR) with 3D GANs under arbitrary illumination conditions in the form of environment maps and unconstrained camera navigation, including 360◦ rotations. We achieve this with three main contributions: a method to estimate poses of a dataset of car images, a generative pipeline for PBR, and an improved generative network architecture and training solution. We only require a dataset of images of the desired object class (cars in our case) and a dataset of environment maps for training. Our method learns a disentangled representation of shading, enabling relighting with high-frequency reflections on shiny car bodies.