arXiv preprint ·
Physically Based Rendering in the Latent Space
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1111/cgf.70633
- Original
- Read it at arXiv
- Topics
Abstract
Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Graphics
FootprintRAG: Visual Analytics for Evidence Context Refinement in RAG-based Scientific Literature Exploration
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
LYRIC: Language-Driven Physics-Based Character Control for Contact-Rich Whole-Body Object Interaction
LINGO: Latent Initialization and Gradient Optimization for Sparse-view X-ray Novel View Synthesis and CT Reconstruction with 3D Gaussian Splatting
Printing the Underdetermined: Materializing Multi-solutionness in Figurative Paintings
Get the few papers worth your time in Graphics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.