A Generative Model for Volume Rendering
TL;DR: This work uses the Generative Adversarial Network framework to compute a model from a large collection of volume renderings, conditioned on viewpoint and transfer functions for opacity and color, and transforms transfer functions into a view-invariant latent space specifically designed to synthesize volume-rendered images.
read more
Abstract: We present a technique to synthesize and analyze volume-rendered images using generative models. We use the Generative Adversarial Network (GAN) framework to compute a model from a large collection of volume renderings, conditioned on (1) viewpoint and (2) transfer functions for opacity and color. Our approach facilitates tasks for volume analysis that are challenging to achieve using existing rendering techniques such as ray casting or texture-based methods. We show how to guide the user in transfer function editing by quantifying expected change in the output image. Additionally, the generative model transforms transfer functions into a view-invariant latent space specifically designed to synthesize volume-rendered images. We use this space directly for rendering, enabling the user to explore the space of volume-rendered images. As our model is independent of the choice of volume rendering process, we show how to analyze volume-rendered images produced by direct and global illumination lighting, for a variety of volume datasets.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Fig. 4: The architecture for our opacity GAN. Numbers indicate the feature output dimension for fully connected layers, or the number of channels produced in convolutional layers. 1D convolutions have width 5 / stride 2, 2D convolutions in the discriminator and generator have width 4 / stride 2 and width 3 / stride 1, respectively. 
Fig. 5: The generator for the opacity-to-color translation GAN, with symbols and notation consistent with Fig. 4. Skip connections, or the concatenation of the opacity image’s 2D convolutional encodings onto the input of the color image’s decoding, help enforce spatial consistency in the synthesized color image. 
Fig. 12: We show qualitative results comparing synthesized images to ground truth volume renderings produced without illumination. The bottom row shows typical artifacts, such as incorrect color mapping and lack of detail preservation. 
Fig. 1: We cast volume rendering as training a deep generative model to synthesize images, conditioned on viewpoint and transfer function. In (a) we show images synthesized with our model, compared to a ground truth volume renderer. Our model also enables novel ways to interact with volumetric data. In (b) we show the transfer function (blue curve) augmented by a sensitivity function (red curve) that quantifies expected image change, guiding the user to only edit regions of the transfer function that are impactful on the output. In (c) we show the projection of a learned transfer function latent space that enables the user to explore the space of transfer functions. 
Fig. 6: We illustrate the computation of opacity TF sensitivity. The input parameters are pushed through the network to obtain an image, then the l2 norm of a user-specified image region is computed, and last the opacity TF gradient is obtained by backpropagation. 
Fig. 11: A user’s browsing through the projected latent TF space (bottom) can aid in their understanding of the space of opacity TFs (middle) based on the synthesized images (top).
Citations
InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations
Wenbin He,Junpeng Wang,Hanqi Guo,Ko-Chih Wang,Han-Wei Shen,Mukund Raj,Youssef S. G. Nashed,Tom Peterka +7 more
TL;DR: This work proposes InSituNet, a deep learning based surrogate model to support parameter space exploration for ensemble simulations that are visualized in situ, designed as a convolutional regression model to learn the mapping from the simulation and visualization parameters to the visualization results.
A Survey on ML4VIS: Applying MachineLearning Advances to Data Visualization.
TL;DR: In this paper, the authors systematically survey 88 ML4VIS studies, aiming to answer two motivating questions: what visualization processes can be assisted by ML and how ML techniques can be used to solve visualization problems.
99
SSR-VFD: Spatial Super-Resolution for Vector Field Data Analysis and Visualization
Li Guo,Shaojie Ye,Jun Han,Hao Zheng,Han Gao,Danny Z. Chen,Jian-Xun Wang,Chaoli Wang +7 more
- 01 Jun 2020
TL;DR: SSR-VFD is the first work that advocates a machine learning approach to generate high-resolution vector fields from low-resolution ones, and lies in the use of three separate neural nets that take the three components of a low- Resolution vector field as input and jointly output a synthesized high- resolution vector field.
73
Volumetric Isosurface Rendering with Deep Learning-Based Super-Resolution
TL;DR: A fully convolutional neural network is introduced, to learn a latent representation generating smooth, edge-aware depth andnormal fields as well as ambient occlusions from a low resolution depth and normal field, by adding a frame-to-frame motion loss into the learning stage, so upscaling can consider temporal variations and achieves improved frame- to-frame coherence.
Compressive Neural Representations of Volumetric Scalar Fields
TL;DR: In this article, a neural network maps a point in the domain to an output scalar value, where the number of weights of the neural network is smaller than the input size, thus framing compression as a type of function approximation.
51
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
•Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
- 01 Jan 2015
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
138.5K
•Posted Content
Deep Residual Learning for Image Recognition
TL;DR: This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
117.9K
ImageNet classification with deep convolutional neural networks
TL;DR: A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
•Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky,Ilya Sutskever,Geoffrey E. Hinton +2 more
- 03 Dec 2012
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.