Open AccessPosted Content
DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
TL;DR: In this article, a differentiable sphere tracing algorithm is proposed to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function, which can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images.
read more
Abstract: We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function. Due to the nature of the implicit function, the rendering process requires tremendous function queries, which is particularly problematic when the function is represented as a neural network. We optimize both the forward and backward passes of our rendering layer to make it run efficiently with affordable memory consumption on a commodity graphics card. Our rendering method is fully differentiable such that losses can be directly computed on the rendered 2D observations, and the gradients can be propagated backwards to optimize the 3D geometry. We show that our rendering method can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images, through inverse optimization. With the geometry based reasoning, our 3D shape prediction methods show excellent generalization capability and robustness against various noises.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Figure 1. Illustration of our proposed differentiable renderer for continuous signed distance function. Our method enables geometric reasoning with strong generalization capability. With a random shape code z0 initialized in the learned shape space, we can acquire high-quality 3D shape prediction by performing iterative optimization with various 2D supervisions. 
Figure 8. Our method can render information encoded in the implict function other than depth. With a pre-trained network encoding textured meshes, we can render high resolution color images under various resolution, camera viewpoints, and illumination. 
Figure 6. Illustration of the optimization process over the camera extrinsic parameters. Our differentiable renderer is able to propagate the error from the image plane to the camera. Top row: rendered surface normal. Bottom row: error map on the silhouette. 
Figure 7. Effects on choices of different convergence thresholds. Under the same marching step, a very large threshold can incur dilation around boundaries while a small threshold may lead to erosion. We pick 5× 10−5 for all of our experiments. ![Figure 2. Illustration on the sphere tracing algorithm [13]. A ray is initiated at each pixel and marching along the viewing direction. The front end moves with a step size equals to the signed distance value of the current location. The algorithm converges when the current absolute SDF is smaller than a threshold, which indicates that the surface has been found.](/figures/figure2-1-5pd2kzd3elb9.png)
Figure 2. Illustration on the sphere tracing algorithm [13]. A ray is initiated at each pixel and marching along the viewing direction. The front end moves with a step size equals to the signed distance value of the current location. The algorithm converges when the current absolute SDF is smaller than a threshold, which indicates that the surface has been found. 
Table 3. Quantitative results on 3D shape prediction from multiview images under the metric of Chamfer Distance. We randomly picked 50 instances from the PMO test set to perform the evaluation. 10000 points are sampled from meshes for evaluation.
![Figure 2. Illustration on the sphere tracing algorithm [13]. A ray is initiated at each pixel and marching along the viewing direction. The front end moves with a step size equals to the signed distance value of the current location. The algorithm converges when the current absolute SDF is smaller than a threshold, which indicates that the surface has been found.](/figures/figure2-1-5pd2kzd3elb9.webp)
Citations
•Posted Content
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Matthew Tancik,Pratul P. Srinivasan,Ben Mildenhall,Sara Fridovich-Keil,Nithin Raghavan,Utkarsh Singhal,Ravi Ramamoorthi,Jonathan T. Barron,Ren Ng +8 more
TL;DR: An approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities is suggested.
2K
IBRNet: Learning Multi-View Image-Based Rendering
Qianqian Wang,Zhicheng Wang,Kyle Genova,Pratul P. Srinivasan,Howard Zhou,Jonathan T. Barron,Ricardo Martin-Brualla,Noah Snavely,Thomas Funkhouser +8 more
- 20 Jun 2021
TL;DR: A method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views using a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations.
•Posted Content
Neural Sparse Voxel Fields
TL;DR: This work introduces Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering that is over 10 times faster than the state-of-the-art (namely, NeRF) at inference time while achieving higher quality results.
825
•Posted Content
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
Michael Niemeyer,Andreas Geiger +1 more
TL;DR: The key hypothesis is that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis and a fast and realistic image synthesis model is proposed.
•Posted Content
Convolutional Occupancy Networks
TL;DR: Convolutional Occupancy Networks is proposed, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes that enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.
668
References
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
R. Qi Charles,Hao Su,Mo Kaichun,Leonidas J. Guibas +3 more
- 21 Jul 2017
TL;DR: This paper designs a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input and provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing.
•Posted Content
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
TL;DR: A hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set and proposes novel set learning layers to adaptively combine features from multiple scales to learn deep point set features efficiently and robustly.
•Proceedings Article
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
Charles R. Qi,Li Yi,Hao Su,Leonidas J. Guibas +3 more
- 07 Jun 2017
TL;DR: PointNet++ as discussed by the authors applies PointNet recursively on a nested partitioning of the input point set to learn local features with increasing contextual scales, and proposes novel set learning layers to adaptively combine features from multiple scales.
3D ShapeNets: A deep representation for volumetric shapes
Zhirong Wu,Shuran Song,Aditya Khosla,Fisher Yu,Linguang Zhang,Xiaoou Tang,Jianxiong Xiao +6 more
- 07 Jun 2015
TL;DR: This work proposes to represent a geometric 3D shape as a probability distribution of binary variables on a 3D voxel grid, using a Convolutional Deep Belief Network, and shows that this 3D deep representation enables significant performance improvement over the-state-of-the-arts in a variety of tasks.
•Posted Content
ShapeNet: An Information-Rich 3D Model Repository
Angel X. Chang,Thomas Funkhouser,Leonidas J. Guibas,Pat Hanrahan,Qixing Huang,Zimo Li,Silvio Savarese,Manolis Savva,Shuran Song,Hao Su,Jianxiong Xiao,Li Yi,Fisher Yu +12 more
TL;DR: ShapeNet contains 3D models from a multitude of semantic categories and organizes them under the WordNet taxonomy, a collection of datasets providing many semantic annotations for each 3D model such as consistent rigid alignments, parts and bilateral symmetry planes, physical sizes, keywords, as well as other planned annotations.