NeuralSlice

NeuralSlice: Neural 3D Triangle Mesh Reconstruction via Slicing 4D Tetrahedral Meshes

Accepted by Proceedings of the 40th International Conference on Machine Learning (ICML 2023)

Our method is inspired by an intriging geometry property, which is different slices of a higher dimensional shape may have different topologies in these slices.

Figure 1: Our work is based on slicing a 4D tetrahedral mesh with a 4D hyperplane, which can be difficult to imagine. To help with understanding, we illustrate a lower-dimensional situation, i.e., slicing a 3D surface with a 3D plane, which bears similar principles. (Left) slicing a cone surface to obtain different conic section curves (hyperbola, circle, ellipse, and parabola). Although the 3D surface is fixed, the 2D intersection curves can have different topologies. (Right) a more complex example that deforms a sphere homomorphically to a cow. Although the 3D surface does not change topology, the 2D sliced curves have changed topology during deformation. We use different colors to indicate slices with different planes.

Learning-based high-fidelity reconstruction of 3D shapes with varying topology is a fundamental problem in computer vision and computer graphics. Recent advances in learning 3D shapes using explicit and implicit representations have achieved impressive results in 3D modeling. However, the template-based explicit representation is limited by fixed topology, and the implicit representation, although flexible with arbitrary topology, requires a large number of sampled points to regress the surface, which is computationally expensive. In this work, we propose a novel 3D shape representation named NeuralSlice, which represents a 3D shape as the intersection of a 4D tetrahedral mesh and a 4D hyperplane. A novel network is designed to incorporate the proposed representation flexibly, which learns a deformable 4D template and a parameter for slicing 4D hyperplane to reconstruct the 3D object. To learn the local deformation of the 4D template, we further propose a spatial-aware network to locate the 4D points within the 3D feature volume of input shape via positional encoding, which leverages the local geometrical feature to guide the 4D deformation. By addressing the 3D problem in a higher 4D space, our method supports flexible topology changes while being highly efficient. Our method is guaranteed to produce manifold meshes. NeuralSlice outperforms the state-of-the-art explicit-based approaches in terms of reconstruction quality. Compared with implicit approaches, by avoiding point sampling, our method is 10 times faster than the implicit approaches, and better preserves thin structures. NeuralSlice has the capability of representing various shapes and topologies using a single 4D tetrahedral mesh. The corresponding code can be found on GitHub at https://github.com/IGLICT/NEURALSLICE

NeuralSlice: Neural 3D Triangle Mesh Reconstruction via Slicing 4D Tetrahedral Meshes

TL;DR: NeuralSlice is a novel approach to representing 3D shapes with flexible topology in a pure explicit fashion, which represents a 3D shape as the intersection of a 4D deformable tetrahedral mesh and a learnable 4D hyperplane.

Construct 4D Tetrahedral Meshes

Figure 2: We map a 3D tetrahedral mesh into a higher 4D space to construct 4D tetrahedral mesh.

Slice 4D Tetrahedral Meshes

Figure 3: There are two possible situation when a 4D tetrahedron sliced by a hyper-plane: one vertex above the hyper-plane and three vertices below the hyper-plane; two vertices above the hyper-plane and two vertices below the hyper-plane(Left). Both these situation need to compute the intersection between the edge of the Tetrahedron and the hyper-plane(Right).

Network Archtecture

Figure 4: Network Architecture: The pipeline of 3D mesh reconstruction via slicing a deformed 4D template tetrahedral mesh. To better predict suitable deformations, we employ both global and local features extracted from the input point cloud. To achieve this, based on PointNet, we build a Global Encoder that extracts the global feature that characterizes the input point cloud as a whole. We also construct a Local Encoder to extract features at each grid point of a 3D Feature Volume. To extract local feature at each vertex of the 4D template mesh, we develop Embedding Net Embed to map 4D coordinates along with global feature to 3D coordinates [x_l,y_l,z_l] and use them for trilinear interpolation on the 3D Feature Volume to obtain the local feature. These are then used to deform 4D template through Deformation Net Deform, to obtain deformed 4D tetrahedral mesh. The global feature passes through a fully connected network to predict a hyperplane parameter a that determines the hyperplane w=a to slice our deformed 4D tetrahedral mesh, producing the output 3D mesh.

Compared with explicit methods

Figure 5: 3D reconstruction results of our method compared with explicit methods. Our method has advantages in both topology and accuracy. Patch-based methods (MNA-25, Atlas-25) may cause cracks. Sphere-based methods cannot represent different topologies.

Compared with implicit methods

Figure 6: 3D reconstruction results comparing our method with implicit methods. Implicit methods need occupancy or signed distance values to train the networks, and it is hard for them to fit very thin surfaces: thin structures may be reconstructed too thick (Occ method) or lost entirely (ConvOcc method).

NeuralSlice can represent different 3D shapes and topologies in one 4D tetrahedral mesh. We can learn a Deformation Net and learnable hyper-planes with fixed feature volume. It means we have a fixed deformed 4D tetrahedral mesh and different 3D meshes from different slices. In DMTet (Shen et al., 2021), one tetrahedron grid only represents a single 3D mesh, as shown in Figure 7,8,9.

Figure 7: NeuralSlice is capable of fitting sphere deformation sequences.

Figure 8: NeuralSlice is capable of fitting animal deformation sequences.

Figure 9: 3D meshes in each row are sliced from the same 4D tetrahedral mesh, they have different shapes and topologies.