This study connects ReLU neural networks to toric geometry to analyze function realization.
problem Determining which continuous piecewise linear functions can be realized by ReLU neural networks.
method Established a connection between toric geometry and ReLU neural networks, defining key structures like the ReLU fan, toric variety, and Cartier divisor.
result Proved a criterion for functions realizable by unbiased shallow ReLU networks using intersection numbers.
Finslerian graph neural networks recover nonlinear diffusion geometry
problem Graph neural networks on point clouds
method Estimates of the Finsler Laplacian
result Recovery of Finsler geometry
Hyperbolic geometry reveals financial network structure and systemic importance.
problem Understanding the structure and importance of financial networks.
method Data from European banking stress tests, hyperbolic geometry analysis.
result Latent dimensions of `popularity' and `similarity' are strongly associated with systemic importance.
Training shapes the geometry of neural network feature maps, revealing local area magnification.
problem Understanding how training affects the geometric structure of neural network feature maps.
method Analyzing the Riemannian geometry induced by neural network feature maps at infinite width and after training.
result Training breaks the symmetry of the geometry induced by random neural network feature maps, magnifying local areas along decision boundaries.
MSA compares neural representations' intrinsic geometry for better understanding.
problem Existing similarity measures fail to capture subtle distinctions between neural network solutions.
method Metric similarity analysis (MSA) using Riemannian geometry.
result MSA can disentangle features of neural computations and compare nonlinear dynamics.
Regularizers change the geometric properties of loss functions in neural networks.
problem Understanding how different regularizers affect the geometric properties of loss functions in neural networks.
method Examined several regularizers, including weight decay, to determine if the regularized loss function becomes Morse.
result For certain regularizers, the regularized loss function becomes Morse, indicating a change in geometric properties.
Natural gradient simplification for deep learning networks.
problem Efficiency in training deep Bayesian networks.
method Analysis of two geometries of Fisher information matrix and development of a method to simplify natural gradient for the second geometry.
result A method to simplify natural gradient for deep networks using an auxiliary recognition model.
In a graph convolutional network, we assume that the graph G is generated wrt some observation noise. During learning, we make small random perturbations ΔG of the graph and try to improve generalization. Based on quantum information geometry, ΔG can be characterized by the eigendecomposition of the graph Laplaci…
Neural networks' feature geometry evolves like discrete Ricci flow.
problem Understanding neural feature representations and their geometric transformations.
method Approximating input manifold with geometric graphs and analyzing their evolution during training.
result Neural feature geometry evolves like discrete Ricci flow, with nonlinear activations playing a crucial role.
We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden laye…
Neural networks learn discrete tasks on continuous data via emergent geometry.
problem Understanding how neural networks perform discrete computations on continuous data.
method Analysis of Riemannian pullback metric across neural network layers.
result Neural networks learn to discretize continuous inputs and perform logical operations on these discretized variables.
This work uses tropical geometry to understand neural network decision boundaries.
problem Characterizing neural network decision boundaries with piecewise linear activations.
method Tropical geometry applied to a simple neural network model.
result Decision boundaries are a subset of a tropical hypersurface related to a polytope formed by zonotopes.
The Fisher information metric is an important foundation of information geometry, wherein it allows us to approximate the local geometry of a probability distribution. Recurrent neural networks such as the Sequence-to-Sequence (Seq2Seq) networks that have lately been used to yield state-of-the-art performance on speech…
The paper explores the geometry and topology of DNN decision boundaries.
problem Understanding the geometric and topological properties of DNN decision boundaries.
method Differential geometry and the Gauss-Bonnet-Chern theorem.
result Computed the Euler characteristics of compact decision boundaries.
By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous sy…
Geodesics found in deep linear networks.
problem Finding shortest paths in deep neural networks.
method Derived ODEs and explicit solutions for geodesics.
result Horizontal straight lines are geodesics in invariant manifold.
Study the expressivity and training complexity of polynomial neural networks.
problem Understanding the expressivity and training complexity of polynomial neural networks.
method Use algebraic geometry to describe neuromanifolds and neurovarieties, analyzing their dimension and learning degree.
result Characterized the dimension and learning degree of neuromanifolds, providing geometric and complexity measures.
Geodesics connect model modes in neural network loss landscapes.
problem Connecting modes in neural network loss landscapes.
method Reframed mode connectivity in Information Geometry, hypothesized geodesics as mode-connecting paths, proposed algorithm to approximate geodesics.
result Geodesics achieve mode connectivity in neural networks.
Rewiring networks using discrete geometry improves GNN training accuracy and reduces runtime.
problem Inefficient information propagation between distant nodes in graph neural networks.
method Discrete analogues of classical geometric curvature to model and rewire networks.
result Classical geometric notions achieve state-of-the-art GNN training accuracy and significantly reduce runtime.
Why and how that deep learning works well on different tasks remains a mystery from a theoretical perspective. In this paper we draw a geometric picture of the deep learning system by finding its analogies with two existing geometric structures, the geometry of quantum computations and the geometry of the diffeomorphic…
BN refines local partition geometry in piecewise-affine networks during training.
problem Understanding the effect of BN on the function realized during training in piecewise-affine networks.
method Analyzing the geometry of switching hyperplanes and affine-region partition conditioned on a mini-batch.
result BN increases expected local partition refinement in ReLU and piecewise-affine networks.
Riemannian metric matching learns the geometry of high-dimensional datasets using neural networks.
problem Estimating the geometry of high-dimensional datasets from samples
method Riemannian metric matching using neural networks
result Riemannian metric matching rivals or improves k-NN-based diffusion geometry estimators TRNN combines tensor geometry with neural network nonlinearity for HD data.
problem Modeling high-dimensional data with preserved tensor geometry and nonlinear interactions.
method Introduces TRNN that integrates tensor geometry and neural network nonlinearity.
result TRNN preserves tensor geometry while offering nonlinearity.
The paper explores how data geometry influences generalization in neural networks.
problem Understanding generalization in overparameterized neural networks.
method Theoretical exploration of overparametrized two-layer ReLU networks trained below the edge of stability.
result Generalization bounds adapt to the intrinsic dimension of data distributions and deteriorate as data concentrates towards the unit sphere.
The paper explores deep learning through algebra and geometry, highlighting geometric structures and differential processes.
problem Understanding the geometric and algebraic foundations of deep learning.
method Investigates neural networks from perceptron to transformer, emphasizing geometric structures and differential processes.
result A coordinate-free formulation of backpropagation equations using canonical scalar products on matrix spaces.
RNNs compute by warping neural representations over time.
problem Understanding how RNNs perform task computations.
method Developed a Riemannian geometric framework to derive the manifold topology and geometry of RNNs.
result Dynamic warping is a fundamental feature of RNN computations.
Unified framework for Riemannian deep learning across manifold-valued representations.
problem Deep learning on manifold-valued representations often relies on Euclidean approximations or costly geometric operations.
method Develops reusable neural modules, manifold-specific network architectures, and geometric designs.
result Generalizes batch normalization and multinomial logistic regression to broader classes of manifolds.
Unified framework for Riemannian deep learning across manifold-valued representations.
problem Deep learning on manifold-valued data lacks reusable modules, specific network architectures, and efficient geometric operations.
method Develops reusable neural modules, manifold-specific network architectures, and geometric designs for broad classes of Lie groups and gyrogroups.
result Generalizes batch normalization and multinomial logistic regression to Riemannian manifolds, including SPD and hyperbolic spaces.
Dropout technique is analyzed using information geometry.
problem Understanding the regularization performance of dropout in neural networks.
method Unified analysis from information geometry viewpoint.
result Dropout flattens the model manifold and its performance depends on curvature.
This paper extends geometric study of neural networks to non-differentiable layers and random walks.
problem Understanding the geometric properties of neural networks, especially those with non-differentiable activation functions.
method Singular Riemannian geometry approach to convolutional, residual, and recursive neural networks.
result Illustrated geometric findings with numerical experiments on image classification and thermodynamic problems.
Develops MENT for interpreting and detecting changes in network trajectories.
problem Distortion of network geometry and invalidation of temporal comparisons in dynamic network analysis.
method Develops Multiscale Euclidean Network Trajectories (MENT) framework based on second-moment geometry.
result Validates and interprets network trajectories through isotropic normalization and orthogonal transformations.
Estimates curvature of network manifolds to understand community structure.
problem Understanding the geometry of network models to infer community structure.
method Develops hypothesis tests to determine manifold type, dimension, and curvature from noisy distance matrices.
result Consistently estimates manifold type, dimension, and curvature from Riemannian manifolds of constant curvature.
New saddle network architectures preserve convex-concave geometry in optimization problems.
problem Optimization models with convex x and concave y components.
method Structured separable decomposition and saddle network architectures.
result Proven one-dimensional approximation theorem and high accuracy on various test functions.
The paper sparsifies networks by finding efficient paths in their functional space.
problem Sparsifying neural networks to improve performance and efficiency.
method The authors use the geometry of weight spaces and functional manifolds to find efficient paths (geodesics) in the functional space of neural networks.
result The proposed framework can sparsify networks and improve performance on various tasks.
Study robustness of polynomial neural networks using algebraic geometry.
problem Certify robustness radius of polynomial neural networks.
method Metric algebraic geometry, Euclidean distance degree, symbolic elimination, homotopy-continuation methods.
result Found decision boundaries with lower ED degree than generic cubic hypersurfaces.
Unsupervised domain mapping has attracted substantial attention in recent years due to the success of models based on the cycle-consistency assumption. These models map between two domains by fooling a probabilistic discriminator, thereby matching the probability distributions of the real and generated data. Instead of…
The paper uses geometry to understand how neural networks learn.
problem Understanding the learning capability of neural networks.
method Statistical and differential geometric analysis of neural networks performing simple regression.
result Neural networks with higher generalization capability have a slower convergence rate.
We explore the use of graph networks to deal with irregular-geometry detectors in the context of particle reconstruction. Thanks to their representation-learning capabilities, graph networks can exploit the full detector granularity, while natively managing the event sparsity and arbitrarily complex detector geometries…
We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity --- the Fisher-Rao norm --- that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of…
How to understand deep learning systems remains an open problem. In this paper we propose that the answer may lie in the geometrization of deep networks. Geometrization is a bridge to connect physics, geometry, deep network and quantum computation and this may result in a new scheme to reveal the rule of the physical w…
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
Monotonic Linear Interpolation property in neural networks persists despite non-convexity.
problem Understanding the geometric properties of neural network loss landscapes.
method Tools from differential geometry to analyze the monotonicity of neural network weights.
result Sufficient conditions for the Monotonic Linear Interpolation property under mean squared error.
Neural networks approximate unit spheres as polytopes.
problem Approximating unit spheres with neural networks.
method Using ReLU activation in neural networks to generate polytopes.
result Neural networks can approximate unit spheres as polytopes.
Study fractal and regular geometry in deep neural networks.
problem Investigate geometric properties of neural networks.
method Analyze boundary volumes of excursion sets for different activations.
result Hausdorff dimension increases with depth for non-regular activations.
Igeood detects out-of-distribution samples using information geometry.
problem Out-of-distribution (OOD) detection in machine learning systems.
method Igeood uses the Fisher-Rao geodesic distance to detect OOD samples from any pre-trained neural network.
result Igeood outperforms state-of-the-art methods on various network architectures and datasets.
Develops a smooth operator framework for analyzing neural network representations.
problem Analyzing the geometry of feedforward neural network representations.
method Introduces a smooth operator-theoretic approach based on diffusion Markov operators derived from feature clouds.
result Establishes a stable operator-geometric framework for tracking training, width, and perturbation stability.
This paper improves dependency networks using information geometry.
problem Technical disadvantage in dependency networks' learned distribution.
method Interpret pseudo-Gibbs sampling as iterative m-projections onto manifolds.
result Dependency networks can learn faster and have similar performance to Bayesian networks.
Entropy formula derived for deep linear networks using geometric analysis.
problem Thermodynamic description of learning in deep linear networks.
method Group actions, Riemannian submersion, foliation, Jacobi matrices.
result Entropy formula defined on the balanced manifold of DLN.