New method efficiently learns positive-definite curvature for neural nets.
problem Efficiently learn positive-definite curvature for neural net training.
method Spectral-factorized positive-definite curvature learning approach.
result Efficiently applies arbitrary matrix roots and generic curvature learning.
This paper reviews discrete curvature models for geometric data analysis.
problem Capturing intrinsic geometric structure in diverse data representations.
method Comprehensive review of discrete curvature models from Riemannian and metric geometry perspectives.
result Systematic pipeline for curvature-driven data analysis and learning.
This paper improves HNNs by learning optimal curvature for better generalization.
problem Inappropriate curvatures in HNNs lead to suboptimal performance.
method Sharpness-aware curvature learning method to smooth loss landscape.
result Proposed method improves HNNs' generalization across various settings.
MCBP detects boundaries in high-dimensional data using curvature.
problem Boundary detection in high-dimensional data.
method MCBP uses mean curvature to model data manifold curvature.
result MCBP improves clustering performance in complex scenarios.
Novel coarse extrinsic curvature for Riemannian submanifolds.
problem Understanding extrinsic curvature of submanifolds.
method Derived from Wasserstein 1-distance between probability measures.
result New insights and approximation of mean curvature from data.
Spectro-Riemannian Graph Neural Networks integrate spectral and curvature signals for better graph representation learning.
problem Enhance graph representation learning by leveraging spectral and curvature signals.
method Proposes Spectro-Riemannian Graph Neural Networks (CUSP) that combines spectral and curvature insights.
result Empirical evaluation shows CUSP outperforms state-of-the-art models by up to 5.3%.
Deep learning predicts curvature of 2D interfaces in level-set method.
problem Estimating curvature in level-set method for complex interfaces.
method Deep learning using feed-forward neural networks trained on synthetic data.
result Deep learning models approximate curvature with comparable precision to traditional methods.
New method approximates curvature from symmetries in deep networks.
problem Hard to approximate curvature in large deep networks.
method Analytically averaging over group actions that leave the loss invariant to construct structured Hessian approximations.
result Structured Hessian approximations from single gradients can be estimated, stored, and inverted.
CurvSSL improves SSL by aligning local manifold curvature.
problem Improving self-supervised learning by capturing local manifold geometry.
method CurvSSL augments Barlow Twins with a curvature-based regularizer to align and decorrelate embeddings across augmentations.
result Curvature-regularized SSL yields competitive or improved linear evaluation performance.
CAMEL embeds data into a manifold using curvature-augmented forces.
problem Data visualization and dimensionality reduction.
method Formulates DR as a physics model with curvature-augmented forces.
result CAMEL outperforms existing methods on benchmark datasets.
Warm-up improves training by adapting learning rate based on curvature.
problem Improving training efficiency in deep learning models.
method Introducing a curvature condition to explain warm-up, and showing empirically that it leads to faster convergence.
result Adapting learning rate based on curvature condition naturally induces warm-up-like schedule, leading to faster convergence.
Study shows how discrete graph curvature relates to manifold curvature.
problem Relating discrete graph curvature to intrinsic manifold curvature.
method Continuum limits of Ollivier's Ricci curvature on data clouds.
result Random geometric graphs inherit global curvature properties of manifolds.
Curvature penalties improve interpretability of KANs without sacrificing accuracy.
problem Pathologically high-curvature oscillations in KANs activations make them hard to interpret.
method Derived a curvature penalty and proved an upper bound on model curvature.
result KANs with curvature penalties achieve substantially smoother activations while maintaining accuracy.
Enhanced Markov chain sampler learns network statistics faster.
problem Learning network statistics efficiently.
method Integrates graph Forman curvature into Markov chain transition probabilities and stationary distribution.
result Curved Markov chain Monte Carlo achieves faster convergence.
Curvature regularization prevents distortion in graph embeddings.
problem Graph topology patterns distort in Euclidean space, making detection difficult.
method Proposes curvature regularization to enforce flatness in embedding manifolds.
result Significant improvements in five embedding methods on open graph datasets.
Graph networks struggle with multi-task learning due to varying property loss surface curvatures.
problem Graph networks underperform in multi-task learning for crystal and molecule properties.
method Assessed curvature of property loss surfaces via spectral properties of Hessians, matrix-free using randomized numerical linear algebra.
result Varying curvature of property loss surfaces explains graph networks' multi-task learning inefficiency.
We propose meta-curvature (MC), a framework to learn curvature information for better generalization and fast model adaptation. MC expands on the model-agnostic meta-learner (MAML) by learning to transform the gradients in the inner optimization such that the transformed gradients achieve better generalization performa…
A new model for graph clustering using curvature spaces.
problem Graph clustering from a geometric perspective.
method Introducing a heterogeneous curvature space and a contrastive learning approach.
result CONGREGATE model outperforms state-of-the-art competitors.
CAMEL enhances manifold embedding and learning with curvature metrics.
problem High-dimensional data classification, dimension reduction, and visualization.
method CAMEL uses a Riemannian manifold with curvature metrics for enhanced expressibility and interpretability.
result CAMEL outperforms state-of-the-art methods on high-dimensional datasets.
New neural network solves Nirenberg problem for curvature on sphere.
problem Prescribing Gaussian curvature on S2 for metrics conformal to the round metric. method Mesh-free physics-informed neural network (PINN) that directly parametrises the conformal factor.
result Neural network achieves very low losses for realisable curvatures, distinguishing them from non-realisable ones.
New method uses curvature to improve graph neural networks.
problem Graph Neural Networks struggle with over-smoothing and over-squashing.
method Augmented Forman-Ricci curvature (AFRC) for scalable rewiring.
result AFRC effectively mitigates over-smoothing and over-squashing.
This thesis explores Ollivier-Ricci curvature in graphs and manifolds, with applications to graph neural networks.
problem Understanding curvature in metric spaces and graphs.
method Combines optimal transport theory, Riemannian manifolds, and graph theory to define and analyze Ollivier-Ricci curvature.
result Extensions of Ollivier-Ricci curvature to directed graphs and applications in network science.
We give a relationship that yields an effective geometric way of evaluating mean curvature of surfaces. The approach is reminiscent of the Gauss's contour based evaluation of intrinsic curvature. The presented formula may have a number of potential applications including estimating the normal vector and mean curvature …
New bounds for neural networks on curved manifolds improve generalization.
problem Existing generalization theories fail to account for non-Euclidean manifold structures.
method Derive covering number bounds incorporating manifold-specific properties like curvature.
result Sharp Rademacher complexity bounds for neural networks on compact manifolds.
New methods estimate curvature, tangent spaces, and dimension of noisy data.
problem Estimating geometric properties of noisy or sparse data.
method Diffusion geometry tools for Riemannian manifold analysis.
result Significantly outperforms existing methods in noisy or sparse data.
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
New framework to understand and exploit curvature in deep learning loss landscapes.
problem Understanding and optimizing the loss landscape in deep learning models.
method New conceptual framework and techniques to estimate and exploit curvature of expected loss changes.
result Alice algorithm optimizes training by incorporating curvature terms and step bounds.
Efficiently trains forward processes to minimize generative trajectories curvature.
problem High curvature of generative trajectories slows down sampling speed.
method Trains forward process to minimize curvature without ODE/SDE simulation.
result Lower curvature than previous models, decreased sampling costs.
Euclidean geometry has historically been the typical "workhorse" for machine learning applications due to its power and simplicity. However, it has recently been shown that geometric spaces with constant non-zero curvature improve representations and performance on a variety of data types and downstream tasks. Conseque…
Focal loss reduces model curvature for better calibration.
problem Improving model confidence in classification problems.
method Geometric interpretation of focal loss to reduce curvature.
result Focal loss reduces the curvature of the loss surface, enhancing model calibration.
This paper speeds up mean curvature computation for high-dimensional data.
problem Efficiently computing mean curvature in high-dimensional datasets.
method Two contributions: algebraic identity and truncated SVD approximation.
result Mean curvature computation reduced from O(m4) to O(k2m+kmp2). In machine learning (ML) security, attacks like evasion, model stealing or membership inference are generally studied in individually. Previous work has also shown a relationship between some attacks and decision function curvature of the targeted model. Consequently, we study an ML model allowing direct control over t…
A key challenge for gradient based optimization methods in model-free reinforcement learning is to develop an approach that is sample efficient and has low variance. In this work, we apply Kronecker-factored curvature estimation technique (KFAC) to a recently proposed gradient estimator for control variate optimization…
The paper introduces curvature-based clustering algorithms for graph analysis.
problem Identifying densely connected substructures in graphs for community detection.
method Discrete Ricci curvatures and geometric flows to reveal community structure.
result The curvature-based approach can identify overlapping communities in graphs.
RicciNets prunes neural networks by removing edges of low importance based on Ricci curvature, reducing FLOPs by 35%.
problem Pruning neural networks to reduce computational load and improve efficiency.
method RicciNets uses Ricci curvature to prune edges of low importance in a randomly wired neural network, reducing FLOPs.
result Reduction of almost 35% in FLOPs with no performance degradation.
The purpose of these notes is to provide an introduction to those who want to learn more about translating solitons for the mean curvature flow in R3, particularly those which are complete graphs over domains in R2. In this paper we describe a full classification of complete translating graphs i…
New approach ties loss curvature to model performance in deep learning.
problem Understanding the relationship between loss curvature and model performance in deep learning.
method Empirical analysis of loss Hessians and theoretical results on input-output Jacobians.
result Novel generalization bound in terms of empirical Jacobian.
Stochastic gradient algorithms have been the main focus of large-scale learning problems and they led to important successes in machine learning. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose a new ad…
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these mat…
New research shows hyperbolic embeddings are useful for global consistency tasks in graphs.
problem The usefulness of hyperbolic representations in graph learning tasks.
method Computed hyperbolic embeddings for node classification and link prediction tasks, addressing optimization issues at zero curvature.
result Hyperbolic embeddings are more effective for tasks requiring global consistency, while Euclidean models are superior for other tasks.
Paper investigates hardness of learning neural networks under manifold hypothesis.
problem Hardness of learning neural networks under the manifold hypothesis.
method Extending proofs of hardness in the SQ and cryptographic settings to the geometric setting.
result Learning is hard under input manifolds of bounded curvature but learnable with additional assumptions on manifold volume.
In topological data analysis, persistent homology is used to study the "shape of data". Persistent homology computations are completely characterized by a set of intervals called a bar code. It is often said that the long intervals represent the "topological signal" and the short intervals represent "noise". We give ev…
Many real-world sequential decision-making problems can be formulated as optimal control with high-dimensional observations and unknown dynamics. A promising approach is to embed the high-dimensional observations into a lower-dimensional latent representation space, estimate the latent dynamics model, then utilize this…
Improved continual learning for neural networks with BN layers using K-FAC extension.
problem Continual learning challenges in neural networks with BN layers.
method Extended K-FAC method to account for inter-example relations, weight merging, and reparameterization for BN layers; proposed weight merging and reparameterization for BN layers; proposed method to select hyperparameters without source task data.
result Better performance in continual learning tasks with BN layers compared to baselines.
New method improves structure learning on sparse graphs.
problem Structure learning on sparse directed acyclic graphs (DAGs).
method Bregman proximal gradient method to address non-convex, high-curvature problem.
result Significantly improved convergence and efficiency.
Consider a sample of n points taken i.i.d from a submanifold Σ of Euclidean space. We show that there is a way to estimate the Ricci curvature of Σ with respect to the induced metric from the sample. Our method is grounded in the notions of Carré du Champ for diffusion semi-groups, the theory of Empirical process…
The paper reveals that baselines significantly impact RL algorithms' convergence.
problem Understanding the true impact of baselines on policy optimization.
method Theoretical analysis of bandit and RL problems, focusing on natural policy gradient and EXP3.
result Baselines can determine algorithm convergence, contradicting traditional optimization theory.
The question of how to incorporate curvature information in stochastic approximation methods is challenging. The direct application of classical quasi- Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper…