Proposes learning invariances in neural networks using a weight-space approach.
problem Learning invariances from data in neural networks remains an open problem.
method Minimizes a lower bound on the marginal likelihood in weight space.
result Results in higher performing models with naturally learned invariances.
New method improves deep learning performance without weight symmetry.
problem Challenges in scaling non-symmetric learning methods to deep convolutional networks.
method Introduced techniques to mitigate scalability issues, including a modified feedback alignment method.
result Demonstrated competitive performance with backpropagation using a weaker form of weight symmetry.
Variational inference struggles with weight symmetries in neural networks, leading to biased posteriors.
problem Weight space symmetries in neural networks cause multimodal posteriors, challenging variational inference.
method Developed a symmetrization mechanism to create permutation invariant variational posteriors.
result Symmetrized variational posteriors have a better fit to the true posterior and improved predictive performance.
New method approximates curvature from symmetries in deep networks.
problem Hard to approximate curvature in large deep networks.
method Analytically averaging over group actions that leave the loss invariant to construct structured Hessian approximations.
result Structured Hessian approximations from single gradients can be estimated, stored, and inverted.
Symmetry proven for positive solutions of a weighted p-Laplace operator inequality.
problem Proving symmetry of positive solutions to a specific type of inequality.
method Analyzing positive critical points of Caffarelli-Kohn-Nirenberg inequalities with a weighted p-Laplace operator.
result Complete classification and symmetry result for positive solutions in a range of parameters.
We bound the symmetry algebra of a vector distribution, possibly equipped with an additional structure, by the corresponding Tanaka algebra. The main tool is the theory of weighted jets.
New rank 3 distributions with exponentially growing symmetries.
problem Constructing distributions with very large symmetries.
method Using rational normal curves and symplectic flags.
result Symmetry groups grow exponentially in dimension.
A new test method improves goodness-of-fit tests for copulas.
problem Developing robust tests for copula goodness-of-fit.
method Binary Expansion Approximation of UniformiTY (BEAUTY) and Binary Expansion Adaptive Symmetry Test (BEAST).
result The BEAST method improves empirical power against various alternatives.
LoRAs enable efficient adaptation of large models; this paper explores processing LoRA weights with machine learning.
problem Efficient processing of low-rank weight decompositions in large finetuned models.
method Developed symmetry-aware invariant and equivariant LoL models to process LoRA weights.
result LoL models can predict CLIP scores, finetuning data attributes, and accuracy on downstream tasks.
Biologically plausible learning algorithms can match BP on large datasets.
problem Learning algorithms that are biologically plausible often perform poorly on large datasets.
method Evaluation of sign-symmetry and feedback alignment algorithms on ImageNet and MS COCO.
result Sign-symmetry algorithm can match BP performance on ImageNet and MS COCO.
The paper explores symmetry in solutions of semilinear PDEs on Riemannian domains.
problem Symmetry phenomena in solutions of semilinear PDEs on Riemannian domains.
method General framework for formulating the symmetry problem; evidence from stable solutions; consideration of manifolds with density.
result Evidence that the framework is natural, with results for stable solutions.
New method shows random, diverse initializations are not essential for deep neural networks.
problem The necessity of random, diverse initializations in deep neural networks.
method Constructed a deep convolutional network with identical features by initializing weights to 0, enabling signal propagation and stable gradients.
result Random, diverse initializations are not necessary for training neural networks.
Novel symmetry found in colored HOMFLY polynomials from superalgebras.
problem Understanding symmetries in colored HOMFLY polynomials.
method Exploring the sl(N∣M) superalgebra to find a symmetry. result A symmetry relating polynomials colored by different representations.
Symmetry in loss functions constrains model parameters, leading to specific learning outcomes.
problem Understanding and leveraging symmetries in neural networks to improve learning outcomes.
method Analyzing the impact of loss function symmetries on model parameters and learning behavior.
result Mirror-reflection symmetries in loss functions lead to constraints on model parameters, influencing learning outcomes.
Symmetry in neural networks reduces parameter count without sacrificing accuracy.
problem Improving parameter usage and efficiency in deep neural networks.
method Imposing symmetry constraints on neural network parameters, especially in convolutional and recurrent networks.
result Symmetry can have little or no negative effect on network accuracy, even in deep overparameterized networks.
On locally conformally flat manifolds we describe a construction which maps generalised conformal Killing tensors to differential operators which may act on any conformally weighted tensor bundle; the operators in the range have the property that they are symmetries of any natural conformally invariant differential ope…
We construct a natural framed weight system on chord diagrams from the curvature tensor of any pseudo-Riemannian symmetric space. These weight systems are of Lie algebra type and realized by the action of the holonomy Lie algebra on a tangent space. Among the Lie algebra weight systems, they are exactly characterized b…
This work explores how overparametrization and priors affect Bayesian neural network posteriors.
problem Symmetries, non-identifiabilities, and weight-space priors fragment and inflate BNN posteriors.
method We study the interplay between overparametrization and priors in BNN posteriors, deriving key phenomena and validating through experiments.
result Overparametrization induces structured, prior-aligned weight posterior distributions.
Generative model creates diverse neural network weights efficiently.
problem Creating high-performance and diverse weights for neural networks.
method Trains a hypernetwork mapping latent vectors to high-performance weights, balancing accuracy and diversity.
result Generated weights form a diverse manifold, improving classification accuracy.
Deep neural networks favor symmetric structures, enabling multilevel symmetries.
problem Understanding and optimizing deep neural networks.
method Formulating DNN training as convex Lasso problems with geometric algebra.
result Deep networks inherently favor symmetric structures, enabling multilevel symmetries.
Graphs of neural networks are represented to preserve symmetry, improving performance across various tasks.
problem Lack of equivariance in neural network representations of other neural networks.
method Represent neural networks as computational graphs and use graph neural networks to preserve permutation symmetry.
result Single model encodes diverse neural architectures, outperforming state-of-the-art methods.
This paper explores Bayesian Neural Network posteriors, uncovering symmetries and their impact.
problem Understanding the complex posterior distribution of deep Bayesian Neural Networks.
method Investigates optimal approaches for approximating posteriors, analyzes modes, and explores visualizations.
result Uncovered weight-space symmetries and their impact on the posterior, particularly scaling symmetries.
DeepWeightFlow generates diverse neural network weights efficiently.
problem Generating complete neural network weights efficiently and accurately.
method Flow Matching in weight space with Git Re-Basin and TransFusion.
result DeepWeightFlow generates high-accuracy neural networks without fine-tuning.
New Lipschitz bound for ReLU networks resists weight rescaling.
problem Lack of robustness guarantees for ReLU networks under weight perturbations.
method Rescaling-invariant Lipschitz bound based on path-metrics.
result The new bound applies to various ReLU-DAG architectures and resists neuron-wise rescalings.
We provide a complete classification of hexagonal singular 3-web germs in the complex plane, satisfying the following two conditions: 1) the Chern connection remains holomorphic at the singular point, 2) the web admits at least one infinitesimal symmetry at this point. As a by-product, a classification of hexagonal wei…
We use a new approach that we call unification to prove that standard weighted double bubbles in n-dimensional Euclidean space minimize immiscible fluid surface energy, that is, surface area weighted by constants. The result is new for weighted area, and also gives the simplest known proof to date of the (unit weight…
Proposes neuron alignment to optimize mode connectivity in neural networks.
problem Understanding and optimizing mode connectivity in deep neural networks.
method Introduces neuron alignment to approximate optimal weight permutations and improve mode connectivity.
result Neuron alignment significantly alleviates robust loss barriers and improves model robustness and accuracy.
Automatically learns flexible symmetry constraints in neural networks using gradients.
problem Fixed hard constraints on neural network functions that cannot be adapted.
method Improves parameterisations of soft equivariance and optimizes marginal likelihood using differentiable Laplace approximations.
result Achieves equivalent or improved performance on image classification tasks compared to baselines with hard-coded symmetry.
The study examines optimal synthesis in a radially symmetric Grushin space with conditions on the weight function.
problem Optimal synthesis in a radially symmetric Grushin space with a weight function.
method Analysis of the geometry of R3 with a weighted Carnot-Carathéodory metric, providing conditions for Grushin-like structure, and describing optimal synthesis. result Sufficient conditions on the weight function ensure a Grushin-like structure, and the candidate cut time coincides with the true cut time in the integrable case.
Empirical study shows removing neural parameter symmetries impacts model performance.
problem Understanding the impact of neural parameter symmetries on model performance.
method Developed two methods to reduce parameter space symmetries in neural networks.
result Removing parameter symmetries can lead to faster and more effective Bayesian neural network training.
New findings on neural network identifiability using affine symmetries.
problem Identifying all neural networks that produce a given function.
method Examined affine symmetries of nonlinearities and their impact on neural network identifiability.
result Symmetries can be used to find a rich set of networks giving rise to the same function, except for a special case.
Noether's theorem clarifies how symmetries in neural networks influence learning.
problem Understanding how symmetries in neural networks affect learning.
method Systematic study of symmetry interactions with learning algorithms using Noether's theorem.
result Symmetries impose restrictions on the optimization path, leading to conserved quantities.
Paper presents a method to summarize HMC samples for neural networks, providing meaningful uncertainty estimates.
problem Lack of interpretable summary statistics for HMC samples in neural networks due to permutation symmetry.
method Introducing a transpositions metric to quantify permutations and using rebasin method to summarize HMC samples.
result Compact representation of HMC samples provides meaningful uncertainty estimates for each weight in a neural network.
In this paper we study sectional curvature bounds for Riemannian manifolds with density from the perspective of a weighted torsion free connection introduced recently by the last two authors. We develop two new tools for studying weighted sectional curvature bounds: a new weighted Rauch comparison theorem and a modifie…
Least symmetry breaking principle explains SGD's local minima in shallow ReLU networks.
problem Understanding the structure of local minima in two-layer ReLU networks.
method Analyzing the squared loss optimization problem for ReLU networks with Gaussian inputs and applying the principle of least symmetry breaking.
result The principle of least symmetry breaking explains the structure of spurious local minima detected by SGD.
Study symmetry groups and curves from sums of exponentials.
problem Understanding the geometry and symmetry of curves from sums of exponentials.
method Analysis of symmetry groups, winding numbers, and parametrization of the unit circle.
result Unified method for constructing curves with specific properties.
Two local learning rules are investigated to avoid weight transport in neural networks.
problem Local learning rules that avoid weight transport are unstable and require tuning.
method Investigated two non-local learning rules and a more robust local rule.
result Non-local learning rules match state-of-the-art performance and operate effectively in noisy updates.
We propose a semantic segmentation model that exploits rotation and reflection symmetries. We demonstrate significant gains in sample efficiency due to increased weight sharing, as well as improvements in robustness to symmetry transformations. The group equivariant CNN framework is extended for segmentation by introdu…
Symmetry in neural networks affects generalization, as shown by CLT and RG transformations.
problem Improving generalization in neural networks by incorporating physical symmetries.
method Evaluation of symmetry constraints and expressivity in MLPs and GNNs using the CLT as a test case.
result Overly complex or overconstrained models generalize poorly, revealing a competition between symmetry constraints and expressivity.
In this paper we analyse financial implications of exchangeability and similar properties of finite dimensional random vectors. We show how these properties are reflected in prices of some basket options in view of the well-known put-call symmetry property and the duality principle in option pricing. A particular atten…
New algorithms avoid weight transport, outperforming current deep learning methods.
problem Current deep learning algorithms rely on weight transport, which is biologically implausible.
method Two mechanisms: weight mirror and modified Kolen-Pollack algorithm, using random feedback weights.
result These mechanisms outperform feedback alignment and other methods on visual recognition tasks.
The paper proves an infinite double bubble theorem in higher dimensions.
problem Characterizing minimizing partitions of infinite and finite volumes in Rn. method Proves a variant of the double bubble theorem for configurations with infinite and finite chambers.
result Locally minimizing (1,2)-clusters are unique in Rn for n≤7 and n≥8 under certain conditions. Establishes Yau-Tian-Donaldson conjecture for weighted metrics.
problem Constant scalar curvature Kähler metrics on polarized projective manifolds.
method Extends Chi Li's work to weighted case, uses a priori estimates and slope formulas.
result Proves Yau-Tian-Donaldson conjecture for weighted extremal Kähler metrics.
In this paper, we give a new generalization of positive sectional curvature called positive weighted sectional curvature. It depends on a choice of Riemannian metric and a smooth vector field. We give several simple examples of Riemannian metrics which do not have positive sectional curvature but support a vector field…
Study of quantum spaces on Kähler manifolds with T-symmetry converging to a mixed polarization.
problem Quantum spaces on Kähler manifolds with T-symmetry and their convergence.
method Construction of a one-parameter family of Kähler structures and study of quantum spaces.
result Quantum spaces corresponding to different polarizations converge to a mixed polarization as the parameter goes to infinity.
Hypernetworks are neural networks that generate weights for another neural network. We formulate the hypernetwork training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We explain how this simple formulation gener…
New optimizer designs respect symmetry, improving deep learning models.
problem Optimizers lack respect for symmetry in neural networks.
method Introduce symmetry-compatible principle for optimizer design.
result Symmetry-compatible optimizers improve model performance.
Study eigenvalues of drift Laplacian on symmetric self-shrinkers in R^3.
problem Estimating the first eigenvalue of the drift Laplacian on symmetric self-shrinkers.
method Analyzing the dihedral and prismatic groups to prove the first eigenvalue is 1/2.
result Proved that the first eigenvalue of the drift Laplacian is 1/2 for symmetric self-shrinkers.