AGOP mechanism explains deep neural collapse in neural networks.
problem Explaining the rigid structure of data representations in deep neural networks.
method Introducing AGOP and Deep RFM to demonstrate DNC.
result AGOP mechanism causes deep neural collapse in neural networks.
Recursive Feature Machines show grokking in modular arithmetic without neural networks.
problem Grokking in modular arithmetic tasks.
method Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP).
result RFM and neural networks learn block-circulant features to solve modular arithmetic.
Paper improves SDR estimation speed and conditions.
problem Improving sufficient dimension reduction for multi-index models.
method Estimating expected smoothed gradient outer product.
result Achieves fast parametric convergence rate of Cd⋅n−1/2. SMAVE optimizes SDR by projecting onto a low-dimensional subspace on a Riemannian manifold.
problem High-dimensional regression challenges due to the curse of dimensionality.
method SMAVE combines nearest-neighbor localization and Riemannian stochastic gradient ascent.
result SMAVE achieves almost-sure convergence and matches RMAVE's synthetic subspace recovery rate.
AGOP from KRR recovers central subspace in fewer samples than needed for prediction.
problem Recovering low-dimensional structure in multi-index polynomial functions.
method Fit kernel ridge regression and compute AGOP from the fitted predictor.
result AGOP's top r eigenspace recovers the central subspace in n≍dp+δ samples. Paper identifies how neural networks learn features.
problem Understanding how neural networks automatically select features.
method Deep Neural Feature Ansatz, implementing average gradient outer product.
result Recursive Feature Machines achieve state-of-the-art performance.
Outer automorphisms of free products are represented by CTs.
problem Representing outer automorphisms of free products by relative train track maps.
method Developed theory of attracting laminations and CTs for free products.
result Outer automorphisms of free products satisfy an index inequality.
Optimal preconditioning improves Langevin sampling efficiency.
problem Improving sampling efficiency in high-dimensional target distributions.
method Optimal preconditioning using Fisher information, applied to MALA.
result Adaptive MCMC scheme significantly outperforms other methods.
Outer automorphism groups of Coxeter groups are trivial except for small ranks.
problem Triviality of outer automorphism groups of Coxeter groups.
method Using Guirardel-Levitt outer space for free products, proving triviality and cyclic order for specific ranks.
result Outer automorphism groups are trivial except for small ranks.
In this work, we contribute a new multi-layer neural network architecture named ONCF to perform collaborative filtering. The idea is to use an outer product to explicitly model the pairwise correlations between the dimensions of the embedding space. In contrast to existing neural recommender models that combine user em…
This paper aims at achieving a "good" estimator for the gradient of a function on a high-dimensional space. Often such functions are not sensitive in all coordinates and the gradient of the function is almost sparse. We propose a method for gradient estimation that combines ideas from Spall's Simultaneous Perturbation …
The paper explains spikes in training loss as catapults, improving feature learning and generalization.
problem Understanding and improving the training process of neural networks.
method Analysis of training loss spikes in SGD, empirical evidence of catapults, and demonstration of AGOP alignment.
result Catapults in training loss promote feature learning and better generalization.
xRFM improves tabular data inference with better accuracy and scalability.
problem Inference from tabular data remains challenging and underdeveloped compared to other AI areas.
method Combines feature learning kernel machines with a tree structure.
result xRFM outperforms other methods across 100 regression and 200 classification datasets.
We show that the horoboundary of outer space for the Lipschitz metric is a quotient of Culler and Morgan's classical boundary, two trees being identified whenever their translation length functions are homothetic in restriction to the set of primitive elements of FN. We identify the set of Busemann points with the s…
The paper shows symmetries of a geometric space for Coxeter groups.
problem Understanding symmetries in the Outer space of a Coxeter group.
method Analyzing the geometric rigidity of the universal Coxeter group of rank n.
result For n ≥ 4, the symmetries of the spine of the outer space are only the outer automorphisms.
Study embeddings of free group products into automorphism groups.
problem Embeddings of direct products of free groups into automorphism groups.
method Proved existence of canonical fixed points in the boundary of Outer space.
result Complete description of embeddings of direct products of free groups.
Breaking symmetry in training data is key for generalization in feature learning kernels.
problem Grokking in algebraic tasks, where models perform well on training but fail on unseen data.
method Used Recursive Feature Machine (RFM) with AGOP to learn task-relevant features, breaking symmetry in training data.
result Generalization occurs only when symmetry in the training set is broken, and RFM generalizes by recovering underlying invariance group action.
Distributionally robust optimization (DRO) problems are increasingly seen as a viable method to train machine learning models for improved model generalization. These min-max formulations, however, are more difficult to solve. We therefore provide a new stochastic gradient descent algorithm to efficiently solve this DR…
Outer automorphism group of hyperbolic groups is HHG under certain conditions.
problem Characterizing the outer automorphism group of hyperbolic groups.
method Proving finite-index subgroups are central extensions of orbifold mapping class groups with bounded Euler class.
result Outer automorphism group of a one-ended hyperbolic group is virtually a hierarchically hyperbolic group.
Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.
problem Nonnegative function estimation in settings like MLE and trajectory optimization.
method First and second-order pseudo-mirror descent with pseudo-gradients and projections.
result Establishes tradeoffs and non-asymptotic bounds on model complexity.
New insights into optimizing Local SGD's outer optimizer for faster convergence.
problem Understanding the impact of outer optimizer and its hyperparameters in Local SGD.
method Analyzing convergence guarantees with new outer learning rates and momentum.
result Tuning the outer learning rate can improve convergence and handle inner learning rate ill-tuning.
In this paper we develop the metric theory for the outer space of a free product of groups. This generalizes the theory of the outer space of a free group, and includes its relative versions. The outer space of a free product is made of G-trees with possibly non-trivial vertex stabilisers. The strategies are the same…
Gradient descent aligns neural feature matrices with pre-activation tangent features.
problem Understanding neural feature learning mechanisms.
method Analytical proof of alignment between weight matrices and pre-activation tangent features.
result Derivative alignment occurs almost surely in high-dimensional settings.
Improves MARS for nonparametric multivariate regression with dimension reduction.
problem High number of basis functions in MARS for high-order interactions.
method Linear combinations of covariates for dimension reduction, facilitating gradient calculation and eigen-analysis for estimation.
result Asymptotic theory and numerical studies show improved performance over MARS.
The paper shows that for Coxeter groups, the commensurator of outer automorphisms is rigid.
problem The rigidity of commensurator of outer automorphisms of Coxeter groups.
method Study of the abstract commensurator of the outer automorphism group of a universal Coxeter group.
result For n≥5, the natural map is an isomorphism and every isomorphism between finite index subgroups is conjugation. In this paper, we show that feedforward and recurrent neural networks exhibit an outer product derivative structure but that convolutional neural networks do not. This structure makes it possible to use higher-order information without needing approximations or infeasibly large amounts of memory, and it may also provid…
The study examines how automorphism growth rates of a group can be deduced from its simpler decompositions.
problem Determine automorphism growth rates of a group from its simpler decompositions.
method Analyze group decompositions into simpler pieces (direct products, free products, graph of groups) and deduce growth rates.
result Information about automorphism growth rates of a group can be deduced from its simpler decompositions.
TrIM improves gradient-based dimension reduction and regression.
problem Efficiently identifying relevant feature subspace for high-dimensional regression.
method Introduced TrIM forest, an iterative approach using Mondrian forest and EGOP estimate.
result Consistency guarantees and convergence rates for EGOP matrix and random forest estimator.
We prove central limit theorems for the random walks on either the mapping class group of a closed, connected, orientable, hyperbolic surface, or on Out(FN), each time under a finite second moment condition on the measure (either with respect to the Teichmüller metric, or with respect to the Lipschitz metric …
We generalise the Karrass-Pietrowski-Solitar and the Nielsen realisation theorems from the setting of free groups to that of free products. As a result, we obtain a fixed point theorem for finite groups of outer automorphisms acting on the relative free splitting complex of Handel--Mosher and on the outer space of a fr…
Let G=G1∗⋯∗Gk∗F be a countable group which splits as a free product, where all groups Gi are freely indecomposable and not isomorphic to Z, and F is a finitely generated free group. If for all i∈{1,…,k}, both Gi and its outer automorphism group Out(Gi) satisfy t…
We investigate the combinatorial and geometric properties of automorphism groups of universal right-angled Coxeter groups, which are the automorphism groups of free products of copies of Z_2. It is currently an open question as to whether or not these automorphism groups have non-positive curvature. Analogous to Outer …
The stochastic variance-reduced gradient method (SVRG) and its accelerated variant (Katyusha) have attracted enormous attention in the machine learning community in the last few years due to their superior theoretical properties and empirical behaviour on training supervised machine learning models via the empirical ri…
Given a real matrix A with n columns, the problem is to approximate the Gram product AA^T by c << n weighted outer products of columns of A. Necessary and sufficient conditions for the exact computation of AA^T (in exact arithmetic) from c >= rank(A) columns depend on the right singular vector matrix of A. For a Monte-…
PDA method optimizes neural networks with global convergence rate analysis.
problem Quantitative convergence rate for neural network optimization in mean field regime.
method Particle dual averaging (PDA) method, combining Langevin algorithm and outer loop optimization.
result Established quantitative global convergence for two-layer mean field neural networks.
The paper explores how gradient descent trains associative memories, revealing oscillations and convergence issues.
problem Training dynamics of associative memories in overparameterized and underparameterized settings.
method Reduction to particle system dynamics, theory, and experiments.
result Oscillatory transitory regimes and benign loss spikes in overparameterized settings, suboptimal memorization in underparameterized settings.
We associate a contractible ``outer space'' to any free product of groups G=G_1*...*G_q. It equals Culler-Vogtmann space when G is free, McCullough-Miller space when no G_i is Z. Our proof of contractibility (given when G is not free) is based on Skora's idea of deforming morphisms between trees. Using the action of Ou…
New mechanism discovered for feature learning in CNNs.
problem Understanding how CNNs learn features from images.
method Proposed Convolutional Neural Feature Ansatz linking filter covariances to patch-based AGOPs.
result Deep ConvRFM algorithm learns features similar to deep CNNs, improving performance.
In this paper we study obstructions to presentability by products for finitely generated groups. Along the way we develop both the concept of acentral subgroups, and the relations between presentability by products on the one hand, and certain geometric and measure or orbit equivalence invariants of groups on the other…
Gradient descent dynamics in quadratic regression models are analyzed, revealing five phases: monotonic, catapult, periodic, chaotic, and divergent.
problem Analyzing the dynamics of gradient descent in quadratic regression models.
method Fine-grained bifurcation analysis of gradient descent dynamics using a cubic map parameterized by the step-size.
result Gradient descent dynamics in quadratic regression models exhibit five distinct phases: monotonic, catapult, periodic, chaotic, and divergent.
Study uses outer metrics for PDE-constrained shape optimization over diffeomorphism group.
problem Optimizing shapes governed by PDEs over the diffeomorphism group.
method Outer metrics on diffeomorphism group, Riemannian steepest descent method.
result Riemannian approach outperforms other metrics in solving PDE-constrained shape optimization problems.
We study aspherical manifolds that do not support Anosov diffeomorphisms. Weakening conditions of Gogolev and Lafont, we show that the product of an infranilmanifold with finitely many aspherical manifolds whose fundamental groups have trivial center and finite outer automorphism group does not support Anosov diffeomor…
Outer billiards defined on geodesics surfaces in 3D space forms.
problem Defining and analyzing outer billiards on geodesics in 3D space forms.
method Defined an outer billiard map on the space of oriented geodesics, showing diffeomorphism and symplectomorphism properties.
result Outer billiard map is a diffeomorphism and symplectomorphism under certain conditions.
Delta method applied to deep nets for uncertainty quantification.
problem Quantifying epistemic uncertainty in deep learning models.
method Low-cost variant of Delta method for L2-regularized deep neural networks. result Approximation error close to zero for meaningful rankings of images.
Let G be a countable group that splits as a free product of groups of the form G=G1∗⋯∗Gk∗FN, where FN is a finitely generated free group. We identify the closure of the outer space PO(G,{G1,…,Gk}) for the axes topology with the space of projective minimal, \emph{very small} …
We consider a class of a nested optimization problems involving inner and outer objectives. We observe that by taking into explicit account the optimization dynamics for the inner objective it is possible to derive a general framework that unifies gradient-based hyperparameter optimization and meta-learning (or learnin…
Calibration in 16D disproves Federer's product question.
problem Proving Federer's product question in 16D.
method Adapted Cayley calibration in R^16, constructing a specific calibration Φ.
result Product of two orthogonally supported calibrations is not always a calibration.
RFM reduces feature space for linear models, improving sparse recovery.
problem Sparse linear regression and low-rank matrix recovery.
method Recursive Feature Machines (RFM) that alternates between reweighting feature vectors by AGOP and learning prediction function.
result RFM generalizes IRLS and outperforms deep linear networks.