arXiv research

AGOP mechanism explains deep neural collapse in neural networks.

Recursive Feature Machines show grokking in modular arithmetic without neural networks.

Paper improves SDR estimation speed and conditions.

SMAVE optimizes SDR by projecting onto a low-dimensional subspace on a Riemannian manifold.

AGOP from KRR recovers central subspace in fewer samples than needed for prediction.

Paper identifies how neural networks learn features.

Outer automorphisms of free products are represented by CTs.

Optimal preconditioning improves Langevin sampling efficiency.

In this work, we contribute a new multi-layer neural network architecture named ONCF to perform collaborative filtering. The idea is to use an outer product to explicitly model the pairwise correlations between the dimensions of the embedding space. In contrast to existing neural recommender models that combine user em…

This paper aims at achieving a "good" estimator for the gradient of a function on a high-dimensional space. Often such functions are not sensitive in all coordinates and the gradient of the function is almost sparse. We propose a method for gradient estimation that combines ideas from Spall's Simultaneous Perturbation …

The paper explains spikes in training loss as catapults, improving feature learning and generalization.

xRFM improves tabular data inference with better accuracy and scalability.

We show that the horoboundary of outer space for the Lipschitz metric is a quotient of Culler and Morgan's classical boundary, two trees being identified whenever their translation length functions are homothetic in restriction to the set of primitive elements of $F_N$. We identify the set of Busemann points with the s…

The paper shows symmetries of a geometric space for Coxeter groups.

Using the Guirardel-Levitt outer space of a free product, we prove that the outer automorphism group of the outer automorphism group of the universal Coxeter group of rank $n \geq 5$ is trivial, and that it is a cyclic group of order 2 if $n=4$. In addition we prove that the outer automorphism group of the automorphism…

Study embeddings of free group products into automorphism groups.

Breaking symmetry in training data is key for generalization in feature learning kernels.

Distributionally robust optimization (DRO) problems are increasingly seen as a viable method to train machine learning models for improved model generalization. These min-max formulations, however, are more difficult to solve. We therefore provide a new stochastic gradient descent algorithm to efficiently solve this DR…

Outer automorphism group of hyperbolic groups is HHG under certain conditions.

Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.

New insights into optimizing Local SGD's outer optimizer for faster convergence.

In this paper we develop the metric theory for the outer space of a free product of groups. This generalizes the theory of the outer space of a free group, and includes its relative versions. The outer space of a free product is made of $G$-trees with possibly non-trivial vertex stabilisers. The strategies are the same…

Gradient descent aligns neural feature matrices with pre-activation tangent features.

Improves MARS for nonparametric multivariate regression with dimension reduction.

The paper shows that for Coxeter groups, the commensurator of outer automorphisms is rigid.

In this paper, we show that feedforward and recurrent neural networks exhibit an outer product derivative structure but that convolutional neural networks do not. This structure makes it possible to use higher-order information without needing approximations or infeasibly large amounts of memory, and it may also provid…

The study examines how automorphism growth rates of a group can be deduced from its simpler decompositions.

TrIM improves gradient-based dimension reduction and regression.

We prove central limit theorems for the random walks on either the mapping class group of a closed, connected, orientable, hyperbolic surface, or on $\text{Out}(F_N)$, each time under a finite second moment condition on the measure (either with respect to the Teichmüller metric, or with respect to the Lipschitz metric …

We generalise the Karrass-Pietrowski-Solitar and the Nielsen realisation theorems from the setting of free groups to that of free products. As a result, we obtain a fixed point theorem for finite groups of outer automorphisms acting on the relative free splitting complex of Handel--Mosher and on the outer space of a fr…

Let $G=G_1\ast\dots\ast G_k\ast F$ be a countable group which splits as a free product, where all groups $G_i$ are freely indecomposable and not isomorphic to $\mathbb{Z}$, and $F$ is a finitely generated free group. If for all $i\in\{1,\dots,k\}$, both $G_i$ and its outer automorphism group $\text{Out}(G_i)$ satisfy t…

We investigate the combinatorial and geometric properties of automorphism groups of universal right-angled Coxeter groups, which are the automorphism groups of free products of copies of Z_2. It is currently an open question as to whether or not these automorphism groups have non-positive curvature. Analogous to Outer …

The stochastic variance-reduced gradient method (SVRG) and its accelerated variant (Katyusha) have attracted enormous attention in the machine learning community in the last few years due to their superior theoretical properties and empirical behaviour on training supervised machine learning models via the empirical ri…

Given a real matrix A with n columns, the problem is to approximate the Gram product AA^T by c << n weighted outer products of columns of A. Necessary and sufficient conditions for the exact computation of AA^T (in exact arithmetic) from c >= rank(A) columns depend on the right singular vector matrix of A. For a Monte-…

The Delta method is a classical procedure for quantifying epistemic uncertainty in statistical models, but its direct application to deep neural networks is prevented by the large number of parameters $P$. We propose a low cost variant of the Delta method applicable to $L_2$-regularized deep neural networks based on th…

PDA method optimizes neural networks with global convergence rate analysis.

The paper explores how gradient descent trains associative memories, revealing oscillations and convergence issues.

We associate a contractible ``outer space'' to any free product of groups G=G_1*...*G_q. It equals Culler-Vogtmann space when G is free, McCullough-Miller space when no G_i is Z. Our proof of contractibility (given when G is not free) is based on Skora's idea of deforming morphisms between trees. Using the action of Ou…

New mechanism discovered for feature learning in CNNs.

In this paper we study obstructions to presentability by products for finitely generated groups. Along the way we develop both the concept of acentral subgroups, and the relations between presentability by products on the one hand, and certain geometric and measure or orbit equivalence invariants of groups on the other…

Gradient descent dynamics in quadratic regression models are analyzed, revealing five phases: monotonic, catapult, periodic, chaotic, and divergent.

Study uses outer metrics for PDE-constrained shape optimization over diffeomorphism group.

We study aspherical manifolds that do not support Anosov diffeomorphisms. Weakening conditions of Gogolev and Lafont, we show that the product of an infranilmanifold with finitely many aspherical manifolds whose fundamental groups have trivial center and finite outer automorphism group does not support Anosov diffeomor…

Outer billiards defined on geodesics surfaces in 3D space forms.

Let $G$ be a countable group that splits as a free product of groups of the form $G=G_1\ast\dots\ast G_k\ast F_N$, where $F_N$ is a finitely generated free group. We identify the closure of the outer space $P\mathcal{O}(G,\{G_1,\dots,G_k\})$ for the axes topology with the space of projective minimal, \emph{very small} …

We consider a class of a nested optimization problems involving inner and outer objectives. We observe that by taking into explicit account the optimization dynamics for the inner objective it is possible to derive a general framework that unifies gradient-based hyperparameter optimization and meta-learning (or learnin…

Calibration in 16D disproves Federer's product question.

RFM reduces feature space for linear models, improving sparse recovery.