arXiv research

New method $μP^2$ improves neural network training by scaling perturbations layerwise.

New framework explains fast transfer of hyperparameters across model scales.

This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.

New geometric interpretation explains over-parameterized models and adversarial perturbations.

mSAM improves generalization by making models flatter.

Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…

Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.

Framework for safely updating machine learning models.

New scaling framework for MoE architectures ensures stability and optimal performance at scale.

Graph Neural Networks (GNNs) achieve an impressive performance on structured graphs by recursively updating the representation vector of each node based on its neighbors, during which parameterized transformation matrices should be learned for the node feature updating. However, existing propagation schemes are far fro…

AdvPCA uses robust optimization to achieve sparse PCA without tuning.

We develop a general framework for proving rigorous guarantees on the performance of the EM algorithm and a variant known as gradient EM. Our analysis is divided into two parts: a treatment of these algorithms at the population level (in the limit of infinite data), followed by results that apply to updates based on a …

ODS improves adversarial attacks by maximizing output diversity.

We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then con…

Unified spectral framework for μP under joint width-depth scaling.

While several feature scoring methods are proposed to explain the output of complex machine learning models, most of them lack formal mathematical definitions. In this study, we propose a novel definition of the feature score using the maximally invariant data perturbation, which is inspired from the idea of adversaria…

A new principle for optimizer selection improves training speed and performance.

Introduces Causal Energy Minimization to understand Transformer layers.

Accelerates optimization in asynchronous systems with sparse updates.

Gradient perturbation, widely used for differentially private optimization, injects noise at every iterative update to guarantee differential privacy. Previous work first determines the noise level that can satisfy the privacy requirement and then analyzes the utility of noisy gradient updates as in the non-private cas…

In this paper we investigate the usage of adversarial perturbations for the purpose of privacy from human perception and model (machine) based detection. We employ adversarial perturbations for obfuscating certain variables in raw data while preserving the rest. Current adversarial perturbation methods are used for dat…

Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing. However, recent works have shown that deep networks can be vulnerable to adversarial perturbations, which raised a serious robustness issue of deep networks. Adversarial training, typically formulated as a …

New variational method solves submodular maximum coverage problem efficiently.

Gradient-based methods introduce noise that penalizes models sensitive to weight perturbations.

A-FADMM improves FL scalability and privacy via wireless channel perturbations and interference.

Study improves policy search in continuous control by using heavy-tailed distributions.

We propose a general framework for increasing local stability of Artificial Neural Nets (ANNs) using Robust Optimization (RO). We achieve this through an alternating minimization-maximization procedure, in which the loss of the network is minimized over perturbed examples that are generated at each parameter update. We…

This short note reviews so-called Natural Gradient Descent (NGD) for multivariate Gaussians. The Fisher Information Matrix (FIM) is derived for several different parameterizations of Gaussians. Careful attention is paid to the symmetric nature of the covariance matrix when calculating derivatives. We show that there ar…

Under-parameterization hinders deep RL's efficiency.

Learnable token perturbations boost extrapolation in LLMs.

Expectation Maximization (EM) is among the most popular algorithms for maximum likelihood estimation, but it is generally only guaranteed to find its stationary points of the log-likelihood objective. The goal of this article is to present theoretical and empirical evidence that over-parameterization can help EM avoid …

Muon optimizes training efficiency by improving data retention at large batch sizes.

Conformal surface parameterization is useful in graphics, imaging and visualization, with applications to texture mapping, atlas construction, registration, remeshing and so on. With the increasing capability in scanning and storing data, dense 3D surface meshes are common nowadays. While meshes with higher resolution …

Constructing an efficient parameterization of a large, noisy data set of points lying close to a smooth manifold in high dimension remains a fundamental problem. One approach consists in recovering a local parameterization using the local tangent plane. Principal component analysis (PCA) is often the tool of choice, as…

This paper re-examines the problem of parameter estimation in Bayesian networks with missing values and hidden variables from the perspective of recent work in on-line learning [Kivinen & Warmuth, 1994]. We provide a unified framework for parameter estimation that encompasses both on-line learning, where the model is c…

New insights into model robustness for random features and NTK models.

In this paper we describe the space of maximal components of the character variety of surface group representations into PSp(4,R) and Sp(4,R). For every rank 2 real Lie group of Hermitian type, we construct a mapping class group invariant complex structure on the maximal components. For the groups PSp(4,R) and Sp(4,R),…

Gradient EM converges globally for over-parameterized Gaussian mixtures.

Proposes a probabilistic digital twin for dynamical systems using sparse Bayesian learning.

RSO uses random weight perturbations to train deep networks without gradients.

Let $S$ be a closed surface of genus at least $2$. For each maximal representation $ρ: π_1(S)\rightarrow\mathsf{Sp}(4,\mathbb{R})$ in one of the $2g-3$ exceptional connected components, we prove there is a unique conformal structure on the surface in which the corresponding equivariant harmonic map to the symmetric spa…

Improves BBVI for high-dimensional Gaussian approximations by using low-rank approximations.

We prove generic regularity and Uhlenbeck-type compactification theorems for the moduli spaces of PU(2)-monopoles. Generic regularity is NOT obtained in the usual way (by applying Sard theorem to a smooth parameterized moduli space), since the parameterized moduli space can be a priori singular. We explain why, using t…

Noise in SGD affects overparameterized models, favoring sparse solutions.

We construct and study a family of double-periodic almost entire solutions of the maximal surface equation. The solutions are parameterized by a submanifold of $3\times 3$-matrices (the so-called generating matrices). We show that the constructed solutions are either space-like or of mixed type with the light-cone type…

A framework for federated adversarial learning with convergence analysis.

Study examines infinite limits of transformer dynamics, identifying key parameterizations.