This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.
problem Quantifying optimal hyperparameters for large language models across scales.
method Developed three metrics to quantify hyperparameter transfer and investigated the importance of embedding layer learning rate.
result Maximal Update (μP) parameterization offers high-quality learning rate transfer compared to standard parameterization (SP).
New method μP2 improves neural network training by scaling perturbations layerwise.
problem Improving neural network performance as models scale up.
method Layerwise perturbation scaling in the infinite-width limit of neural networks.
result Layerwise perturbation scaling ensures all layers are effectively perturbed in the limit.
Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…
Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…
Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.
problem Challenges in choosing an appropriate learning rate for deep networks, especially as depth increases.
method Introduces Arithmetic-Mean μP (AM-μP), constraining network-wide average pre-activation second moment to a constant scale, combined with residual-aware He fan-in initialization. result Demonstrates a −3/2 scaling law for learning rates across depths, enabling zero-shot learning-rate transfer. Framework for safely updating machine learning models.
problem Continuous updates to machine learning models can lead to unintended consequences.
method Formalizes the problem as computing the largest locally invariant domain (LID), uses tractable primal-dual formulation.
result Matches or exceeds heuristic baselines for avoiding forgetting while providing formal safety guarantees.
New scaling framework for MoE architectures ensures stability and optimal performance at scale.
problem Lack of principled understanding of how hyperparameters should scale in MoE architectures.
method Developed a novel Dynamical Mean Field Theory (DMFT) for three scaling regimes of MoE architectures.
result Derived Maximally Scale-Stable Parameterization (MSSP) for SGD and Adam, providing robust learning rate transfer and monotonic improvement with scale.
Graph Neural Networks (GNNs) achieve an impressive performance on structured graphs by recursively updating the representation vector of each node based on its neighbors, during which parameterized transformation matrices should be learned for the node feature updating. However, existing propagation schemes are far fro…
New framework explains fast transfer of hyperparameters across model scales.
problem Understanding and optimizing hyperparameters for large-scale models.
method Developed a conceptual framework for HP transfer across scale, showing fast transfer is equivalent to useful transfer for compute-optimal grid search.
result Fast transfer of hyperparameters is equivalent to useful transfer for compute-optimal grid search, offering asymptotic computational advantage.
We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then con…
Unified spectral framework for μP under joint width-depth scaling.
problem Challenges in stable feature learning and HP transfer for width-depth scaled models.
method Developed a simple and unified spectral framework for μP under joint width-depth scaling.
result Unified and generalized μP formulation for practical architectures with multi-transformation branches.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
New variational method solves submodular maximum coverage problem efficiently.
problem Submodular maximum coverage problem in various applications.
method Variational optimization using Nemhauser divergence, alternating E and M steps.
result Efficient solution to submodular maximum coverage problem.
Under-parameterization hinders deep RL's efficiency.
problem Implicit under-parameterization impairs data-efficiency in deep RL.
method Characterized and mitigated the rank collapse of value network features.
result Controlling rank collapse improves deep RL performance.
Expectation Maximization (EM) is among the most popular algorithms for maximum likelihood estimation, but it is generally only guaranteed to find its stationary points of the log-likelihood objective. The goal of this article is to present theoretical and empirical evidence that over-parameterization can help EM avoid …
Muon optimizes training efficiency by improving data retention at large batch sizes.
problem Improving training efficiency and data retention at large batch sizes.
method Introducing Muon, a second-order optimizer, and combining it with muP for efficient hyperparameter transfer.
result Muon outperforms AdamW in retaining data efficiency at large batch sizes, enabling more economical training.
Conformal surface parameterization is useful in graphics, imaging and visualization, with applications to texture mapping, atlas construction, registration, remeshing and so on. With the increasing capability in scanning and storing data, dense 3D surface meshes are common nowadays. While meshes with higher resolution …
This paper re-examines the problem of parameter estimation in Bayesian networks with missing values and hidden variables from the perspective of recent work in on-line learning [Kivinen & Warmuth, 1994]. We provide a unified framework for parameter estimation that encompasses both on-line learning, where the model is c…
In this paper we describe the space of maximal components of the character variety of surface group representations into PSp(4,R) and Sp(4,R). For every rank 2 real Lie group of Hermitian type, we construct a mapping class group invariant complex structure on the maximal components. For the groups PSp(4,R) and Sp(4,R),…
Gradient EM converges globally for over-parameterized Gaussian mixtures.
problem Recovering ground truth Gaussian mixtures with over-parameterized models.
method Gradient EM with over-parameterization, using Hermite polynomials and tensor decomposition.
result Gradient EM globally converges to ground truth with n=Ω(mlogm) over-parameterization. Let S be a closed surface of genus at least 2. For each maximal representation ρ:π1(S)→Sp(4,R) in one of the 2g−3 exceptional connected components, we prove there is a unique conformal structure on the surface in which the corresponding equivariant harmonic map to the symmetric spa…
Improves BBVI for high-dimensional Gaussian approximations by using low-rank approximations.
problem Scalability issues with BBVI for high-dimensional multivariate Gaussian approximations.
method Extends BaM framework to handle full covariance matrices by integrating patch step for low-rank parameterization.
result Shows improved efficiency and scalability on synthetic and real-world high-dimensional inference problems.
We construct and study a family of double-periodic almost entire solutions of the maximal surface equation. The solutions are parameterized by a submanifold of 3×3-matrices (the so-called generating matrices). We show that the constructed solutions are either space-like or of mixed type with the light-cone type…
A framework for federated adversarial learning with convergence analysis.
problem Unique vulnerabilities to adversarial attacks in federated learning.
method Formulates a general federated adversarial learning framework with inner and outer loops for client-side adversarial training and server-side model aggregation.
result The minimum loss under the proposed algorithm can converge to ε with chosen learning rate and communication rounds.
NGD improves multivariate Gaussian inference by optimizing Fisher information.
problem Efficiently optimizing multivariate Gaussian models.
method Natural Gradient Descent applied to multivariate Gaussian parameters.
result NGD updates are more efficient for symmetric covariance matrices.
Study examines infinite limits of transformer dynamics, identifying key parameterizations.
problem Understanding the training dynamics of transformer models in the feature learning regime.
method Analysis of infinite scaling limits using dynamical mean field theory.
result Identified parameterizations that admit well-defined infinite width and depth limits.
Study on learning rates in neural networks of varying depth.
problem Dependence of maximal update learning rate on network depth.
method Analysis of random fully connected ReLU networks with mean-field weight initialization.
result Maximal update learning rate scales like L−3/2 with network depth. The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
In this paper, we introduce a novel, non-recursive, maximal matching algorithm for double auctions, which aims to maximize the amount of commodities to be traded. It differs from the usual equilibrium matching, which clears a market at the equilibrium price. We compare the two algorithms through experimental analyses, …
Generative adversarial network improves geosteering in fluvial reservoirs.
problem Improving geosteering in complex reservoirs with high uncertainties.
method Generative adversarial deep neural network (GAN) trained to model fluvial successions.
result Reduces uncertainty and correctly predicts geological features up to 500 meters ahead of drill-bit.
Onflow optimizes portfolio allocation with gradient flows, robust to transaction fees.
problem Optimizing portfolio allocation with transaction costs.
method Gradient flow reinforcement learning method for dynamic asset allocation.
result Onflow outperforms benchmarks in high transaction cost regimes.
A determinantal point process (DPP) is a probabilistic model of set diversity compactly parameterized by a positive semi-definite kernel matrix. To fit a DPP to a given task, we would like to learn the entries of its kernel matrix by maximizing the log-likelihood of the available data. However, log-likelihood is non-co…
Revisits PPO design choices, exposing failure modes and proposing alternatives.
problem Failure modes of standard PPO in new environments.
method Revisits standard PPO design choices, exposes failure modes, and proposes alternative approaches.
result Alternative design choices prevent failure modes in new environments.
Gradient EM converges globally for over-parameterized Gaussian mixtures.
problem Global convergence of gradient EM for Gaussian mixtures with more than 2 components.
method Likelihood-based convergence analysis framework.
result Gradient EM converges globally with a sublinear rate of O(1/√t).
Distributed optimization often consists of two updating phases: local optimization and inter-node communication. Conventional approaches require working nodes to communicate with the server every one or few iterations to guarantee convergence. In this paper, we establish a completely different conclusion that each node…
Data assimilation for subsurface flow using latent diffusion models shows that ensemble Kalman methods may overestimate posterior uncertainty, while Monte Carlo sampling is more reliable.
problem Data assimilation for subsurface flow
method Ensemble Kalman smoother and Markov chain Monte Carlo sampling
result Monte Carlo sampling is more reliable than ensemble Kalman methods
New learning rule simplifies Bayesian updates for deep learning.
problem Bayesian learning rule's complexity and manifold constraints.
method Lie-group approach to simplify Bayesian updates.
result New algorithm learns sparse features in deep learning.
Proof of learning rate transfer in MLPs with μP parameterization.
problem Understanding and optimizing learning rates in neural networks with different parameterizations.
method Theoretical analysis and empirical validation of learning rate transfer in MLPs with μP, SP, and NTP parameterizations. result The optimal learning rate converges to a non-zero constant as width goes to infinity under μP, explaining learning rate transfer. The paper analyzes generalization of noisy, iterative algorithms using maximal leakage.
problem Analyzing the generalization behavior of noisy, iterative learning algorithms.
method Information-theoretic framework with maximal leakage metric.
result Explicit upper bounds on maximal leakage for various scenarios.
We augment recurrent neural networks with an external memory mechanism that builds upon recent progress in metalearning. We conceptualize this memory as a rapidly adaptable function that we parameterize as a deep neural network. Reading from the neural memory function amounts to pushing an input (the key vector) throug…
Paper proposes a math-inspired L2O model for better generalization.
problem Overfitting and poor generalization of generic L2O models.
method Derived mathematical conditions for successful update rules and proposed a novel L2O model.
result Proposed model outperforms generic L2O models in out-of-distribution settings.
This paper proposes CSADA to make DNNs cost-sensitive.
problem Over-parameterization challenges cost-sensitive classification in DNNs.
method CSADA framework using adversarial data augmentation.
result CSADA effectively minimizes overall cost and reduces critical errors.
PAC-Net prunes deep models for better transfer learning.
problem Improving transfer learning with over-parameterized models.
method Prune, Allocate, Calibrate (PAC) approach.
result PAC-Net achieves state-of-the-art performance in inductive transfer learning.
Study on maximizing submodular functions with limited updates, achieving tight bounds and poly-time algorithms.
problem Online submodular maximization with constant recourse.
method Information-theoretic bounds and poly-time randomized algorithms.
result Achieved tight bounds of 2/3 and 3/4 for general and coverage functions, respectively, with a 0.51 approximation.
MADE improves exploration in RL by maximizing deviation from explored regions.
problem Efficient exploration in high-dimensional RL tasks with sparse rewards.
method Proposes a new exploration approach via maximizing the deviation of the occupancy of the next policy from explored regions, adding it as an adaptive regularizer to the RL objective.
result Significantly improves sample efficiency in navigation and locomotion tasks.
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
Study reveals neural scaling laws in random graphs and natural language models.
problem Understanding the origin of neural scaling laws in complex systems.
method Examined scaling laws in transformers trained on random walks and simplified natural language models.
result Neural scaling laws emerge in the absence of power law structure in data correlations.
A new algorithm balances fairness in clustering to avoid discrimination.
problem Clustering data can unfairly discriminate against different demographic groups.
method Designing a stochastic alternating balance fair k-means algorithm (SAfairKM) that alternates between k-means updates and group swap updates.
result The algorithm efficiently constructs well-spread and high-quality Pareto fronts on synthetic and real datasets.