A higher-order Runge-Kutta optimizer performs poorly compared to Adam when evaluated fairly.
problem Evaluating the performance of adaptive Runge-Kutta optimizers under strict conditions.
method Built and evaluated a representative Adam variant using a Bogacki-Shampine 3(2) RK pair, FSAL reuse, and local-error step control.
result The adaptive nature of the RK optimizer is illusory; it behaves like a fixed-step optimizer with gradient averaging.
Thompson Sampling is at most twice as bad as any other policy in Bayesian bandit models.
problem Optimizing selection of the best arm in Bayesian bandit models with independent latent processes.
method Thompson Sampling approach applied to models with independent latent arm processes.
result Thompson Sampling makes at most twice the expected number of mistakes compared to any other policy.
Develops a power-calibrated framework for LLM watermarking, optimizing tradeoffs between detectability and distortion.
problem The trade-off between detectability and semantic distortion in logit-based watermarking.
method Power-calibrated statistical framework for watermark hyperparameters, establishing explicit relationships.
result Derives practical parameter selection procedures achieving optimal tradeoffs under constraints.
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
Theoretical study explains grokking in neural networks.
problem Understanding the abrupt transition from fitting to generalizing in neural networks.
method Characterized a shell-core topological configuration of the solution space induced by Adam's optimization dynamics.
result Derived grokking scaling laws for learning rate, batch size, and regularization coefficient.
A new optimizer DDC improves deep learning models by respecting symmetries.
problem Deep networks' loss is invariant to continuous symmetries, leading to optimization issues.
method DDC builds a Dead-Direction Conditioner that lifts a base optimizer into a G-equivariant one, preserving the quotient geometry.
result DDCAdam and DDCMuon outperform standard optimizers in various tasks, improving validation-train loss gaps and learning dynamics.
PACE optimizes training for averaged language models, improving performance.
problem How to optimize training for averaged language model iterates.
method Formulated as an optimal-control problem, solved for minimizing error of the average with a penalty on intervention size.
result PACE improves the limiting squared error of the iterate-average estimator by an arbitrarily large factor on some instances.
Is AdamW effective under heavy-tailed noise?
problem Stochastic gradient noise in LLM pretraining is typically heavy-tailed.
method Formulate as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism.
result No rigorous convergence theory for AdamW established in heavy-tailed regime.
Adam's convergence rate in nonsmooth nonconvex optimization is analyzed.
problem Understanding Adam's convergence behavior in nonsmooth nonconvex optimization.
method Finite-time analysis of Adam's classical form.
result Proven convergence rate of 1 / T 2 / 13 1/T^{2/13} 1/ T 2/13 for nonsmooth nonconvex optimization. Trajectory-level supervision allows efficient offline reinforcement learning.
problem Offline reinforcement learning
method Developing a statistical theory for offline policy optimization from trajectory-level labels
result Proving a high-probability guarantee of order O ~ ( H 2 C s a ( π ⋆ ) / n ) \widetilde O(H^2\sqrt{C_{sa}(\pi^\star)/n}) O ( H 2 C s a ( π ⋆ ) / n ) Adapting functional gradients improves FGD's practicality and theoretical guarantees.
problem Implementing FGD in practice due to infinite-dimensional functional gradients.
method Adapting the representation of functional gradients.
result Establishes convergence to a stationary point for smooth losses and a global minimizer under smoothness + Polyak-Lojasiewicz condition.
Muon replaces matrix gradient with polar factor, optimizing flat spectrum updates
problem Optimization bias in matrix updates
method Using polar factor of gradient
result Muon update maximizes entropy among bounded updates
A method for finding most influential sets reduces a complex problem to a sequence of simpler top- k k k problems.
problem Identifying most influential subsets in complex models.
method Reduces the problem to a sequence of top- k k k problems using Dinkelbach's method. result The method returns a globally optimal set for the univariate ratio objective, including partial linear models.
EoS selectively shapes learning, affecting some groups more than others.
problem EoS affects learning differently across the data distribution.
method Branching intervention to enter or exit EoS regime, controlled perturbation to isolate mechanisms.
result EoS redistributes learning, amplifying progress on some groups and suppressing others.
New optimization method helps models generalize better after achieving near-perfect training performance.
problem Models can achieve near-perfect training performance but fail to generalize well to unseen examples.
method GROKtimizer combines rapid convergence to interpolation with post-interpolation norm minimization using Critically Damped Momentum.
result GROKtimizer provides a quadratic speedup over classical gradient descent, offering a natural solution for selecting low-norm interpolating solutions.
A new learning framework reduces PV-Battery system costs by 3.6%.
problem Optimizing PV-Battery systems for cost savings with accurate forecasts.
method Decision-focused learning framework integrating optimization and prediction.
result Decision-focused method reduces average electricity costs by 3.6%.
New saddle network architectures preserve convex-concave geometry in optimization problems.
problem Optimization models with convex x and concave y components.
method Structured separable decomposition and saddle network architectures.
result Proven one-dimensional approximation theorem and high accuracy on various test functions.
This study reveals the critical role of scale vectors in large language models, improving optimization and expressivity.
problem Understanding and optimizing the scale vectors in large language models.
method Systematic study of scale vectors from expressivity, optimization, and architectural perspectives; theoretical and empirical analysis of weight decay; proposing and evaluating improvements.
result Scale vectors improve optimization through a self-amplifying preconditioning effect and are beneficial for expressivity in certain architectures.
No policy can simultaneously be fully autonomous, optimally calibrated, and helpful, proving a trilemma.
problem Proving impossibility of a policy achieving maximum helpfulness, optimal calibration, and full autonomy.
method Geometric proof showing that adding any non-affine autonomy incentive to a strictly proper scoring rule destroys strict properness.
result The Behavioral Credibility Trilemma: no policy can achieve all three goals simultaneously.
Develops a gradient flow for Muon optimizer, a method for optimization.
problem Optimization of complex systems with matrix-valued parameters.
method Gradient flow on probability measures induced by regularized Muon optimizer.
result Derives continuous-time limits and proves Hamiltonian dissipation.
New methods learn DAGs from noisy data, adapting to noise levels.
problem Inferring causal relationships from observational data with noise and confounding.
method Reformulate DAG learning as a continuous optimization problem over adjacency matrices, jointly inferring structure and noise levels.
result Improved robustness to heteroscedasticity and distribution shifts.
New optimizer SF-NorMuon matches tuned AdamW across various horizons.
problem Fixed learning-rate schedules in neural network training lead to strong path dependence and costly re-tuning.
method Schedule-Free Spectral Optimization (SF-NorMuon)
result SF-NorMuon outperforms tuned AdamW on 125M and 772M parameter models across different horizons.
New optimizer designs respect symmetry, improving deep learning models.
problem Optimizers lack respect for symmetry in neural networks.
method Introduce symmetry-compatible principle for optimizer design.
result Symmetry-compatible optimizers improve model performance.
Integrates fairness guarantees into deep learning models.
problem Ensuring fairness in deep learning models.
method Integrates a differentiable fairness layer into neural models and uses an online primal-dual algorithm for provable fairness guarantees.
result Guarantees a chosen notion of output parity in deep learning models.
Optimizes trading policies using future price forecasts.
problem Static reinforcement learning agents lack mechanisms for using price forecasts at inference time.
method FPILOT framework inspired by Model Predictive Control (MPC). Uses a predictive model to construct an allocation-based imagined return objective at each decision step.
result Consistent improvements in total return and risk-adjusted metrics across various policy learning algorithms.
New bounds for KANs trained with DP-SGD, addressing correlated noise.
problem Risk bounds for Kolmogorov-Arnold Networks trained by DP-SGD with correlated noise.
method Established new optimization and population risk analysis for KANs trained with DP-SGD, addressing correlated noise.
result First optimization and population risk analysis of correlated-noise mechanisms for DP training in non-convex settings, including neural networks.
Pion optimizes LLMs by preserving weight matrix singular values.
problem Training large language models (LLMs) with standard optimizers leads to unstable weight matrices.
method Pion uses orthogonal transformations to update weight matrices, preserving their singular values.
result Pion offers a stable alternative to standard optimizers for LLM pretraining and finetuning.
New algorithm reduces regret for single-index bandits to nearly optimal.
problem Optimizing rewards from unknown projections of high-dimensional contexts.
method Two-phase algorithm: estimate projection direction, reduce to 1D bandit, use UCB.
result Achieved optimal regret of i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) . Generative models learn distributions, new method finds inputs matching desired conditional distributions.
problem Designing inputs that produce specific conditional distributions, not just points.
method Conditional Distribution Matching (CDM) and MLGD-F algorithm.
result MLGD-F reliably recovers inputs matching diverse user-specified conditional distributions.
High-dimensional statistics advances in complex data domains.
problem Complex, rich datasets challenge traditional methods.
method Evolved to address sophisticated estimation and inference problems.
result Deepened connections with optimization, concentration, and information theory.
SOO uses bandit theory to optimize functions with limited evaluations.
problem Optimizing functions with a limited number of evaluations.
method SOO is a machine learning approach that uses domain partitioning and a tradeoff between exploration and exploitation.
result SOO provides guarantees on the returned solution and is numerically efficient.
New algorithm POO optimizes noisy, unknown-smooth functions.
problem Optimizing functions with unknown smoothness and noisy evaluations.
method Adaptive optimization algorithm POO.
result POO performs nearly as well as known algorithms with smoothness knowledge, and works for broader classes of functions.
Paper analyzes sample complexity of offline MABs with KL regularization.
problem Optimizing sample complexity for offline decision-making with KL-regularized metrics.
method Sharp analysis of KL-PCB, providing upper and lower bounds.
result Characterizes sample complexity for offline MABs with KL regularization.
Algorithm optimizes ε-SVR with MAPE loss and sample-dependent constraints.
problem Optimizing ε-SVR with MAPE loss and sample-dependent constraints.
method Sequential Minimal Optimization (SMO) for ε-SVR with MAPE loss and sample-dependent box constraints.
result Algorithm achieves lowest median runtime on every tested configuration.
Develops accelerated methods for optimization using low-dimensional projected-gradient information.
problem Optimization with low-dimensional projected-gradient information and Nesterov acceleration.
method Randomized-subspace Nesterov accelerated gradient methods for smooth convex and strongly convex optimization.
result Established accelerated oracle-complexity guarantees and unified basis for comparing sketch families.
StoSOO optimistically maximizes noisy, locally smooth functions.
problem Global maximization of noisy, locally smooth functions with unknown semi-metric.
method StoSOO uses optimistic upper confidence bounds to iteratively decide on the next evaluation point.
result StoSOO performs almost as well as the best tuned algorithms, even without knowing the semi-metric.
BOOOM optimizes orthonormal matrices without needing gradients.
problem Optimizing over the Stiefel manifold in non-convex, non-smooth settings.
method Global Givens rotation-based parametrization and Recursive Modified Pattern Search.
result BOOOM achieves strong performance across various optimization problems.
New method improves calibration of neural networks by targeting robust margins and local smoothness.
problem Poor calibration of neural networks, leading to unreliable confidence estimates.
method Intervene on training procedure by targeting robust margins and local smoothness.
result Improved out-of-sample calibration without sacrificing accuracy.
Paper presents a new way to analyze machine learning generalization without probabilistic assumptions.
problem Traditional generalization analysis assumes i.i.d. data, which is often unverifiable.
method Uses sensitivity analysis of optimization problems to derive deterministic generalization bounds.
result Obtains generalization bounds that relate in-sample and out-of-sample evaluations through an error term quantifying data similarity.
StrEBM learns distinct latent components for better source separation.
problem Blind source separation with identifiable and decoupled latent components.
method Structured latent energy-based model with learnable structural biases.
result The model effectively recovers source components from mixed signals.
A lightweight framework improves convergence and stability of PINNs for complex PDEs.
problem Training instability and reduced accuracy in PINNs for complex PDEs.
method Adaptive curvature correction using secant information to optimize first-order optimizers.
result Consistent improvements in convergence speed, stability, and accuracy over standard optimizers.
The paper studies how to allocate human validation in AI-assisted tasks to minimize errors.
problem Heterogeneous reliability of AI-generated signals across tasks, products, and customer segments.
method Tuned prediction-powered inference, upper confidence bounds policy, Neyman square-root rule.
result The proposed policy outperforms uniform and epsilon-greedy allocation, closing most of the gap to the oracle when reliability is heterogeneous.
New method uses SLL to create masks for PX in noisy optimization problems.
problem Effective optimization in noisy problems with hidden variable dependencies.
method Statistical Linkage Learning (SLL) for decomposition and mask construction.
result Proposed method maintains effectiveness in noisy conditions and outperforms state-of-the-art.
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
mlr3mbo is a modular R toolbox for Bayesian optimization.
problem Efficiently solving optimization problems with multiple objectives and constraints.
method Bayesian optimization with support for multi-objective, multi-point proposals, parallelization, and custom algorithms.
result mlr3mbo performs competitively with state-of-the-art optimizers and robustly handles various optimization regimes.
Machine learning improves high-dimensional matrix estimation.
problem Efficient estimation of high-dimensional matrices.
method Integrates machine learning with classical optimization algorithms for high-dimensional matrix estimation.
result The reparameterized LADMM achieves faster convergence and higher accuracy.
This paper extends the Risk Quadrangle framework for risk management and optimization.
problem Integrating risk management, optimization, and statistical estimation.
method Review and extension of the Risk Quadrangle framework with new quadrangles.
result New quadrangles offer novel approaches to risk-sensitive decision-making.
Paper reveals hidden convexities in deep learning models using sparse signal processing.
problem Non-convex loss functions in deep learning models complicate optimization and theoretical understanding.
method Developed convex equivalences of ReLU NNs and their connections to sparse signal processing models.
result Recent research has uncovered hidden convexities in certain NN architectures, notably two-layer ReLU networks and other architectures.
Survey on optimizing topological descriptors for machine learning.
problem Optimizing topological priors in machine learning models.
method Minimizing topologically-informed losses using gradient descent.
result Various techniques enable optimization of persistence-based loss functions.
Proposes MGCE for improved classification performance.
problem Optimizing between robustness and optimization difficulty in classification.
method Minimax formulation of GCE leading to convex optimization over margins.
result MGCE achieves strong accuracy and better calibration, especially in noisy labels.
This paper extends Mirror Descent to Riemannian manifolds for optimization.
problem Optimization on Riemannian manifolds.
method Developed a Riemannian Mirror Descent (RMD) framework and a stochastic variant.
result Established non-asymptotic convergence guarantees for RMD and stochastic RMD.
The paper proposes a method to learn and leverage contextual preference distributions for better decision-making.
problem Heterogeneous and context-dependent human preferences in decision-making problems.
method A sequential learning-and-optimization pipeline using a bounded-variance score function gradient estimator to train a predictive model mapping contextual features to preference distributions.
result The approach reduces average post-decision surprise by up to 25 times compared to risk-averse baselines in a ridesharing environment.
New method improves optimization algorithms without Lipschitz smoothness.
problem Improving optimization algorithms in the absence of Lipschitz smoothness.
method Dual kernel conditioning (DKC) to provide dual Lipschitz continuity.
result First complexity bounds and iterate convergence for random reshuffling mirror descent.
New method solves optimization problems with stochastic objectives and constraints.
problem Optimization problems with stochastic objectives and deterministic constraints.
method Trust-region interior-point stochastic sequential quadratic programming (TR-IP-SSQP) method.
result Global almost-sure convergence to first-order stationary points under standard assumptions.
Improved Adam for time series forecasting with distributional drift.
problem Non-stationary data challenges Adam's effectiveness.
method Proposed TS_Adam, removing Adam's second-order bias correction.
result TS_Adam achieves 12.8% reduction in MSE and 5.7% in MAE on ETT datasets.
Accelerates MMLE using SVGD with Nesterov acceleration.
problem Maximum Marginal Likelihood Estimation optimization.
method Stein variational gradient descent with Nesterov acceleration.
result Consistently accelerates convergence across various tasks.
Paper introduces a new optimization method for imbalanced datasets.
problem Overfitting in imbalanced datasets, especially in financial fraud detection.
method Fractional Weyl Integral optimization algorithm.
result Significantly improved performance in financial fraud detection (40% improvement in PR-AUC).
New algorithms improve generalization of Adam and AdamW optimizers.
problem Adam and AdamW optimizers generalize worse than SGD.
method Algorithmic stability analysis and clever momentum-based SGD integration.
result HomeAdam(W) algorithms achieve better generalization and faster convergence.
Paper proposes S-BOMM for optimization with multiple models, focusing on consistency.
problem Optimization challenges with multiple models of varying fidelity and accuracy.
method Set-Based Optimization with Multiple Models (S-BOMM) focusing on model consistency.
result Empirical results show S-BOMM's effectiveness in identifying good solutions across multiple models.
BL learns interpretable optimization structures from data.
problem Learning interpretable optimization structures from data.
method BL parameterizes a compositional utility function from intrinsically interpretable modular blocks.
result BL supports architectures from single to hierarchical compositions, modeling hierarchical optimization structures.