Improved Frank-Wolfe algorithms for large-scale optimization.
problem Efficiently solving large-scale optimization problems.
method Modifications to Frank-Wolfe algorithm using stochastic gradients, approximate solutions, and sketched variables.
result Achieves optimal convergence rate of O ( 1 k ) \mathcal{O}(\frac{1}{k}) O ( k 1 ) for large problems. Efficiently solves large-scale robust portfolio optimization problems.
problem High computational demands in large-scale robust portfolio optimization.
method Extended supporting hyperplane approximation for distributionally robust portfolio problems.
result Significantly reduces computational time from several thousand seconds to just a few.
BanditLP optimizes personalized recommendations for large-scale systems.
problem Optimizing personalized recommendations for large-scale systems with constraints.
method Unified neural Thompson Sampling for learning and large-scale linear programming for action selection.
result Consistent gains over strong baselines in experiments and business win in LinkedIn's email marketing system.
A new framework speeds up submodular optimization for large problems.
problem Large scale submodular optimization problems are hard to solve.
method Memoization framework for submodular functions with precomputed statistics.
result Significant speedups for various submodular optimization problems.
Serverless runtimes boost large-scale optimization efficiency.
problem Efficiently solving large-scale optimization problems.
method Master-worker setup with AWS Lambda, parallel optimization algorithm.
result Relative speedups up to 256 workers and efficiencies above 70% up to 64 workers.
The paper reviews optimization methods for large-scale machine learning.
problem Challenges in optimizing machine learning models, especially in large-scale applications.
method Case studies and a comprehensive theory of the stochastic gradient (SG) method.
result The SG method is a versatile and effective approach for large-scale machine learning.
Proposes MamBO for efficient high-dimensional large-scale optimization.
problem High-dimensional and large-scale optimization problems in machine learning and simulation.
method Combines subsampling and subspace embeddings with model aggregation to address uncertainty in surrogate models.
result Improves robustness of Bayesian optimization algorithm and achieves superior performance.
New algorithm for large-scale nonsmooth convex optimization with robust convergence.
problem Minimizing the average of many nonsmooth and convex functions in machine learning.
method Developed a new algorithm called Randomized Smoothing SVRG that achieves robust linear convergence.
result Achieves robust linear convergence rate and superior time and gradient complexity compared to state-of-the-art methods.
A new L-BFGS method tackles large-scale optimization with fewer evaluations.
problem Efficiently solving large-scale unconstrained optimization problems.
method Proposes a regularized L-BFGS method with line search techniques.
result Shows global convergence and robust performance in numerical tests.
FALKON efficiently processes large datasets using kernel methods.
problem Limited applicability of kernel methods in large scale scenarios.
method Combining stochastic subsampling, iterative solvers, and preconditioning.
result Optimal statistical accuracy achieved with O ( n ) O(n) O ( n ) memory and O ( n n ) O(n\sqrt{n}) O ( n n ) time. Paper proposes PPMM for fast estimation of large-scale OTM.
problem Estimation of large-scale optimal transport maps (OTM) is challenging due to the curse of dimensionality.
method Combines projection pursuit regression and sufficient dimension reduction to adaptively select projection directions.
result PPMM consistently estimates the most informative projection direction and weakly converges to the target OTM.
New algorithms optimize risk for large datasets, improving efficiency.
problem Optimizing risk for large datasets with robust methods.
method Proposed algorithms for distributionally robust optimization with CVaR and χ² divergence uncertainty sets.
result Algorithms require independent gradient evaluations of training set size and parameters, suitable for large-scale applications.
This study reveals the critical role of scale vectors in large language models, improving optimization and expressivity.
problem Understanding and optimizing the scale vectors in large language models.
method Systematic study of scale vectors from expressivity, optimization, and architectural perspectives; theoretical and empirical analysis of weight decay; proposing and evaluating improvements.
result Scale vectors improve optimization through a self-amplifying preconditioning effect and are beneficial for expressivity in certain architectures.
Large batch training improves deep learning performance without needing warmup.
problem Slow convergence at early epochs in large batch training.
method Proposes CLARS algorithm and analyzes convergence rate.
result Proposed algorithm outperforms gradual warmup and state-of-the-art large-batch optimizers.
Optimizer choice affects neural scaling laws, changing the exponent α \alpha α .
problem The exponent α \alpha α in neural scaling laws L ( N ) ∝ N − α L(N) \propto N^{-\alpha} L ( N ) ∝ N − α varies with the optimizer used. method Controlled random-feature regression experiments with five optimizer variants and six spectral conditions.
result Preconditioned optimizers yield steeper scaling (larger α \alpha α ), with the α \alpha α -shift increasing across most of the tested spectral range. Adaptive scan Gibbs sampler improves large-scale inference performance.
problem Efficiently updating large-scale online inference problems.
method Derives an adaptive scan Gibbs sampler that optimizes mini-batch size selection.
result Demonstrates superior performance compared to collapsed Gibbs sampler.
Paper proposes a faster SPIDER-EM variant for large-scale nonconvex optimization.
problem High computational cost of EM algorithm in large-scale learning.
method Extension of SPIDER-EM for nonconvex finite-sum optimization problems.
result Achieves state-of-the-art complexity bounds and linear convergence under certain conditions.
A new method, k-SVRG, speeds up large-scale optimization.
problem Efficiently solving large-scale optimization problems with variance reduction.
method k-SVRG, which uses available memory and minimizes stalling phases.
result Proves linear convergence on strongly convex problems and convergence to stationary points on non-convex problems.
We optimize saddle-point problems for large-scale Markov decision processes.
problem Optimizing policies in large-scale Markov decision processes.
method Characterized conditions for convergence and designed an optimization algorithm.
result Our algorithm converges faster and is state-space independent.
Paper tackles large-scale optimal transport and mapping estimation.
problem Learning optimal maps between large distributions.
method Two-step approach: first, stochastic dual regularized OT; second, Monge map estimation.
result The method scales better with large samples and converges to optimal maps.
Proposes a probabilistic optimization method for large-scale problems.
problem Large-scale regularized optimization problems.
method Develops a probabilistic interpretation of the incremental proximal gradient algorithm and uses Bayesian filtering.
result Makes it possible to solve large-scale problems using well-known Bayesian filters.
Memory-efficient learning for large-scale imaging systems.
problem Memory limitations in GPUs for real-world large-scale inverse problems.
method Exploits reversibility of network layers to enable data-driven design.
result Demonstrated on small-scale and large-scale real-world systems.
Unified small and large scale geometry concepts.
problem Combining topology and geometry for large and small scale spaces.
method Introducing orthogonality relations to unify topology and geometry.
result Optimal orthogonality relations for compactifications.
This paper optimizes slate decision systems for large action spaces.
problem Optimizing large-scale decision systems with arbitrary reward functions.
method A policy optimization framework with a novel relaxation of decision functions.
result Demonstrates the effectiveness of the proposed method on large action spaces.
Optimal scaling found to depend on operator norm across large models and datasets.
problem Lack of unifying principle for optimal hyperparameter scaling across models and datasets.
method Discovered that optimal scaling is conditioned on the operator norm of the output layer.
result The optimal learning rate/batch size pair ( η ∗ , B ∗ ) (η^{\ast}, B^{\ast}) ( η ∗ , B ∗ ) consistently has the same operator norm value. We derive scaling laws for optimizing neural networks in hardware.
problem Optimizing the large parameter space of neural networks in hardware.
method Analytical derivation of scaling laws for Coordinate Descent optimization.
result Convergence is exponential and scales linearly with the number of neurons.
Ensembles of random-feature models can't outperform a single large model.
problem Finding the optimal balance between model size and ensemble size.
method Deterministic equivalent risk estimates and scaling laws analysis.
result Ensembles of random-feature models achieve near-optimal performance only under specific conditions.
EBO optimizes expensive high-dimensional functions with thousands of queries.
problem Optimizing expensive functions with high-dimensional inputs.
method Ensemble Bayesian Optimization (EBO) using additive Gaussian process models.
result Achieved unprecedented scalability to tens of thousands of observations.
Robust and fast method for large-scale stochastic optimization.
problem Large-scale stochastic optimization problems.
method Auxiliary variable construction coupled with adaptive inverse Hessian approximation.
result Encouraging performance on real-world problems with millions of observations and unknowns.
We propose HAMSI (Hessian Approximated Multiple Subsets Iteration), which is a provably convergent, second order incremental algorithm for solving large-scale partially separable optimization problems. The algorithm is based on a local quadratic approximation, and hence, allows incorporating curvature information to sp…
RL solves large-scale MV portfolio allocation with high returns.
problem Large-scale mean-variance portfolio optimization.
method Continuous-time reinforcement learning with a multivariate Gaussian policy.
result Our method outperforms econometric and deep RL methods by significant margins.
A distributed algorithm learns patterns in large images and signals.
problem High-dimensional optimization in large images and signals.
method Distributed asynchronous algorithm with locally greedy coordinate descent.
result Patterns can be learned on large scales images from the Hubble Space Telescope.
Efficiently updates classifiers after small dataset modifications.
problem Updating classifiers quickly after small dataset changes in large-scale problems.
method Proposes a method to bound optimal classifiers without re-training.
result Provides bounds on optimal classifiers with low computational cost.
New framework optimizes deep learning training by deferring large batch sizes to late stages.
problem Optimizing batch size scheduling for deep learning training efficiency.
method Introduced the functional scaling law (FSL) framework to analyze and optimize batch size scheduling.
result Large batch sizes can be deferred to late training stages without sacrificing performance.
Adaptive IHS improves sketching for large-scale data.
problem Efficiently modeling large-scale data with iterative Hessian sketch.
method Deterministic A-optimal subsampling for improved IHS.
result A-optimal IHS outperforms existing accelerated IHS methods.
AcceleratedLiNGAM speeds up causal discovery methods for large datasets.
problem Slow causal discovery methods for large-scale datasets.
method Parallelized LiNGAM method with GPU acceleration.
result Up to 32-fold speed-up on benchmark datasets.
Paper presents an efficient algorithm for learning minimax risk classifiers with large-scale data.
problem Efficient learning of minimax risk classifiers for large-scale data with multiple classes.
method Combination of constraint and column generation for efficient learning.
result 10x speedup for general large-scale data and 100x speedup with many classes.
NLCG optimizes DNN training, especially with large mini-batches.
problem Improving convergence speed in large-scale DNN training.
method Stochastic Preconditioned Nonlinear Conjugate Gradient (SP-NLCG) algorithm.
result NLCG improves DNN training accuracy by over 10 percentage points at large mini-batch sizes.
This paper analyzes convergence of large-scale Transformers with weight decay.
problem Understanding optimization guarantees in large-scale Transformer training.
method Construct mean-field limit, show gradient flow convergence to PDE, demonstrate global minimum consistency.
result Gradient flow reaches global minimum in large-scale Transformers with small weight decay.
SOFAR learns large-scale association networks efficiently.
problem Efficiently understanding large-scale response-predictor association networks.
method Sparse Orthogonal Factor Regression (SOFAR) via sparse singular value decomposition with orthogonality constraints.
result SOFAR achieves statistical efficiency and scientific insights.
ParK efficiently solves kernel ridge regression for large datasets.
problem Large-scale kernel ridge regression efficiency and accuracy.
method Partitioning feature space with random projections and iterative optimization.
result Provably maintains statistical accuracy with reduced space and time complexity.
New tuning rules for Metropolis algorithms derived from Bayesian large-sample asymptotics.
problem Optimal scaling in random-walk Metropolis algorithms under realistic assumptions.
method Large-sample asymptotics to derive weak convergence results and tuning guidelines.
result Tuning guidelines consistent with previous ones when target density is product form, accounting for correlation structure.
ADVGP scales up Gaussian process regression to large datasets efficiently.
problem Expensive computational cost of traditional GP inference for large datasets.
method Asynchronous Distributed Variational Gaussian Process (ADVGP) with weight space augmentation and asynchronous proximal gradient optimization.
result ADVGP achieves superior prediction accuracy for large-scale regression tasks.
Ensemble AL searches large datasets for optimal training samples.
problem Improving DNN performance and reducing training time with optimal training data.
method Ensemble Active Learning (AL) for large-scale training data subset search.
result Our approach finds favorable subsets of training data that improve DNN accuracy.
Many real-world regression problems demand a measure of the uncertainty associated with each prediction. Standard decision forests deliver efficient state-of-the-art predictive performance, but high-quality uncertainty estimates are lacking. Gaussian processes (GPs) deliver uncertainty estimates, but scaling GPs to lar…
This work improves Gaussian process model selection for large datasets.
problem Prohibitively high computational cost in Gaussian process model selection.
method Linear-time scaling and computational uncertainty tradeoff.
result Computation-aware Gaussian processes can be trained on large datasets efficiently.
New scalable methods for robust model learning from large datasets.
problem Training robust models resistant to data distribution shifts.
method Composite optimization for distributionally robust optimization (DRO).
result Scalable methods for learning robust models from large datasets.
Optimization of neural networks scales with γ, revealing unique loss curves and optimal learning rates.
problem Understanding the impact of feature learning strength on neural network optimization.
method Empirical investigation of neural networks with varying γ, analyzing the γ γ γ - η η η plane, and examining loss curves. result Optimal learning rate scales non-trivially with γ, with η ∗ ∝ γ 2 η^* \propto γ^2 η ∗ ∝ γ 2 for small γ and η ∗ ∝ γ 2 / L η^* \propto γ^{2/L} η ∗ ∝ γ 2/ L for large γ.