A new method combines extrapolation and line search for solving nonconvex, nonsmooth optimization problems.
problem Nonconvex, nonsmooth optimization problems in machine learning and image processing.
method Proximal gradient method with extrapolation and line search (PGels).
result The method reduces to existing algorithms under proper parameter choices and converges to stationary points.
GOALS improves learning rate selection for dynamic MBSS in deep learning.
problem Challenges in selecting learning rates for dynamic MBSS in deep learning.
method Gradient-only approximation line search (GOALS) for dynamic MBSS loss functions.
result GOALS reduces model errors in multimodal cases.
Develops a new method for matrix factorization problems.
problem Solving a general matrix factorization model with potential function.
method Non-monotone alternating updating method based on a potential function.
result The method can outperform existing methods for specific applications.
This study addresses the challenges of dynamic mini-batch sub-sampling in neural network training.
problem Challenges in training neural networks due to dynamic mini-batch sub-sampling.
method Distinguishes between static and dynamic sub-sampling, recasting optimization to find SNN-GPPs.
result SNN-GPPs are less susceptible to sub-sampling-induced discontinuities and better approximate true optima.
This research introduces a line search method for deep learning that uses parabolic approximations.
problem Finding optimal step sizes for deep learning optimization.
method Parabolic approximation line search approach.
result The batch loss over lines in negative gradient direction is mostly convex locally and suitable for parabolic approximations.
Probabilistic line search improves stochastic optimization efficiency.
problem Lack of direct line search methods for stochastic optimization.
method Combines deterministic line search structure with Bayesian optimization concepts.
result Effective removal of learning rate definition for SGD.
Improved SGD with line-search achieves fast convergence rates for various models.
problem Achieving fast convergence rates for stochastic gradient descent (SGD) in over-parameterized models.
method Proposes using line-search techniques to automatically set the step-size in SGD, proving convergence rates for convex, strongly-convex, and non-convex functions.
result SGD with Armijo line-search attains deterministic convergence rates for convex and strongly-convex functions, and linear convergence for non-convex functions.
Introduces GOLS-I for automatically determining step sizes in neural networks without surrogates.
problem Determining step sizes in neural network training using predetermined rules or expensive global optimization strategies.
method Gradient-Only Line Searches (GOLS-I) that are Inexact.
result GOLS-I is a competitive strategy for reliably determining step sizes in stochastic loss functions.
New criterion for solving inverse Hessian equations, including J-equation.
problem Existence of solutions to inverse Hessian equations, including J-equation.
method Stability of pairs in the sense of Paul, formulated in terms of GIT criterion.
result New numerical criterion for existence of solutions to inverse Hessian equations.
A new method for feature selection in high-dimensional data reduces search cost while maintaining performance.
problem Feature selection in very-high-dimensional datasets is computationally expensive and challenging.
method Stochastic Sequential Search (SSS) using temperature-controlled softmax sampling and dependency-aware statistics.
result The method significantly reduces search cost while maintaining or improving performance.
Study examines line search approximations for neural networks using MBSS.
problem Reducing computational cost in training large-scale neural networks.
method Empirical study of quadratic line search approximations for dynamic MBSS loss functions, enforcing different types of function and derivative information.
result Selectively enforcing information in approximations reduces the variance of predicted step sizes.
Parallel algorithm finds sparse solutions for nonconvex problems.
problem Nonconvex sparsity-regularized rank minimization.
method Parallel best-response algorithm with exact line search.
result Guaranteed convergence to a stationary point.
In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a prob…
Adaptive gradient methods converge faster with over-parameterization and line-search.
problem Training over-parameterized models using adaptive gradient methods.
method Simplified setting of smooth, convex losses with over-parameterized models, proving convergence rates and demonstrating improvements with line-search techniques.
result Adaptive gradient methods, particularly AMSGrad, converge faster with line-search techniques.
GOLS-I automatically determines learning rates for various neural network training algorithms.
problem Adapting learning rates in stochastic training algorithms for neural networks.
method Gradient-Only Line Search (GOLS-I) for automatically setting learning rates.
result GOLS-I learning rate schedules are competitive with manually tuned rates across multiple algorithms, architectures, datasets, and loss functions.
Improved SGD methods converge faster for nonconvex optimization.
problem Nonconvex optimization challenges in machine learning.
method Adaptive SGD with line-search and Polyak stepsizes.
result Unified convergence rates for various nonconvex functions.
Study proves a criterion for curve diffusion flow blow-up.
problem Analyzing curve diffusion flow with contact angle constraints.
method Contradiction proof using compactness and short time existence.
result Proves blow-up criterion for L2 curvature bound. Armijo line-search speeds up gradient descent for various functions.
problem Improving convergence rate of gradient descent.
method Applying Armijo line-search to adjust step-size in gradient descent.
result GD with Armijo line-search converges faster than GD with a fixed step-size.
New algorithm optimizes AUC in binary classification and changepoint detection.
problem Difficult to optimize AUC in binary classification and changepoint detection.
method Proposes efficient path-following algorithms for choosing optimal learning rate.
result Proposed line search algorithm computes complete AUM/AUC representation.
A new line search rule improves support recovery in high-dimensional data.
problem Support recovery in high-dimensional data analysis with ℓ0 penalty. method Data-driven line search rule for adaptive step size determination.
result Proves ℓ2 error bound without restrictions on cost functional. The group lasso is a penalized regression method, used in regression problems where the covariates are partitioned into groups to promote sparsity at the group level. Existing methods for finding the group lasso estimator either use gradient projection methods to update the entire coefficient vector simultaneously at e…
New methods solve saddle point problems without line search.
problem Solving saddle point problems efficiently and adaptively.
method Auto-conditioned primal-dual hybrid gradient (AC-PDHG) and auto-conditioned ADMM (AC-ADMM) methods.
result Methods achieve optimal complexity and convergence guarantees.
New method finds optimal learning rates for neural nets.
problem Finding optimal learning rates in stochastic neural networks.
method Gradient-only line searches using Non-negative Associative Gradient Projection Points (NN-GPPs).
result Learning rates can be reliably resolved as step sizes along search directions.
ITCA optimizes label combination for ambiguous outcomes in multi-class classification.
problem Ambiguous outcome labels in real-world datasets hinder accurate multi-class classification.
method Information-theoretic classification accuracy (ITCA) and search strategies (greedy, breadth-first) guide label combination.
result ITCA improves prediction accuracy and identifies ambiguous labels across diverse applications.
Stochastic quasi-Newton tackles noisy gradients in optimization.
problem Optimizing with noisy data in stochastic settings.
method Extends quasi-Newton methods to handle stochastic gradients through flexible Hessian modeling and line-search regularization.
result Demonstrates superior performance in maximum likelihood estimation for complex models.
Paper introduces a privacy-preserving line search method for optimization.
problem Optimization performance depends on step size tuning, which is difficult and privacy-sensitive.
method Introduces a stochastic adaptive line search algorithm that satisfies differential privacy.
result The algorithm efficiently uses privacy budget and outperforms existing private optimizers.
Online boosting method improves weak to strong learner.
problem Online learning of weak to strong learner.
method Extends batch GentleAdaBoost to online approach with line search.
result Online boosting performs better than other methods.
Simple DP algorithms find approximate solutions for nonconvex ERM.
problem Finding approximate solutions to nonconvex ERM problems with privacy.
method Differential privacy, descent directions, line search, mini-batching, two-phase strategy.
result Effective algorithms for nonconvex ERM with privacy guarantees.
Paper introduces FoMoH for optimization without backpropagation.
problem Optimizing machine learning models without backpropagation.
method Second-order hyperplane search, forward-mode stochastic gradient method, hyper-dual numbers, FoMoH.
result Developed a novel optimization algorithm that avoids backpropagation.
A new method approximates expected empirical loss for stochastic deep learning tasks.
problem Determining optimal step sizes for stochastic gradient descent in deep learning.
method Applying one-dimensional function fitting to noisy losses of vertical cross sections to approximate expected empirical loss.
result The method leads to a robust and straightforward optimization method that performs well across datasets and architectures.
Optimal regret achieved in stochastic, discrete multi-armed bandits using information-theoretic exploration.
problem Optimal exploration vs. exploitation in stochastic, discrete multi-armed bandits.
method Proposes an information-theoretic strategy based on the value of information criterion, using simulated-annealing-like updates of a parameter.
result Achieves logarithmic optimal regret with respect to the number of episodes.
A new L-BFGS method tackles large-scale optimization with fewer evaluations.
problem Efficiently solving large-scale unconstrained optimization problems.
method Proposes a regularized L-BFGS method with line search techniques.
result Shows global convergence and robust performance in numerical tests.
SALSA automatically adjusts learning rates in stochastic gradient methods.
problem Automatic adjustment of learning rates in stochastic gradient methods.
method SALSA uses a line-search procedure to gradually increase the learning rate, then a statistical test to decrease it.
result SALSA matches the performance of best hand-tuned learning rate schedules in deep learning tasks.
Improved L-BFGS for large-scale ML with progressive batching.
problem Combining fast generalization with reliable line search in large-scale ML.
method Progressive batching, stochastic line search, stable quasi-Newton updating.
result Performs well on training logistic regression and deep neural networks.
Reduced SVM training time by limiting solver time for model selection.
problem Time-consuming parameter tuning in SVM model selection.
method Introduced a stopping criterion for SVM training time during parameter tuning.
result Significantly reduced model selection times by an order of magnitude.
Paper proposes algorithms for sparse signal estimation with nonconvex regularization.
problem Sparse signal estimation with nonconvex regularization.
method Successive convex approximation framework combining majorization-minimization and line search.
result Flexibility, fast convergence, low complexity, guaranteed convergence to stationary point.
Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.
problem Analyzing risk and learning rate dynamics in high-dimensional optimization problems.
method Developed a framework to give exact expressions for risk and learning rate curves using ODEs.
result Exact expressions for risk and learning rate curves, with detailed analysis of two adaptive learning rates.
Novel SAAG variants reduce variance in large-scale learning.
problem Reduction of variance in noisy gradient approximations for large-scale machine learning.
method Proposed SAAG-III and IV variants with SBAS for step size determination.
result Proved linear convergence of SAAG-IV for all smoothness and strong-convexity combinations.
The paper optimizes grid search for machine learning parameters with improved complexity.
problem Challenging tuning of machine learning regularization parameters.
method Unified framework for approximating regularization paths with refined complexity.
result Optimized grid search with improved complexity for various loss functions.
Max-value Entropy Search improves Bayesian optimization efficiency.
problem Expensive computation in maximizing entropy for Bayesian optimization.
method MES, a new criterion that uses information about the maximum function value.
result MES maintains or improves empirical performance while significantly reducing computational cost.
New preconditioners speed up SGD on Lie groups.
problem Accelerating SGD convergence on Lie groups.
method Online updating of low-rank and matrix-free preconditioners.
result Preconditioners maintain symmetry and invariance.
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
Develops an SSBO algorithm for global optimization of expensive models.
problem Global optimization of expensive black-box models.
method Asynchronous hybrid-criterion with interval reduction.
result Improves global search ability and local search efficiency.
Ortho-MADS optimizes SVM hyperparameters for better accuracy.
problem Optimizing hyperparameters for SVM with Gaussian kernel.
method Deterministic Mesh Adaptive Direct Search (MADS) with orthogonal directions (Ortho-MADS).
result Ortho-MADS consistently finds comparable or better solutions than other methods.
New method ranks power grid contingencies for faster security assessment.
problem Maintain high voltage power transmission networks in security.
method Neural network-based ranking of higher order contingencies.
result Residual risk of contingencies decreases dramatically compared to considering only N-1 cases.
Novel approach simplifies VI problems with faster performance.
problem Black-box VI optimization problems.
method Sample Average Approximation (SAA) combined with quasi-Newton methods and line search.
result Achieves faster performance than existing methods.
GOLS finds activation functions affect training robustness, especially ReLU.
problem Investigate how different activation functions impact GOLS in neural network training.
method Identify SNN-GPPs for GOLS, analyze activation function effects on gradient continuity.
result GOLS robust for most activation functions but sensitive to ReLU.
Twistor lines connect complex tori in their period domain.
problem Connecting complex tori in their period domain.
method Analyzing twistor lines in the period domain of complex tori.
result Periods of complex tori can be joined by generic chains of twistor lines.