The paper explores parabolic regularity in geometric variational analysis.
problem Developing calculus rules and computation formulas for second-order generalized differential constructions.
method Introducing and applying the concept of parabolic regularity to geometric aspects of second-order variational analysis.
result Established new calculus rules and computation formulas for second-order generalized differential constructions.
A new algorithm for solving constrained convex optimization problems efficiently.
problem Constrained convex optimization problems requiring high accuracy solutions.
method Second-Order Conditional Gradient Sliding (SOCGS) algorithm, using projection-free methods to solve quadratic subproblems inexactly.
result Converges quadratically in primal gap after a finite number of linearly convergent iterations.
New Lagrangian approach for optimal control of second-order systems.
problem Optimal control of second-order differential equations derived from force-controlled Lagrangian systems.
method Proposes a new hyperregular control Lagrangian and control Hamiltonian, providing necessary optimality conditions.
result Defines an extended Tulczyjew's triple with controls and studies the relationship between Noether symmetries.
Two new algorithms improve federated optimization under second-order similarity.
problem Federated learning under communication constraints and second-order similarity.
method SVRP and Catalyzed SVRP algorithms combining proximal point evaluations, client sampling, and variance reduction.
result Achieves superior performance and uniformly improves upon existing algorithms for federated optimization under second-order similarity and strong convexity.
New optimality conditions for sub-Riemannian geodesics derived.
problem Optimality conditions for sub-Riemannian geodesics.
method Geometric translation and ODE derivation.
result New second-order necessary optimality conditions.
Investigates optimal execution under time-varying liquidity, preventing price manipulation.
problem Optimal execution with time-varying liquidity impacts and price manipulation prevention.
method Almgren-Chriss framework, deterministic time variation, well-posedness, second-order conditions, price manipulation prevention.
result Sufficient conditions for a unique solution and prevention of price manipulation.
Paper discusses optimal CP for second-order predictions.
problem How to incorporate second-order predictions into conformal prediction.
method Introduces Bernoulli prediction sets (BPS) for second-order predictions and applies conformal risk control for compromised validity.
result BPS provides the smallest prediction sets with conditional coverage.
A new method for faster optimization in high dimensions.
problem Slow convergence in high-dimensional optimization problems.
method Subspace cubic regularized Newton method within Krylov subspace.
result Achieves a dimension-independent convergence rate of O(1/mk + 1/k^2).
Paper explores challenges in training PINNs and loss landscape effects.
problem Challenges in training Physics-Informed Neural Networks (PINNs) due to loss landscape issues.
method Examined gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, and introduced NysNewton-CG (NNCG).
result Adam+L-BFGS outperforms other optimizers, and NysNewton-CG significantly improves PINN performance.
Disputes the empirical Fisher approximation for natural gradient descent.
problem The empirical Fisher approximation fails to capture second-order information in general.
method Comparison of empirical Fisher and Fisher information matrices.
result The empirical Fisher does not generally approximate the Fisher or Hessian.
Negative step sizes improve second-order methods for neural networks.
problem Second-order methods discard negative curvature, limiting their effectiveness.
method Introduce negative step sizes in second-order methods combined with Wolfe line search.
result Negative step sizes lead to global convergence and improved performance.
The (global) Lipschitz smoothness condition is crucial in establishing the convergence theory for most optimization methods. Unfortunately, most machine learning and signal processing problems are not Lipschitz smooth. This motivates us to generalize the concept of Lipschitz smoothness condition to the relative smoothn…
Adaptive norms improve neural network optimization with second-order methods.
problem Optimizing neural networks with second-order methods.
method Regularized Newton methods with adaptive norms, interpreting adaptive gradient methods as trust region methods.
result Ellipsoidal constraints outperform spherical constraints in terms of backpropagations and asymptotic loss value.
Stochastic second-order methods converge fast under interpolation conditions.
problem Minimizing smooth and strongly-convex functions efficiently.
method Regularized subsampled Newton method (R-SSN) and stochastic BFGS algorithms.
result R-SSN achieves global linear convergence and quadratic rate in a local neighbourhood.
Second-order optimizers retain residual information after data deletion, affecting machine unlearning.
problem Residual information in second-order optimizers after data deletion.
method Comparison of first-order and second-order learners, eigendecomposition analysis.
result Second-order optimizers retain residual information, not detectable by first-order analysis.
New method finds stationary points in bilevel optimization problems.
problem Solving nonconvex-strongly-convex bilevel optimization problems.
method Restarted Accelerated HyperGradient Descent (RAHGD) method.
result Achieves best-known theoretical guarantees for finding stationary points in bilevel optimization.
SONIA optimizes machine learning problems with a novel algorithm.
problem Empirical risk minimization in machine learning.
method Symmetric Blockwise Truncated Optimization (SONIA) algorithm combining second-order and steepest descent steps.
result SONIA converges to stationary points in both convex and nonconvex cases.
PWGF escapes saddle points in nonconvex optimization.
problem Escaping saddle points in nonconvex optimization.
method PWGF uses noisy perturbations via Gaussian process to escape saddle points.
result PWGF achieves second-order optimality for nonconvex objectives.
SLEDGE algorithm reduces gradient computation errors in optimization.
problem Accumulated errors in gradient estimation methods for large-scale optimization.
method Single-loop method for finite-sum nonconvex optimization without periodic gradient refresh.
result Achieves nearly optimal gradient complexity and second-order optimality.
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the r…
Exact second-order optimization for deep learning reduces computational cost and improves performance.
problem Inadequate use of second-order optimization methods in deep learning due to high computational cost and non-convexity.
method Developed an exact stochastic second-order Newton method that addresses the non-convexity issue and provides an expression for the stochastic Hessian.
result Exact second-order Newton direction formula and its application in deep learning datasets.
New method solves stochastic optimization problems with random models.
problem Optimizing stochastic objectives with deterministic constraints.
method Trust-Region Sequential Quadratic Programming with random model.
result Global convergence guarantees for first- and second-order stationary points.
In this paper, we study stochastic non-convex optimization with non-convex random functions. Recent studies on non-convex optimization revolve around establishing second-order convergence, i.e., converging to a nearly second-order optimal stationary points. However, existing results on stochastic non-convex optimizatio…
Paper examines risk measure expansions under FGM dependence, improving accuracy at extreme levels.
problem Capturing higher-order tail behavior and dependence effects in risk measures.
method Second-order asymptotic expansions using extreme value theory and regular variation theory.
result Second-order approximations reduce approximation errors, especially at extreme confidence levels.
Develops unbiased averaging methods for second order optimization in distributed systems.
problem Computing the Hessian is challenging and communication is a bottleneck in distributed optimization.
method Unbiased parameter averaging methods using sampling and sketching of the Hessian.
result Provably minimizes bias for sketched Newton directions.
Paper introduces FoMoH for optimization without backpropagation.
problem Optimizing machine learning models without backpropagation.
method Second-order hyperplane search, forward-mode stochastic gradient method, hyper-dual numbers, FoMoH.
result Developed a novel optimization algorithm that avoids backpropagation.
New method learns BN structures from data efficiently.
problem Learning sparse DAG structure of BN from continuous data.
method Consistent second-order conic integer programming with early stopping criterion.
result Near-optimal solutions to medium-size problems within reasonable time.
Second-order methods improve differential privacy in convex optimization.
problem Improving differential privacy in convex optimization.
method Developed a private variant of the regularized cubic Newton method for strongly convex loss functions.
result Achieves quadratic convergence and optimal excess loss for strongly convex loss functions.
Improved robustness in optimization methods using second-order information.
problem Scalability and sensitivity to mini-batch size in optimization methods.
method Mini-Batch Stochastic Variance-Reduced Newton ( e x t t t M b − S V R N exttt{Mb-SVRN} e x ttt M b − S V R N ) algorithm incorporating partial second-order information. result Achieves a fast linear convergence rate independent of mini-batch size for large data sizes.
AdaSub optimizes with second-order info in low-dims subspace.
problem Efficiently use second-order optimization methods with low computational cost.
method Adaptive subspace selection for second-order optimization.
result AdaSub outperforms other stochastic optimizers in time and iterations.
Second-order guarantees for federated learning algorithms.
problem Non-convex optimization in federated learning with saddle-points as bottlenecks.
method Drawing on recent results on second-order optimality in centralized and decentralized settings, establish second-order guarantees for federated learning algorithms.
result Established second-order guarantees for federated learning algorithms.
Second-order optimization speeds up deep hedging for complex options.
problem Hedging exotic options with market frictions in realistic markets.
method Second-order optimization scheme leveraging pathwise differentiability and Kronecker-factoring.
result Our method optimizes the policy in 1/4 the steps of standard optimization.
Modern proximal and stochastic gradient descent (SGD) methods are believed to efficiently minimize large composite objective functions, but such methods have two algorithmic challenges: (1) a lack of fast or justified stop conditions, and (2) sensitivity to the objective function's conditioning. In response to the firs…
Study finds risk sharing without convexity assumptions.
problem Finding fair risk allocations among agents with heterogeneous beliefs.
method Combines local comonotone improvement with Dieudonné-type argument.
result Existence of Pareto optima without convexity assumption.
The paper explores arbitrage opportunities in derivative markets under specific conditions.
problem Arbitrage opportunities in derivative markets under different conditions.
method Analyzes the relationship between pricing kernel monotonicity and stochastic arbitrage opportunities.
result Pricing kernel nonmonotonicity is equivalent to stochastic arbitrage opportunities under adequacy.
Paper develops a TR-SSQP method for noisy optimization with heavy-tailed noise.
problem Optimization problems with stochastic objectives and heavy-tailed noise.
method Trust-Region Stochastic Sequential Quadratic Programming (TR-SSQP) method.
result Achieves high-probability first-order and second-order stationarity bounds for heavy-tailed noise.
Paper proposes a second-order method for faster SVI convergence.
problem Poor convergence rate of first-order SVI algorithms.
method Derives Hessian matrix and implements two numerical schemes for efficient second-order SVI.
result Proposed approach achieves faster convergence compared to first-order SVI.
Data whitening and second order optimization harm generalization by reducing access to dataset information.
problem Harmful effects of data whitening and second order optimization on generalization in machine learning.
method Analysis of fully connected models and experimental verification.
result Data whitening and second order optimization reduce or prevent generalization by limiting access to dataset information.
First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …
A new method for faster optimization of noisy functions.
problem Optimizing noisy functions efficiently.
method A universal and adaptive second-order method for convex functions.
result Achieves O ( σ / T ) O(σ/ \sqrt{T}) O ( σ / T ) convergence for stochastic oracles and O ( 1 / T 3 ) O( 1 / T^3) O ( 1/ T 3 ) for deterministic oracles. New insights into adversarial vulnerability linked to manifold separability issues.
problem Adversarial vulnerability in machine learning models.
method Characterized data distribution as a low-dimensional manifold, focusing on on/off manifold directions and the impact of first-order vs. second-order optimization methods.
result First-order optimization methods lead to poor convergence in the off-manifold direction, causing adversarial vulnerability in inseparable datasets.
We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method. When the function is smooth enough, we show that acceleration can be achieved by a stable discretization of this O…
New algorithm finds approximate stationary points in non-convex optimization.
problem Finding approximate stationary points in non-convex stochastic optimization.
method Design of an algorithm using O ( ε − 3 ) O(ε^{-3}) O ( ε − 3 ) stochastic gradient and Hessian-vector products. result Optimal rate of O ( ε − 3 ) O(ε^{-3}) O ( ε − 3 ) for finding ε ε ε -approximate stationary points, matching lower bounds. Study second order integrability of Einstein deformations on Riemannian and Kähler manifolds.
problem Integrability condition for second order infinitesimal Einstein deformations.
method New expression for integrability condition, Koiso obstruction simplification.
result Complete description of integrable deformations on complex 2-plane Grassmannian.
The paper classifies second-order superintegrable systems with torsion and semi-degeneracy.
problem Classifying second-order superintegrable systems with torsion and semi-degeneracy.
method Information-geometric structure and geometric conditions for non-degeneracy.
result A ( n + 1 ) (n+1) ( n + 1 ) -parameter potential is non-degenerate if a certain trace-free tensor field vanishes. Integrates side information for robust portfolio optimization.
problem Portfolio optimization under uncertainty and side information.
method Distributionally robust optimization with optimal transport ambiguity set.
result The problem can be reformulated as a finite-dimensional optimization problem.
Estimates for harmonic functions in curved spaces.
problem Quantifying harmonic functions in curved spaces.
method Quantitative Sobolev estimates for p p p -harmonic functions in manifolds with curvature conditions. result Established a quantitative second order Sobolev estimate for p p p -harmonic functions. New method speeds up deep learning optimization.
problem Scalable second-order optimization for deep learning.
method Second-order optimization with algorithmic and numerical improvements.
result Significant convergence and wall-clock time improvements.