Study KKT conditions for multi-objective optimization on Hadamard manifolds.
problem Optimizing multi-objective interval-valued functions on Hadamard manifolds.
method Developed KKT conditions for Pareto optimal solutions under different ordering and convexity notions.
result Results are more general than on Euclidean spaces.
This work improves data reconstruction methods by ensuring unique solutions and refining optimization.
problem Ensuring unique solutions and optimizing reconstruction from KKT conditions.
method Discussion of sufficient conditions for unique solutions and introduction of sample splitting for optimization.
result Sample splitting improves reconstruction performance across various methods.
New insights into how linear classifiers and leaky ReLU networks can overfit without harming generalization.
problem Understanding conditions for benign overfitting in linear classifiers and leaky ReLU networks.
method Utilizing Karush--Kuhn--Tucker (KKT) conditions for margin maximization.
result Satisfaction of KKT conditions leads to benign overfitting in linear classifiers and leaky ReLU networks.
The paper refines NOTEARS for learning Bayesian networks, improving accuracy and efficiency.
problem Learning Bayesian networks from continuous optimization.
method Generalized algebraic characterizations and Karush-Kuhn-Tucker (KKT) conditions for optimization.
result Local search post-processing improves structural Hamming distance by a factor of 2 or more.
Early training of deep neural networks leads to small, directionally converging weights.
problem Training dynamics of deep homogeneous neural networks with small initializations.
method Gradient flow analysis and study of KKT points for neural correlation function.
result Weights converge in direction to KKT points during early training stages.
The paper tackles partial inference in structured prediction using a convex optimization approach.
problem Maximizing a score function with unary and pairwise potentials in graph label spaces.
method Generative model approach with two-stage convex optimization for label recovery.
result Conditions for recovering a majority of labels with provable guarantees.
We embed KKT points in neural networks of different sizes.
problem Classifying data using homogeneous neural networks.
method Introducing KKT point embedding principle and proving it for different network types.
result KKT points of a smaller network can be mapped to those of a larger network via linear transformations.
Study optimality conditions for interval-valued optimization problems on Riemannian manifolds.
problem Optimizing interval-valued functions on Riemannian manifolds under a total order relation.
method Generalized Hukuhara directional differentiability to derive KKT-type optimality conditions.
result Derives optimality conditions for interval-valued optimization problems on Riemannian manifolds.
A scalable framework optimizes multi-asset portfolios with constraints.
problem Optimizing multi-asset portfolios with inequality constraints.
method Integrates neural policies with Pontryagin's Maximum Principle, enforcing feasibility via log-barrier regularization.
result Recover KKT-optimal policies in high-dimensional problems without violating constraints.
Develops a new screening method called Newton screening for faster and more accurate sparse learning.
problem Sparse learning problems with large-scale optimization.
method Newton screening method with built-in working set and dual variable updates.
result Newton screening achieves one-step local convergence and sharp estimation error bound.
New estimator for tensor weights with improved bias.
problem Estimating tensor weights from noisy data.
method Random matrix theory and KKT conditions.
result Asymptotically unbiased estimator for tensor rank.
Paper tackles bilevel optimization problems using penalty methods.
problem Unconstrained and constrained bilevel optimization problems with nonsmooth lower levels.
method Introduces first-order penalty methods and O(ε−4logε−1) and O(ε−7logε−1) operation complexities. result Establishes operation complexities for finding ε-KKT solutions. Method solves nonconvex constrained optimization problems with a new augmented Lagrangian approach.
problem Nonconvex composite functional constraints with inequality constraints.
method First-order augmented Lagrangian method with smoothed prox-linear reformulation.
result Explicit convergence rates for the proposed method in terms of KKT residual.
New algorithms minimize dynamic regret for strongly convex losses.
problem Minimizing dynamic regret for strongly convex losses.
method Developed Strongly Adaptive algorithms exploiting KKT conditions.
result Achieved near optimal dynamic regret of O(d1/3n1/3extTV[u1:n]2/3∨d). New algorithms minimize dynamic regret in non-stationary online learning.
problem Universal dynamic regret minimization under exp-concave and smooth losses.
method Strongly Adaptive algorithms with a path variational based on second order differences of the comparator sequence.
result Achieve a dynamic regret of ildeO(d2n1/5Cn2/5∨d2), optimal modulo dependencies. New algorithm reduces dynamic regret for exp-concave losses.
problem Minimizing dynamic regret in online learning with exp-concave losses.
method Integrates KKT conditions to achieve optimal dynamic regret.
result Achieves dynamic regret of ildeO∗(n1/3Cn2/3∨1). We propose a semismooth Newton algorithm for pathwise optimization (SNAP) for the LASSO and Enet in sparse, high-dimensional linear regression. SNAP is derived from a suitable formulation of the KKT conditions based on Newton derivatives. It solves the semismooth KKT equations efficiently by actively and continuously s…
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
A new method solves complex constrained minimax problems.
problem Solving constrained minimax optimization problems.
method First-order augmented Lagrangian method.
result Established an operation complexity of O(ε−4logε−1). Study shows how steepest descent algorithms' geometric margin increases during training.
problem Understanding implicit bias in steepest descent algorithms for neural networks.
method Analysis of steepest descent algorithms with infinitesimal learning rates in homogeneous neural networks.
result Limit points of training trajectories correspond to KKT points of margin-maximization problems.
For rational homology 3-spheres, there exist two universal finite-type invariants: the Le-Murakami-Ohtsuki invariant and the Kontsevich-Kuperberg-Thurston invariant. These invariants take values in the same space of "Jacobi diagrams", but it is not known whether they are equal. In 2004, Lescop proved that the KKT invar…
We consider rules for discarding predictors in lasso regression and related problems, for computational efficiency. El Ghaoui et al (2010) propose "SAFE" rules that guarantee that a coefficient will be zero in the solution, based on the inner products of each predictor with the outcome. In this paper we propose strong …
A new method solves a complex optimization problem efficiently.
problem Nonconvex-strongly-concave constrained minimax optimization.
method First-order augmented Lagrangian method with a first-order subproblem solver.
result Achieves improved operation complexity for finding solutions.
Enhanced PC2 improves surrogate modeling for high-dimensional problems.
problem Degrading performance and efficiency of PC2 in high-dimensional parameter spaces. method Integrates SULM solver and D-optimal sampling strategy into PC2 framework. result Enhanced PC2 demonstrates better comprehensive capability and efficiency. New methods solve complex optimization problems without strong convexity assumptions.
problem Complex bilevel optimization problems with minimax lower-level structures.
method Penalty-based first-order methods for bilevel minimax optimization.
result Achieves ε-KKT point with improved oracle complexity. In this paper, we propose ℓp-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the ℓp-norm regularized models…
This study explains and mitigates inflated returns and turnover in SPO-based portfolio optimization.
problem Inflated returns and excessive turnover in SPO-based portfolio optimization.
method KKT-based interpretation of portfolio decisions as ranking over adjusted scores, empirical evaluation of stabilization mechanisms.
result Realistic output constraints and portfolio-level turnover control improve SPO-based strategies.
Algorithm improves SVM classification in non-Euclidean spaces.
problem Limitations of traditional SVM in non-Euclidean spaces.
method Covariance-adjusted SVM using Cholesky Decomposition.
result Cholesky-SVM outperforms traditional SVM in non-Euclidean spaces.
New findings on the max margin problem in neural networks.
problem Understanding the max margin problem in neural networks.
method Analyzing gradient flow and max margin problem in linear and ReLU networks.
result The KKT point is not always an optimum of the max margin problem.
Paper introduces MKL-L0/1-SVM for SVM with (0,1) loss.
problem Optimization of SVM with (0,1) loss function. method MKL framework combined with ADMM algorithm for solving the optimization problem.
result Performance of MKL-L0/1-SVM comparable to SimpleMKL. In this paper we study a broad class of structured nonlinear programming (SNLP) problems. In particular, we first establish the first-order optimality conditions for them. Then we propose sequential convex programming (SCP) methods for solving them in which each iteration is obtained by solving a convex programming pro…
A Bayesian factor graph reduced to normal form consists in the interconnection of diverter units (or equal constraint units) and Single-Input/Single-Output (SISO) blocks. In this framework localized adaptation rules are explicitly derived from a constrained maximum likelihood (ML) formulation and from a minimum KL-dive…
Symmetric nonnegative matrix factorization (SymNMF) has important applications in data analytics problems such as document clustering, community detection and image segmentation. In this paper, we propose a novel nonconvex variable splitting method for solving SymNMF. The proposed algorithm is guaranteed to converge to…
The support vector machine (SVM) is a powerful and widely used classification algorithm. This paper uses the Karush-Kuhn-Tucker conditions to provide rigorous mathematical proof for new insights into the behavior of SVM. These insights provide perhaps unexpected relationships between SVM and two other linear classifier…
We introduce in this paper a novel strategy for efficiently approximating the Sinkhorn distance between two discrete measures. After identifying neglectable components of the dual solution of the regularized Sinkhorn problem, we propose to screen those components by directly setting them at that value before entering t…
This study presents a rapid multiple incremental and decremental mechanism based on Weight-Error Curves (WECs) for support-vector analysis. Recursion-free computation is proposed for predicting the Lagrangian multipliers of new samples. This study examines Ridge Support Vector Models, subsequently devising a recursion-…
Paper proposes SMO for solving bilevel optimization problems efficiently.
problem Solving bilevel optimization problems with nonsmooth convex lower-level and nonconvex upper-level objectives.
method Sequential minimax optimization (SMO) method using modified augmented Lagrangian and penalty schemes.
result Improves operation complexity for finding ε-KKT solutions. New single-loop algorithm tackles weakly convex constraints in stochastic optimization.
problem Optimization with weakly convex constraints in machine learning.
method Single-loop penalty-based stochastic algorithm using hinge-based penalty.
result Achieves state-of-the-art complexity for finding approximate KKT solutions.
We analyze the necessary and sufficient conditions for exact inference of a latent model. In latent models, each entity is associated with a latent variable following some probability distribution. The challenging question we try to solve is: can we perform exact inference without observing the latent variables, even w…
We present foundations for using Model Predictive Control (MPC) as a differentiable policy class for reinforcement learning in continuous state and action spaces. This provides one way of leveraging and combining the advantages of model-free and model-based approaches. Specifically, we differentiate through MPC by usin…
Develops a method for learning sparse generalized linear models in high-dimensional data.
problem Feature selection in high-dimensional data with many variables.
method GSDAR method based on KKT conditions for ℓ0-penalized maximum likelihood estimations. result The errors of the proposed estimate decay exponentially to the optimal order under certain conditions.
GD iterates for non-homogeneous deep nets increase margin and converge in direction.
problem Understanding implicit bias in non-homogeneous deep networks.
method Characterization of GD iterates' properties starting from small empirical risk.
result GD iterates converge in direction despite diverging norms, satisfying KKT conditions.
We create consistent option surfaces without arbitrage.
problem Constructing consistent option surfaces free of arbitrage across different maturities.
method Combining PCA-Smolyak approximation with chain-consistent diffusion and c-EMOT bridge.
result Computable certificates for strong convexity, solver correctness, and Dupire/Greeks stability.
Gradient ascent method successfully removes specific data points from neural networks without retraining.
problem Addressing privacy and ethical concerns by removing specific data points from trained models.
method Gradient ascent approach to unlearning, leveraging the implicit bias of gradient descent towards margin maximization conditions.
result Gradient ascent method can successfully unlearn specific data points from two-layer ReLU neural networks without retraining.
In this paper, we propose exact passive-aggressive (PA) online algorithms for learning to rank. The proposed algorithms can be used even when we have interval labels instead of actual labels for examples. The proposed algorithms solve a convex optimization problem at every trial. We find exact solution to those optimiz…
Symmetric nonnegative matrix factorization has found abundant applications in various domains by providing a symmetric low-rank decomposition of nonnegative matrices. In this paper we propose a Frank-Wolfe (FW) solver to optimize the symmetric nonnegative matrix factorization problem under a simplicial constraint, whic…
New algorithm tackles stochastic optimization with inequality constraints.
problem Stochastic optimization with inequality constraints in various applications.
method Active-set stochastic sequential quadratic programming (StoSQP) with a differentiable exact augmented Lagrangian.
result Global convergence for any initialization, KKT residuals converge to zero almost surely.
DCCNNs reduce computational overhead and ambiguity in convolutional neural networks.
problem Reducing computational overhead and ambiguity in convolutional neural networks.
method Introducing a primal learning problem and constructing a dual convex training program, using Fenchel conjugates and Karush-Kuhn-Tucker conditions.
result Eliminates ambiguity and reduces computational overhead in constructing a large kernel matrix.