In the article the necessary and sufficient conditions for a representation of Lipschitz function of two variables as a difference of two convex functions are formulated. An algorithm of this representation is given. The outcome of this algorithm is a sequence of pairs of convex functions that converge uniformly to a p…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “difference of convex algorithm”
In this paper, we study a family of non-convex and possibly non-smooth inf-projection minimization problems, where the target objective function is equal to minimization of a joint function over another variable. This problem include difference of convex (DC) functions and a family of bi-convex functions as special cas…
Unified framework for analyzing online convex optimization across various settings.
problem Analyzing online convex optimization in different settings and feedback types.
method Unified framework allowing systematic proposal and analysis of meta-algorithms.
result Comparable regret bounds for various feedback types and adversary types.
Unified stability bounds for noisy SGD across convex and non-convex losses.
problem Deriving generalization bounds for noisy stochastic gradient descent.
method Unified approach using Lyapunov functions and applied probability.
result Time-uniform stability bounds for SGD on various loss functions.
In this two-part work, we propose an algorithmic framework for solving non-convex problems whose objective function is the sum of a number of smooth component functions plus a convex (possibly non-smooth) or/and smooth (possibly non-convex) regularization function. The proposed algorithm incorporates ideas from several…
A New Randomized Primal-Dual Algorithm for Convex Optimization with Optimal Last Iterate Ratesmath.OC
Unified algorithm solves convex optimization problems with optimal rates.
problem Solving nonsmooth constrained convex optimization problems.
method Unified randomized block-coordinate primal-dual algorithm.
result Achieves optimal convergence rates of and .
Boosted Difference of Convex Functions Algorithm solves VaR constrained portfolio optimization.
problem Designing VaR optimal portfolios under financial regulations.
method Boosted Difference of Convex Functions Algorithm (BDCA) with a novel line search framework.
result BDCA linearly converges to a Karush-Kuhn-Tucker point for VaR constrained portfolio problems.
Revisits PCA with new formulations and insights.
problem Improving PCA formulations and understanding.
method Difference-of-convex (DC) framework, kernelizability, out-of-sample applicability, simultaneous iteration, DCA perspective.
result PCA-like problems are kernelizable and have new optimization perspectives.
New algorithm solves complex non-convex problems efficiently.
problem Non-smooth non-convex problems with weakly convex and strongly concave components.
method Stochastic Moreau envelope approximate gradient method (SMAG).
result First single-loop algorithm with state-of-the-art convergence rate.
In this paper, we propose a successive convex approximation framework for sparse optimization where the nonsmooth regularization function in the objective function is nonconvex and it can be written as the difference of two convex functions. The proposed framework is based on a nontrivial combination of the majorizatio…
We propose a DC proximal Newton algorithm for solving nonconvex regularized sparse learning problems in high dimensions. Our proposed algorithm integrates the proximal Newton algorithm with multi-stage convex relaxation based on the difference of convex (DC) programming, and enjoys both strong computational and statist…
Paper solves high-order portfolio optimization with cardinality constraint.
problem Solving non-convex cardinality constrained high-order portfolio optimization.
method Transformed cardinality constraint into penalty term, proposed pDCA, pDCAe, and SCA algorithms.
result Proposed algorithms achieve high utility and sparse solutions efficiently.
Paper develops probabilistic bounds for a stochastic gradient algorithm in non-convex problems.
problem Stochastic optimization in non-convex finite sum problems.
method Develops a new dimension-free Azuma-Hoeffding type bound for a martingale difference sequence.
result Empirical results show superior probabilistic performance of Prob-SARAH compared to other algorithms.
Enforcing Fair Predicted Scores on Intervals of Percentiles by Difference-of-Convex Constraintscs.LG
Proposes a framework for partially fair machine learning models.
problem Achieving full fairness across all score ranges compromises predictive performance.
method Formulates model training as constrained optimization with difference-of-convex constraints, solvable by IDCA.
result Demonstrates high predictive performance while enforcing partial fairness in specific percentile intervals.
In this paper, we consider the problem of learning high-dimensional tensor regression problems with low-rank structure. One of the core challenges associated with learning high-dimensional models is computation since the underlying optimization problems are often non-convex. While convex relaxations could lead to polyn…
Paper proposes DC functions for better regularization of inverse problems with theoretical guarantees.
problem Improving regularization for ill-posed inverse problems.
method Introduces difference-of-convex (DC) functions and uses them with optimization algorithms like DCA and PSM.
result DC functions yield improved performance and theoretical guarantees compared to weakly convex functions.
New method improves MAP inference for CGMs on path graphs, avoiding approximation and maintaining integrality.
problem Improving MAP inference for aggregated count data in CGMs with small values.
method Formulated as a minimum cost flow problem, solved using DCA with efficient subroutines.
result Outputs higher quality solutions than conventional methods.
Perturbed Proximal Descent to Escape Saddle Points for Non-convex and Non-smooth Objective Functionscs.LG
We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…
We introduce a novel algorithm for solving learning problems where both the loss function and the regularizer are non-convex but belong to the class of difference of convex (DC) functions. Our contribution is a new general purpose proximal Newton algorithm that is able to deal with such a situation. The algorithm consi…
CCMM efficiently solves large-scale convex clustering problems.
problem Scalability and hierarchical structure in convex clustering.
method Majorization-minimization algorithm with cluster fusions and efficient updating.
result CCMM achieves efficient solutions for large datasets.
This paper reports applications of Difference of Convex functions (DC) programming to Learning from Demonstrations (LfD) and Reinforcement Learning (RL) with expert data. This is made possible because the norm of the Optimal Bellman Residual (OBR), which is at the heart of many RL and LfD algorithms, is DC. Improvement…
Randomized Geometric Algebra for Convex Neural Networks Optimizes Transfer Learning.
problem Training neural networks to global optimality via convex optimization.
method Randomized algorithms in Clifford's Geometric Algebra for hypercomplex vector spaces.
result Convex optimization and geometric algebra improve LLMs' robustness and reliability in transfer learning.
We propose a new stochastic L-BFGS algorithm and prove a linear convergence rate for strongly convex and smooth functions. Our algorithm draws heavily from a recent stochastic variant of L-BFGS proposed in Byrd et al. (2014) as well as a recent approach to variance reduction for stochastic gradient descent from Johnson…
By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of th…
In this paper, we study adaptive online convex optimization, and aim to design a universal algorithm that achieves optimal regret bounds for multiple common types of loss functions. Existing universal methods are limited in the sense that they are optimal for only a subclass of loss functions. To address this limitatio…
New algorithm adapts to optimize non-convex problems efficiently.
problem Inefficient hyperparameter tuning in existing optimization methods.
method Combining geometrization and SARAH algorithms.
result Achieves adaptivity to both accuracy and PL constant.
Unified SVM algorithm for various losses with fast training.
problem Training SVM models with different convex or nonconvex losses.
method Introducing LS-DC loss, proposing DCA-based UniSVM algorithm.
result Unified algorithm solves SVM models with any convex or nonconvex LS-DC loss efficiently.
A new algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with finite-sum structure.
method Nesterov Accelerated Shuffling Gradient (NASG) integrating Nesterov's acceleration with different shuffling schemes.
result Improved convergence rate of O(1/T) for unified shuffling schemes.
New algorithms reduce regret in online convex optimization with heavy-tailed gradients.
problem Challenges in online convex optimization with heavy-tailed gradients.
method Examined and analyzed old algorithms for online convex optimization in the heavy-tailed setting.
result Established new regret bounds for classical methods without algorithmic modification.
A fast sketching algorithm solves regularized least squares problems efficiently.
problem Solving large-scale optimization problems with convex or nonconvex regularization.
method Sketching for Regularized Optimization (SRO) algorithm that generates a sketch of the original data matrix and solves the sketched problem.
result General theoretical results for the approximation error between the original and sketched problems, including minimax rates for sparse signal estimation.
In this paper, we consider first-order convergence theory and algorithms for solving a class of non-convex non-concave min-max saddle-point problems, whose objective function is weakly convex in the variables of minimization and weakly concave in the variables of maximization. It has many important applications in mach…
Sparse additive modeling is a class of effective methods for performing high-dimensional nonparametric regression. In this work we show how shape constraints such as convexity/concavity and their extensions, can be integrated into additive models. The proposed sparse difference of convex additive models (SDCAM) can est…
Adaptive methods improve gradient descent and proximal gradient for convex optimization.
problem Improving efficiency of gradient descent and proximal gradient methods.
method Adaptive versions of GD and ProxGD using local curvature information.
result Proved convergence with local Lipschitz gradient assumptions.
New algorithms improve submodular minimization via DC programming.
problem Minimizing the difference of two submodular functions.
method Introducing variants of the DC algorithm (DCA) and its complete form (CDCA) for DC programs corresponding to DS minimization.
result Our algorithms outperform existing baselines on speech corpus selection and feature selection.
This paper proposes a new family of algorithms for training neural networks (NNs). These are based on recent developments in the field of non-convex optimization, going under the general name of successive convex approximation (SCA) techniques. The basic idea is to iteratively replace the original (non-convex, highly d…
We describe and analyze a simple algorithm for principal component analysis and singular value decomposition, VR-PCA, which uses computationally cheap stochastic iterations, yet converges exponentially fast to the optimal solution. In contrast, existing algorithms suffer either from slow convergence, or computationally…
In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared …
This paper is a follow up to the previous author's paper on convex optimization. In that paper we began the process of adjusting greedy-type algorithms from nonlinear approximation for finding sparse solutions of convex optimization problems. We modified there three the most popular in nonlinear approximation in Banach…
Two-stage nonconvex algorithm and convex relaxation both achieve optimal accuracy in noisy blind deconvolution.
problem Solving bilinear systems of equations with random noise under different designs.
method Two-stage nonconvex algorithm and convex relaxation.
result Both methods achieve minimax-optimal accuracy in the presence of random noise.
We study dual-based algorithms for distributed convex optimization problems over networks, where the objective is to minimize a sum of functions over in a network. We provide complexity bounds for four different cases, namely: each function is strongly convex and smooth, each function is ei…
Stochastic-gradient-based optimization has been a core enabling methodology in applications to large-scale problems in machine learning and related areas. Despite the progress, the gap between theory and practice remains significant, with theoreticians pursuing mathematical optimality at a cost of obtaining specialized…
Paper develops a discounted algorithm for online convex optimization that adapts to unknown discount factors.
problem Developing an algorithm that can adapt to an unknown discount factor in online convex optimization.
method Smoothed Online Gradient Descent (SOGD) with Discounted-Normal-Predictor (DNP).
result Achieves a uniform discounted regret across a continuous interval of discount factors.
Matrix factorization (MF) is a versatile learning method that has found wide applications in various data-driven disciplines. Still, many MF algorithms do not adequately scale with the size of available datasets and/or lack interpretability. To improve the computational efficiency of the method, an online (streaming) M…
Paper introduces a new framework for optimizing non-convex functions.
problem Optimizing non-convex functions, especially DR-submodular and concave functions.
method Developed a general meta-algorithm to convert linear/quadratic optimization to optimization of upper-linearizable/quadratizable functions.
result Unified approach to concave and DR-submodular optimization problems.
Improved algorithm finds second-order stationary points in non-convex optimization.
problem Minimizing non-convex objectives while preserving training data privacy.
method SpiderBoost framework with two gradient oracles: precise and less precise.
result Improved rates for finding second-order stationary points.
Clustering is a fundamental problem in many scientific applications. Standard methods such as -means, Gaussian mixture models, and hierarchical clustering, however, are beset by local minima, which are sometimes drastically suboptimal. Recently introduced convex relaxations of -means and hierarchical clustering s…
We show that the herding procedure of Welling (2009) takes exactly the form of a standard convex optimization algorithm--namely a conditional gradient algorithm minimizing a quadratic moment discrepancy. This link enables us to invoke convergence results from convex optimization and to consider faster alternatives for …
We investigate online convex optimization in changing environments, and choose the adaptive regret as the performance measure. The goal is to achieve a small regret over every interval so that the comparator is allowed to change over time. Different from previous works that only utilize the convexity condition, this pa…