We study online convex optimization under stochastic sub-gradient observation faults, where we introduce adaptive algorithms with minimax optimal regret guarantees. We specifically study scenarios where our sub-gradient observations can be noisy or even completely missing in a stochastic manner. To this end, we propose…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “sub-gradient descent”
FPCA optimizes fairness in target vectors' span.
problem Fairness in principal component analysis for multiple target vectors.
method Non-concave maximization of worst projected target norm using sub-gradient descent.
result Optimization landscape is benign with globally optimal local minima.
Computationally Efficient and Statistically Optimal Robust High-Dimensional Linear Regressionmath.ST
New algorithm for robust high-dimensional linear regression is both fast and statistically optimal.
problem Challenges in high-dimensional linear regression under heavy-tailed noise or outliers.
method Projected sub-gradient descent algorithm for sparse and low-rank regression problems.
result Algorithm achieves linear convergence and statistical optimality under various noise conditions.
The paper establishes sub-gradient estimates and entropy formulas for quaternionic contact geometry heat equations.
problem Developing sub-gradient estimates and entropy formulas for quaternionic contact geometry.
method Establishing sub-gradient estimates and entropy formulas for the quaternionic contact heat equation.
result Two Perelman-type entropy formulas and sub-gradient estimates for the quaternionic contact heat equation.
GeoAdaLer enhances geometric understanding of Adam for stochastic optimization.
problem Understanding geometric principles behind Adam's success in stochastic optimization.
method Introduces GeoAdaLer, an adaptive learning method based on geometric properties.
result Extends interpretability and effectiveness in complex optimization scenarios.
Computationally Efficient and Statistically Optimal Robust Low-rank Matrix and Tensor Estimationmath.ST
A new algorithm improves both computational efficiency and statistical optimality for robust low-rank matrix and tensor estimation.
problem Challenges in low-rank matrix estimation under heavy-tailed noise, both computationally and statistically.
method Riemannian sub-gradient (RsGrad) algorithm, which is computationally efficient and statistically optimal.
result RsGrad achieves linear convergence and statistical optimality for robust loss functions under Gaussian and heavy-tailed noise.
ICCNLS models complex relationships as convex and concave components.
problem Complex input-output relationships with affine ambiguity.
method Sub-gradient constrained affine functions, global orthogonality constraints, L1, L2, and elastic net regularisation.
result Improved predictive accuracy and model simplicity compared to conventional methods.
New framework learns labels at both bag and graph levels.
problem Learning multi-label classifiers from multi-graph bags.
method Designing scoring functions and rank-loss objective for graph and bag levels; developing sub-gradient descent algorithm.
result Superior performance over state-of-the-art algorithms.
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assump…
In this work and the supporting Part II, we examine the performance of stochastic sub-gradient learning strategies under weaker conditions than usually considered in the literature. The new conditions are shown to be automatically satisfied by several important cases of interest including SVM, LASSO, and Total-Variatio…
Sub-gradient method recovers low-rank matrices robustly from noisy measurements.
problem Recovering low-rank matrices from noisy measurements with unknown rank.
method Sub-gradient method with small initialization, robust to over-parameterization and noise.
result Sub-gradient method converges exponentially fast to the true solution under noisy and over-parameterized conditions.
Paper relaxes stability and generalization assumptions for SGD.
problem Stability and generalization for SGD under restrictive assumptions.
method Introduces on-average model stability and develops novel bounds.
result First-ever-known fast bounds in low-noise setting using stability approach.
In this paper, we derive a sub-gradient estimate for pseudoharmonic maps from noncompact complete Sasakian manifolds which satisfy CR sub-Laplace comparison property, to simply-connected Riemannian manifolds with nonpositive sectional curvature. As its application, we obtain some Liouville theorems for pseudoharmonic m…
In the era of big data, an important weapon in a machine learning researcher's arsenal is a scalable Support Vector Machine (SVM) algorithm. SVMs are extensively used for solving classification problems. Traditional algorithms for learning SVMs often scale super linearly with training set size which becomes infeasible …
Binary classification is a common statistical learning problem in which a model is estimated on a set of covariates for some outcome indicating the membership of one of two classes. In the literature, there exists a distinction between hard and soft classification. In soft classification, the conditional class probabil…
New SPS variant improves non-smooth optimization without small gradients.
problem Improving non-smooth optimization without small gradients.
method Safeguarded Stochastic Polyak Step Size (SPS) for non-smooth optimization.
result Rigorous convergence guarantees for non-smooth convex optimization without strong assumptions.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Binary Iterative Hard Thresholding converges with optimal number of 1-bit measurements.
problem Recovering sparse signals from 1-bit compressed measurements.
method Binary Iterative Hard Thresholding (BIHT) algorithm.
result BIHT converges with only O(k/ε) measurements, optimal for recovery.
We consider the binary classification problem when data are large and subject to unknown but bounded uncertainties. We address the problem by formulating the nonlinear support vector machine training problem with robust optimization. To do so, we analyze and propose two bounding schemes for uncertainties associated to …
CR Sub-Laplacian Comparison and Liouville-type Theorem in a Complete Noncompact Sasakian Manifoldmath.AP
In this paper, we first obtain the sub-Laplacian comparison theorem in a complete noncompact pseudohermitian manifold of vanishing torsion (i.e. Sasakian manifold). Secondly, we derive the sub-gradient estimate for positive pseudoharmonic functions in a complete noncompact pseudohermitian manifold which satisfies the C…
The analysis in Part I revealed interesting properties for subgradient learning algorithms in the context of stochastic optimization when gradient noise is present. These algorithms are used when the risk functions are non-smooth and involve non-differentiable components. They have been long recognized as being slow co…
A new algorithm speeds up sparse-penalized quantile regression solving non-convex penalties.
problem Sparse-penalized quantile regression with non-convex penalties.
method Single-loop smoothing ADMM (SIAD) algorithm for faster convergence.
result SIAD method outperforms existing approaches in solving sparse-penalized quantile regression.
Geodesic convexity generalizes the notion of (vector space) convexity to nonlinear metric spaces. But unlike convex optimization, geodesically convex (g-convex) optimization is much less developed. In this paper we contribute to the understanding of g-convex optimization by developing iteration complexity analysis for …
We describe an approach for incorporating prior knowledge into machine learning algorithms. We aim at applications in physics and signal processing in which we know that certain operations must be embedded into the algorithm. Any operation that allows computation of a gradient or sub-gradient towards its inputs is suit…
Converting an n-dimensional vector to a probability distribution over n objects is a commonly used component in many machine learning tasks like multiclass classification, multilabel classification, attention mechanisms etc. For this, several probability mapping functions have been proposed and employed in literature s…
A neural network learns a convex regularizer for better image reconstruction.
problem Improving image reconstruction in inverse problems.
method Adversarial training of a data-adaptive ICNN as a convex regularizer.
result The convex regularizer leads to better convergence and error reduction in image reconstruction.
Linear optimization is many times algorithmically simpler than non-linear convex optimization. Linear optimization over matroid polytopes, matching polytopes and path polytopes are example of problems for which we have simple and efficient combinatorial algorithms, but whose non-linear convex counterpart is harder and …
Motivation: A major challenge in the development of machine learning based methods in computational biology is that data may not be accurately labeled due to the time and resources required for experimentally annotating properties of proteins and DNA sequences. Standard supervised learning algorithms assume accurate in…
Stochastic (sub)gradient methods require step size schedule tuning to perform well in practice. Classical tuning strategies decay the step size polynomially and lead to optimal sublinear rates on (strongly) convex problems. An alternative schedule, popular in nonconvex optimization, is called \emph{geometric step decay…
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
Reparameterizes mirror descent as gradient descent for efficient sparse learning.
problem Efficiently training small sparse networks with mirror descent.
method Develops a framework to convert mirror descent updates into gradient descent updates on different parameters.
result Mirror descent can be reparameterized as gradient descent on modified parameters, facilitating standard backpropagation.
In this paper we propose a randomized primal-dual proximal block coordinate updating framework for a general multi-block convex optimization model with coupled objective function and linear constraints. Assuming mere convexity, we establish its convergence rate in terms of the objective value and feasibility m…
Derives Mirror Descent from gradient flow on a Riemannian manifold.
problem No specific problem stated; focuses on derivation.
method Derives Mirror Descent from gradient flow on a Riemannian manifold with a natural discretization.
result Generalizes Mirror Descent to non-Hessian metrics.
Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
In this note, we observe the behavior of gradient flow and discrete and noisy gradient descent in some simple settings. It is commonly noted that addition of noise to gradient descent can affect the trajectory of gradient descent. Here, we run some computer experiments for gradient descent on some simple functions, and…
Graph-Structured Multi-task Regression and an Efficient Optimization Method for General Fused Lassostat.ML
We consider the problem of learning a structured multi-task regression, where the output consists of multiple responses that are related by a graph and the correlated response variables are dependent on the common inputs in a sparse but synergistic manner. Previous methods such as l1/l2-regularized multi-task regressio…
New analysis shows GMD can converge linearly under PL-like conditions.
problem Establishing linear convergence for generalized mirror descent.
method PL-based analysis for time-dependent mirrors, Taylor-series approach for stochastic GMD.
result Linear convergence of stochastic GMD under PL-like conditions.
A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
Double descent phenomenon explained in simple terms.
problem Understanding the surprising drop in test error in overparameterized models.
method Informal explanation using linear algebra and probability, visual intuition with polynomial regression, mathematical analysis with ordinary linear regression.
result Three factors create double descent: data undersampling, model size, and parameter count. Ablating any one of these factors prevents double descent.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
New insights into double descent phenomenon in neural networks.
problem Understanding the double descent behavior in deep learning models.
method Linear teacher-student setup and tools from statistical physics.
result Distinct features are learned at different scales, leading to epoch-wise double descent.
Gradient descent dynamics in nonconvex models explained with universality.
problem Understanding long-time behavior of nonconvex gradient descent.
method Developed a state evolution system for tracking gradient descent iterates.
result Gradient descent iterates are approximately independent of data and strongly incoherent with feature vectors.
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcond-mat.dis-nn
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Overview of non-stochastic-gradient SA algorithms in signal processing and ML.
problem Dealing with large data sets and uncertainties in signal processing and machine learning.
method General framework of SA algorithms using Lyapunov functions.
result Unified convergence properties of non-stochastic-gradient algorithms.
The paper connects tempering and entropic mirror descent for sampling.
problem Sampling from a target distribution with known unnormalized density.
method Establishes the connection between tempering SMC and entropic mirror descent, deriving convergence rates and geometric insights.
result Tempering SMC iterates correspond to entropic mirror descent on the reverse KL divergence, providing new optimization perspectives.
We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had …
Develops DP-SCD for stochastic coordinate descent, making it differentially private.
problem Privacy leak in auxiliary information during stochastic coordinate descent training.
method Develops DP-SCD, leveraging independent noise addition and decoupling/parallelizing coordinate updates.
result Demonstrates competitive performance against DP-SGD with less tuning.