The paper establishes sub-gradient estimates and entropy formulas for quaternionic contact geometry heat equations.
problem Developing sub-gradient estimates and entropy formulas for quaternionic contact geometry.
method Establishing sub-gradient estimates and entropy formulas for the quaternionic contact heat equation.
result Two Perelman-type entropy formulas and sub-gradient estimates for the quaternionic contact heat equation.
Optimal algorithms for online convex optimization with missing sub-gradient observations.
problem Online convex optimization with noisy or missing sub-gradient observations.
method Adaptive algorithms using sub-gradient descent with minimax optimal regret guarantees.
result Achieves tight minimax optimal regret bounds with empirical property estimation.
ICCNLS models complex relationships as convex and concave components.
problem Complex input-output relationships with affine ambiguity.
method Sub-gradient constrained affine functions, global orthogonality constraints, L1, L2, and elastic net regularisation.
result Improved predictive accuracy and model simplicity compared to conventional methods.
In this work and the supporting Part II, we examine the performance of stochastic sub-gradient learning strategies under weaker conditions than usually considered in the literature. The new conditions are shown to be automatically satisfied by several important cases of interest including SVM, LASSO, and Total-Variatio…
GADGET SVM uses gossip-based distributed learning for scalable SVMs.
problem Scalability issues in traditional SVM algorithms for large datasets.
method Gossip-based distributed learning for the primal SVM formulation.
result Performance comparable to centralized and online SVM algorithms.
Sub-gradient method recovers low-rank matrices robustly from noisy measurements.
problem Recovering low-rank matrices from noisy measurements with unknown rank.
method Sub-gradient method with small initialization, robust to over-parameterization and noise.
result Sub-gradient method converges exponentially fast to the true solution under noisy and over-parameterized conditions.
FPCA optimizes fairness in target vectors' span.
problem Fairness in principal component analysis for multiple target vectors.
method Non-concave maximization of worst projected target norm using sub-gradient descent.
result Optimization landscape is benign with globally optimal local minima.
New algorithm for robust high-dimensional linear regression is both fast and statistically optimal.
problem Challenges in high-dimensional linear regression under heavy-tailed noise or outliers.
method Projected sub-gradient descent algorithm for sparse and low-rank regression problems.
result Algorithm achieves linear convergence and statistical optimality under various noise conditions.
In this paper, we derive a sub-gradient estimate for pseudoharmonic maps from noncompact complete Sasakian manifolds which satisfy CR sub-Laplace comparison property, to simply-connected Riemannian manifolds with nonpositive sectional curvature. As its application, we obtain some Liouville theorems for pseudoharmonic m…
A new algorithm improves both computational efficiency and statistical optimality for robust low-rank matrix and tensor estimation.
problem Challenges in low-rank matrix estimation under heavy-tailed noise, both computationally and statistically.
method Riemannian sub-gradient (RsGrad) algorithm, which is computationally efficient and statistically optimal.
result RsGrad achieves linear convergence and statistical optimality for robust loss functions under Gaussian and heavy-tailed noise.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
The analysis in Part I revealed interesting properties for subgradient learning algorithms in the context of stochastic optimization when gradient noise is present. These algorithms are used when the risk functions are non-smooth and involve non-differentiable components. They have been long recognized as being slow co…
We consider the binary classification problem when data are large and subject to unknown but bounded uncertainties. We address the problem by formulating the nonlinear support vector machine training problem with robust optimization. To do so, we analyze and propose two bounding schemes for uncertainties associated to …
In this paper, we first obtain the sub-Laplacian comparison theorem in a complete noncompact pseudohermitian manifold of vanishing torsion (i.e. Sasakian manifold). Secondly, we derive the sub-gradient estimate for positive pseudoharmonic functions in a complete noncompact pseudohermitian manifold which satisfies the C…
New framework learns labels at both bag and graph levels.
problem Learning multi-label classifiers from multi-graph bags.
method Designing scoring functions and rank-loss objective for graph and bag levels; developing sub-gradient descent algorithm.
result Superior performance over state-of-the-art algorithms.
Geodesic convexity generalizes the notion of (vector space) convexity to nonlinear metric spaces. But unlike convex optimization, geodesically convex (g-convex) optimization is much less developed. In this paper we contribute to the understanding of g-convex optimization by developing iteration complexity analysis for …
GeoAdaLer enhances geometric understanding of Adam for stochastic optimization.
problem Understanding geometric principles behind Adam's success in stochastic optimization.
method Introduces GeoAdaLer, an adaptive learning method based on geometric properties.
result Extends interpretability and effectiveness in complex optimization scenarios.
This work analyzes and improves stochastic gradient methods for GAN training.
problem Understanding the training dynamics of GANs, particularly their convergence.
method Continuous-time analysis using differential equations, focusing on simGD and its variants.
result The methods converge under different assumptions, providing new insights into GAN training.
Reduces error bounds by incorporating known operations into deep nets.
problem Maximizing training error in machine learning models.
method Incorporating known operations into deep learning models to reduce error bounds.
result Known operations reduce the maximum training error bounds for deep nets.
Binary classification is a common statistical learning problem in which a model is estimated on a set of covariates for some outcome indicating the membership of one of two classes. In the literature, there exists a distinction between hard and soft classification. In soft classification, the conditional class probabil…
A neural network learns a convex regularizer for better image reconstruction.
problem Improving image reconstruction in inverse problems.
method Adversarial training of a data-adaptive ICNN as a convex regularizer.
result The convex regularizer leads to better convergence and error reduction in image reconstruction.
New SPS variant improves non-smooth optimization without small gradients.
problem Improving non-smooth optimization without small gradients.
method Safeguarded Stochastic Polyak Step Size (SPSsafe) for non-smooth optimization. result Rigorous convergence guarantees for non-smooth convex optimization without strong assumptions.
Unified framework for sparse alternatives to softmax with control over sparsity.
problem Lack of understanding and explicit control over sparsity in probability mapping functions.
method Unified framework encompassing softmax, sum-normalization, spherical softmax, and sparsemax. Two novel sparse formulations (sparsegen-lin and sparsehourglass) and convex loss functions developed.
result Improved performance in multilabel classification and seq2seq tasks like neural machine translation and abstractive summarization.
Linear optimization is many times algorithmically simpler than non-linear convex optimization. Linear optimization over matroid polytopes, matching polytopes and path polytopes are example of problems for which we have simple and efficient combinatorial algorithms, but whose non-linear convex counterpart is harder and …
Motivation: A major challenge in the development of machine learning based methods in computational biology is that data may not be accurately labeled due to the time and resources required for experimentally annotating properties of proteins and DNA sequences. Standard supervised learning algorithms assume accurate in…
Paper relaxes stability and generalization assumptions for SGD.
problem Stability and generalization for SGD under restrictive assumptions.
method Introduces on-average model stability and develops novel bounds.
result First-ever-known fast bounds in low-noise setting using stability approach.
A new algorithm speeds up sparse-penalized quantile regression solving non-convex penalties.
problem Sparse-penalized quantile regression with non-convex penalties.
method Single-loop smoothing ADMM (SIAD) algorithm for faster convergence.
result SIAD method outperforms existing approaches in solving sparse-penalized quantile regression.
In this paper we propose a randomized primal-dual proximal block coordinate updating framework for a general multi-block convex optimization model with coupled objective function and linear constraints. Assuming mere convexity, we establish its O(1/t) convergence rate in terms of the objective value and feasibility m…
Geometric step decay schedules improve stochastic algorithms' convergence on sharp nonconvex problems.
problem Convergence of stochastic algorithms on sharp nonconvex problems.
method Geometric step decay schedule applied to stochastic algorithms.
result Geometric step decay schedules lead to local linear convergence rates for sharp nonconvex problems.
We consider the problem of learning a structured multi-task regression, where the output consists of multiple responses that are related by a graph and the correlated response variables are dependent on the common inputs in a sparse but synergistic manner. Previous methods such as l1/l2-regularized multi-task regressio…
Overview of non-stochastic-gradient SA algorithms in signal processing and ML.
problem Dealing with large data sets and uncertainties in signal processing and machine learning.
method General framework of SA algorithms using Lyapunov functions.
result Unified convergence properties of non-stochastic-gradient algorithms.
A new neural network model identifies hysteresis universally.
problem Inability of existing models to simulate hysteresis universally.
method Inspired by the Preisach model, an Extended Preisach Neural Network (EPNN) is introduced with two hidden layers and a hybrid training algorithm.
result EPNN successfully identifies various hysteresis phenomena from different fields.
Binary Iterative Hard Thresholding converges with optimal number of 1-bit measurements.
problem Recovering sparse signals from 1-bit compressed measurements.
method Binary Iterative Hard Thresholding (BIHT) algorithm.
result BIHT converges with only O(k/ε) measurements, optimal for recovery.