This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.
problem Theoretical convergence properties of SGD algorithms for non-convex smooth functions.
method Analysis of SGD algorithms with arbitrary data ordering for non-convex smooth functions.
result Enhanced convergence guarantees for incremental gradient and single shuffle SGD, improving the optimization term of convergence guarantee.
Unified analysis of multi-attribute graph learning with non-convex penalties.
problem Graph inference from multi-attribute data.
method Penalized log-likelihood objective function with ADMM and local linear approximation.
result Local consistency in support recovery and precision matrix estimation for non-convex penalties.
This paper examines the role and efficiency of the non-convex loss functions for binary classification problems. In particular, we investigate how to design a simple and effective boosting algorithm that is robust to the outliers in the data. The analysis of the role of a particular non-convex loss for prediction accur…
Note on the computational complexity of Gromov-Wasserstein distance.
problem Computational difficulty of Gromov-Wasserstein distance.
method Analysis of the optimization problem structure and providing explicit examples.
result Gromov-Wasserstein distance optimization problem is non-convex quadratic.
We rigorously prove statistical physics predictions for non-convex GLMs in high dimensions.
problem Analyzing high-dimensional optimization problems in non-convex Generalized Linear Models.
method Developed a systematic framework using the Gaussian Min-Max Theorem and AMP to rigorously prove replica-symmetric formulas.
result Validated statistical physics predictions for non-convex GLMs, aligning with physicist's conjectures.
Paper estimates differences in multi-attribute Gaussian graphical models using non-convex penalties.
problem Estimating differences in multi-attribute Gaussian graphical models with similar structure.
method Penalized D-trace loss function with non-convex (log-sum and SCAD) penalties, proximal gradient descent methods.
result Theoretical analysis and numerical examples support consistency in support recovery and estimation.
We consider online forecasting problems for non-convex machine learning models. Forecasting introduces several challenges such as (i) frequent updates are necessary to deal with concept drift issues since the dynamics of the environment change over time, and (ii) the state of the art models are non-convex models. We ad…
Paper proposes a working set algorithm for non-convex sparse regression with provable convergence.
problem Estimating sparse linear models from high-dimensional data using non-convex regularizers.
method FireWorks algorithm based on non-convex reformulation and leveraging residual geometry.
result Convergence to a stationary point of the full problem with provable guarantees.
New approach for distributed online optimization of non-convex losses with sublinear regret.
problem Regret evaluation and consensus in distributed, multi-agent systems with non-convex losses.
method Composite regret metric and consensus-based online normalized gradient (CONGD) approach for pseudo-convex losses; offline optimization oracle for general non-convex losses.
result First sublinear regret bound for general distributed online non-convex learning.
New method estimates GGLM parameters, overcoming non-convexity.
problem Estimating parameters in GGLM with dependencies.
method Monotone operator-based variational inequality method.
result Guarantees for parameter recovery in GLM and GGLM.
AEGD optimizes non-convex functions with dynamic energy updates.
problem Optimizing non-convex functions efficiently and robustly.
method Adaptive Gradient Descent (AEGD) with a dynamically updated energy variable.
result AEGD achieves energy-dependent convergence rates for both convex and non-convex objectives.
A fast method for decentralized non-convex optimization over networks.
problem Decentralized non-convex optimization problems over a network of nodes.
method GT-SAGA, a randomized incremental gradient method that evaluates one component gradient per node per iteration.
result GT-SAGA achieves almost sure and mean-squared convergence to a first-order stationary point for general smooth non-convex problems.
Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-conv…
Proves convergence of PSGLA for sampling non-convex potentials.
problem Sampling from non-convex potentials with stability.
method Combines ULA and proximal optimization with stability analysis.
result First proof of convergence for PSGLA on non-convex potentials.
Compact, non-convex curve flows are created.
problem Creating compact, non-convex ancient solutions for curve shortening flow.
method Constructed an ancient solution asymptotic to Yin-Yang curve.
result Compact, non-convex ancient solutions for curve shortening flow are demonstrated.
Several recently proposed architectures of neural networks such as ResNeXt, Inception, Xception, SqueezeNet and Wide ResNet are based on the designing idea of having multiple branches and have demonstrated improved performance in many applications. We show that one cause for such success is due to the fact that the mul…
A new depth measure for non-convex data supports, faster than halfspace depth.
problem Non-convex data supports in multivariate statistics.
method Extending halfspace depth to Reproducing Kernel Hilbert Space (RKHS).
result The new depth measure is consistent and can be computed faster.
New bounds found for optimizing non-convex functions with noisy data.
problem Limits of first-order stochastic optimization in non-convex settings.
method Divergence decomposition to construct challenging subclasses.
result Sharp lower bounds on noisy gradient queries for various non-convex classes.
New methods improve convergence in non-convex non-smooth learning problems.
problem Sparse learning from high-dimensional data with non-convex, non-smooth regularizers.
method Stochastic proximal gradient methods with arbitrary sampling.
result Independent sampling improves performance over uniform sampling.
Paper uses integer programming for non-convex boosting in classification.
problem Improving classification performance using non-convex optimization.
method Non-convex boosting via integer programming.
result Results comparable to or better than state-of-the-art.
New optimizer G-AdaGrad improves upon AdaGrad for non-convex machine learning problems.
problem Solving non-convex machine learning problems efficiently.
method Proposes a new optimizer G-AdaGrad and analyzes its convergence using state-space models.
result Empirical results show G-AdaGrad performs better than AdaGrad and Adam.
SGD converges to global minimum for structured non-convex functions.
problem Optimizing non-convex functions using SGD with slow convergence rates.
method Convergence theorems for SGD on structured non-convex functions, including Quasar and PL conditions.
result SGD converges to global minimum for specific non-convex functions under certain conditions.
Paper checks SSC for matrix factorizations using Gurobi.
problem Checking the SSC for various matrix factorizations.
method Formulated as a non-convex quadratic optimization problem over a bounded set, solved with Gurobi.
result SSC can be checked in reasonable time for realistic scenarios.
First order methods can take extremely long to find global minima of non-convex functions.
problem Finding global minimizers of non-convex functions.
method Designing a family of non-convex functions and using statistical lower bounds for parameter estimation.
result First order methods can take exponential time to converge to a global minimizer.
New algorithm improves convergence for non-convex problems with boundaries.
problem Optimizing non-convex problems with constraints.
method Reflected Gradient Langevin Dynamics with probabilistic representation.
result Promising convergence rates, faster than existing methods.
Paper improves stability analysis of SGD for various loss functions and data distributions.
problem Improving stability analysis of SGD for non-convex loss functions and data distributions.
method Analyzes stability of SGD for convex and non-convex loss functions, and improves data-dependent bounds.
result Improved stability bounds for non-convex loss functions and convex regularized loss functions.
CDP reduces point cloud dimensions by preserving detour-induced local non-convexity.
problem Preserving local non-convexity in point cloud dimensionality reduction.
method CDP builds a k-NN graph, identifies admissible pairs, aggregates normalized directions, and uses top-k eigenvectors for projection.
result CDP provides verifiable guarantees on post-projection distortion and direction energy.
This thesis tackles non-convex Bayesian learning via scalable dynamic importance sampling algorithms.
problem Non-convex Bayesian learning problem in deep neural networks.
method Replica exchange Langevin Monte Carlo, control variates method, population-chain replica exchange, scalable dynamic importance sampling.
result Control variates method reduces variance and accelerates convergence in non-convex Bayesian learning.
Paper tackles non-convex constrained DRO with a stochastic algorithm for large-scale applications.
problem Training robust models against data distribution shifts with non-convex loss functions.
method Developed a stochastic algorithm for non-convex constrained DRO with a complexity independent of dataset size.
result Algorithm finds ε-stationary points with computational complexity of O(ε^(-3k_*-5)) for general Cressie-Read divergence.
New SGD method uses adaptive sampling to converge faster in non-convex problems.
problem Non-convex optimization problems with noisy gradients.
method Adaptive coordinate sampling in stochastic gradient descent (SGD).
result Almost sure convergence and non-asymptotic bounds established.
This work studies the denoising of piecewise smooth graph signals that exhibit inhomogeneous levels of smoothness over a graph, where the value at each node can be vector-valued. We extend the graph trend filtering framework to denoising vector-valued graph signals with a family of non-convex regularizers, which exhibi…
ProxSkip achieves linear speedup in distributed non-convex optimization.
problem Achieving linear speedup in distributed non-convex optimization.
method Unified convergence analysis for stochastic non-convex, convex, and strongly convex problems.
result ProxSkip achieves linear speedup in the number of nodes under stochastic gradients.
Paper extends RPD for better handling multiple modalities and non-convexity.
problem Handling multiple modalities and non-convexity in data clouds.
method Computes RPD in a reproducing kernel Hilbert space using kernel principal component analysis.
result The method outperforms RPD and is comparable to other models on benchmark datasets.
MAGIC method optimally estimates model predictions changes.
problem Estimating how training data affects model predictions in large-scale settings.
method Combines classical methods and recent advances in metadifferentiation.
result MAGIC method nearly optimally estimates model predictions changes.
New PAC-Bayesian bounds for online learning with data streams.
problem Challenges of traditional PAC-Bayesian bounds in dynamic data collection.
method Developed new PAC-Bayesian bounds in online learning framework, using updated regret definition and batch-to-online conversion.
result PAC-Bayesian bounds hold for online learning with dependent data and non-convex losses.
Improved DP algorithms for non-convex optimization with tighter generalization bounds.
problem Private stochastic non-convex optimization in high-dimensional spaces.
method Differential privacy techniques, including adaptive algorithms like DP RMSProp and DP Adam, combined with adaptive data analysis.
result Achieved a sharper rate of p 4 / n \sqrt[4]{p}/\sqrt{n} 4 p / n for population loss, improving upon previous bounds. Survey of advances in non-convex min-max optimization for applications.
problem Finding optimal solutions in non-convex, non-concave min-max problems.
method Selective review of theoretical and algorithmic advances.
result Exciting recent advances in solving non-convex min-max problems.
This work shows neural networks can solve non-convex constraints problems.
problem Training neural networks under non-convex constraints.
method Project stochastic gradient descent with no-regret analysis of online learning.
result Overparameterized neural networks achieve near-optimal and near-feasible solutions.
In this paper, we consider the problem of learning high-dimensional tensor regression problems with low-rank structure. One of the core challenges associated with learning high-dimensional models is computation since the underlying optimization problems are often non-convex. While convex relaxations could lead to polyn…
Improved algorithm finds second-order stationary points in non-convex optimization.
problem Minimizing non-convex objectives while preserving training data privacy.
method SpiderBoost framework with two gradient oracles: precise and less precise.
result Improved rates for finding second-order stationary points.
New algorithms optimize non-smooth, non-convex objectives with improved complexity.
problem Optimizing non-smooth, non-convex stochastic objectives.
method Reduction to online learning, applying optimistic online learning techniques.
result Improved complexity for finding ( δ , ε ) (δ,ε) ( δ , ε ) -stationary points. New bounds for SGLD show error decreases with more data.
problem Establishing generalization error bounds for SGLD in non-convex settings.
method Using dissipativity, smoothness, and uniform stability, time-independent bounds are derived.
result Error bounds decay to zero as sample size increases.
Kernel k-Means algorithm improves clustering of non-linear data.
problem Non-convexity of kernel k-Means objective function leads to local minima.
method Generalizes MM approach to solve non-convex problem in kernel and multi-kernel settings.
result Establishes strong consistency guarantees for Kernel Power k-Means.
SGD's uncertainty quantified in non-convex learning problems.
problem Uncertainty quantification in non-convex learning problems.
method Asymptotic normality of SGD iterates and bias characterization.
result SGD iterates are asymptotically normally distributed around the expected value of the invariant distribution.
Optimizers find approximate global minima in non-convex problems.
problem Understanding why local methods solve non-convex optimization problems.
method Formalizing the hypothesis that many local minima are approximately global minima.
result Most local minima of practical non-convex objectives are approximately global minima.
Convex optimization with sparsity-promoting convex regularization is a standard approach for estimating sparse signals in noise. In order to promote sparsity more strongly than convex regularization, it is also standard practice to employ non-convex optimization. In this paper, we take a third approach. We utilize a no…
New insights into using momentum for non-convex optimization.
problem Improving training of non-convex models like deep neural networks.
method Developed a Lyapunov analysis of SGD with momentum using stochastic primal averaging.
result Precise conditions under which SGD+M outperforms SGD and optimal hyper-parameter schedules.
CDL index improves clustering validation for non-convex data.
problem Selecting clustering algorithms and hyperparameters without labeled data.
method CDL uses compactness, centers, and covariances to compute a probabilistic description length bound.
result CDL outperforms conventional CVIs on synthetic and image benchmarks.