arXiv research

This paper tackles the computational complexity of finding approximate stationary points in non-convex optimization.

New algorithm finds approximate stationary points faster under differential privacy constraints.

New method finds points for approximating distributions faster.

Develops a deep non-stationary kernel for non-stationary spatio-temporal point processes.

New algorithm finds approximate stationary points in non-convex optimization.

Memory-based models can learn to approximate Bayes-optimal predictors for non-stationary data.

Novel methods for accelerating optimization in complex bilevel and minimax problems.

Develops consistent approximations for composite optimization problems.

Detect changes in noisy dynamical systems using empirical approximations and finite-sample bounds.

Neural networks' weights don't converge to stationary points but training loss stabilizes.

Simple gradient descent algorithm escapes saddle points efficiently.

New method improves stability of soft FQI for offline RL.

We consider the case of derivative-free algorithms for non-convex optimization, also known as zero order algorithms, that use only function evaluations rather than gradients. For a wide variety of gradient approximators based on finite differences, we establish asymptotic convergence to second order stationary points u…

Paper proposes a method to find approximate SOSP for nonconvex conic optimization problems.

New method stabilizes FQE by reweighting Bellman targets.

We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…

Gradient-based methods find saddle points, not critical points, in neural networks.

Exact Gaussian Processes for massive datasets using non-stationary sparsity-discovering kernels.

New method solves stochastic optimization problems with random models.

DR-submodular continuous functions are important objectives with wide real-world applications spanning MAP inference in determinantal point processes (DPPs), and mean-field inference for probabilistic submodular models, amongst others. DR-submodularity captures a subclass of non-convex functions that enables both exact…

ELBO of VAEs converges to a sum of three entropies.

The method approximates stationary distributions of Markov models by truncating irrelevant states.

The Hessian-vector product has been utilized to find a second-order stationary solution with strong complexity guarantee (e.g., almost linear time complexity in the problem's dimensionality). In this paper, we propose to further reduce the number of Hessian-vector products for faster non-convex optimization. Previous a…

Paper proposes a new method to find approximate SOSP for nonconvex constrained optimization problems.

New method reduces variance and bias in approximating indefinite kernels.

The paper develops a stationary-distribution theory for Random Forest ensemble size selection.

The Bivariate Dynamic Contagion Processes (BDCP) are a broad class of bivariate point processes characterized by the intensities as a general class of piecewise deterministic Markov processes. The BDCP describes a rich dynamic structure where the system is under the influence of both external and internal factors model…

In this paper we consider the dictionary learning problem for sparse representation. We first show that this problem is NP-hard by polynomial time reduction of the densest cut problem. Then, using successive convex approximation strategies, we propose efficient dictionary learning schemes to solve several practical for…

New sampling method guarantees approximate first-order stationary points for non-convex functions.

The paper simplifies arguments for stationary varifolds results.

Robust estimation methods find global minima efficiently via quasi-gradients.

New algorithm tackles non-stationary reinforcement learning with general function approximation.

New findings show Bregman proximal algorithms can get stuck near non-stationary points.

We analyze stochastic gradient algorithms for optimizing nonconvex problems. In particular, our goal is to find local minima (second-order stationary points) instead of just finding first-order stationary points which may be some bad unstable saddle points. We show that a simple perturbed version of stochastic recursiv…

This paper proposes a novel kernel-based optimization scheme to handle tasks in the analysis, e.g., signal spectral estimation and single-channel source separation of 1D non-stationary oscillatory data. The key insight of our optimization scheme for reconstructing the time-frequency information is that when a nonparame…

Paper proposes a method to find approximate SOSP for nonconvex conic optimization problems.

We present a joint message passing approach that combines belief propagation and the mean field approximation. Our analysis is based on the region-based free energy approximation method proposed by Yedidia et al. We show that the message passing fixed-point equations obtained with this combination correspond to station…

Polyconvex energies with conformal invariance have smooth stationary points outside a discrete set.

This work aims to provide understandings on the remarkable success of deep convolutional neural networks (CNNs) by theoretically analyzing their generalization performance and establishing optimization guarantees for gradient descent based training algorithms. Specifically, for a CNN model consisting of $l$ convolution…

In this paper we study general $l_p$ regularized unconstrained minimization problems. In particular, we derive lower bounds for nonzero entries of first- and second-order stationary points, and hence also of local minimizers of the $l_p$ minimization problems. We extend some existing iterative reweighted $l_1$ (IRL1) a…

The application of Gaussian processes (GPs) to large data sets is limited due to heavy memory and computational requirements. A variety of methods has been proposed to enable scalability, one of which is to exploit structure in the kernel matrix. Previous methods, however, cannot easily deal with non-stationary process…

New method finds stationary points in bilevel optimization problems.

Stochastic optimization naturally arises in machine learning. Efficient algorithms with provable guarantees, however, are still largely missing, when the objective function is nonconvex and the data points are dependent. This paper studies this fundamental challenge through a streaming PCA problem for stationary time s…

HA-SME models SGD dynamics with Hessian info for better escaping behaviors.

Enhances DGPs with adaptive RKHS Fourier features for better non-stationary pattern modeling.

Transformers achieve near-optimal dynamic regret in non-stationary reinforcement learning.

Many optimization algorithms converge to stationary points. When the underlying problem is nonconvex, they may get trapped at local minimizers and occasionally stagnate near saddle points. We propose the Run-and-Inspect Method, which adds an "inspect" phase to existing algorithms that helps escape from non-global stati…

GraphGP: Scalable Gaussian Processes with Vecchia's Approximation