arXiv research

Gradient descent with early stopping achieves optimal sparse recovery.

This paper converts ADMM to proximal gradient for efficient sparse estimation.

New DP optimization methods for sparse gradients, improving on existing algorithms.

Neural network learning is usually time-consuming since backpropagation needs to compute full gradients and backpropagate them across multiple layers. Despite its success of existing works in accelerating propagation through sparseness, the relevant theoretical characteristics remain under-researched and empirical stud…

Sparse GCA finds linear relationships in multiple datasets, using gradient descent.

Law derived for neural networks with sparse connections.

New method controls gradient error for sparse MRFs.

Paper solves k-sparse parity problem with sign SGD, matching SQ lower bound.

Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find …

A new algorithm estimates sparse gradients on graphs with improved risk bounds.

Stochastic gradient descent (SGD) is commonly used for optimization in large-scale machine learning problems. Langford et al. (2009) introduce a sparse online learning method to induce sparsity via truncated gradient. With high-dimensional sparse data, however, the method suffers from slow convergence and high variance…

A new algorithm solves sparse optimization problems on measures efficiently.

Paper tackles NAS problem by modeling it as a sparse supernet.

Variable selection and dimension reduction are two commonly adopted approaches for high-dimensional data analysis, but have traditionally been treated separately. Here we propose an integrated approach, called sparse gradient learning (SGL), for variable selection and dimension reduction via learning the gradients of t…

Sparse perturbations improve convergence in SZO methods for faster training.

Online method learns sparse models efficiently in large scale settings.

A scalable gradient-based framework for sparse portfolio selection.

Sparse principal component analysis (PCA) and sparse canonical correlation analysis (CCA) are two essential techniques from high-dimensional statistics and machine learning for analyzing large-scale data. Both problems can be formulated as an optimization problem with nonsmooth objective and nonconvex constraints. Sinc…

Rotation invariant algorithms fail on sparse problems even with noise.

Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.

ETGL-DDPG improves DDPG for sparse reward control with new exploration and replay techniques.

New method improves structure learning on sparse graphs.

A fast method for discrete OT with group-sparse regularization for class label preservation.

Sparse matrices simplify computation of GP variances and likelihoods.

The paper improves sparse Gaussian processes by optimizing predictive loss.

A simple 2-layer linear network outperforms neural networks in learning sparse targets.

Accelerated gradient method tackles nonconvex penalties in sparse learning.

Paper proposes a new method to separate low rank and sparse matrices without bias.

Proposes efficient Bayesian logistic regression for large sparse datasets.

A tuning-free method recovers jointly sparse signals in MMV using implicit regularization.

Hadamard Wirtinger Flow recovers sparse signals from fewer measurements.

Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time. Coding-theoretic techniques have been recently proposed to mitigate stragglers via algorithmic redundancy. Prior work in coded computation and gradient coding has mainly focuse…

We consider estimating a piecewise-constant image, or a gradient-sparse signal on a general graph, from noisy linear measurements. We propose and study an iterative algorithm to minimize a penalized least-squares objective, with a penalty given by the "l_0-norm" of the signal's discrete graph gradient. The method proce…

Efficiently performs robust and sparse kernel regression.

With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep models. Due to the latency and limited bandwidth of the network, communication has become the bottleneck of distributed learning. Communication c…

New method tackles high-dimensional SBL without covariance matrices.

This paper explores a new framework for reinforcement learning based on online convex optimization, in particular mirror descent and related algorithms. Mirror descent can be viewed as an enhanced gradient method, particularly suited to minimization of convex functions in highdimensional spaces. Unlike traditional grad…

We study a hybrid conditional gradient - smoothing algorithm (HCGS) for solving composite convex optimization problems which contain several terms over a bounded set. Examples of these include regularization problems with several norms as penalties and a norm constraint. HCGS extends conditional gradient methods to cas…

This paper deals with unsupervised clustering with feature selection. The problem is to estimate both labels and a sparse projection matrix of weights. To address this combinatorial non-convex problem maintaining a strict control on the sparsity of the matrix of weights, we propose an alternating minimization of the Fr…

Model shows loss curve with two distinct exponents due to sparse activations.

Reinforcement learning usually uses the feedback rewards of environmental to train agents. But the rewards in the actual environment are sparse, and even some environments will not rewards. Most of the current methods are difficult to get good performance in sparse reward or non-reward environments. Although using shap…

Variance reduction methods such as SVRG and SpiderBoost use a mixture of large and small batch gradients to reduce the variance of stochastic gradients. Compared to SGD, these methods require at least double the number of operations per update to model parameters. To reduce the computational cost of these methods, we i…

Optimizes sparse fine-tuning for privacy in neural networks.

Autoencoders fail to capture sparse structure in 1-bit data compression.

Hard Thresholding Pursuit (HTP) is an iterative greedy selection procedure for finding sparse solutions of underdetermined linear systems. This method has been shown to have strong theoretical guarantee and impressive numerical performance. In this paper, we generalize HTP from compressive sensing to a generic problem …

Pruning method removes less important features in linear models.

Recent work has established an empirically successful framework for adapting learning rates for stochastic gradient descent (SGD). This effectively removes all needs for tuning, while automatically reducing learning rates over time on stationary problems, and permitting learning rates to grow appropriately in non-stati…

Study iterative regularization for linear models with convex bias, improving robust sparse recovery.