arXiv research

Paper analyzes sparse aggregation in GLMs with Kullback-Leibler risk bounds.

Paper introduces kernel deformed exponential families for sparse continuous attention.

We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to…

Novel Bayesian method for high-dimensional count data prediction.

This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected traject…

Gradient descent on normalized networks reveals sparsity preferences.

Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that …

Catapult phase in neural nets shows exponential loss growth before quick decrease.

Bayesian learning has been recently considered as an effective means of accounting for uncertainty in trained deep network parameters. This is of crucial importance when dealing with small or sparse training datasets. On the other hand, shallow models that compute weighted sums of their inputs, after passing them throu…

In this paper we propose a novel framework for the construction of sparsity-inducing priors. In particular, we define such priors as a mixture of exponential power distributions with a generalized inverse Gaussian density (EP-GIG). EP-GIG is a variant of generalized hyperbolic distributions, and the special cases inclu…

New method reduces Bayesian inference time by optimizing Hamiltonian flows.

Extends online learning to metric spaces using exponential weights.

A new method for differentiable structured sparsity improves neural network performance and sparsity.

Improves variational inference for sparse models using mixtures of exponential families.

We consider the following multi-component sparse PCA problem: given a set of data points, we seek to extract a small number of sparse components with disjoint supports that jointly capture the maximum possible variance. These components can be computed one by one, repeatedly solving the single-component problem and def…

Sparse linear regression is hard to solve efficiently, even with k-sparse solutions.

Minimizing a convex function of a measure with a sparsity-inducing penalty is a typical problem arising, e.g., in sparse spikes deconvolution or two-layer neural networks training. We show that this problem can be solved by discretizing the measure and running non-convex gradient descent on the positions and weights of…

Guarantees sparse recovery for neural networks with iterative hard thresholding.

Exponential Lasso improves Lasso's robustness to outliers and heavy-tailed noise.

We introduce a covariance matrix estimator that both takes into account the heteroskedasticity of financial returns (by using an exponentially weighted moving average) and reduces the effective dimensionality of the estimation (and hence measurement noise) via techniques borrowed from random matrix theory. We calculate…

A new method for exponentially weighted moving models using approximations.

A scalable gradient-based framework for sparse portfolio selection.

Paper extends sparse alternatives to softmax for continuous domains, enabling efficient attention mechanisms.

Introduces Exponentially Weighted Signature for better path representation.

Neural Optimal Design of Experiments improves inverse problem solving efficiency.

Deep weight factorization improves neural network training through smooth optimization of sparse penalties.

The Kaczmarz algorithm is popular for iteratively solving an overdetermined system of linear equations. The traditional Kaczmarz algorithm can approximate the solution in few sweeps through the equations but a randomized version of the Kaczmarz algorithm was shown to converge exponentially and independent of number of …

Paper introduces DP methods for high-dimensional variable selection.

Sparse random networks reduce communication in federated learning.

SWAT reduces training efficiency and speed for CNNs.

Fast classification for sparse models, even with correlated features.

Researchers extend asymptotic analysis to Bergman projections with Gevrey weights.

The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.

We consider a decomposition method for compressive streaming data in the context of online compressive Robust Principle Component Analysis (RPCA). The proposed decomposition solves an $n$-$\ell_1$ cluster-weighted minimization to decompose a sequence of frames (or vectors), into sparse and low-rank components, from com…

Soft diamond regularizers improve deep learning performance and sparsity.

Regularization improves stability and consistency of sparse autoencoders.

Optimizes sparse mean-reverting portfolios for higher returns.

Paper explores exact recovery of communities in weighted graphs using Gaussian and exponential distributions.

Optimizes sparse fine-tuning for privacy in neural networks.

A new accelerated method with simpler momentum update rules.

Paper develops sparse learning for heavy-tailed time series with locally stationary dynamics.

Given two data matrices $X$ and $Y$, sparse canonical correlation analysis (SCCA) is to seek two sparse canonical vectors $u$ and $v$ to maximize the correlation between $Xu$ and $Yv$. However, classical and sparse CCA models consider the contribution of all the samples of data matrices and thus cannot identify an unde…

The paper extends a prediction method to curved spaces.

This paper explores efficient neural networks by identifying sparse structures.

Stochastic gradient descent (SGD) is commonly used for optimization in large-scale machine learning problems. Langford et al. (2009) introduce a sparse online learning method to induce sparsity via truncated gradient. With high-dimensional sparse data, however, the method suffers from slow convergence and high variance…

New method improves tensor completion by selectively preserving important elements.

Decentralized algorithm reduces regret and converges to Nash equilibrium in online congestion games.

Dynamic sparseness reduces neural network computation by selectively omitting parts of computations.