arXiv research

Dynamic Sparse Training finds efficient sparse networks from scratch.

New pruning methods improve dynamic sparse training performance.

ADAPT method controls balance between sparse generator and discriminator in GANs.

FedDST trains sparse sub-networks to improve efficiency in federated learning.

Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy. Recently, techniques based on dynamic reallocation of non-zero parameters have emerged, allowing direct training of sparse networks without…

Predictive Sparse Manifold Transform learns dynamic video sequences.

Large SGD step sizes lead to sparse feature learning in neural networks.

Dynamic model pruning improves performance on deep neural networks without retraining.

Neural networks have achieved state of the art performance across a wide variety of machine learning tasks, often with large and computation-heavy models. Inducing sparseness as a way to reduce the memory and computation footprint of these models has seen significant research attention in recent years. In this paper, w…

We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the prev…

We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that a…

Study shows how specialized attention circuits emerge during transformer training.

Graph neural networks have become increasingly popular in recent years due to their ability to naturally encode relational input data and their ability to scale to large graphs by operating on a sparse representation of graph adjacency matrices. As we look to scale up these models using custom hardware, a natural assum…

A new method reduces feature size in CRFs for faster training.

Efficiently constructs sparse ROMs for high-dimensional data using causation entropy.

A new distributed method speeds up sparse model training.

dLDS models neural dynamics as sparse combinations of simpler components.

New model leads to optimal test loss in sparse linear regression.

Proposes a flexible MGP model for dynamic, sparse correlations.

DLNs dynamics change with variance, leading to saddle-to-saddle training phases.

DistGP models multi-robot mapping with distributed Gaussian process learning.

Deep weight factorization improves neural network training through smooth optimization of sparse penalties.

Dynamic models learn from sparse, interacting sub-systems.

Improved trajectory prediction for team sports using sparse outputs.

New method finds sparse networks without labels, improving performance.

Prototype for adaptive electron microscopy scans reduces dose and time.

We present Distributed Equivalent Substitution (DES) training, a novel distributed training framework for large-scale recommender systems with dynamic sparse features. DES introduces fully synchronous training to large-scale recommendation system for the first time by reducing communication, thus making the training of…

UCB-TQL learns from multiple tasks with shared dynamics and adapts to task-specific variations.

Powerpropagation makes neural networks inherently sparse.

This study explains RL training dynamics in LLMs, focusing on token-level optimization and reasoning pattern reshaping.

Paper improves reinforcement learning in multi-scene tasks.

Deep learning predicts dynamics from sparse data.

We address the issue of estimating the topology and dynamics of sparse linear dynamic networks in a hyperparameter-free setting. We propose a method to estimate the network dynamics in a computationally efficient and parameter tuning-free iterative framework known as SPICE (Sparse Iterative Covariance Estimation). The …

Early alignment in neural networks leads to sparse representations but hinders convergence.

New method learns dynamics from sparse data using geometric constraints.

FLASH-MAX predicts electromagnetic fields from sparse data in seconds.

New techniques improve the accuracy of identifying nonlinear systems from noisy data.

We consider the problem of modeling multivariate time series with parsimonious dynamical models which can be represented as sparse dynamic Bayesian networks with few latent nodes. This structure translates into a sparse plus low rank model. In this paper, we propose a Gaussian regression approach to identify such a mod…

HierGP improves emulator efficiency for sparse, structured data.

Parsimonious Dynamic Mode Decomposition selects sparse modes robustly.

LD-EnSF speeds up data assimilation with sparse observations.

Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). Recently, surprise has been used as an intrinsic reward that encourages systematic and efficient exploration. We introduce a new definition of surprise and its RL implementation named Variation…

Model shows loss curve with two distinct exponents due to sparse activations.

Paper improves DOA estimation in sparse arrays using Siamese neural networks.

Framework learns continuous dynamics from sparse trajectories.

Improved SINDy autoencoder for identifying noisy dynamical systems.

Despite the importance of sparsity signal models and the increasing prevalence of high-dimensional streaming data, there are relatively few algorithms for dynamic filtering of time-varying sparse signals. Of the existing algorithms, fewer still provide strong performance guarantees. This paper examines two algorithms f…

A nonparametric Bayesian sparse graph linear dynamical system (SGLDS) is proposed to model sequentially observed multivariate data. SGLDS uses the Bernoulli-Poisson link together with a gamma process to generate an infinite dimensional sparse random graph to model state transitions. Depending on the sparsity pattern of…