Sparse DNNs face scalability issues; MIT/IEEE/Amazon challenge analyzes best solutions.
problem Scalability issues in Sparse Deep Neural Networks (DNNs).
method Mathematically defined DNN inference computation, community submissions from various fields.
result Sparse DNN execution time, TmDNN, is strongly dependent on the number of operations, Nmop. Hierarchical Block Sparse Neural Networks improve both accuracy and runtime efficiency of sparse DNNs.
problem Inefficiency of sparse DNNs on regular parallel hardware due to irregular computation.
method Introducing HBsNN, a structured sparse neural network that balances accuracy and runtime efficiency.
result HBsNN achieves better runtime performance and accuracy than unstructured and highly structured sparse models.
Sparse DNNs simplify DNN mathematics and reveal exact solutions.
problem Sparse DNNs to overcome memory limitations in large, complex models.
method Associative array algebra to simplify DNN mathematics and construct exact solutions.
result Construct exact solutions and perturbation models for ReLU DNN equations.
New algorithm generates diverse sparse neural network topologies.
problem Training deep neural networks requires significant hardware resources.
method Deterministic algorithm to generate sparse neural network topologies.
result Generated topologies train to the same precision as dense DNNs at lower cost.
RadiX-Net generates diverse sparse neural topologies.
problem Sparse neural networks require more efficient storage and training.
method Deterministically generates RadiX-Nets from sparse topologies.
result RadiX-Nets can train to the same precision as dense DNNs at lower cost.
Sparse DNN challenge aims to improve graph data analysis.
problem Scalability issues in sparse data analysis.
method Mathematically defined DNN inference, vertex-centric and array-based implementations.
result Performance predictions based on simple hardware models.
Sparse DNNs improve adversarial robustness by reducing computation and memory usage.
problem Adversarial attacks on DNNs increase computation and memory requirements.
method Pruning dense DNNs to create sparse models, analyzing their robustness.
result Higher sparsity in nonlinear DNNs correlates with better adversarial robustness.
Proposes a method to learn sparse deep neural networks with theoretical guarantees.
problem Over-parameterized deep neural networks cause training, prediction, and interpretation difficulties.
method Frequentist-like method for sparse DNNs under Bayesian framework.
result Consistent sparse DNNs with at most O(n/log(n)) connections. Paper proposes energy-efficient DNN training methods.
problem Energy-constrained deployment of deep neural networks.
method Weighted sparse projection and layer input masking integrated into DNN training.
result Framework provides higher accuracy with same or lower energy budgets.
Paper proposes a new sparse Bayesian neural network for simpler, more efficient DNNs.
problem Complex and large DNN architectures require simplification for better performance and efficiency.
method Masked Bayesian Neural Networks (BNN) with nodewise sparsity and optimal posterior distributions.
result The proposed BNN yields well-condensed DNN architectures with similar accuracy and uncertainty quantification to large DNNs.
Deep neural network estimates support of sparse signals for improved phase retrieval.
problem Sparse phase retrieval from Fourier magnitudes with support estimation.
method Trained deep neural network (DNN) provides extended support estimate E larger than the support T. result DNN-based support estimation improves signal reconstruction performance with lower complexity.
The paper develops adaptive deep learning methods for nonlinear time series models.
problem Estimating mean functions of non-stationary and nonlinear time series models.
method Develops non-penalized and sparse-penalized DNN estimators for general non-stationary time series, derives minimax lower bounds, and shows the sparse-penalized DNN estimator is adaptive and optimal.
result Sparse-penalized DNN estimator achieves minimax optimal rates for many nonlinear AR models.
New method learns sparse distributions by thresholding samples, improving performance and efficiency.
problem Sparse coding optimization in high-dimensional problems is computationally expensive and inefficient.
method Proposes a new variational sparse coding approach that learns sparse distributions by thresholding samples.
result Shows superior performance, statistical efficiency, and gradient estimation compared to other sparse distributions.
NGSLL combines DNN accuracy with linear model interpretability.
problem Combining high accuracy of DNNs with interpretability of linear models.
method Neural generators of sparse local linear models (NGSLL) using DNNs to approximate non-linear functions.
result Effective in real-world datasets, achieving high predictive performance and interpretability.
Paper proposes a new method to optimize deep neural networks with sparse regularization.
problem Difficulty in achieving optimal convergence rates for deep neural networks due to sparsity constraints.
method Introduces a novel penalized estimation method for sparse DNNs, resolving computational and theoretical issues.
result Establishes an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator and derives convergence rates.
Physics-informed neural networks improve subsurface transport parameter estimation from sparse data.
problem Estimating subsurface transport parameters from sparse measurements.
method Physics-informed neural networks (DNNs) for joint inversion of conductivity, hydraulic head, and concentration fields.
result Physics-informed DNNs yield significantly more accurate parameter estimates than standard DNNs.
Paper improves robustness and sparsity in adversarially trained DNNs.
problem Developing efficient compression algorithms for robustly trained DNNs.
method Pruning weights using relaxed augmented Lagrangian algorithms for both structured and unstructured levels, leveraging Feynman-Kac formalism.
result At least doubles channel sparsity of adversarially trained ResNet20 for CIFAR10 classification.
New pruning technique reduces index size for DNNs.
problem Irregular index form in fine-grained pruning limits parallelism and memory usage.
method Proposes a low-rank binary index matrix for efficient compression and decompression.
result Fine-grained pruning with binary matrices achieves lower memory footprint and higher parallelism.
Derives moments of PL networks for robust DNNs.
problem Analyzing robustness of DNNs to adversarial attacks.
method Derives exact analytic expressions for moments of PL networks, generalizes variance expression, and constructs sparse and smooth adversarial attacks.
result New variance expressions are tighter and can be efficiently approximated.
Post-processes deep networks with StoNet to quantify uncertainty.
problem Uncertainty quantification in predictions from large-scale deep neural networks.
method Feeds DNN output into StoNet, trains StoNet with sparse penalty, constructs prediction intervals.
result Proposed approach constructs honest confidence intervals with shorter lengths and better calibration.
We develop a sparse representation method for neural network uncertainty.
problem Estimating model uncertainty in neural networks.
method Sparse representation of model uncertainty using inverse Multivariate Normal Distribution (MND), with a novel sparsification algorithm and analytical sampler.
result The information form of neural networks can be effectively applied for model uncertainty representation, showing competitive performance.
SmartExchange trades memory for computation in neural networks.
problem Energy-efficient inference of deep neural networks.
method Algorithm-hardware co-design framework integrating sparsification, decomposition, and quantization.
result SmartExchange reduces energy consumption and improves performance.
PAC-Bayesian bounds show fully connected DNNs with Gaussian priors match minimax rates.
problem Theoretical limits of fully connected deep neural networks with Gaussian priors.
method PAC-Bayesian bounds for fully connected Bayesian DNNs with Gaussian priors.
result PAC-Bayesian bounds match minimax-optimal rates in Besov space for nonparametric regression and binary classification.
Deep learning improves nearshore bathymetry estimation from sparse data.
problem Estimating nearshore bathymetry from limited and often sparse data.
method Deep Neural Network (DNN) and Conditional Generative Adversarial Network (cGAN) for posterior estimates; Kriging for comparison.
result DNN-based methods outperform traditional Kriging in predicting nearshore bathymetry with sharp gradients.
Novel approach reduces DNN redundancy and enhances computational efficiency.
problem Redundant parameters in large-scale DNNs pose hardware deployment challenges.
method WGSEF regularization technique for structured sparsity.
result Reduces redundancy and enhances computational efficiency.
BEAN models neuronal correlations to create interpretable representations.
problem Hard interpretation of dense-layer representations in DNNs.
method Inspired by neuroscience, BEAN models neuronal correlations and dependencies.
result BEAN enables formation of interpretable neuronal clusters without sacrificing model performance.
We propose to model the acoustic space of deep neural network (DNN) class-conditional posterior probabilities as a union of low-dimensional subspaces. To that end, the training posteriors are used for dictionary learning and sparse coding. Sparse representation of the test posteriors using this dictionary enables proje…
Paper reveals hidden convexities in deep learning models using sparse signal processing.
problem Non-convex loss functions in deep learning models complicate optimization and theoretical understanding.
method Developed convex equivalences of ReLU NNs and their connections to sparse signal processing models.
result Recent research has uncovered hidden convexities in certain NN architectures, notably two-layer ReLU networks and other architectures.
New theory for BNNs with Gaussian priors achieves optimal posterior concentration rates.
problem Lack of theoretical results for BNNs with Gaussian priors.
method New approximation theory for non-sparse DNNs with bounded parameters.
result BNNs with non-sparse general priors can achieve near-minimax optimal posterior concentration rates.
A new framework for interpretable models using sparse linear layers.
problem Performance degradation and lower interpretability in concept bottleneck models.
method Contrastive Language Image models and a single sparse linear layer with Bayesian inference.
result Our framework outperforms recent CBM approaches in accuracy and concept sparsity.
It is by now well-known that small adversarial perturbations can induce classification errors in deep neural networks (DNNs). In this paper, we make the case that sparse representations of the input data are a crucial tool for combating such attacks. For linear classifiers, we show that a sparsifying front end is prova…
This paper compresses deep neural networks for efficient learning on embedded systems.
problem Large memory and computation requirements of deep neural networks.
method Sparse coding with proximal point algorithms and debiasing for model compression.
result Minimal learning models suitable for small embedded devices are produced.
DSG activates only a small amount of neurons for efficient deep learning.
problem Efficient deployment of deep neural networks on embedded devices.
method Dynamic and sparse graph structure with dimension-reduction search and double-mask selection.
result Significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss.
CREX makes deep neural networks more credible by focusing on relevant evidence.
problem Deep neural networks often use incorrect evidence for decisions, leading to mistrust and poor generalization.
method CREX regularizes DNN training with rationales to encourage correct local explanations.
result DNNs trained with CREX are more credible and perform better on unseen data.
EC2T creates sparse and ternary neural networks for resource-constrained devices.
problem Deploying deep neural networks on resource-constrained devices.
method Entropy-Constrained Trained Ternarization (EC2T) framework.
result EC2T creates sparse and ternary neural networks that are efficient in terms of storage and computation.
This work uses differential topology to address challenges in DNNs.
problem Challenges in Deep Neural Networks: expressibility, optimisability, and generalisability.
method Modeling the dataset as a smooth manifold and applying differential topology to loss landscape, expressibility, and generalisability.
result A differential topological view offers new insights into DNNs' challenges.
PCONV combines fine-grained and coarse-grained pruning for efficient DNN inference on mobile devices.
problem Achieving high sparsity and accuracy in DNN weight pruning for real-time mobile execution.
method PCONV introduces a new sparsity dimension by combining fine-grained pruning patterns inside coarse-grained structures.
result PCONV outperforms state-of-the-art frameworks in speed and efficiency without accuracy loss.
SmartDeal reduces energy and storage costs for deep neural networks.
problem Heavy parameterization of deep neural networks leads to inefficient use of DRAM.
method SmartDeal decomposes weights into a small basis matrix and a structurally sparse coefficient matrix, quantized to power-of-2.
result Up to 2.44x energy efficiency improvement in inference and 10.56x reduction in training energy.
Deep learning predicts mass from images with sparse ground truth.
problem Accurately estimating mass from images with limited ground truth.
method Semi-supervised deep learning with gradient aggregation and sparse ground truth.
result Deep neural network accurately predicts mass from images.
Iterative shrinkage/thresholding algorithm (ISTA) is a well-studied method for finding sparse solutions to ill-posed inverse problems. In this letter, we present a data-driven scheme for learning optimal thresholding functions for ISTA. The proposed scheme is obtained by relating iterations of ISTA to layers of a simpl…
Deep learning models need better theoretical understanding to avoid costly mistakes.
problem Deep learning models often make costly mistakes in sensitive applications.
method Theoretical analysis of deep learning models using various mathematical and psychological perspectives.
result Theoretical understanding of deep learning models is limited and requires convergence of different perspectives.
Enhances deep neural networks with fixed-mean Gaussian processes for uncertainty estimation.
problem Post-hoc uncertainty estimation of pre-trained deep neural networks.
method Fixed-mean Gaussian processes with variational inference for efficient stochastic optimization.
result FMGP improves uncertainty estimation and computational efficiency compared to state-of-the-art methods.
Enhances MRI image quality with Conditional WGAN and adaptive balancing.
problem Struggles to reconstruct sharp images with fine detail.
method Conditional Wasserstein Generative Adversarial Network (WGAN) with Adaptive Gradient Balancing.
result Produces sharper images than other techniques.
CLEANN detects and mitigates neural network Trojans without labeled data.
problem Trojans in embedded neural networks that bypass detection during inference.
method Dictionary learning and sparse approximation for identifying Trojan triggers, lightweight algorithm/hardware co-design.
result Efficient real-time execution on resource-constrained platforms, competitive attack resiliency.
New method quantifies uncertainty in physics-informed neural networks for stochastic problems.
problem Uncertainty in physics-informed neural networks for solving stochastic PDEs.
method Combining DNNs for residual and data mismatch, using stochastic data and dropout for uncertainty quantification, and active learning.
result Effective uncertainty quantification for both parametric and approximation uncertainties in PINNs.
A new method trains deep networks by separating weight locations from values.
problem Training deep networks efficiently and effectively.
method Lookahead Permutation (LaPerm) to train DNNs by reconnecting weights.
result LaPerm can train DNNs with random and dense, sparse, or single-valued initial weights.
A new framework compresses neural networks using sparse optimization.
problem Efficiently reducing the size of deep neural networks for practical deployment.
method Sparse optimization for model compression, tailored for stochastic learning.
result Up to 7.2 and 2.9 times FLOPs reduction with comparable accuracy.
Sparse algorithms reduce model size for faster keyword spotting.
problem Large models are hard to deploy on resource-limited devices.
method Apply sparse algorithms to reduce parameter count in DNN KWS models.
result Sparse models perform better with minimal parameter loss.