SDAMI enhances interpretable high-dimensional regression with sparse deep learning and footprint principle.
problem Personalized models for small samples and high-dimensional features with interpretability.
method Sparse Deep Additive Model with Interactions (SDAMI) combining sparsity-driven feature selection and deep subnetworks.
result SDAMI successfully identifies pure interactions with near-zero false positive rates.
RNNs solve modular addition tasks using low rank and sparse Fourier structures.
problem Solving modular addition tasks with recurrent neural networks.
method Identified low rank structures and sparse Fourier representations in RNN weights.
result RNNs robust to removing individual frequencies but degrade with more ablation.
SIAN bridges simple models to neural networks by identifying necessary feature combinations.
problem The gap between simple models and powerful neural networks in performance.
method Feature interaction detection and sparse selection algorithm.
result Competitive performance across multiple tabular datasets with optimal tradeoff.
SNAM improves NAM's accuracy and feature selection via group sparsity.
problem Improving interpretability and accuracy in deep learning models.
method Employing group sparsity regularization in neural additive models (SNAM).
result SNAM provably converges to zero training loss and achieves exact support recovery.
Dynamic model pruning improves performance on deep neural networks without retraining.
problem High memory and latency requirements for deep neural networks on low-end devices.
method Dynamic allocation of sparsity pattern and feedback signal to reactivate pruned weights.
result Sparse models achieve state-of-the-art performance with no additional retraining.
The paper provides statistical guarantees for sparse deep learning.
problem Understanding the potential and limitations of sparse deep learning.
method Develops statistical guarantees for different types of sparsity in sparse deep learning.
result Statistical guarantees for sparse deep learning with mild dependence on network widths and depths.
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
The paper develops adaptive deep learning methods for nonlinear time series models.
problem Estimating mean functions of non-stationary and nonlinear time series models.
method Develops non-penalized and sparse-penalized DNN estimators for general non-stationary time series, derives minimax lower bounds, and shows the sparse-penalized DNN estimator is adaptive and optimal.
result Sparse-penalized DNN estimator achieves minimax optimal rates for many nonlinear AR models.
This paper accelerates sparse CNN layers on GPUs by using unstructured sparsity.
problem Efficiency of sparse CNN layers on GPUs.
method Direct sparse operation and reduced precision.
result Achieving up to 90% sparsity in deep CNN models improves efficiency.
Study improves theoretical understanding of Bayesian deep learning for classification tasks.
problem Theoretical gap in understanding Bayesian approaches in deep learning for classification.
method PAC-Bayes bounds techniques and Spike-and-Slab priors for sparse deep learning.
result Established non-asymptotic results for prediction error, achieving minimax optimal rates.
A simple self-supervised model for tensor RPCA using deep unfolding.
problem Tensor robust principal component analysis (RPCA) challenges in practical applications.
method Deep unfolding with only four hyperparameters.
result Competitive or superior performance compared to supervised methods, even in data-starved scenarios.
New method recovers PDEs from noisy data, even when conditions are violated.
problem Discovering PDEs from noisy, limited data.
method Randomized adaptive Lasso integrated into DeepMod.
result Recovery of PDEs with higher noise-to-sample ratios and single hyperparameters.
New estimators improve sparse semiparametric additive modeling.
problem Sparse semiparametric additive modeling with structured sparsity.
method Combines group subset selection with shrinkage for nonconvex optimization.
result New estimators outperform alternatives in synthetic and real-world data.
Sparse additive modeling is a class of effective methods for performing high-dimensional nonparametric regression. In this work we show how shape constraints such as convexity/concavity and their extensions, can be integrated into additive models. The proposed sparse difference of convex additive models (SDCAM) can est…
A new distributed algorithm for fitting sparse additive models with feature division and decorrelation.
problem Fitting high-dimensional sparse additive models efficiently and accurately.
method Divide, decorrelate, and conquer approach.
result Effective and efficient recovery of sparsity patterns and statistical inference for each component.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
Model-free reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG) often require additional exploration strategies, especially if the actor is of deterministic nature. This work evaluates the use of model-based trajectory optimization methods used for exploration in Deep Deterministic Policy…
Theoretical analysis of deep neural networks for time series data.
problem Theoretical development for deep neural networks on temporally dependent observations is lacking.
method Established non-asymptotic bounds for prediction error of deep neural networks under mixing-type assumptions.
result Deep neural networks can model non-linear time series data with additional logarithmic factors due to dependence.
New BNN model proves optimal posterior concentration and enables practical inference.
problem Improving generalization and uncertainty quantification in deep neural networks.
method Proposes a new node-sparse BNN model with theoretical guarantees and a novel MCMC algorithm for inference.
result Proves near minimax optimal posterior concentration rate and adaptiveness to true model smoothness.
Deep learning uses layers of transformations to predict structured data with uncertainty.
problem Predicting structured high-dimensional data efficiently and with uncertainty.
method Applying layers of semi-affine input transformations to find features for probabilistic statistical methods.
result Achieves scalable prediction rules with uncertainty quantification and feature selection.
Study on signal detection in sparse additive models with nonasymptotic minimax rates.
problem Signal detection in sparse additive models.
method Nonasymptotic minimax analysis of signal detection in sparse additive models.
result Established minimax separation rate for signal detection.
A unique challenge in predictive model building for omics data has been the small number of samples (n) versus the large amount of features (p). This "n≪p" property brings difficulties for disease outcome classification using deep learning techniques. Sparse learning by incorporating external gene network info…
Sparse generalized additive models (GAMs) are an extension of sparse generalized linear models which allow a model's prediction to vary non-linearly with an input variable. This enables the data analyst build more accurate models, especially when the linearity assumption is known to be a poor approximation of reality. …
New pruning method for sparse additive models speeds up causal structure learning.
problem Efficiently prune spurious edges from fully-connected DAG induced by estimated topological order.
method Sparse additive models combined with randomized tree embedding and group-wise sparse regression.
result Significantly faster than existing pruning methods while maintaining comparable accuracy.
Deep Gaussian processes (DGPs) can model complex marginal densities as well as complex mappings. Non-Gaussian marginals are essential for modelling real-world data, and can be generated from the DGP by incorporating uncorrelated variables to the model. Previous work on DGP models has introduced noise additively and use…
We consider the problem of sparse variable selection in nonparametric additive models, with the prior knowledge of the structure among the covariates to encourage those variables within a group to be selected jointly. Previous works either study the group sparsity in the parametric setting (e.g., group lasso), or addre…
Although deep learning has shown great success in recent years, researchers have discovered a critical flaw where small, imperceptible changes in the input to the system can drastically change the output classification. These attacks are exploitable in nearly all of the existing deep learning classification frameworks.…
Chromatic Learning reduces feature dimensions for sparse datasets.
problem Sparse, high-dimensional data challenges traditional learning methods.
method Graph coloring over co-occurrence graph to create dense feature representation.
result Compresses sparse datasets significantly while maintaining model accuracy.
New framework tackles deep learning issues like local traps and miscalibration.
problem Local traps and miscalibration in deep neural networks.
method Sparse deep learning framework with prior annealing algorithms.
result Proposed method successfully addresses local traps and miscalibration.
We introduce a new algorithm, called adaptive sparse backfitting algorithm, for solving high dimensional Sparse Additive Model (SpAM) utilizing symmetric, non-negative definite smoothers. Unlike the previous sparse backfitting algorithm, our method is essentially a block coordinate descent algorithm that guarantees to …
Study identifies methods for handling sparse data in web settings with contextual bandits.
problem Handling sparse data in web settings with contextual bandits.
method Identified and categorized methods for addressing sparse data issues.
result Updated understanding of sparse data problems using contextual bandits in web settings.
Develops SGP-VAE for efficient sparse GP inference in multi-dimensional datasets.
problem Sparse GP approximations and missing data in multi-dimensional spatio-temporal datasets.
method Leverages partial inference networks for sparse GP approximations and amortized variational inference.
result Outperforms multi-output GPs and structured VAEs in various experiments.
Sparse linear models improve neural network debuggability.
problem Improving neural network interpretability and debugging.
method Using sparse linear models over learned deep feature representations.
result The approach leads to more debuggable and accurate neural networks.
Proposes a new model for high-dimensional data analysis with unknown link function.
problem Estimating link function, component functions, and variable interactions in high-dimensional data.
method Generalized Sparse Additive Model with Unknown Link Function (GSAMUL) using B-spline basis and MLP network for link estimation, with ℓ2,1-norm regularizer for variable selection. result Can realize both variable selection and hidden interaction.
Study on deep learning for speckle noise reduction in imaging modalities.
problem Multiplicative speckle noise challenges conventional deep learning methods for speckle denoising.
method Likelihood-based deep neural network (DNN) estimators for nonparametric regression under speckle noise.
result Established minimax rates for speckle denoising, matching those for additive Gaussian noise alone.
Despite their impressive performance, deep convolutional neural networks (CNNs) have been shown to be sensitive to small adversarial perturbations. These nuisances, which one can barely notice, are powerful enough to fool sophisticated and well performing classifiers, leading to ridiculous misclassification results. In…
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.
Probabilistic NDVI forecasting from sparse satellite data.
problem Challenges in short-term NDVI forecasting due to sparse and irregular satellite data.
method Probabilistic forecasting framework using historical NDVI and meteorological observations, with temporal-distance weighted quantile loss and extreme-weather feature engineering.
result The proposed method outperforms baselines on pointwise and probabilistic evaluation metrics.
We develop a novel procedure for constructing confidence bands for components of a sparse additive model. Our procedure is based on a new kernel-sieve hybrid estimator that combines two most popular nonparametric estimation methods in the literature, the kernel regression and the spline method, and is of interest in it…
Fast classification for sparse models, even with correlated features.
problem Sparse classification with many correlated features.
method Linear and quadratic surrogate cuts, priority queue, and analytical solution for exponential loss.
result 2 to 5 times faster than previous approaches, interpretable models with comparable accuracy.
New model leads to optimal test loss in sparse linear regression.
problem Sparse linear regression with low test loss despite interpolating training data.
method Developed a new parametrization of the model that combines benefits of ℓ1 and ℓ2 norms.
result Training via gradient descent leads to an interpolator with near-optimal test loss.
Meta Additive Model learns auto-weighting for robust sparse learning.
problem Sparse models' sensitivity to atypical data and noise.
method Bilevel optimization framework with meta data-driven weighting.
result MAM outperforms state-of-the-art models under various data corruptions.
Deep learning predicts dynamics from sparse data.
problem Predicting spatiotemporal dynamics from sparse data.
method Spatially dimension-independent deep learning framework.
result Predicts dynamics from sparse data sites.
Integrates momentum into SMoE for improved stability and robustness.
problem Stable training and robustness to data contamination in SMoE models.
method Established a connection between SMoE dynamics and gradient descent, then integrated momentum.
result MomentumSMoE is more stable and robust than SMoE.
Proposes a method to learn sparse deep neural networks with theoretical guarantees.
problem Over-parameterized deep neural networks cause training, prediction, and interpretation difficulties.
method Frequentist-like method for sparse DNNs under Bayesian framework.
result Consistent sparse DNNs with at most O(n/log(n)) connections. Much recent work has concerned sparse approximations to speed up the Gaussian process regression from the unfavorable O(n3) scaling in computational time to O(nm2). Thus far, work has concentrated on models with one covariance function. However, in many practical situations additive models with multiple covariance func…
New regularization techniques improve stability of deep neural networks.
problem Improving stability of deep neural networks in high-dimensional data.
method Apply manifold regularization to develop new regularizers based on graph Laplacian sparsification.
result Empirically, networks achieve high stability in various perturbation models, including adversarial attacks.
Develops methods for estimating and providing confidence bands in sparse high-dimensional additive models.
problem Estimating and providing reliable confidence bands for nonparametric components in high-dimensional additive models.
method Integrates sieve estimation into a high-dimensional Z-estimation framework, employing a multiplier bootstrap procedure.
result Constructs uniformly valid confidence bands for the target component f1 in sparse high-dimensional additive models.