Sparse neural networks can match dense models on Lipschitz functions.
problem Sparse networks are more efficient but lack theoretical guarantees.
method Formal model of sparse networks, LSH-based routing function, Lipschitz function approximation.
result Sparse networks can approximate dense networks on Lipschitz functions.
Evidential Softmax preserves multimodality in sparse probability distributions for generative models.
problem Sparse probability distributions in deep generative models make exact marginalization computationally intractable.
method Introduce ev-softmax, a sparse normalization function that preserves multimodality and can be trained with probabilistic loss functions.
result ev-softmax outperforms existing techniques in distributional accuracy and dimensionality reduction.
Efficiently estimates sparse functionals robustly in high dimensions.
problem Statistical procedures are sensitive to minor deviations in high-dimensional settings.
method Proposes a computationally and statistically efficient algorithm for robust estimation of sparse functionals.
result Guarantees accurate recovery of sparse functionals under certain deterministic conditions.
Study confirms sparse coding in whole brain using MRI data.
problem Sparse coding in the whole brain's neural activities.
method Applied various matrix factorization methods to fMRI data.
result Sparse coding hypothesis in information representation in the whole human brain is confirmed.
Scalable multi-task regression via sparse Gaussian process priors.
problem Efficiently modeling and predicting multiple related tasks.
method Direct Cholesky factorization for sparse parameterization of Gaussian process priors.
result Sparse parameterization improves scalability and accuracy in multi-task regression.
This paper addresses the problem of sparsity penalized least squares for applications in sparse signal processing, e.g. sparse deconvolution. This paper aims to induce sparsity more strongly than L1 norm regularization, while avoiding non-convex optimization. For this purpose, this paper describes the design and use of…
Convex-constrained sparse additive models improve regression performance.
problem High-dimensional nonparametric regression with shape constraints.
method Sparse difference of convex additive models (SDCAM) with regularization and efficient backfitting algorithm.
result SDCAM estimates functions without smoothness assumptions and outperforms existing methods.
Develops methods for selecting and estimating smooth functional coefficients in high-dimensional multivariate functional data.
problem Functional predictor selection and estimation of smooth functional coefficients in high-dimensional multivariate functional data.
method Functional group-sparse regression methods in a generic Hilbert space of infinite dimension.
result Consistency of estimation and selection (oracle property) under infinite-dimensional Hilbert spaces.
Sparse Hopfield model improves memory retrieval with fewer connections.
problem Memory retrieval efficiency with fewer connections.
method Sparse extension of Hopfield model, derived from sparse entropic regularizer.
result Sparse Hopfield model achieves tighter error bounds and better performance.
Self-attention prefers sparse functions of input sequences, reducing sample complexity.
problem Understanding the inductive biases of self-attention in modeling long-range dependencies.
method Theoretical analysis and synthetic experiments to probe sample complexity of learning sparse functions with Transformers.
result Bounded-norm Transformer networks can represent sparse functions of the input sequence with logarithmic sample complexity.
Optimal sketching bounds for sparse linear regression under various loss functions are established.
problem Sparse linear regression under different loss functions.
method Distribution over oblivious sketches for sparse ℓ 2 \ell_2 ℓ 2 norm regression and hinge-like loss functions. result Optimal sketching bounds with O ( k log ( d / k ) / ε 2 ) O(k\log(d/k)/\varepsilon^2) O ( k log ( d / k ) / ε 2 ) rows for sparse ℓ 2 \ell_2 ℓ 2 norm regression and O ( μ 2 k log ( μ n d / ε ) / ε 2 ) O(μ^2 k\log(μn d/\varepsilon)/\varepsilon^2) O ( μ 2 k log ( μ n d / ε ) / ε 2 ) rows for hinge-like loss functions. HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
Unified framework for constructing nonconvex sparse recovery methods.
problem Constructing valid nonconvex regularization functions remains open.
method Unified framework based on probability density function, using Weibull distribution.
result New nonconvex sparse recovery method based on Weibull distribution.
Optimizes sparse functions with gradient-free methods, reducing complexity.
problem Optimizing high-dimensional functions without gradients.
method Sparse Stochastic Zeroth-Order Optimization (SZO) with theoretical and experimental validation.
result Iteration complexity reduced to expected number of active features.
New insights into brain networks show they can approximate complex functions efficiently.
problem Understanding how brain networks learn and approximate functions.
method Characterized function spaces induced by sparse random features in brain networks.
result Sparse brain networks can approximate functions of high dimensionality.
Reduces function approximation dimensions from high to low with sparse data.
problem Function approximation from sparse data.
method Nonlinear Level Set Learning (NLL) with geometric information.
result Reduces input dimension to theoretical lower bound with minor accuracy loss.
Paper finds sparse representation of functions using inverse scale space flow.
problem Finding sparse representation of L 2 L^2 L 2 functions. method Inverse scale space flow to minimize L 2 L^2 L 2 loss. result Convergence to optimal solution in ideal and noisy cases.
New algorithms learn sparse set functions in non-orthogonal Fourier bases.
problem Learning sparse set functions in non-orthogonal Fourier bases.
method Novel algorithms using non-orthogonal Fourier transforms.
result At most n k − k log 2 k + k nk - k \log_2 k + k nk − k log 2 k + k queries for k k k non-zero Fourier coefficients. Proposes a new sparse recovery method using generalized error function.
problem Sparse recovery in signal processing and imaging.
method Introduces a penalty function with shape and scale parameters for sparse recovery.
result The method improves MRI reconstruction and is theoretically sound.
SANs use sparse activation functions to compress data representations.
problem Learning meaningful features without considering compression.
method Introduce φ metric, define activation functions, and present SANs.
result SANs achieve small description length and interpretable kernels.
The paper shows that g-convex functions on manifolds are sparse.
problem Characterizing and understanding the sparseness of g-convex functions.
method Established criteria for g-convexity and used them to prove sparseness results.
result Most g-convex functions on compact manifolds have few critical points.
SANs use sparse activation functions to minimize model complexity.
problem Model complexity in unsupervised learning.
method Introduce φ metric, define activation functions, present Sparsely Activated Networks (SANs).
result SANs with selected activation functions have small description length and interpretable kernels.
Transformers learn sparse Boolean functions through RL and SFT, revealing distinct learning behaviors.
problem Learning sparse Boolean functions with Transformers.
method Reinforcement Learning (RL) with process rewards and Supervised Fine-Tuning (SFT).
result RL learns the whole CoT chain simultaneously, while SFT learns step by step.
Majority voting neural networks improve binary compressed sensing for sparse signal recovery.
problem Sparse signal recovery in binary compressed sensing.
method Majority voting neural networks with a cross entropy-like term and L1 regularization.
result The majority voting neural network achieves excellent recovery performance, approaching optimal performance as the number of component nets grows.
Efficiently prices American options with multiple assets using sparse grids.
problem Pricing American options with multiple underlying assets efficiently.
method Dynamic programming formulation followed by sparse grid interpolation.
result Sparse grids reduce the number of interpolation points and maintain function smoothness.
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
problem Offline reinforcement learning with sparse decision dynamics and estimation sparsity.
method Reward-filtered least-squares policy evaluation using thresholded lasso.
result The method provides theoretical guarantees with sample complexity dependent on sparse component size.
Standard sparse pseudo-input approximations to the Gaussian process (GP) cannot handle complex functions well. Sparse spectrum alternatives attempt to answer this but are known to over-fit. We suggest the use of variational inference for the sparse spectrum approximation to avoid both issues. We model the covariance fu…
Sparse perturbations improve convergence in SZO methods for faster training.
problem Dependency of SZO methods on function dimensionality limits their convergence speed.
method Sparse perturbations reduce the effective dimensionality of the optimization problem.
result Sparse SZO optimization leads to faster convergence in training loss and test accuracy.
New insights into the top-K sparse softmax gating function for deep learning.
problem Understanding the theoretical effects of the top-K sparse softmax gating function on density and parameter estimations.
method Using a Gaussian mixture of experts, novel loss functions, and theoretical analysis.
result The convergence rates of density and parameter estimations are parametric under certain conditions, but slow under over-specified models.
Sparse Transformers can approximate dense Transformers with only O(n) connections.
problem Can sparse Transformers approximate arbitrary sequence-to-sequence functions?
method Proposed sufficient conditions for universal approximation and proved that sparse Transformers with O(n) connections can approximate dense models.
result Sparse Transformers with O(n) connections can approximate the same function class as dense models with n^2 connections.
Unified approach for estimating causal contrasts robust to model misspecification.
problem Estimating causal contrasts in non-parametric models with potential misspecification.
method A unifying approach using doubly-robust ℓ 1 \ell_1 ℓ 1 regularized estimation with cross-fitting and sample splitting. result Estimators are robust to misspecification of either nuisance function, ensuring consistent and normal asymptotic properties.
Regularized variants of Principal Components Analysis, especially Sparse PCA and Functional PCA, are among the most useful tools for the analysis of complex high-dimensional data. Many examples of massive data, have both sparse and functional (smooth) aspects and may benefit from a regularization scheme that can captur…
SLIMs hybridize sparse linear and isotonic models for high-dimensional data.
problem High-dimensional data with non-linear relationships.
method Hybridizing sparse linear models and isotonic models, proposing a two-step algorithm for estimation.
result The algorithm accurately estimates sparse parameters and monotone functions.
Sparsity-constrained optimization has wide applicability in machine learning, statistics, and signal processing problems such as feature selection and compressive Sensing. A vast body of work has studied the sparsity-constrained optimization from theoretical, algorithmic, and application aspects in the context of spars…
Paper tackles imitation learning with sparse rewards and heterogeneous actions.
problem Challenges of imitation learning with sparse rewards and different actions.
method Proposes a method that balances imitation and reinforcement learning objectives.
result Agent efficiently leverages sparse rewards and learns from different actions.
Sparse curves on surfaces grow at a specific intermediate rate.
problem Understanding growth patterns of sparse curve systems on surfaces.
method Analyzing the intersection numbers and sizes of sparse curve systems.
result Sparse curve systems grow roughly like c g c^{\sqrt{g}} c g . Sparse optimization refers to an optimization problem involving the zero-norm in objective or constraints. In this paper, nonconvex approximation approaches for sparse optimization have been studied with a unifying point of view in DC (Difference of Convex functions) programming framework. Considering a common DC appro…
Deep networks learn sparse hierarchical features without CoD.
problem Overparameterized deep networks struggle with the curse of dimensionality.
method Norm-constrained neural networks for sparse compositional functions.
result Deep networks can learn sparse hierarchical features efficiently.
RO-TD learns sparse value functions efficiently.
problem Learning sparse value functions efficiently.
method RO-TD integrates off-policy convergent gradient TD methods and online convex regularization.
result RO-TD learns sparse value functions with low computational complexity.
Proposes a new model for high-dimensional data analysis with unknown link function.
problem Estimating link function, component functions, and variable interactions in high-dimensional data.
method Generalized Sparse Additive Model with Unknown Link Function (GSAMUL) using B-spline basis and MLP network for link estimation, with ℓ 2 , 1 \ell_{2,1} ℓ 2 , 1 -norm regularizer for variable selection. result Can realize both variable selection and hidden interaction.
The paper introduces methods to learn smooth functions on hypergraphs with sparsity.
problem Learning smooth functions on hypergraphs with sparsity.
method General framework for smoothness measures, sparse learning on hypergraphs.
result Proposes sparsely smooth formulations that induce sparsity on hypergraphs and show benefits in handling irrelevant or noisy data.
Proposes interpretable set functions for sparse categorical features.
problem Automating the creation of interpretable features from sparse categorical data.
method Deep lattice network model with monotonicity constraints for permutation-invariant feature vectors.
result Achieved similar accuracy to deep sets or neural networks, but with enhanced interpretability.
In high-dimensional data, many sparse regression methods have been proposed. However, they may not be robust against outliers. Recently, the use of density power weight has been studied for robust parameter estimation and the corresponding divergences have been discussed. One of such divergences is the γ γ γ -divergence a…
Sparse Gaussian processes with compact kernels for faster inference.
problem Efficient Gaussian process inference with high computational complexity.
method Parametric families of compactly-supported kernels for sparse matrix representations.
result Sub-quadratic inference complexity and improved performance on real-world tasks.
Sparse ELM classifier predicts brain ages from adolescent multimodal brain data.
problem Predicting brain ages from adolescent multimodal brain data with high accuracy.
method Sparse ELM classifier using residual errors for feature pruning.
result RES-ELM classifier outperforms conventional and sparse Bayesian learning ELM.
sBayFDNN bridges deep learning and functional data analysis for complex, structured data.
problem Challenges in functional data analysis, especially for complex, continuously structured data.
method Sparse Bayesian functional deep neural network (sBayFDNN) that learns adaptive functional embeddings and interpretable region selection.
result First theoretical guarantees for a Bayesian deep functional model, ensuring reliability and statistical rigor.
Sparse random features improve accuracy in data-scarce settings.
problem Limited accuracy of random feature methods in data-scarce applications.
method Sparse random feature expansion using compressive sensing.
result Improved generalization bounds for sparse random features.
SDIFT generates full-field dynamics from sparse, irregular data.
problem Modeling and reconstructing physical dynamics from sparse, off-grid observations.
method SDIFT uses a functional Tucker model and sequential diffusion for generating full-field evolution from irregular sparse observations.
result Significant improvements in reconstruction accuracy and computational efficiency compared to state-of-the-art approaches.