arXiv research

Predictive Sparse Manifold Transform learns dynamic video sequences.

We present a signal representation framework called the sparse manifold transform that combines key ideas from sparse coding, manifold learning, and slow feature analysis. It turns non-linear transformations in the primary sensory signal space into linear interpolations in a representational embedding space while maint…

SGPA calibrates transformer uncertainty for safety-critical tasks.

Minimalistic unsupervised learning with sparse manifold transform achieves SOTA performance.

Deep learning uses layers of transformations to predict structured data with uncertainty.

Transformers can learn optimal variable selection in group-sparse classification.

Sparse representations using data dictionaries provide an efficient model particularly for signals that do not enjoy alternate analytic sparsifying transformations. However, solving inverse problems with sparsifying dictionaries can be computationally expensive, especially when the dictionary under consideration has a …

SPARTAN learns sparse interaction graphs between objects in scenes.

Generates high-quality images using sparse DCT representations.

In this paper, we consider the sparse regularization of manifold-valued data with respect to an interpolatory wavelet/multiscale transform. We propose and study variational models for this task and provide results on their well-posedness. We present algorithms for a numerical realization of these models in the manifold…

Transformers exhibit sparse activation maps, reducing computational load and improving robustness.

Sparse Transformers can approximate dense Transformers with only O(n) connections.

Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.

Multi-output prediction deals with the prediction of several targets of possibly diverse types. One way to address this problem is the so called problem transformation method. This method is often used in multi-label learning, but can also be used for multi-output prediction due to its generality and simplicity. In thi…

The paper offers generalization bounds for Transformers that ignore sequence length.

E$^3$ combines sparse MoEs and ensembles to improve model efficiency and performance.

Sparse transformer architecture improves accuracy and speed in generative modeling and inverse problems.

DRFormer uses dynamic tokenization and multi-scale transformer to forecast long time series.

Study improves paddy rice yield predictions in Peru using sparse regression and climatic variables.

Sparse Transformers degrade semantic information first, with early layers encoding more.

FEDformer combines Transformer with seasonal-trend decomposition for efficient long-term forecasting.

The presence of a sparse "truth" has been a constant assumption in the theoretical analysis of sparse PCA and is often implicit in its methodological development. This naturally raises questions about the properties of sparse PCA methods and how they depend on the assumption of sparsity. Under what conditions can the r…

Many transformations in deep learning architectures are sparsely connected. When such transformations cannot be designed by hand, they can be learned, even through plain backpropagation, for instance in attention mechanisms. However, during learning, such sparse structures are often represented in a dense form, as we d…

In drug-discovery-related tasks such as virtual screening, machine learning is emerging as a promising way to predict molecular properties. Conventionally, molecular fingerprints (numerical representations of molecules) are calculated through rule-based algorithms that map molecules to a sparse discrete space. However,…

Combining LETKF and RC improves chaotic system prediction from noisy, sparse data.

Transformers learn sparse Boolean functions through RL and SFT, revealing distinct learning behaviors.

Nonnegative matrix factorization (NMF) is a popular method for audio spectral unmixing. While NMF is traditionally applied to off-the-shelf time-frequency representations based on the short-time Fourier or Cosine transforms, the ability to learn transforms from raw data attracts increasing attention. However, this adds…

New framework improves generative models with prediction and consistency constraints.

Combines deep and statistical learning for structured data.

DGPFM uses deep Gaussian processes to map functions accurately and quantify uncertainty.

Neural networks compress and sample WDN contamination dynamics efficiently.

Self-attention prefers sparse functions of input sequences, reducing sample complexity.

A new model improves CT image quality from low-dose scans.

The paper explores algorithms to transform 3-manifold triangulations while controlling sparsity.

SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.

The study of healthy brain development helps to better understand the brain transformation and brain connectivity patterns which happen during childhood to adulthood. This study presents a sparse machine learning solution across whole-brain functional connectivity (FC) measures of three sets of data, derived from resti…

Let $Φ\in\mathbb{R}^{m\times n}$ be a sparse Johnson-Lindenstrauss transform [KN14] with $s$ non-zeroes per column. For a subset $T$ of the unit sphere, $\varepsilon\in(0,1/2)$ given, we study settings for $m,s$ required to ensure $\mathop{\mathbb{E}}_Φ\sup_{x\in T} \left|\|Φx\|_2^2 - 1 \right| < \varepsilon ,$ i.e…

It is difficult to find the optimal sparse solution of a manifold learning based dimensionality reduction algorithm. The lasso or the elastic net penalized manifold learning based dimensionality reduction is not directly a lasso penalized least square problem and thus the least angle regression (LARS) (Efron et al. \ci…

Global Memory Augmentation (GMAT) improves Transformer performance on long documents.

A new iterative low complexity algorithm has been presented for computing the Walsh-Hadamard transform (WHT) of an $N$ dimensional signal with a $K$-sparse WHT, where $N$ is a power of two and $K = O(N^α)$, scales sub-linearly in $N$ for some $0 < α< 1$. Assuming a random support model for the non-zero transform domain…

Efficiently computes sparse signature coefficients using kernels.

Transformers learn to integrate information from past positions incrementally, specializing heads in distinct patterns.

New auto-encoder handles varying noise levels without retraining.

Hybrid approach combines transformer and Bayesian filtering for robust multiple particle tracking.

New algorithms learn sparse set functions in non-orthogonal Fourier bases.

CoT improves transformer sample efficiency by reducing input token dependencies and attention sparsity.

New method predicts aphasia severity with narrower uncertainty intervals.

New GPU kernels boost deep learning speed and memory efficiency.