The article provides formulas for homological blocks of Seifert fibered homology 3-spheres.
problem Calculating Witten-Reshetikhin-Turaev invariants for Seifert fibered homology 3-spheres.
method Explicit modular transformation formulas of homological blocks.
result New proof of Witten asymptotic conjecture for Seifert fibered homology 3-spheres.
This paper proposes network recasting as a general method for network architecture transformation. The primary goal of this method is to accelerate the inference process through the transformation, but there can be many other practical applications. The method is based on block-wise recasting; it recasts each source bl…
DiffusionBlocks trains neural networks by breaking them into independent blocks, reducing memory usage.
problem Memory bottlenecks in end-to-end neural network training.
method Transforming transformer-based networks into independent trainable blocks via a denoising process.
result Independent block-wise training matches end-to-end training performance while reducing memory requirements.
Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
Paper finds sharpness differences in transformer blocks accelerating LLM training.
problem Understanding and accelerating large language model pre-training.
method Uncovering sharpness disparity across transformer blocks and proposing Blockwise Learning Rate.
result Blockwise Learning Rate strategy accelerates LLM pre-training with lower loss and speedup.
Improved singular value approximation for convolutional layers.
problem Improving accuracy of singular value approximation for linear convolutional layers.
method Developed a new spectral density matrix method for singular value approximation with improved accuracy and reduced computational complexity.
result Obtained moderate improvement in singular value distribution compared to circular approximation.
SympFormer accelerates attention blocks using inertial dynamics on density spaces.
problem Improving the efficiency of self-attention blocks in Transformers.
method Introduced accelerated attention blocks derived from inertial Nesterov dynamics on density spaces.
result Accelerated attention blocks converge faster than classical blocks while preserving oracle calls.
Training-free looped transformers improve model performance without additional training.
problem Improving model performance without additional training or fine-tuning.
method A lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning.
result Our method improves model performance across various model families.
DEN learns diverse tasks to generalize to unseen tasks.
problem Generalization from a diverse set of classification tasks with limited data.
method Three-block architecture: covariate transformation, distribution embedding, and classification.
result DEN outperforms existing methods in various synthetic and real tasks.
Improved hyperparameter optimization using simplified Transformer blocks.
problem Discovering optimal architectures in high-dimensional search spaces with limited exploration budgets.
method Simplified Transformer block for modeling hyper-parameter dependencies, actor-critic style algorithm, ensembling.
result Outperformed most algorithms on NAS-Bench-101 and Random Search in discovering more accurate model architectures.
This paper precisely estimates transformer derivatives for explicit learning guarantees.
problem Computing fully-explicit generalization bounds for transformers with precise higher-order derivative estimates.
method Analyzes and estimates all higher-order derivatives of transformers with multiple attention heads and layer normalization.
result Obtains explicit pathwise generalization bounds for transformers learning from non-i.i.d. samples.
A new quantization strategy reduces Transformer model size and inference time.
problem Heavy computation load and memory overhead in Transformer models for mobile devices.
method Mixed precision quantization with varying bits per word in embedding blocks.
result 11.8x smaller model size and 3.5x speed up for on-device NMT.
Filters in convolutional networks are typically parameterized in a pixel basis, that does not take prior knowledge about the visual world into account. We investigate the generalized notion of frames designed with image properties in mind, as alternatives to this parametrization. We show that frame-based ResNets and De…
Study isotropy groups for complex orthogonal and skew-symmetric matrices.
problem Understanding isotropy subgroups of orthogonal similarity transformations.
method Analysis of group structure of nonsingular block matrices.
result Group structure of isotropy subgroups related to block Toeplitz matrices.
In many state-of-the-art compression systems, signal transformation is an integral part of the encoding and decoding process, where transforms provide compact representations for the signals of interest. This paper introduces a class of transforms called graph-based transforms (GBTs) for video compression, and proposes…
Linear Transformer Block combines MLP and linear attention for near-optimal ICL in linear regression.
problem Achieving near-optimal in-context learning (ICL) risk for linear regression with a Gaussian prior.
method Combines linear attention and MLP components in a Linear Transformer Block (LTB). Establishes correspondence with one-step gradient descent estimators (GDext−β). result LTB achieves nearly Bayes optimal ICL risk for linear regression with a Gaussian prior.
WavPool improves deep neural networks with wavelet-based pooling.
problem Improving efficiency and performance of deep neural networks.
method Introducing WavPool, a wavelet-transform-based pooling layer.
result WavPool outperforms existing network architectures by 10% on CIFAR-10.
The paper considers the block sampling method for long-range dependent processes. Our theory generalizes earlier ones by Hall, Jing and Lahiri (1998) on functionals of Gaussian processes and Nordman and Lahiri (2005) on linear processes. In particular, we allow nonlinear transforms of linear processes. Under suitable c…
HaKAN uses Hahn-KAN blocks to forecast multivariate time series.
problem Long-term time series forecasting challenges with high complexity and spectral bias.
method HaKAN integrates channel independence, patching, and a stack of Hahn-KAN blocks with residual connections. It uses Hahn polynomial-based learnable activation functions.
result HaKAN consistently outperforms state-of-the-art methods on various forecasting benchmarks.
We propose unitary group convolutions (UGConvs), a building block for CNNs which compose a group convolution with unitary transforms in feature space to learn a richer set of representations than group convolution alone. UGConvs generalize two disparate ideas in CNN architecture, channel shuffling (i.e. ShuffleNet) and…
In many video coding systems, separable transforms (such as two-dimensional DCT-2) have been used to code block residual signals obtained after prediction. This paper proposes a parametric approach to build graph-based separable transforms (GBSTs) for video coding. Specifically, a GBST is derived from a pair of line gr…
PARD generates graphs efficiently and invariantly to node ordering.
problem Graph generation sensitivity to node ordering.
method Integrates autoregressive and diffusion models with a partial order for nodes and edges.
result PARD achieves state-of-the-art performance on molecular and non-molecular datasets.
New ODE-Block handles stateful layers with continuous-in-depth functions using basis functions.
problem Handling stateful layers in ODE-Nets.
method Formulate ODE-Block using continuous-in-depth functions with basis function expansions.
result Enables state-of-the-art performance and reduces memory footprint.
We introduce block-tree graphs as a framework for deriving efficient algorithms on graphical models. We define block-tree graphs as a tree-structured graph where each node is a cluster of nodes such that the clusters in the graph are disjoint. This differs from junction-trees, where two clusters connected by an edge al…
Singing Voice Separation (SVS) tries to separate singing voice from a given mixed musical signal. Recently, many U-Net-based models have been proposed for the SVS task, but there were no existing works that evaluate and compare various types of intermediate blocks that can be used in the U-Net architecture. In this pap…
TNP-KR improves scalability of NPs with Transformer blocks and attention mechanisms.
problem Scalability bottleneck in NPs and GPs, especially for large datasets.
method Introduces TNP-KR with KRBlock, kernel-based attention, and two attention mechanisms.
result TNP-KR with DKA outperforms Performer and achieves state-of-the-art results.
The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we fir…
Global Memory Augmentation (GMAT) improves Transformer performance on long documents.
problem Large memory requirements of Transformer pairwise dot-product attention for long sequences.
method Integrates a dense global memory of length M into sparse Transformer blocks.
result Significant improvement on various tasks, including synthetic tasks, masked language modeling, and reading comprehension.
Transformers interpreted as probabilistic Laplacian Eigenmaps steps.
problem Improving transformer performance through probabilistic interpretation.
method Probabilistic Laplacian Eigenmaps model derivation and graph diffusion step.
result Subtracting identity from attention matrix improves transformer performance.
Method estimates number of clusters in Block Markov Chain trajectories.
problem Challenges in choosing number of clusters for sequential data.
method Spectral embedding and density-based clustering.
result Asymptotically consistent method for estimating clusters.
Improved graph embedding through refined linear transformation and community recovery.
problem Identifying meaningful latent communities in graph data.
method Refined graph encoder embedding via linear transformation, self-training, and latent community recovery.
result Improved vertex embedding and better decision boundaries for vertex classification.
THieF improves day-ahead electricity price prediction accuracy by reconciling hourly and block forecasts.
problem Improving accuracy in predicting day-ahead electricity prices.
method Temporal hierarchy forecasting (THieF) reconciling hourly and block forecasts.
result THieF significantly improves accuracy (up to 13%) at all levels of prediction.
Generates high-quality images using sparse DCT representations.
problem Challenges in generating images due to high dimensionality.
method Transformers trained on sparse DCT block sequences.
result Competitive image generation quality with state-of-the-art methods.
New invertible transformations improve flow-based generative models.
problem Improving flow-based generative models for better performance.
method Proposed new invertible transformations and coupling layers.
result New coupling layers achieve better results in IDF.
In this paper, we propose several dictionary learning algorithms for sparse representations that also impose specific structures on the learned dictionaries such that they are numerically efficient to use: reduced number of addition/multiplications and even avoiding multiplications altogether. We base our work on facto…
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
Proposes σ-PCA to learn identifiable linear transformations without whitening.
problem Cannot identify axes with equal variances in PCA.
method Unified model for linear and nonlinear PCA, introducing a missing piece to eliminate rotational indeterminacy.
result Eliminates subspace rotational indeterminacy in PCA.
The study explains how transformer components enable in-context learning.
problem Understanding how transformer components contribute to in-context learning.
method Analyzed a two-attention-layer transformer model trained on Markov chain data.
result Gradient flow converges to a limiting model with a copier, selector, and classifier mechanism.
Many recent invertible neural architectures are based on coupling block designs where variables are divided in two subsets which serve as inputs of an easily invertible (usually affine) triangular transformation. While such a transformation is invertible, its Jacobian is very sparse and thus may lack expressiveness. Th…
Develops a cross-lingual hate speech detection model using pre-trained Transformers.
problem Detecting hate speech in low-resource languages.
method Utilizes frozen Transformer language models and AXEL attention-based classification block for zero-shot and few-shot learning.
result Demonstrates highly competitive results on English and Spanish subsets of the HatEval challenge.
Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforci…
SGPA calibrates transformer uncertainty for safety-critical tasks.
problem Uncertainty estimation in transformer models for safety-critical domains.
method Bayesian inference in transformer's output space using sparse Gaussian processes.
result SGPA-based Transformers improve in-distribution calibration and out-of-distribution robustness.
DEFINED improves wireless symbol detection with limited pilot data.
problem Efficient symbol detection over block-fading channels with scarce pilot data.
method In-context learning with decision feedback mechanism.
result Significant performance improvements, often needing only a single pilot pair.
TSLANet improves time series models by capturing long-term and short-term interactions.
problem Noise sensitivity, computational efficiency, and overfitting in Transformer-based models for time series data.
method Adaptive Spectral Block and Interactive Convolution Block for robust feature representation and noise mitigation.
result TSLANet outperforms state-of-the-art models in various time series tasks.
Transformers tend to learn more symmetric functions in sequence data.
problem Understanding inductive bias in Transformers with infinitely over-parameterized models.
method Analyzing Transformers in the Gaussian process limit, using representation theory of the symmetric group.
result Transformers are biased towards more permutation symmetric functions, and this can be quantitatively predicted.
Signal models based on sparsity, low-rank and other properties have been exploited for image reconstruction from limited and corrupted data in medical imaging and other computational imaging applications. In particular, sparsifying transform models have shown promise in various applications, and offer numerous advantag…
New Transformers maintain Lipschitz continuity for robustness.
problem Ensuring robustness in Transformers for safety-sensitive applications.
method Introducing gradient-descent-type in-context Transformers with explicit Euler steps of negative gradient flows.
result Universal approximation theorem for Lipschitz continuous Transformers.
Transformers learn to solve modular arithmetic tasks by in-context learning and skill composition.
problem Understanding how large language models generalize to unseen tasks in modular arithmetic.
method Pre-training on a set of modular arithmetic tasks and evaluating out-of-distribution performance.
result Transformers require two transformer blocks for out-of-distribution generalization, and deeper models exhibit transient out-of-distribution performance.