Previous research has shown that computation of convolution in the frequency domain provides a significant speedup versus traditional convolution network implementations. However, this performance increase comes at the expense of repeatedly computing the transform and its inverse in order to apply other network operati…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “transformer activations”
Basis Manifoldmath.DG
In this paper we study a symmetry group of vector space. Basis manifold is a homogeneous space of a symmetry group. This concept leads us to the definition of active and passive transformations on basis manifold. Active transformation can be expressed as a transformation of vector space. Passive transformation gives ab…
Spark Transformer achieves high sparsity in FFN and attention without sacrificing model quality.
problem Inactivity of most neurons in FFN of Transformers, leading to inefficiency.
method Top-k masking, statistical top-k algorithm, and predictor for activated entries.
result Only 8% of FFN neurons are activated, achieving up to 2.5x wall-time speedup.
Transformers exhibit sparse activation maps, reducing computational load and improving robustness.
problem Sparse activation in Transformer models.
method Extensive experiments on various Transformer architectures and tasks.
result Sparsity in Transformers is a prevalent phenomenon, reducing FLOP count and improving model robustness.
DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcs.LG
DFRot improves LLMs by reducing outlier and massive activation effects.
problem Reducing outlier and massive activation effects in rotated LLMs.
method Weighted loss function and orthogonal Procrustes transforms for rotation matrix refinement.
result DFRot achieves dual free (Outlier-Free and Massive Activation-Free) with significant improvements in perplexity.
We study the theoretical advantages of active learning over passive learning. Specifically, we prove that, in noise-free classifier learning for VC classes, any passive learning algorithm can be transformed into an active learning algorithm with asymptotically strictly superior label complexity for all nontrivial targe…
NAST generalizes scattering transform for non-stationary time series analysis.
problem Analyzing non-stationary time series data.
method Neural activation of scattering transform with various activation functions and high pass filters.
result Central and non-central limit theorems for NAST of Gaussian processes.
Sensitive inferences and user re-identification are major threats to privacy when raw sensor data from wearable or portable devices are shared with cloud-assisted applications. To mitigate these threats, we propose mechanisms to transform sensor data before sharing them with applications running on users' devices. Thes…
Recent seminal work at the intersection of deep neural networks practice and random matrix theory has linked the convergence speed and robustness of these networks with the combination of random weight initialization and nonlinear activation function in use. Building on those principles, we introduce a process to trans…
The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate wh…
Token-adaptive FFN design improves LLM expressivity.
problem Fixed activation functions limit FFN expressivity in LLMs.
method Mixture of Activations (MoA) and learnable activations (LA).
result MoA achieves lower terminal loss and better scaling than baselines.
Theory of MoE Transformers' generalization and scaling.
problem Understanding the generalization and scaling of Mixture-of-Experts (MoE) Transformers.
method Developed a theory that separates active capacity from routing combinatorics, derived a sup-norm covering-number bound, and proved a constructive approximation theorem.
result Generalization and scaling laws for MoE Transformers, showing how active capacity and routing structure affect performance.
The paper investigates polynomial alternatives to softmax in transformer models.
problem The effectiveness of softmax attention in transformers is questioned.
method The authors explore polynomial activations as alternatives to softmax, focusing on their ability to regularize the attention matrix.
result Certain polynomials can serve as effective substitutes for softmax in transformer applications, achieving strong performance.
A new KAN variant uses sinusoidal activations to approximate functions.
problem Approximating multivariable functions using neural networks.
method Replacing inner and outer functions in Kolmogorov-Arnold representation with weighted sinusoidal functions.
result The new KAN variant outperforms fixed-frequency Fourier transform and achieves comparable performance to MLPs.
TransFall uses transfer learning to improve activity recognition from mobile sensors.
problem Performance degradation due to platform and user movement differences.
method Two-tier data transformation, label estimation, and model generation layers.
result TransFall enhances activity recognition accuracy for new scenarios.
This paper studies activation sparsity in large language models, finding key trends and implications.
problem Activation sparsity in large language models (LLMs) can be improved for efficiency and interpretability.
method Proposes PPL- sparsity, analyzes trends with training data, width-depth ratio, and parameter scale.
result ReLU is more efficient for sparsity than SiLU, and deeper architectures can improve sparsity.
Sparse Transformers degrade semantic information first, with early layers encoding more.
problem Understanding how sparse Transformers affect learned representations and semantic information.
method Probed Transformers with progressively pruned weights to observe changes in semantic information and model behavior.
result Complex semantic information is first to degrade in sparse Transformers, with early layers encoding more.
MLPs can approximate any function in context, challenging the importance of in-context universality.
problem Understanding why transformers are more effective than classical models.
method Proved MLPs with trainable activation functions are universal in context.
result Transformer success is likely due to factors other than in-context universality.
A Power Transformcs.LG
Novel power transform unifies various mathematical functions.
problem Normalizing and standardizing datasets.
method Presented a novel power transform.
result Unified various mathematical functions.
BiTAT improves neural network quantization for edge devices by focusing on weight dependencies and disentangling them.
problem Performance degradation of compact neural networks under extreme quantization.
method Task-dependent Aggregated Transformation (BiTAT) method that orthonormalizes weights and progressively quantizes them.
result BiTAT effectively preserves model performance on ImageNet and CIFAR-100 with compact backbones.
S2D selectively decays large singular values to improve quantization of neural activations.
problem Large activation outliers in transformer models cause accuracy drops during quantization.
method Selective Spectral Decay () that surgically regularizes only the largest singular values.
result Significantly reduces activation outliers and produces well-conditioned representations.
We show LLMs can be locally linear, enabling better control of activations.
problem Suboptimal control of LLM activations during generation.
method Model LLM inference as a linear dynamical system, compute feedback controllers using Jacobians, and adapt classical control theory.
result Robust, fine-grained control of LLM activations across models and tasks.
Spectral gradient methods outperform Euclidean in certain deep learning scenarios.
problem When do spectral gradient updates outperform Euclidean in deep learning?
method Layerwise condition comparing squared nuclear-to-Frobenius ratio to stable rank of activations.
result Spectral updates can be more effective than Euclidean in deep networks and transformers.
Transformationally invariant processors constructed by transformed input vectors or operators have been suggested and applied to many applications. In this study, transformationally identical processing based on combining results of all sub-processes with corresponding transformations at one of the processing steps or …
Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unitystat.ML
Paper studies Transformer learning theory for Euclidean and Riemannian domains.
problem Understanding and optimizing Transformer networks for regression tasks.
method Constructive approximation framework using softmax partition of unity and attention mechanism.
result Transformer can achieve uniform ε-approximation error with minimal parameters.
Paper hypothesizes MLP layers in LLMs can be approximated by sparse Mixture of Experts.
problem Understanding dense MLP layers in LLMs.
method Theoretical connection between MoE models and SAE structure in activation space.
result MLP layers in LLMs can be well approximated by sparse Mixture of Experts.
Zero-Direction Probing: A Linear-Algebraic Framework for Deep Analysis of Large-Language-Model Driftcs.LG
ZDP detects drift in large language models without labels, proving key theorems and metrics.
problem Detecting drift in large language models without task labels or output evaluations.
method Zero-Direction Probing (ZDP) framework based on null directions of transformer activations, proving theoretical guarantees.
result Proves the Variance--Leak Theorem, Fisher Null-Conservation, Rank--Leak bound, and logarithmic-regret guarantee.
This paper addresses anisotropy in Transformer models, providing geometric insights and empirical support.
problem Anisotropy phenomenon in Transformer models, challenging their geometric interpretation.
method Derive geometric arguments and use concept-based mechanistic interpretability during training.
result Activation-derived directions capture large gradient energy and a larger share of gradient anisotropy than normal controls.
We simplify neural networks to 3D to study their topological changes.
problem Understanding how neural network layers affect low-dimensional topological invariants.
method Limiting each layer to a width of 3D space, tracking changes in linking numbers.
result ResNets and transformers are equally powerful in changing linking numbers.
This paper proposes network recasting as a general method for network architecture transformation. The primary goal of this method is to accelerate the inference process through the transformation, but there can be many other practical applications. The method is based on block-wise recasting; it recasts each source bl…
ALINE optimizes Bayesian inference and data acquisition by strategically querying informative data.
problem Strategic acquisition of informative data for Bayesian inference in challenging tasks.
method Unified framework combining amortized Bayesian inference and active data acquisition using a transformer architecture trained via reinforcement learning.
result ALINE delivers both instant and accurate inference along with efficient selection of informative points.
Researchers relax the CVF's smoothness requirement to create more flexible flow models.
problem Challenges in constructing flexible density models due to the CVF's smoothness requirement.
method Introduce -diffeomorphisms as generalized transformations that may violate smoothness on zero Lebesgue-measure sets.
result The relaxation allows for the use of non-smooth activation functions like ReLU in residual flows.
IGSD separates task-specific content channels in transformer components by comparing activation replacement with zero ablation.
problem Mechanistic interpretability of transformer components
method IGSD: paired-intervention framework for comparing activation replacement with zero ablation
result IGSD identifies an early-layer content channel in transformer components that standard importance methods underestimate.
The paper tackles matching a desired mean in causal systems through shift interventions.
problem Matching a desired mean in causal systems.
method Defining Markov equivalence classes, proposing active learning strategies, deriving lower bounds.
result Proposed active learning strategies require fewer interventions than previous approaches, especially for certain graph classes.
We introduce switched linear projections for expressing the activity of a neuron in a deep neural network in terms of a single linear projection in the input space. The method works by isolating the active subnetwork, a series of linear transformations, that determine the entire computation of the network for a given i…
A framework and benchmark for deep batch active learning in neural networks.
problem Efficiently acquiring labels for neural network regression.
method Framework of base kernels, transformations, and selection methods; use of sketched finite-width neural tangent kernels and clustering.
result Proposed method outperforms state-of-the-art on benchmark, scales to large data sets.
We prove that the global minimum of the backpropagation (BP) training problem of neural networks with an arbitrary nonlinear activation is given by the ridgelet transform. A series of computational experiments show that there exists an interesting similarity between the scatter plot of hidden parameters in a shallow ne…
Model predicts epileptic seizures by detecting preictal state using wavelet transform and PCA.
problem Predicting epileptic seizures before onset.
method Common spatial pattern filtering, wavelet transform for preprocessing, PCA for feature extraction, SVM for classification.
result Average sensitivity of 93.1% for 84 seizures in 23 subjects.
Automatic music transcription (AMT) aims to infer a latent symbolic representation of a piece of music (piano-roll), given a corresponding observed audio recording. Transcribing polyphonic music (when multiple notes are played simultaneously) is a challenging problem, due to highly structured overlapping between harmon…
Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
problem Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
method Formal study of PAC learnability of MLP layers with activation sparsity.
result Classes of functions with activation sparsity lead to provable computational and statistical advantages over their non-sparse counterparts.
We provide a theoretical explanation of the role of the number of nodes at each layer in deep neural networks. We prove that the largest variation of a deep neural network with ReLU activation function arises when the layer with the fewest nodes changes its activation pattern. An important implication is that deep neur…
Active learning reduces labeling efforts for QDE models.
problem Efficiently labeling data for QDE models with minimal human effort.
method Active learning approach using PowerVariance acquisition function.
result Active learning achieves performance close to fully supervised models with only 10% labeled data.
Deep learning transforms data geometrically, akin to Ricci flow, improving classification accuracy.
problem Understanding geometric transformations in non-smooth activation functions.
method Developed a computational framework to quantify geometric changes in DNNs and introduced the concept of `global Ricci network flow`.
result Global Ricci network flow correlates with DNN accuracy, independent of network architecture and data set.
Anomalies are intuitively easy for human experts to understand, but they are hard to define mathematically. Therefore, in order to have performance guarantees in unsupervised anomaly detection, priors need to be assumed on what the anomalies are. By contrast, active learning provides the necessary priors through approp…
Automatic trajectory recognition in Active Target Time Projection Chambers data by means of hierarchical clusteringphysics.ins-det
The automatic reconstruction of three-dimensional particle tracks from Active Target Time Projection Chambers data can be a challenging task, especially in the presence of noise. In this article, we propose a non-parametric algorithm that is based on the idea of clustering point triplets instead of the original points.…
Transformers can perform well with less long-range memory.
problem The need for deep long-range memory in Transformers for language modeling.
method Performed interventions to show performance can be achieved with fewer long-range memories and by limiting attention range.
result Comparable performance can be achieved with 6X fewer long-range memories and better performance with limited attention range.
New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
The paper shows neural networks can approximate functions over non-compact domains with non-polynomial activation.
problem Approximating functions over non-compact domains using neural networks.
method Using single-hidden-layer feedforward neural networks with non-polynomial activation functions over non-compact subsets of Euclidean spaces.
result Neural networks can approximate functions in weighted -spaces and weighted Sobolev spaces over unbounded domains.