As the Portable Document Format (PDF) file format increases in popularity, research in analysing its structure for text extraction and analysis is necessary. Detecting headings can be a crucial component of classifying and extracting meaningful data. This research involves training a supervised learning model to detect…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “classifier heads”
Headless attacks bypass classification heads to fool transfer learning models.
problem Adversarial attacks against transfer learning models without access to the classification head.
method Label-blind adversarial attacks that do not require class-label information.
result Transfer attack lowers ResNet18 accuracy on CIFAR10 by over 40%.
We introduce collaborative learning in which multiple classifier heads of the same network are simultaneously trained on the same training data to improve generalization and robustness to label noise with no extra inference cost. It acquires the strengths from auxiliary training, multi-task learning and knowledge disti…
Improved robustness of BERT through orthogonal classifier heads.
problem Improving robustness of language models for smaller datasets.
method Enforces orthogonal classifier heads and adaptively eliminates weaker ones.
result Improves accuracy by +9% on average across datasets.
Hessian alignment improves OOD generalization in deep learning.
problem Improving deep learning models' ability to generalize to out-of-distribution data.
method Analyzed Hessian and gradient alignment for domain generalization using recent OOD theory.
result Hessian alignment methods achieve promising performance on various OOD benchmarks.
Optimizes neural networks by removing unnecessary layers, improving performance and speed.
problem Finding the optimal depth of neural networks to improve performance and speed.
method Develops a fast end-to-end method for training lightweight neural networks with multiple classifier heads, allowing the model to determine the importance of each head and choosing a single shallow classifier.
result Significantly reduces the number of parameters and accelerates inference, outperforming many standard pruning methods.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcs.LG
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
Research classifies knots based on sliceness and amphichirality.
problem Classifying odd-stranded Turk's head knots based on sliceness and amphichirality.
method Constructing commuting pairs of ambient involutions and analyzing the equivariant Fox-Milnor square condition.
result Established a sharp parity dichotomy for equivariant rational sliceness and Klein amphichirality of odd-stranded Turk's head knots.
EERO optimizes resource usage for efficient classification.
problem Managing computational resources in complex machine learning models.
method EERO uses multiple classifiers with a reject option to adaptively shorten processing paths.
result EERO effectively manages budget allocation and enhances accuracy in complex scenarios.
GWHD dataset offers 4,700 high-res images of wheat heads.
problem Challenges in wheat head detection from high-resolution imagery.
method Large, diverse dataset with detailed metadata.
result Benchmark for wheat head detection methods.
Multi-head attention outperforms single-head in in-context linear regression tasks.
problem Comparing performance of transformer with single-/multi-head attention in in-context learning.
method Theoretical analysis of performance of transformers with different attention mechanisms in linear regression tasks.
result Multi-head attention with a substantial embedding dimension outperforms single-head attention in in-context linear regression tasks.
New theory shows how multi-head attention reduces variance and decorrelates outputs.
problem Understanding and optimizing multi-head attention in neural networks.
method Developed a statistical theory linking multi-head attention to ensemble Nadaraya-Watson estimators.
result MHA variance reduction depends on head decorrelation, not just head count.
Geometric theory of projection heads in self-supervised learning.
problem Dimensional collapse and information invariance trade-off in projection heads.
method Geometric modeling of projection heads as Riemannian metrics, analyzing Hessian eigenvalues, and tracking optimization geometry.
result Smooth nonlinear heads induce negative curvature, preventing collapse; linear and ReLU heads cannot.
Improved language models with talking-heads attention.
problem Language model perplexity and quality issues.
method Added linear projections in multi-head attention.
result Better perplexities and quality in language tasks.
New approach improves multi-head attention by making heads less similar.
problem Multi-head attention can lead to similar features, reducing model expressiveness.
method Proposes a non-parametric approach using Bayesian techniques to make heads repel each other.
result Improves feature diversity, leading to better representations and performance.
Deep forecasting models show output heads significantly improve performance on fat-tailed financial returns.
problem Improving deep learning models for forecasting fat-tailed financial returns.
method Comparison of backbone architectures and output heads (point, Gaussian, Gaussian mixture) on S&P 500 monthly log-returns.
result Switching from point to Gaussian heads improves CRPS by about 1.3 percent, and from Gaussian to mixture adds another 2.4 percent.
Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables in a spoken word. From the subsequences, it retrieves richer information than a single-head attention which only summarizes the whole sequen…
Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are de…
One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
Survey of Turk's head knots and links properties.
problem Characterize Turk's head knots and links.
method Discussion of various results in the mathematical literature.
result Turk's head links are alternating, fibered, hyperbolic, invertible, non-split, periodic, and prime.
This work proposes a collaborative multi-head attention layer to reduce model size without sacrificing accuracy.
problem Over-parameterization in transformer models trained with large datasets.
method Proposes a collaborative multi-head attention layer that shares key/query projections.
result Reduction in model size by 4 for same accuracy and speed.
Multi-headed ensembles boost model performance with faster training.
problem Limited computational resources hinder ensemble search performance.
method Extend NES to multi-headed ensembles, leveraging end-to-end training and one-shot NAS methods.
result Multi-headed ensemble search finds robust ensembles 3 times faster with comparable performance.
New approach reduces model size for Transformer architectures.
problem Large embedding dimensions limit model applicability.
method Identified low-rank bottleneck in multi-head attention.
result Reducing head size to sequence length improves model performance.
Study on multi-head softmax attention dynamics for in-context learning.
problem Understanding and optimizing multi-head softmax attention models for multi-task linear regression.
method Gradient flow analysis and spectral mapping technique.
result Gradient flow converges to optimal multi-head softmax attention model, with task allocation emerging during training.
Transformer-MGK replaces redundant heads with Gaussian key mixtures, improving efficiency and performance.
problem Redundant attention heads in transformers degrade performance and efficiency.
method Transformer-MGK replaces redundant heads with a mixture of Gaussian keys.
result Transformer-MGK accelerates training and inference, reduces parameters and FLOPs, and achieves comparable or better accuracy.
New method improves text classification without labeled target data.
problem Improving text classification under domain shift without labeled target data.
method Diversity-based generalization using multi-head attention with diversity constraints.
result Method matches state-of-the-art performance without labeled target data.
Minimalistic model captures head direction system properties.
problem Representing head direction system in a high-dimensional space.
method A minimalistic representation model of the rotation group U(1), including fully connected and convolutional versions.
result Emergence of Gaussian-like tuning profiles and 2D circle geometry in both model versions.
Develops a mean-field theory for multi-head self-attention under cross-entropy training.
problem Mean-field analysis of multi-head self-attention under cross-entropy training.
method Mean-field theory for a simplified single-layer causal multi-head self-attention model.
result Proves a static finite-head approximation bound for the optimal risk.
Investigates the benefits of multi-head attention in Transformers, deriving convergence and generalization guarantees.
problem Underexplored dynamics of multi-head attention in Transformer training and generalization.
method Derives convergence and generalization guarantees for gradient-descent training of a multi-head self-attention model.
result Establishes conditions for initialization that ensure multi-head attention's realizability.
We study the head and tail of the colored Jones polynomial while focusing mainly on alternating links. Various ways to compute the colored Jones polynomial for a given link give rise to combinatorial identities for those power series. We further show that the head and tail functions only depend on the reduced checkerbo…
Proposes a new neural head for asymmetric representation learning.
problem Asymmetric representation learning in directed relations.
method Role-aware neural convex divergence head.
result Role-aware projections improve directional accuracy over plain ICNN-Bregman heads.
We develop a multi-task convolutional neural network (CNN) to classify multiple diagnoses from 12-lead electrocardiograms (ECGs) using a dataset comprised of over 40,000 ECGs, with labels derived from cardiologist clinical interpretations. Since many clinically important classes can occur in low frequencies, approaches…
New deep learning model generates accurate personalized human head models for electromagnetic dosimetry.
problem Challenges in generating accurate human head models for personalized electromagnetic dosimetry.
method Proposed ForkNet architecture for segmentation of whole human head structures using deep learning.
result Generated head models exhibit strong matching with manual segmentation results.
A new method for multi-expert learning-to-defer avoids optimization issues.
problem Optimization issues in multi-expert learning-to-defer systems.
method A decoupled surrogate with a softmax classifier head and independent sigmoid heads per expert.
result First multi-expert L2D guarantee with a constant not growing with the expert pool.
BOIL updates model body only, showing better few-shot learning performance.
problem Few-shot learning efficiency with model reuse vs. change.
method Proposes BOIL, updating only model body, freezing head.
result Significantly outperforms MAML on cross-domain tasks.
Investigates optimal parameter allocation in Transformers for efficiency and expressivity.
problem Balancing expressivity and efficiency in Transformer model parameters.
method Mathematical analysis and theoretical characterization of attention heads and head dimensions.
result Later layers can operate more efficiently with reduced parameters due to saturation of softmax activations.
We demystify attention patterns in multi-head softmax models for linear data.
problem Understanding the training dynamics and emergent patterns in multi-head softmax attention models.
method Extensive empirical experiments and rigorous theoretical analysis.
result Multi-head softmax attention models approximate a debiased gradient descent predictor, outperforming single-head attention and achieving near-Bayesian optimality.
New insights into attention mechanisms reveal dramatic trade-offs between rank and heads.
problem Dramatic trade-offs between rank and number of heads in attention mechanisms.
method Presented a simple target function and proved theoretical limits.
result Full-rank attention is necessary for long contexts, while low-rank is sufficient for short ones.
Improved robot navigation using multi-head attention for natural language instructions.
problem Improving robot navigation in unfamiliar environments.
method Proposes a multi-head attention mechanism blending layer in a neural network model.
result Significant performance gains in translating instructions for unseen environments.
Study shows the number of attention heads affects transformer performance.
problem Understanding how the number of attention heads impacts transformer performance.
method Introduced a generalized -retrieval task, established upper and lower bounds on parameter complexity, and validated with experiments.
result Transformers with many heads can efficiently approximate functions, while few heads require a large number of parameters.
New method removes contrastive loss by adding a prediction head, revealing learning mechanisms.
problem Understanding why neural networks learn competitive representations despite trivial optima.
method Empirical and theoretical analysis of a trainable, identity-initialized prediction head.
result The trainable prediction head enables learning all features, preventing dimensional collapse.
We show that the head and tail functions of the colored Jones polynomial of adequate links are the product of head and tail functions of the colored Jones polynomial of alternating links that can be read-off an adequate diagram of the link. We apply this to strengthen a theorem of Kalfagianni, Futer and Purcell on the …
Single-head attention approximates any function under various norms.
problem Universal approximation of functions using attention mechanisms.
method Interpreting attention as partitioning and summing linear transformations.
result Single-head attention can approximate any continuous function under -norm and Lebesgue integrable functions under -norm.
RNNs trained on head direction task mimic brain's compass and shifter neurons.
problem Modeling brain's head direction system using neural networks.
method Optimized recurrent neural networks trained on angular velocity integration.
result RNNs naturally emerge with compass and shifter neuron-like properties.
Study shows pre-trained models can handle long-tailed relations well, improving classifier performance.
problem Challenges in long-tailed relation classification due to class imbalance.
method Used instance-balanced sampling to pre-train models and then improved classifier performance through attentive relation routing.
result Robust classifier with attentive relation routing achieves better long-tailed classification ability.
In this paper we combine one method for hierarchical reinforcement learning - the options framework - with deep Q-networks (DQNs) through the use of different "option heads" on the policy network, and a supervisory network for choosing between the different options. We utilise our setup to investigate the effects of ar…
Deep learning predicts VR head movements for better 360-degree videos.
problem Predicting head movements in 360-degree VR videos.
method Deep learning network using position data and video content for long-term prediction.
result 16.1% improvement in prediction accuracy compared to baseline.
Study assesses CNN model robustness to noise in low-cost CT scans.
problem Evaluate CNN model performance on noisy, artifact-prone low-cost CT images.
method Developed and tested a CNN model for head CT triage, varying tube current and projections.
result Model remains robust to reduced tube current and fewer projections, maintaining AUROC close to original.