New technique reduces bias in DNN models without sensitive attribute annotations.
problem Existing bias mitigation methods require instance-level annotations and do not guarantee removal of all sensitive information.
method Representation Neutralization for Fairness (RNF) debiases only the classification head of DNN models using neutralized representations.
result RNF effectively reduces discrimination of DNN models with minimal performance degradation.
Headless attacks bypass classification heads to fool transfer learning models.
problem Adversarial attacks against transfer learning models without access to the classification head.
method Label-blind adversarial attacks that do not require class-label information.
result Transfer attack lowers ResNet18 accuracy on CIFAR10 by over 40%.
Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables in a spoken word. From the subsequences, it retrieves richer information than a single-head attention which only summarizes the whole sequen…
EERO optimizes resource usage for efficient classification.
problem Managing computational resources in complex machine learning models.
method EERO uses multiple classifiers with a reject option to adaptively shorten processing paths.
result EERO effectively manages budget allocation and enhances accuracy in complex scenarios.
Novel framework for reliable long-tailed classification.
problem Challenges of long-tailed imbalance and specific error risks.
method Bayesian Decision Theory and variational optimization.
result Demonstrates reliability and flexibility in diverse tasks.
Domain adaptation approaches seek to learn from a source domain and generalize it to an unseen target domain. At present, the state-of-the-art unsupervised domain adaptation approaches for subjective text classification problems leverage unlabeled target data along with labeled source data. In this paper, we propose a …
Optimizes neural networks by removing unnecessary layers, improving performance and speed.
problem Finding the optimal depth of neural networks to improve performance and speed.
method Develops a fast end-to-end method for training lightweight neural networks with multiple classifier heads, allowing the model to determine the importance of each head and choosing a single shallow classifier.
result Significantly reduces the number of parameters and accelerates inference, outperforming many standard pruning methods.
Acoustic Scene Classification (ASC) is a challenging task, as a single scene may involve multiple events that contain complex sound patterns. For example, a cooking scene may contain several sound sources including silverware clinking, chopping, frying, etc. What complicates ASC more is that classes of different activi…
Comparison of AutoML and DL shows AutoML outperforms simple DL for binary classification.
problem Choosing between AutoML and DL for binary classification tasks.
method Comparison of AutoML and DL on 6 public datasets, evaluation of a new AutoML tool.
result AutoML outperforms simple DL classifiers but integrating DL into AutoML improves performance further.
GWHD dataset offers 4,700 high-res images of wheat heads.
problem Challenges in wheat head detection from high-resolution imagery.
method Large, diverse dataset with detailed metadata.
result Benchmark for wheat head detection methods.
Multi-head attention outperforms single-head in in-context linear regression tasks.
problem Comparing performance of transformer with single-/multi-head attention in in-context learning.
method Theoretical analysis of performance of transformers with different attention mechanisms in linear regression tasks.
result Multi-head attention with a substantial embedding dimension outperforms single-head attention in in-context linear regression tasks.
New theory shows how multi-head attention reduces variance and decorrelates outputs.
problem Understanding and optimizing multi-head attention in neural networks.
method Developed a statistical theory linking multi-head attention to ensemble Nadaraya-Watson estimators.
result MHA variance reduction depends on head decorrelation, not just head count.
Geometric theory of projection heads in self-supervised learning.
problem Dimensional collapse and information invariance trade-off in projection heads.
method Geometric modeling of projection heads as Riemannian metrics, analyzing Hessian eigenvalues, and tracking optimization geometry.
result Smooth nonlinear heads induce negative curvature, preventing collapse; linear and ReLU heads cannot.
Improved language models with talking-heads attention.
problem Language model perplexity and quality issues.
method Added linear projections in multi-head attention.
result Better perplexities and quality in language tasks.
New approach improves multi-head attention by making heads less similar.
problem Multi-head attention can lead to similar features, reducing model expressiveness.
method Proposes a non-parametric approach using Bayesian techniques to make heads repel each other.
result Improves feature diversity, leading to better representations and performance.
Attention models can overfit without harming test performance.
problem Understanding benign overfitting in single-head attention models.
method Analyzing conditions for benign overfitting in a single-head softmax attention model.
result A single-head attention model can overfit without harming test performance under certain conditions.
Deep forecasting models show output heads significantly improve performance on fat-tailed financial returns.
problem Improving deep learning models for forecasting fat-tailed financial returns.
method Comparison of backbone architectures and output heads (point, Gaussian, Gaussian mixture) on S&P 500 monthly log-returns.
result Switching from point to Gaussian heads improves CRPS by about 1.3 percent, and from Gaussian to mixture adds another 2.4 percent.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are de…
One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
Being able to model correlations between labels is considered crucial in multi-label classification. Rule-based models enable to expose such dependencies, e.g., implications, subsumptions, or exclusions, in an interpretable and human-comprehensible manner. Albeit the number of possible label combinations increases expo…
A new neural network layer integrates graph learning into classification tasks.
problem Lack of relational information in standard deep learning architectures for label predictions.
method Derives backpropagation equations for a differentiable graph learning layer.
result Smooth label transitions, improved generalization, and robustness to adversarial attacks.
Survey of Turk's head knots and links properties.
problem Characterize Turk's head knots and links.
method Discussion of various results in the mathematical literature.
result Turk's head links are alternating, fibered, hyperbolic, invertible, non-split, periodic, and prime.
This work proposes a collaborative multi-head attention layer to reduce model size without sacrificing accuracy.
problem Over-parameterization in transformer models trained with large datasets.
method Proposes a collaborative multi-head attention layer that shares key/query projections.
result Reduction in model size by 4 for same accuracy and speed.
Multi-headed ensembles boost model performance with faster training.
problem Limited computational resources hinder ensemble search performance.
method Extend NES to multi-headed ensembles, leveraging end-to-end training and one-shot NAS methods.
result Multi-headed ensemble search finds robust ensembles 3 times faster with comparable performance.
Study on multi-head softmax attention dynamics for in-context learning.
problem Understanding and optimizing multi-head softmax attention models for multi-task linear regression.
method Gradient flow analysis and spectral mapping technique.
result Gradient flow converges to optimal multi-head softmax attention model, with task allocation emerging during training.
Transformer-MGK replaces redundant heads with Gaussian key mixtures, improving efficiency and performance.
problem Redundant attention heads in transformers degrade performance and efficiency.
method Transformer-MGK replaces redundant heads with a mixture of Gaussian keys.
result Transformer-MGK accelerates training and inference, reduces parameters and FLOPs, and achieves comparable or better accuracy.
Attention based Transformer architecture has enabled significant advances in the field of natural language processing. In addition to new pre-training techniques, recent improvements crucially rely on working with a relatively larger embedding dimension for tokens. Unfortunately, this leads to models that are prohibiti…
Minimalistic model captures head direction system properties.
problem Representing head direction system in a high-dimensional space.
method A minimalistic representation model of the rotation group U(1), including fully connected and convolutional versions.
result Emergence of Gaussian-like tuning profiles and 2D circle geometry in both model versions.
Develops a mean-field theory for multi-head self-attention under cross-entropy training.
problem Mean-field analysis of multi-head self-attention under cross-entropy training.
method Mean-field theory for a simplified single-layer causal multi-head self-attention model.
result Proves a static finite-head approximation bound for the optimal risk.
Exploiting dependencies between labels is considered to be crucial for multi-label classification. Rules are able to expose label dependencies such as implications, subsumptions or exclusions in a human-comprehensible and interpretable manner. However, the induction of rules with multiple labels in the head is particul…
STAD adapts models to evolving time-based data shifts.
problem Gradual distribution shifts over time challenge existing test-time adaptation methods.
method Bayesian filtering method that learns time-varying dynamics in hidden features.
result STAD excels in handling small batch sizes and label shift on real-world data.
Investigates the benefits of multi-head attention in Transformers, deriving convergence and generalization guarantees.
problem Underexplored dynamics of multi-head attention in Transformer training and generalization.
method Derives convergence and generalization guarantees for gradient-descent training of a multi-head self-attention model.
result Establishes conditions for initialization that ensure multi-head attention's realizability.
We study the head and tail of the colored Jones polynomial while focusing mainly on alternating links. Various ways to compute the colored Jones polynomial for a given link give rise to combinatorial identities for those power series. We further show that the head and tail functions only depend on the reduced checkerbo…
New framework removes harmful momentum effect for long-tailed classification.
problem Challenges in maintaining balanced datasets with long-tailed data.
method Causal inference framework to disentangle and remove harmful effects of momentum.
result Achieves state-of-the-art performance on long-tailed visual recognition benchmarks.
Proposes a new neural head for asymmetric representation learning.
problem Asymmetric representation learning in directed relations.
method Role-aware neural convex divergence head.
result Role-aware projections improve directional accuracy over plain ICNN-Bregman heads.
Method learns audio embeddings with contextualized tags.
problem Align audio and tags for cross-modal tasks.
method Audio autoencoder, word embeddings, multi-head self-attention, contrastive loss.
result Multi-head self-attention improves audio representations.
New insights into contrastive learning reveal how projectors affect downstream performance.
problem Understanding how projectors in contrastive learning impact downstream linear classification accuracy.
method Identified and modeled two effects: expansion and shrinkage induced by contrastive loss.
result Linear projectors operating in the shrinkage regime hinder downstream classification accuracy.
Investigates optimal parameter allocation in Transformers for efficiency and expressivity.
problem Balancing expressivity and efficiency in Transformer model parameters.
method Mathematical analysis and theoretical characterization of attention heads and head dimensions.
result Later layers can operate more efficiently with reduced parameters due to saturation of softmax activations.
We demystify attention patterns in multi-head softmax models for linear data.
problem Understanding the training dynamics and emergent patterns in multi-head softmax attention models.
method Extensive empirical experiments and rigorous theoretical analysis.
result Multi-head softmax attention models approximate a debiased gradient descent predictor, outperforming single-head attention and achieving near-Bayesian optimality.
New insights into attention mechanisms reveal dramatic trade-offs between rank and heads.
problem Dramatic trade-offs between rank and number of heads in attention mechanisms.
method Presented a simple target function and proved theoretical limits.
result Full-rank attention is necessary for long contexts, while low-rank is sufficient for short ones.
Improved robot navigation using multi-head attention for natural language instructions.
problem Improving robot navigation in unfamiliar environments.
method Proposes a multi-head attention mechanism blending layer in a neural network model.
result Significant performance gains in translating instructions for unseen environments.
Study shows the number of attention heads affects transformer performance.
problem Understanding how the number of attention heads impacts transformer performance.
method Introduced a generalized D-retrieval task, established upper and lower bounds on parameter complexity, and validated with experiments. result Transformers with many heads can efficiently approximate functions, while few heads require a large number of parameters.
We show that the head and tail functions of the colored Jones polynomial of adequate links are the product of head and tail functions of the colored Jones polynomial of alternating links that can be read-off an adequate diagram of the link. We apply this to strengthen a theorem of Kalfagianni, Futer and Purcell on the …
New method removes contrastive loss by adding a prediction head, revealing learning mechanisms.
problem Understanding why neural networks learn competitive representations despite trivial optima.
method Empirical and theoretical analysis of a trainable, identity-initialized prediction head.
result The trainable prediction head enables learning all features, preventing dimensional collapse.
As the Portable Document Format (PDF) file format increases in popularity, research in analysing its structure for text extraction and analysis is necessary. Detecting headings can be a crucial component of classifying and extracting meaningful data. This research involves training a supervised learning model to detect…
Single-head attention approximates any function under various norms.
problem Universal approximation of functions using attention mechanisms.
method Interpreting attention as partitioning and summing linear transformations.
result Single-head attention can approximate any continuous function under L∞-norm and Lebesgue integrable functions under Lp-norm. The development of personalized human head models from medical images has become an important topic in the electromagnetic dosimetry field, including the optimization of electrostimulation, safety assessments, etc. Human head models are commonly generated via the segmentation of magnetic resonance images into different…