Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptiv…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,786 papers · 148 categories
Trend · papers per month
48 results for “Soft attention”
Paper examines NMT robustness to nonsensical inputs.
problem NMT systems fail when source sentences are altered.
method Soft-attention technique to replace words in source sentences.
result Proposed technique achieves high success rate and outperforms existing methods.
End-to-end TTS framework uses hard alignment to improve accuracy.
problem End-to-end TTS systems struggle with accurate alignment between input text and output acoustic features.
method Proposes a constrained alignment scheme with hard monotonic alignments, marginalized during training.
result Improves alignment learning and prediction in end-to-end TTS systems.
Hierarchical Memory Networksstat.ML
Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which …
Inspired by recent work in attention models for image captioning and question answering, we present a soft attention model for the reinforcement learning domain. This model uses a soft, top-down attention mechanism to create a bottleneck in the agent, forcing it to focus on task-relevant information by sequentially que…
ConViT combines CNN and ViT strengths, improving image classification.
problem Combining the strengths of CNNs and ViTs while avoiding their limitations.
method Introducing GPSA, a form of positional self-attention with a soft convolutional inductive bias.
result ConViT outperforms DeiT on ImageNet while offering improved sample efficiency.
Improved text generation with discourse-aware soft prompts.
problem Efficient fine-tuning methods don't generalize across all generation tasks.
method Hierarchical blocking and attention sparsity on prefix parameters.
result Structured design of prefix parameters yields more coherent generations.
Employers actively look for talents having not only specific hard skills but also various soft skills. To analyze the soft skill demands on the job market, it is important to be able to detect soft skill phrases from job advertisements automatically. However, a naive matching of soft skill phrases can lead to false pos…
Jointly Modeling Heterogeneous Student Behaviors and Interactions Among Multiple Prediction Taskscs.LG
Modeling student behaviors and multiple predictions for early intervention.
problem Predicting student outcomes and interactions among multiple tasks.
method Proposes a variant of LSTM and soft-attention mechanism for heterogeneous behaviors, and co-attention mechanism for task interactions.
result Demonstrated effectiveness in predicting student outcomes and interactions.
In this paper we take a problem of unsupervised nodes clustering on graphs and show how recent advances in attention models can be applied successfully in a "hard" regime of the problem. We propose an unsupervised algorithm that encodes Bethe Hessian embeddings by optimizing soft modularity loss and argue that our mode…
Deep learning detects Crohn's disease in MRI images.
problem Detecting Crohn's disease in MRI images.
method Residual Networks for feature extraction and soft attention mechanisms for interpretability.
result Deep learning algorithms perform comparably to clinical standards with faster inference.
Self-adaptive PINNs improve accuracy in stiff PDEs.
problem Accurate numerical solution of stiff PDEs using neural networks.
method Adaptive weights applied to each training point individually, using a soft attention mechanism.
result SA-PINNs outperform other PINN algorithms in L2 error with fewer training epochs.
Framework tackles OOD challenges in molecule property prediction by modeling environments.
problem Challenges in modeling OOD samples for molecule property prediction.
method Soft causal learning framework incorporating chemistry theories and cross-attention mechanisms.
result Demonstrates well generalization ability on seven datasets.
Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property …
A new SVM classifier using soft-margin loss for improved performance.
problem Improving SVM performance in binary classification tasks.
method Introducing soft-margin loss and using the alternating direction method of multipliers.
result The new -SVM model generates better performance with shorter computational time and fewer support vectors.
There has recently been significant interest in hard attention models for tasks such as object recognition, visual captioning and speech recognition. Hard attention can offer benefits over soft attention such as decreased computational cost, but training hard attention models can be difficult because of the discrete la…
Paper estimates optimal classification error with soft labels and calibration.
problem Estimating the optimal classification error with soft labels and calibration.
method Extends previous work on soft labels to estimate Bayes error, addressing bias and corrupted labels.
result The method provides a statistically consistent estimator of the Bayes error, even with imperfectly calibrated soft labels.
Attention operators have been widely applied in various fields, including computer vision, natural language processing, and network embedding learning. Attention operators on graph data enables learnable weights when aggregating information from neighboring nodes. However, graph attention operators (GAOs) consume exces…
Why Bonds Fail Differently? Explainable Multimodal Learning for Multi-Class Default Predictionq-fin.RM
EMDLOT predicts bond defaults better than traditional methods.
problem Lack of interpretability and irregular temporal dependencies in financial data.
method Integrates time-series and textual data, uses Time-Aware LSTM, soft clustering, and multi-level attention.
result EMDLOT outperforms traditional and deep learning benchmarks in recall, F1-score, and mAP.
Deep metric learning aims to learn a deep embedding that can capture the semantic similarity of data points. Given the availability of massive training samples, deep metric learning is known to suffer from slow convergence due to a large fraction of trivial samples. Therefore, most existing methods generally resort to …
SoftBart improves BART for high-noise modeling in science.
problem High noise in scientific data.
method Soft BART algorithm for Bayesian additive regression trees.
result Improves predictive performance and facilitates larger model integration.
Study on how attention in prompt-tuning affects large language models.
problem Limited theoretical understanding of prompt-tuning and attention in LLMs.
method Exploration of prompt-tuning for one-layer attention architectures, contextual mixture-models, and self-contained prompt-attention model.
result Softmax-prompt-attention is more expressive than self-attention and linear-prompt-attention under contextual data model.
LatFormer improves geometric reasoning by incorporating lattice symmetry priors in attention mechanisms.
problem State-of-the-art models struggle with geometric reasoning tasks in the ARC and LARC datasets.
method Introduced LatFormer, a model that uses lattice symmetry priors in attention masks.
result LatFormer requires 2 orders of magnitude fewer data than standard attention mechanisms.
Hybrid approach combines transformer and Bayesian filtering for robust multiple particle tracking.
problem Challenges in tracking multiple particles in noisy scenes due to combinatorial explosion of hypotheses.
method Attention-Bayesian hybrid framework using transformer for association and Bayesian filtering for pruning hypotheses.
result Improved tracking accuracy and robustness against spurious detections.
Deep learning diagnoses MS from smartphone data.
problem Diagnosing MS with complex clinical assessments and tests.
method Deep-learning approach using smartphone-derived digital biomarkers.
result Deep-learning models distinguish MS with 88% accuracy.
Single-head transformers with a single self-attention layer can approximate any sequence-to-sequence function and are efficient under certain conditions.
problem Statistical and computational limits of prompt tuning for transformer-based models.
method Investigation of single-head transformers with a single self-attention layer, proving universality and efficiency under SETH.
result Existence of almost-linear time prompt tuning inference algorithms under certain conditions.
Neural attention (NA) has become a key component of sequence-to-sequence models that yield state-of-the-art performance in as hard tasks as abstractive document summarization (ADS) and video captioning (VC). NA mechanisms perform inference of context vectors; these constitute weighted sums of deterministic input sequen…
Soft labeling impacts OOD detection in neural networks.
problem Impact of soft labeling on OOD detection in deep neural networks.
method Empirical analysis of how soft labeling affects OOD detection performance.
result Soft labeling can either improve or deteriorate OOD detection performance.
Developed a new thresholding method that connects soft and hard thresholding.
problem Connecting soft and hard thresholding methods in data analysis.
method Scaled soft thresholding method with empirical scaling values.
result Found two sources of over-fitting in the scaled soft thresholding method.
Soft cells fill space without gaps, derived from minimal surfaces and deformed using edge bending.
problem Creating space-filling shapes without sharp corners.
method Edge bending algorithm to deform polyhedral tilings into soft tilings.
result Soft tilings derived from minimal surfaces can be continuously transformed into one another.
Deep neural networks excel in regimes with large amounts of data, but tend to struggle when data is scarce or when they need to adapt quickly to changes in the task. In response, recent work in meta-learning proposes training a meta-learner on a distribution of similar tasks, in the hopes of generalization to novel but…
New analysis shows how cross-entropy training shapes attention in transformers.
problem Understanding how gradient-based learning creates the required internal geometry in transformers.
method Developed a first-order analysis of cross-entropy training effects on attention scores and values in a transformer attention head.
result Introduced an advantage-based routing law and responsibility-weighted update for attention scores and values, respectively.
GAttNHP predicts future events in temporal knowledge graphs by encoding long-range dependencies and handling mutual excitation.
problem Forecasting future events in temporal knowledge graphs due to long-range dependencies, mutual excitation, and heavy-tailed inter-arrival times.
method GAttNHP uses a self-attention encoder, semantic soft-grouping, and NCQ regression to address these issues.
result GAttNHP improves entity and time prediction on six benchmark TKG datasets compared to state-of-the-art baselines.
A new method for RL with continuous actions improves stability and scalability.
problem Stability and scalability issues in existing RL methods.
method Soft policy gradient with entropy regularization, combined with double sampling for soft Bellman equation.
result Outperforms off-policy prior methods in continuous action RL tasks.
ASBART accelerates Soft BART for faster Bayesian regression.
problem Slow computation in Soft BART.
method Proposed ASBART, a variant of Soft BART.
result ASBART is about 10 times faster than Soft BART with similar accuracy.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
Transformers can emulate various algorithms by prompting, proving universality.
problem How to emulate algorithms using fixed-weight Transformers.
method Two modes of in-context algorithm emulation: task-specific and prompt-programmable. Constructing prompts that encode algorithm parameters into token representations.
result Fixed-weight Transformers can emulate a broad class of algorithms via prompts.
Effective understanding of the environment and accurate trajectory prediction of surrounding dynamic obstacles are critical for intelligent systems such as autonomous vehicles and wheeled mobile robotics navigating in complex scenarios to achieve safe and high-quality decision making, motion planning and control. Due t…
Deep learning models for survival analysis have gained significant attention in the literature, but they suffer from severe performance deficits when the dataset contains many irrelevant features. We give empirical evidence for this problem in real-world medical settings using the state-of-the-art model DeepHit. Furthe…
Generalizes soft noncommutative schemes to flag varieties.
problem Applying soft noncommutative schemes to flag varieties.
method Generalization via toric geometry and distinguished affine charts.
result Soft noncommutative schemes can be applied to flag varieties.
New Soft Actor-Critic for discrete actions.
problem Applying reinforcement learning to games with discrete actions.
method Derived an alternative Soft Actor-Critic for discrete actions.
result Competitive with tuned model-free state-of-the-art on Atari games.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
Introduces Soft-SVM for binary classification bridging logistic and SVM.
problem Data separability issues in binary classification.
method Soft-SVM regression using convex relaxation of hinge loss with softness and class-separation parameters.
result Soft-SVM performs well in classification and prediction errors.
A deep latent variable model for semi-supervised multi-unit soft sensing in industrial processesstat.ML
Deep model learns from labeled and unlabeled data for industrial soft sensing.
problem Lack of data limits soft sensor development in industrial processes.
method Hierarchical, generative deep latent variable model for semi-supervised multi-unit soft sensing.
result Model outperforms current methods for soft sensing in industrial processes.
New method constructs Floer homologies without hard analysis.
problem Constructing homological invariants in infinite dimensions.
method Soft, finite-dimensional tools to avoid hard analysis.
result Demonstrates construction of Floer homologies without PDE.
This paper introduces TNTK to study infinite soft tree ensembles.
problem Understanding the behavior of infinite soft tree ensembles.
method Introduced Tree Neural Tangent Kernel (TNTK) to analyze infinite soft tree ensembles.
result Identified several non-trivial properties of infinite soft tree ensembles.
Improves deep neural networks using soft labels through alternating minimization.
problem Improving deep neural networks training with soft labels.
method Co-Learns DNNs and soft labels via Alternating Minimization of two objectives.
result COLAM achieves improved performance on many tasks with better testing classification accuracy.
Study improves fraud detection in e-commerce with a stacked model combining CNNs, GNNs, and confidence gating.
problem Detecting credit card fraud in online transactions.
method Stacking approach with attention and confidence-driven layers, using DOWA and IOWA operators.
result The method achieves high accuracy and robust generalization in CCF detection.