Deep neural network models have recently draw lots of attention, as it consistently produce impressive results in many computer vision tasks such as image classification, object detection, etc. However, interpreting such model and show the reason why it performs quite well becomes a challenging question. In this paper,…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “attention-based models”
Attention-based models have recently shown great performance on a range of tasks, such as speech recognition, machine translation, and image captioning due to their ability to summarize relevant information that expands through the entire length of an input sequence. In this paper, we analyze the usage of attention mec…
AGBoost uses attention weights to improve GBM for regression problems.
problem Improving gradient boosting machine for regression tasks.
method Attention-based modification of GBM with trainable attention weights.
result AGBoost achieves better performance on regression datasets.
Sequence-to-sequence models, such as attention-based models in automatic speech recognition (ASR), are typically trained to optimize the cross-entropy criterion which corresponds to improving the log-likelihood of the data. However, system performance is usually measured in terms of word error rate (WER), not log-likel…
Mathematical study shows post-hoc explanations are better than attention weights alone.
problem Understanding the internal behavior of attention-based models.
method Mathematical analysis of a simple attention-based architecture.
result Post-hoc explanations provide more useful insights than attention weights alone.
Graph Neural Networks (GNNs) are powerful to learn the representation of graph-structured data. Most of the GNNs use the message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information of its neighbors. To achieve a better expressive capability of node influences, attention m…
LARF improves random forests with attention mechanisms and contamination models.
problem Improving accuracy in classification tasks with random forests.
method Introduces a two-level attention mechanism and uses a mixture of contamination models.
result Significantly improved classification performance on various datasets.
Hybrid model predicts flow and pressure in water systems.
problem Predicting flow and pressure in water distribution systems with complex spatial-temporal correlations.
method Hybrid dual-stage spatial-temporal attention-based recurrent neural networks (hDS-RNN).
result Our model outperformed 9 baseline models in flow and pressure series prediction.
Proposes a novel node embedding framework for graphs using Fisher Information.
problem Lack of theoretical understanding of attention-based GNNs.
method Uses hierarchical kernels and Fisher Information to learn node embeddings.
result Proposed method outperforms existing GNNs on node classification benchmarks.
Paper proposes ADC framework to reduce ViT SL training communication overhead.
problem Reducing communication overhead in ViT SL training.
method Two parallel compression strategies: class-agnostic merging and token discarding.
result Significantly reduces communication overhead without sacrificing accuracy.
Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.
problem The issue of oversmoothing in attention-based GNNs.
method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.
result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.
The recent proliferation of knowledge graphs (KGs) coupled with incomplete or partial information, in the form of missing relations (links) between entities, has fueled a lot of research on knowledge base completion (also known as relation prediction). Several recent works suggest that convolutional neural network (CNN…
Models use embeddings and attention for better claim severity prediction.
problem Improving predictive modeling of claim severity with categorical data.
method Developed neural networks and attention-based architectures with embeddings.
result Attention-based models enhance predictive performance with contextual augmentation.
New methods improve integration of external LMs with AED models.
problem Improving performance of AED models by integrating external LMs.
method Comparing and proposing novel methods to estimate implicit LM from AED models.
result Proposed methods outperform previous approaches.
Improves key instance detection in MIL models by using neural network inversion with sparseness constraint.
problem Limited key instance detection performance in attention-based deep MIL models due to skewed attention scores.
method Sparse network inversion with a sparseness constraint incorporated into neural network inversion, solved by proximal gradient method.
result Significantly improved key instance detection performance while maintaining bag-level prediction performance.
ABC learns context-aware representations for clustering.
problem Learning latent representations that adapt to context in machine learning.
method Attention-based neural architecture that learns a similarity kernel.
result Competitive results for clustering Omniglot characters.
Text segmentation plays an important role in various Natural Language Processing (NLP) tasks like summarization, context understanding, document indexing and document noise removal. Previous methods for this task require manual feature engineering, huge memory requirements and large execution times. To the best of our …
This paper introduces new attention-based convolutional neural networks for selecting bands from hyperspectral images. The proposed approach re-uses convolutional activations at different depths, identifying the most informative regions of the spectrum with the help of gating mechanisms. Our attention techniques are mo…
Proposes new methods for interpreting document classification models.
problem Interpretation fragility of attention-based neural networks.
method Corpus-level and concept-based explanation methods using attention weights.
result Extracts semantically meaningful keywords and concepts for model predictions.
This paper presents our system details and results of participation in the RDoC Tasks of BioNLP-OST 2019. Research Domain Criteria (RDoC) construct is a multi-dimensional and broad framework to describe mental health disorders by combining knowledge from genomics to behaviour. Non-availability of RDoC labelled dataset …
Study compares deep learning models for traffic forecasting, highlighting graph elements' impact.
problem Challenges in forecasting spatial-temporal traffic patterns.
method In-depth comparative study of four deep neural network models with different basic elements.
result Graph attention improves long-term predictions in traffic forecasting models.
Study improves stock price prediction using advanced ML models.
problem Improving financial forecasting accuracy in stock markets.
method Evaluation of RNN architectures including LSTM, GRU, and attention-based models.
result Attention-based models outperform others in capturing complex dependencies.
GOBO compresses 99.9% of BERT model parameters to 3 bits, improving inference efficiency.
problem Efficient execution of attention-based NLP models, especially in terms of latency and energy consumption.
method GOBO quantizes 32-bit floating-point parameters to 3 bits without fine-tuning, using hardware compression and co-designed architectures.
result GOBO maintains model accuracy while significantly reducing inference latency and energy consumption.
Objective: To compare different deep learning architectures for predicting the risk of readmission within 30 days of discharge from the intensive care unit (ICU). The interpretability of attention-based models is leveraged to describe patients-at-risk. Methods: Several deep learning architectures making use of attentio…
Improved hyperparameter optimization using simplified Transformer blocks.
problem Discovering optimal architectures in high-dimensional search spaces with limited exploration budgets.
method Simplified Transformer block for modeling hyper-parameter dependencies, actor-critic style algorithm, ensembling.
result Outperformed most algorithms on NAS-Bench-101 and Random Search in discovering more accurate model architectures.
In this paper, we propose a refined multi-stage multi-task training strategy to improve the performance of online attention-based encoder-decoder (AED) models. A three-stage training based on three levels of architectural granularity namely, character encoder, byte pair encoding (BPE) based encoder, and attention decod…
ResGCN detects anomalies in attributed networks by capturing sparsity and nonlinearity.
problem Detecting anomalous nodes in attributed networks.
method Attention-based deep residual modeling using Graph Convolutional Networks.
result ResGCN effectively detects anomalies in attributed networks.
Out-of-vocabulary word translation is a major problem for the translation of low-resource languages that suffer from a lack of parallel training data. This paper evaluates the contributions of target-language context models towards the translation of OOV words, specifically in those cases where OOV translations are der…
In line with recent advances in neural drug design and sensitivity prediction, we propose a novel architecture for interpretable prediction of anticancer compound sensitivity using a multimodal attention-based convolutional encoder. Our model is based on the three key pillars of drug sensitivity: compounds' structure i…
Sequence-to-sequence text-to-speech (TTS) is dominated by soft-attention-based methods. Recently, hard-attention-based methods have been proposed to prevent fatal alignment errors, but their sampling method of discrete alignment is poorly investigated. This research investigates various combinations of sampling methods…
Network embedding represents nodes in a continuous vector space and preserves structure information from the Network. Existing methods usually adopt a "one-size-fits-all" approach when concerning multi-scale structure information, such as first- and second-order proximity of nodes, ignoring the fact that different scal…
Streamable model improves speech recognition performance.
problem Improving speech recognition in long-form audio.
method Chunk-based attention model for streaming speech recognition.
result Streamable model maintains competitive performance and generalizes well to long-form speech.
Hybrid model predicts stock prices with high accuracy.
problem Complex volatility of stock market makes traditional models unsatisfactory.
method Attention-based CNN-LSTM and XGBoost integrated model.
result Hybrid model improves prediction accuracy.
Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attenti…
Unlike images or videos data which can be easily labeled by human being, sensor data annotation is a time-consuming process. However, traditional methods of human activity recognition require a large amount of such strictly labeled data for training classifiers. In this paper, we present an attention-based convolutiona…
Model traffic congestion events using multi-modal data and attention-based neural networks.
problem Capture non-homogeneous temporal and directional spatial dependencies in traffic congestion events.
method Attention-based neural networks for point processes, adapted tail-up model for spatial statistics.
result Superior performance compared to state-of-the-art methods on synthetic and real data.
Determining whether hypotensive patients in intensive care units (ICUs) should receive fluid bolus therapy (FBT) has been an extremely challenging task for intensive care physicians as the corresponding increase in blood pressure has been hard to predict. Our study utilized regression models and attention-based recurre…
Simple attention model outperforms complex sEMG classifiers.
problem Improving myoelectric control for robotic prosthetics.
method Attention-based model for sEMG signal classification.
result Simple model achieves benchmark results on multiple datasets.
FedACS uses attention to select clients with similar data for federated learning.
problem Non-IID data and data scarcity in federated learning.
method FedACS integrates an attention mechanism to prioritize clients with similar data distributions.
result FedACS improves federated learning performance by addressing non-IID data and data scarcity.
A new deep learning framework improves HAR with user adaptation.
problem Sensor-based human activity recognition with long-term dependencies.
method Attention-based deep learning framework with user adaptation.
result Average increment of more than 7% on F1 score over state-of-the-art.
Neural machine translation (NMT) systems have been shown to give undesirable translation when a small change is made in the source sentence. In this paper, we study the behaviour of NMT systems when multiple changes are made to the source sentence. In particular, we ask the following question "Is it possible for an NMT…
ABRF uses attention weights to improve RF performance.
problem Improving the performance of random forest models.
method Attention mechanism applied to random forest with optimization and gradient-based methods.
result The proposed ABRF models outperform standard RF models on various datasets.
Attention-based clusteringstat.ML
Transformers can cluster data from Gaussian mixtures without supervision.
problem Clustering data from Gaussian mixtures without labeled data.
method Theoretical analysis of attention-based layers, focusing on a simplified two-head attention layer and an identity matrix attention layer.
result Attention-based layers can align with true mixture centroids and adapt to input-specific distributions.
Time series prediction with deep learning methods, especially long short-term memory neural networks (LSTMs), have scored significant achievements in recent years. Despite the fact that the LSTMs can help to capture long-term dependencies, its ability to pay different degree of attention on sub-window feature within mu…
FIGARO generates symbolic music with fine-grained control.
problem Minimal control over generated music sequences.
method Description-to-sequence task, learning conditional distribution of sequences given high-level descriptions.
result State-of-the-art controllable symbolic music generation.
Graph representation learning for hypergraphs can be used to extract patterns among higher-order interactions that are critically important in many real world problems. Current approaches designed for hypergraphs, however, are unable to handle different types of hypergraphs and are typically not generic for various lea…
Paper proposes deep learning model for dynamic stock repurchase forecasting.
problem Complex temporal dependencies in corporate financial conditions.
method Hybrid Temporal Convolutional Network (TCN) and Attention-based LSTM.
result Model significantly outperforms static baselines in stock repurchase forecasting.
We present a voice conversion solution using recurrent sequence to sequence modeling for DNNs. Our solution takes advantage of recent advances in attention based modeling in the fields of Neural Machine Translation (NMT), Text-to-Speech (TTS) and Automatic Speech Recognition (ASR). The problem consists of converting be…