Neural processes (NPs) learn stochastic processes and predict the distribution of target output adaptively conditioned on a context set of observed input-output pairs. Furthermore, Attentive Neural Process (ANP) improved the prediction accuracy of NPs by incorporating attention mechanism among contexts and targets. In …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Attentive Neural Processes”
Researchers analyze neural process architectures and their representational capacities.
problem Understanding what functions can be represented by different neural process architectures.
method Analyzing four types of neural process architectures: CNPs, ANPs, TNPs, and their latent variants.
result Prove these architectures form a strict hierarchy and characterize their representational capabilities.
New attention mechanism improves meta-transfer learning in dynamic tasks.
problem Underfitting in meta-transfer learning with dynamic tasks.
method Proposed Recurrent Memory Reconstruction (RMR) attention mechanism.
result ASNP-RMR significantly outperforms baselines in various tasks.
TNP-KR improves scalability of NPs with Transformer blocks and attention mechanisms.
problem Scalability bottleneck in NPs and GPs, especially for large datasets.
method Introduces TNP-KR with KRBlock, kernel-based attention, and two attention mechanisms.
result TNP-KR with DKA outperforms Performer and achieves state-of-the-art results.
BSA-TNP improves NP scalability and accuracy for spatiotemporal data.
problem Scalability and accuracy trade-off in Neural Processes.
method Introduces KRBlocks, group-invariant attention biases, and BSA for scalable spatiotemporal inference.
result BSA-TNP matches or exceeds accuracy of best models while training faster.
Attention mechanisms in deep learning become Gaussian process-like as the number of heads increases.
problem Understanding the behavior of attention mechanisms in deep learning models.
method Extending the equivalence between wide neural networks and Gaussian processes to attention architectures.
result Multi-head attention architectures behave as Gaussian processes as the number of heads tends to infinity.
Knowledge-Driven Modulation of Neural Networks with Attention Mechanism for Next Activity Predictioncs.AI
The paper improves neural network predictions by integrating process knowledge.
problem Improving neural network predictions for process execution data.
method Integrates background process knowledge into neural networks with attention mechanisms.
result Improves prediction accuracy for process execution data.
IAL uses interactive learning to improve model performance with minimal human feedback.
problem Overfitting and high human interaction cost in training neural networks.
method IAL framework with NAP attention generator and reranking algorithm.
result IAL significantly outperforms baselines with less retraining and human interaction.
Models use embeddings and attention for better claim severity prediction.
problem Improving predictive modeling of claim severity with categorical data.
method Developed neural networks and attention-based architectures with embeddings.
result Attention-based models enhance predictive performance with contextual augmentation.
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
Model traffic congestion events using multi-modal data and attention-based neural networks.
problem Capture non-homogeneous temporal and directional spatial dependencies in traffic congestion events.
method Attention-based neural networks for point processes, adapted tail-up model for spatial statistics.
result Superior performance compared to state-of-the-art methods on synthetic and real data.
Attentive Neural Controlled Differential Equations for Time-series Classification and Forecastingcs.LG
ANCDEs improve time-series forecasting and classification using attention in NCDEs.
problem Improving time-series forecasting and classification using neural controlled differential equations.
method Integrating attention into neural controlled differential equations (ANCDEs).
result ANCDEs consistently show the best accuracy in time-series classification and forecasting.
Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property …
Mapping of attention mechanisms to a generalized Potts modelcond-mat.dis-nn
Transformers' self-attention mechanism is mapped to a generalized Potts model.
problem Uncertainty in what type of data distribution self-attention can efficiently learn.
method Decouple word positions and embeddings, then show self-attention learns a generalized Potts model.
result Training self-attention is equivalent to solving the inverse Potts problem.
Unlike images or videos data which can be easily labeled by human being, sensor data annotation is a time-consuming process. However, traditional methods of human activity recognition require a large amount of such strictly labeled data for training classifiers. In this paper, we present an attention-based convolutiona…
Study improves forecasting in betting markets using novel neural networks.
problem Improving short-term price movement predictions in betting exchanges.
method Innovative convolutional attention mechanisms applied to recurrent neural networks and bi-dimensional layers.
result All proposed innovations positively impact classification task performance.
New approach improves multi-head attention by making heads less similar.
problem Multi-head attention can lead to similar features, reducing model expressiveness.
method Proposes a non-parametric approach using Bayesian techniques to make heads repel each other.
result Improves feature diversity, leading to better representations and performance.
Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layerscs.LG
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimiz…
AttentionDDI: Siamese Attention-based Deep Learning method for drug-drug interaction predictionsq-bio.QM
A neural network predicts drug interactions using attention mechanisms.
problem Predicting drug-drug interactions from massive combinations of drugs.
method Siamese self-attention multi-modal neural network integrating drug characteristics.
result The model achieves AUPR scores ranging from 0.77 to 0.92 on various benchmark datasets.
Study reveals attention mechanism's similarity computation parallels traditional machine learning.
problem Understanding the essence and principles of attention mechanism in deep learning.
method Examined classic metrics and vector space properties in manifold learning, clustering, and supervised learning to identify key characteristics of similarity computation and information propagation.
result Self-attention mechanism in deep learning adheres to the same principles but operates more flexibly and adaptively.
Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for att…
We present a novel attention-based model for discrete event data to capture complex non-linear temporal dependence structures. We borrow the idea from the attention mechanism and incorporate it into the point processes' conditional intensity function. We further introduce a novel score function using Fourier kernel emb…
Voice and accompaniment separation in music using self-attention convolutional neural networkeess.AS
A new neural network separates vocals from music accompaniment.
problem Separating vocals from music accompaniment in recordings.
method Self-attention convolutional neural network (CNN) with densely-connected blocks.
result 19.5% relative improvement in vocals separation.
A new model uses attention and Gaussian processes for efficient time-series generation.
problem Computational inefficiency and uncertainty underestimation in sequence transduction.
method Attention-based Gaussian process network for real-valued sequence generation.
result The model improves training efficiency and learns factorized generative distribution.
Proposes GPCA module for channel attention in CNNs using Gaussian processes.
problem Improving performance in visual tasks through effective channel selection.
method Integrates Gaussian processes into channel attention mechanisms for probabilistic modeling of channel correlations.
result Demonstrates improved performance of GPCA module in end-to-end CNN training.
GAttNHP predicts future events in temporal knowledge graphs by encoding long-range dependencies and handling mutual excitation.
problem Forecasting future events in temporal knowledge graphs due to long-range dependencies, mutual excitation, and heavy-tailed inter-arrival times.
method GAttNHP uses a self-attention encoder, semantic soft-grouping, and NCQ regression to address these issues.
result GAttNHP improves entity and time prediction on six benchmark TKG datasets compared to state-of-the-art baselines.
Researchers develop a new spatial process model for non-Gaussian data.
problem Non-Gaussian spatial data with asymmetry and heavy-tailedness.
method Re-parameterized Unified Skew-Normal (SUN) distribution, GSUN process, neural Bayes inference with GATs.
result GSUN process captures non-Gaussian spatial data properties and outperforms conventional models.
Improved weather forecasting with gridded pseudo-token TNPs.
problem Handling large-scale, unstructured spatio-temporal data in weather forecasting.
method Introducing gridded pseudo-token transformer neural processes (TNPs) with efficient attention mechanisms.
result Consistently outperforms baselines on various synthetic and real-world regression tasks involving large-scale data.
Pyramid Attention Networks improve image restoration by leveraging self-similarities across scales.
problem Lack of full exploitation of self-similarities in image restoration by recent deep learning methods.
method Introduces a Pyramid Attention module that processes multi-scale feature correspondences to borrow clean signals from coarser levels.
result Pyramid Attention module achieves state-of-the-art results in various image restoration tasks.
Proposes a novel node embedding framework for graphs using Fisher Information.
problem Lack of theoretical understanding of attention-based GNNs.
method Uses hierarchical kernels and Fisher Information to learn node embeddings.
result Proposed method outperforms existing GNNs on node classification benchmarks.
This paper offers an overview of neural network compression techniques.
problem Overparameterized neural networks are large and resource-intensive.
method Pruning, quantization, tensor decomposition, knowledge distillation.
result A comprehensive review of compression techniques for deep neural networks.
Quaternion self-attention reduces computational cost and improves performance.
problem Existing quaternion self-attention increases computational cost and diverges attention distributions.
method Proposes a shared-score quaternion self-attention mechanism.
result Reduces score-computation multiplications by 75% and softmax operations from four to one.
Tensor-based Sequential Learning via Hankel Matrix Representation for Next Item Recommendationscs.LG
New model mimics neural next item recommendation using Hankel matrices.
problem Next item recommendation efficiency and structural knowledge capture.
method Tensor factorization with Hankel matrix representation.
result Model performs competitively with neural networks but is simpler.
AdaCAD improves semi-supervised classification by focusing on intra-class nodes.
problem Improving semi-supervised classification by addressing inter-class connections in graphs.
method AdaCAD uses a class-attentive diffusion process to adaptively aggregate nodes based on their class similarity.
result AdaCAD significantly outperforms state-of-the-art methods in semi-supervised classification.
Neural Diffusion Processesstat.ML
NDPs learn to sample from complex function distributions using neural networks and diffusion models.
problem Learning rich distributions over functions with neural networks.
method NDPs use denoising diffusion models and custom attention blocks to incorporate stochastic process properties.
result NDPs can capture functional distributions close to true Bayesian posteriors and outperform neural processes.
Improved robot navigation using multi-head attention for natural language instructions.
problem Improving robot navigation in unfamiliar environments.
method Proposes a multi-head attention mechanism blending layer in a neural network model.
result Significant performance gains in translating instructions for unseen environments.
Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attenti…
Enhances SNNs for spatio-temporal feature extraction.
problem Insufficient temporal dependencies in existing SNN synaptic structures.
method Integrates temporal convolution and attention mechanisms into synaptic connections.
result Improves SNN performance on classification tasks.
Neural operators approximate Stackelberg game solutions.
problem Intractability of follower's best-response operator in dynamic Stackelberg games.
method Used attention-based neural operators to approximate the best-response operator.
result Approximate best-response operator yields close game value.
Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcs.LG
Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Transformer is the attention mechanism, which concurrently processes all inputs in the streams. In this …
MQTransformer improves forecast accuracy with context-aware attention.
problem Improving probabilistic demand prediction accuracy.
method Incorporates Transformer architectures for context alignment and feedback-aware attention.
result Significant improvements in forecast accuracy, reducing excess variability.
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
Study improves posterior inference in neural processes with limited data.
problem Improving posterior predictive inference in probabilistic models with scarce conditioning data.
method Examined effects of pooling operators and variational families on posterior quality in neural processes.
result Novel neural process architectures lead to superior posterior predictive samples in image completion/in-painting tasks.
Neural networks are surprisingly good at interpolating and perform remarkably well when the training set examples resemble those in the test set. However, they are often unable to extrapolate patterns beyond the seen data, even when the abstractions required for such patterns are simple. In this paper, we first review …
ISAHP discovers instance-level causal structures in event sequences.
problem Discovering fine-grained causal relationships in asynchronous, interdependent event sequences.
method ISAHP, a novel deep learning framework using self-attention mechanism.
result ISAHP meets Granger causality requirements and discovers complex causal structures.
Bayesian Attention Modulesstat.ML
Bayesian attention modules improve model interpretability and performance.
problem Deterministic attention modules limit model interpretability and optimization.
method Proposes a scalable stochastic attention module using simplex-constrained distributions and Bayesian learning.
result Consistent improvements over baselines in various attention-based models.
Paper analyzes infinite-width attention layers using Tensor Programs.
problem Capturing the infinite-width limit of attention layers.
method Tensor Programs framework to rigorously identify the limit distribution.
result Derives exact form of infinite-width limit distribution without Gaussian approximations.
Perceptual capabilities of artificial systems have come a long way since the advent of deep learning. These methods have proven to be effective, however they are not as efficient as their biological counterparts. Visual attention is a set of mechanisms that are employed in biological visual systems to ease computationa…