Paper explores a new model for non-Markovian control learning.
problem Sequential control learning with delayed rewards in partially observable environments.
method Gated End-to-End Memory Policy Network (GEMPN) using value-based algorithms.
result Encouraging results show the model's capability in non-Markovian decision processes.
Gated attention improves performance by using a hierarchical mixture of experts.
problem Improving performance of self-attention mechanisms in Transformers.
method Rigorously show that gated attention can be modeled as a hierarchical mixture of experts, providing a theoretical justification for its benefits.
result Gated attention is more sample-efficient than multi-head self-attention, requiring fewer data points to achieve the same estimation error.
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
Proposes a new method for deep semantic segmentation learning without needing a pre-trained model.
problem Deep convolutional semantic segmentation learning fails to converge without a pre-trained model.
method Joint cooperative end-to-end learning with improved inception-based architecture and partial attention gating.
result Achieves faster convergence and better accuracy for segmentation tasks.
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
A new method for feature fusion in U-Net decoders using difference-based gating.
problem Precise fusion of high-level semantics and low-level details in U-Net decoder reconstruction.
method Proposes two difference-based gating approaches: Feature-difference gating (FDG) and Entropy-difference gating (EDG).
result Both FDG and EDG methods outperform existing attention-based fusion methods, with EDG showing superior performance.
ExGate improves neural network accuracy with feature-based attention.
problem Efficiency and accuracy in artificial neural networks for feature-based attention.
method Externally controlled neuron gating in artificial neural networks.
result 5% increase in classification accuracy on CIFAR-10 dataset.
Improved molecular property prediction using attention and gate mechanisms.
problem Predicting molecular properties from chemical data.
method Attention and gate mechanisms in graph convolutional networks.
result Improved prediction of molecular properties, including photovoltaic efficiency.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
RAU integrates attention into GRU for better sequence learning.
problem Lack of attention mechanism in GRU leads to information redundancy or loss.
method RAU adds an attention gate to GRU to adaptively focus on regions of interest.
result RAU consistently outperforms GRU and other methods in various tasks.
New model optimizes portfolios with realistic transaction costs.
problem Real-world transaction costs impact portfolio profitability.
method DPGRGT model with 2D relative-attentional Gated Transformer.
result Model outperforms baseline models in U.S. stock market data.
DAGR improves navigation by refining goal representations conditioned on the current state.
problem Goal-conditioned reinforcement learning lacks state awareness, leading to inefficient policy recovery.
method DAGR refines static goal embeddings into state-conditioned ones using gated cross-attention with a state-goal discrepancy map.
result DAGR improves navigation tasks on OGBench, matching or outperforming base methods.
AuGMEnT network struggles with long-term memory for hierarchical tasks.
problem Learning and memory in neural networks, especially hierarchical tasks.
method Introduced hybrid AuGMEnT with leaky and non-leaky memory units.
result Hybrid AuGMEnT solves hierarchical and distractor tasks.
Study improves stock movement prediction using multimodal data.
problem Inaccurate stock movement prediction due to incomplete multimodal data integration.
method Introduces MSGCA framework for robust multimodal fusion.
result MSGCA framework outperforms existing methods by 21.7% on multimodal datasets.
MGpi model predicts social actions in group interactions.
problem Social interaction among multiple agents and groups.
method Deep neural network with Kinesic-Proxemic-Message gate for social signal gating.
result Achieves state-of-the-art performance in group identification.
GA-Net selectively attends to parts of a sequence for text classification.
problem Inefficient global attention mechanisms on long sequences.
method Gated Attention Network (GA-Net) using an auxiliary network to dynamically select and attend to important parts of the sequence.
result GA-Net achieves better performance with less computation and interpretability.
KATA improves associative recall by optimizing feature maps derived from nonnegative attention weights.
problem Linear attention's poor performance on associative recall tasks.
method Formulates attention recall as a spherical-packing problem and introduces Kernelized Linear Attention Activations (KATA).
result KATA features offer a favorable capacity-interference tradeoff, enabling efficient associative recall.
WAVE improves time series forecasting by integrating AR and MA components.
problem Time series forecasting challenges.
method WAVE attention mechanism with AR and MA components.
result WAVE attention consistently improves TSF performance.
Study combines CNNs and LSTMs for ECG classification, improving performance with attention mechanisms.
problem Improving ECG classification accuracy for weakly supervised tasks.
method Investigates ConvLSTM networks and various attention mechanisms to capture both morphological and temporal features.
result Attention mechanisms improve classification performance and provide interpretable saliency maps.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
problem Identifiability issues and complex interactions in Gaussian mixture of experts.
method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.
result Connects convergence rate of MLE to a solvability problem of polynomial equations.
Study improves fraud detection in e-commerce with a stacked model combining CNNs, GNNs, and confidence gating.
problem Detecting credit card fraud in online transactions.
method Stacking approach with attention and confidence-driven layers, using DOWA and IOWA operators.
result The method achieves high accuracy and robust generalization in CCF detection.
Deep Sets improve reinforcement learning agent's object-centered navigation and generalization.
problem Improving reinforcement learning agents' ability to generalize to unseen objects and goals.
method Combining object-wise permutation invariant networks (Deep Sets) and gated-attention mechanisms.
result Agent demonstrates strong generalization to out-of-distribution goals in a procedurally-generated 2D world.
New model improves sentiment analysis by fusing words from audio and video.
problem Improving sentiment analysis in noisy multimodal data.
method Gated Multimodal Embedding LSTM with Temporal Attention.
result State-of-the-art sentiment classification and regression results on CMU-MOSI dataset.
Paper proposes binary-valued gates for better LSTM training.
problem LSTMs struggle with soft gates, leading to unclear information flow.
method Introduces binary-valued gates to control information flow more clearly.
result Binary-valued gates improve LSTM performance and generalization.
Paper shows how LSTM can remember long sequences by attending to persisted information.
problem LSTMs struggle with long sequences due to fading information and bias towards recent data.
method The paper introduces a mechanism that allows LSTMs to attend to information in memory based on how long it was persisted by the gating mechanism.
result The method improves LSTM's ability to process long sequences by retrieving information proportionally to its persistence in memory.
Hierarchical pretraining with slow-fast ODEs
problem Causal self-attention vs. slow-fast ODEs
method Instantiating fast-slow ODE formalism as a concrete neural network
result Equilibrium manifold x=φ(y) is exactly the master-equation (ME) stationary distribution The paper establishes convergence rates for MoE models in classification problems.
problem Understanding the behavior of MoE models in classification settings.
method Established convergence rates for density and parameter estimation in softmax gating multinomial logistic MoE models.
result Parameter estimation rates are significantly improved with a novel modified softmax gating function.
ConViT combines CNN and ViT strengths, improving image classification.
problem Combining the strengths of CNNs and ViTs while avoiding their limitations.
method Introducing GPSA, a form of positional self-attention with a soft convolutional inductive bias.
result ConViT outperforms DeiT on ImageNet while offering improved sample efficiency.
The study analyzes the convergence rates of Gaussian mixtures of experts.
problem Analyzing the convergence rates of Gaussian mixtures of experts.
method The study uses a novel notion of algebraic independence and optimal transport theory to establish convergence rates and minimax lower bounds.
result The study provides theoretical convergence rates for maximum likelihood estimation of over-specified Gaussian mixtures of experts.
Attention-based CNNs improve band selection in hyperspectral images.
problem Selecting informative bands from hyperspectral images for accurate classification.
method Attention-based convolutional neural networks reusing activations at different depths.
result Deep models with attention mechanisms achieve high-quality classification and identify significant bands.
Paper presents a neural network method for automated bug and ticket classification.
problem Automated classification of bug and ticket content in systems.
method Recurrent neural network with hierarchical attention mechanism.
result The method outperforms previous approaches on two datasets.
LSTM improves cross-network recommendations by capturing user preference changes and irregular time intervals.
problem Offline cross-network recommender solutions fail to capture user preference changes and dynamic environments.
method Proposes a multi-layered LSTM network with attention mechanisms, higher order interactions, and time-aware gates.
result The model consistently outperforms state-of-the-art in accuracy, diversity, and novelty.
FleXOR trains fractional quantization for neural networks, improving accuracy and size.
problem Quantization limits to integer bits restricts compression and accuracy.
method Encryption algorithm with XOR gates for fractional bits during inference.
result FleXOR achieves high accuracy with fractional sub-1-bit weights.
Methods from convex optimization are widely used as building blocks for deep learning algorithms. However, the reasons for their empirical success are unclear, since modern convolutional networks (convnets), incorporating rectifier units and max-pooling, are neither smooth nor convex. Standard guarantees therefore do n…
The paper analyzes convergence rates of softmax gating in MoE models.
problem The effectiveness and scalability of machine learning models using MoE.
method Convergence analysis of parameter and expert estimation under MoE with softmax gating and its variants.
result Theoretical results show polynomially many data points are needed for strong identifiability conditions, while exponential points are required for linear experts.
Temporal Functional Circuits explain KAN forecasts with interpretable edge functions.
problem Lack of mechanistic explanations in KAN forecasting.
method Transform KAN edge functions into faithful, temporally grounded explanations using a gated residual KAN.
result Gated KAN achieves lower MSE than linear-only models on regime-switching signals.
A new model improves relation extraction accuracy through relation-gated adversarial learning.
problem Relation extraction from sentences is challenging due to expensive human annotation and noisy distant supervision.
method Proposes relation-gated adversarial learning for relation extraction, extending domain adaptation methods.
result The model outperforms previous domain adaptation methods and improves accuracy of distance supervised relation extraction.
Analyzes convergence rates for Gaussian-gated MoE model.
problem Theoretical understanding of Gaussian-gated MoE model is incomplete.
method Maximum likelihood estimation with novel Voronoi loss functions.
result MLE has distinct behaviors under different settings of Gaussian gating function parameters.
THGFM models dynamic relational systems with cross-type and temporal fusion.
problem Learning on temporal heterogeneous graphs with diverse node and relation types.
method Dual-Path Architecture with Shared-Space and Relational Type-Partitioned Temporal Attention.
result THGFM outperforms baseline models on academic graph benchmarks.
Focused RNNs improve sequence processing tasks with attention mechanisms.
problem Sequence processing tasks benefit from more focused attention mechanisms.
method Focused hierarchical RNNs with discrete gating mechanisms that selectively attend to relevant parts of the input.
result Focused RNNs outperform existing methods on various sequence processing tasks, including MS MARCO and SearchQA.
Extended LSTM improves language modeling performance.
problem Improving LSTM for large-scale language modeling.
method Exponential gating, modified memory structure, and residual stacking.
result xLSTM outperforms state-of-the-art models in performance and scaling.
Entropy regularization improves sparse model discovery in federated learning.
problem Sparse model discovery in federated learning with limited data.
method Entropy regularization of gate distributions for probabilistic sparse model exploration.
result Entropy regularization leads to better sparse model recovery and performance.
Study improves stock price prediction using advanced ML models.
problem Improving financial forecasting accuracy in stock markets.
method Evaluation of RNN architectures including LSTM, GRU, and attention-based models.
result Attention-based models outperform others in capturing complex dependencies.
FEM improves attention mechanisms by applying value-driven log-linear tilts.
problem Standard attention mechanisms read via convex average, limiting channel-wise selection.
method Free Energy Mixer (FEM) applies a value-driven, per-channel log-linear tilt to a fast prior over indices.
result FEM outperforms strong baselines on NLP, vision, and time-series tasks.
Paper uses AI to predict option volatility surfaces with improved accuracy.
problem Difficult to predict dynamic evolution of option volatility smile surface.
method Combines deep learning (LSTM) with attention mechanism.
result Predicted volatility surfaces lead to higher returns and Sharpe ratios.
Deep learning predicts vascular disease from medical history.
problem Predicting high-risk vascular diseases from medical records.
method Medical History-based Prediction using Attention Network (MeHPAN) models.
result MeHPAN models outperform standard classification models.