Transformer attention layers solve single-location regression tasks.
problem Understanding token-wise sparsity and internal linear representations in attention-based models.
method Introduce single-location regression task and a simplified predictor based on self-attention layers.
result Transformer attention layers are asymptotically Bayes optimal and can learn underlying structures effectively.
Bilateral trade relationships in the international level between pairs of countries in the world give rise to the notion of the International Trade Network (ITN). This network has attracted the attention of network researchers as it serves as an excellent example of the weighted networks, the link weight being defined …
This study investigates abrupt learning dynamics in Transformers, revealing plateau formation and internal representation collapse.
problem Abrupt learning in Transformers, particularly during the loss plateau.
method Investigates mechanisms of abrupt learning in shallow Transformers, focusing on attention maps and hidden states.
result Reveals plateau formation, internal representation collapse, and strong repetition bias in outputs.
Mathematical study shows post-hoc explanations are better than attention weights alone.
problem Understanding the internal behavior of attention-based models.
method Mathematical analysis of a simple attention-based architecture.
result Post-hoc explanations provide more useful insights than attention weights alone.
Sparse Transformers degrade semantic information first, with early layers encoding more.
problem Understanding how sparse Transformers affect learned representations and semantic information.
method Probed Transformers with progressively pruned weights to observe changes in semantic information and model behavior.
result Complex semantic information is first to degrade in sparse Transformers, with early layers encoding more.
Framework for explaining CNN predictions using input resampling.
problem Limited model interpretability in neural networks.
method Select neurons by two metrics over perturbed input images.
result Identifies neurons that influence and generalize network output.
Proposes a framework to predict stock movements by integrating multi-order and internal dynamics.
problem Predicting stock movements with multi-order and internal dynamics.
method Temporal generative filters and hypergraph attentions using wavelet basis.
result Framework outperforms state-of-the-art methods in terms of profit and stability.
Softmax attention approximates complex functions and subsumes many known universal approximators.
problem Universal approximation of continuous sequence-to-sequence functions.
method Interpolation-based analysis of attention's internal mechanism, showing its ability to approximate ReLU functions.
result Softmax attention is a universal approximator for continuous sequence-to-sequence functions.
TabPFN model shows strong robustness to noisy data.
problem TabPFN tackles robustness to noisy and imperfect tabular data.
method Empirical robustness analysis of TabPFN's attention mechanisms under various perturbations.
result TabPFN maintains high predictive performance and coherent internal behavior under noisy and imperfect data.
Modeling individual cardiovascular responses from wearable sensor data.
problem Capturing and understanding cardiovascular responses to physical activity and sleep changes.
method Attentional convolutional neural network to learn signatures from minute-level sensor data.
result Generated signatures generalize and outperform baseline models in predicting cardiovascular variables.
Interprets how intrinsic motivation shapes behavior in RL agents.
problem Understanding how intrinsic motivation influences behavior in reinforcement learning agents.
method Analyzed five RL agents in procedurally generated environments using various interpretability techniques.
result Curiosity-driven agents exhibit broader and more dynamic attention than extrinsically motivated agents.
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
RAU integrates attention into GRU for better sequence learning.
problem Lack of attention mechanism in GRU leads to information redundancy or loss.
method RAU adds an attention gate to GRU to adaptively focus on regions of interest.
result RAU consistently outperforms GRU and other methods in various tasks.
This research integrates attention into XAI frameworks for better model explanations.
problem Improving the interpretability of transformer models.
method Developed two novel explanation methods: Shapley value decomposition and token-level directional derivatives.
result Attention weights can be meaningfully incorporated into XAI frameworks, enhancing transformer explainability.
This work proposes an attention mechanism for structured sparsity in neural networks.
problem Uncontrolled sparsity in neural networks leads to significant accuracy drops.
method An attention mechanism that controls sparsity intensity and network pruning.
result The proposed method outperforms baseline methods by 6% on CIFAR-10.
Transformer models align words through attention weights, closely approximating Optimal Transport.
problem Understanding the internal mechanism of transformer models in language processing.
method Empirical evidence and theoretical analysis of attention weights and their relation to Optimal Transport.
result Transformer models can simulate gradient descent on the dual of entropy-regularized OT problem, providing a theoretical foundation for token alignment.
Vision Transformers show different internal representations compared to CNNs.
problem Understanding how Vision Transformers solve image classification tasks.
method Comparative analysis of ViT and CNN architectures on image classification benchmarks.
result ViT has more uniform representations across all layers, while CNNs have more varied representations.
Paper proposes HGAT for detecting fake news on HIN.
problem Viral spread of fake news causing social harm.
method Hierarchical Graph Attention Network (HGAT) for node representation learning in HIN.
result HGAT outperforms text-based and network-based models.
This paper recovers input data from transformer models using attention weights.
problem Recovering input data from transformer models for security and privacy concerns.
method Introducing an algorithm to minimize the loss function between expected and actual outputs of transformers.
result The algorithm successfully recovers input data from attention weights and outputs of transformers.
Team aims to predict particulate matter levels on ISS using Bi-GRU.
problem Early warning system for particulate matter on ISS.
method Bi-GRU algorithm analyzing past 90 minutes of data.
result Bi-GRU predicts particulate matter levels up to 1 minute in advance.
The international trade network (ITN) has received renewed multidisciplinary interest due to recent advances in network theory. However, it is still unclear whether a network approach conveys additional, nontrivial information with respect to traditional international-economics analyses that describe world trade only i…
Paper proposes MTL for weakly labelled SED, improving performance with 2-step attention.
problem Weakly labelled sound event detection.
method Multi-Task Learning framework with 2-step Attention Pooling.
result Improved SED performance with 22.3%, 12.8%, 5.9% gains at 0, 10, 20 dB SNR.
Develops a new framework for temporal anchoring in deep embedding spaces.
problem Temporal anchoring in deep embedding spaces, especially drift and convergence issues.
method Operator-theoretic framework with drift maps and event-indexed blocks, proving convergence theorems and equivalence theorems.
result Proves convergence theorems and equivalence theorems for the proposed framework.
Improved learning to reweight using deep interactions between student and teacher models.
problem Limitation of existing learning to reweight methods in utilizing student model's internal states.
method Proposes an algorithm that uses the student model's internal states to the teacher model, which returns adaptive weights to enhance student model training.
result Significant improvement over previous methods in image classification and neural machine translation experiments.
Study shows Transformer and Neural GPU are Turing complete without external memory.
problem Exploring computational power of modern neural network architectures.
method Analyzing computational properties of Transformer and Neural GPU.
result Transformer and Neural GPU are Turing complete without external memory.
Study predicts stock transaction durations using LSTM and attention mechanism.
problem Estimating the probability density function of transaction durations in financial markets.
method Proposes a hybrid model combining LSTM networks and attention mechanism to extend ACD model.
result Demonstrates superior performance of the hybrid model on large-scale financial data.
EXFormer predicts foreign exchange returns with high accuracy using a multi-scale self-attention mechanism and dynamic variable selection.
problem Accurately forecasting daily exchange rate returns in international finance.
method EXFormer uses a multi-scale trend-aware self-attention mechanism with dynamic variable selection and embedded squeeze-and-excitation blocks.
result EXFormer outperforms other models in forecasting daily exchange rate returns, achieving statistically significant improvements in directional accuracy.
New analysis shows how cross-entropy training shapes attention in transformers.
problem Understanding how gradient-based learning creates the required internal geometry in transformers.
method Developed a first-order analysis of cross-entropy training effects on attention scores and values in a transformer attention head.
result Introduced an advantage-based routing law and responsibility-weighted update for attention scores and values, respectively.
ProHOC detects OOD samples in class hierarchies, predicting them to correct internal nodes.
problem Binary OOD detection ignores semantic relationships between OOD and ID classes.
method Probabilistic hierarchical model using multi-depth networks trained for ID classification.
result ProHOC effectively classifies OOD samples to their correct internal nodes in class hierarchies.
New stabilization methods for exotic surfaces in 4-manifolds.
problem Understanding and stabilizing exotic surfaces in 4-manifolds.
method Relating internal and external stabilization, proving results using this relation.
result Many exotic surfaces become smoothly isotopic after external stabilization.
New methods improve integration of external LMs with AED models.
problem Improving performance of AED models by integrating external LMs.
method Comparing and proposing novel methods to estimate implicit LM from AED models.
result Proposed methods outperform previous approaches.
New GAN models robust against noisy attacks.
problem Robustness of GANs against adversarial interference.
method Introduced a perturbed discriminator to enhance robustness, explored model and training properties, and tested regularization effects.
result Variations of GANs are more robust to noisy attacks and require less regularization.
Improved audio event recognition using audiovisual transformers.
problem Weakly labeled audio event classification with limited temporal information.
method Adapted transformer architecture that combines audio and video inputs.
result State-of-the-art performance on audiovisual data set.
Transformers reduce redundancy by focusing on invariant relational quantities.
problem Substantial internal redundancy in Transformer models due to coordinate-dependent representations and continuous symmetries.
method Reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction.
result Architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.
Deep neural network learns hierarchical language family structure.
problem Language family dependency and external information affect classification accuracy.
method Hierarchical attentive units learn auxiliary tasks for robust internal representation.
result Improved classification accuracy on small and big language corpora.
New method detects careless responding in long surveys.
problem Careless responding in long surveys threatens internal validity.
method Detects a changepoint in combined measurements of carelessness.
result Highly accurate in identifying carelessness onset.
Internal Lagrangians derived from variational principles.
problem Reproducing the principle of stationary action in variational geometry.
method Introducing stationary points of internal Lagrangians, establishing connections with symmetries and conservation laws, and investigating relations between non-degenerate and internal Lagrangians.
result Noether's theorem reformulated in terms of internal Lagrangians.
Model predicts internal fraud in retail banking is cyclical and influenced by corruption.
problem Predicting and mitigating internal fraud losses in retail banking.
method Developed a dynamic model considering internal factors and macroeconomic indicators.
result Internal fraud losses are pro-cyclical and positively affected by corruption perceptions.
This work introduces 'Artificial Entanglement' to understand LLMs' fine-tuning effectiveness.
problem Understanding the effectiveness of parameter-efficient fine-tuning methods for large language models.
method Adopting a quantum-information-inspired perspective, the study measures 'Artificial Entanglement' in neural networks.
result LoRA and FFT induce distinct internal entanglement signatures but not external ones, suggesting a 'no-hair' property.
A new method backtracks through a few key past states to speed up credit assignment in long sequences.
problem Computational inefficiency of back-propagation through time for long sequences.
method Sparse attentive backtracking using learned attention mechanisms to skip connections.
result Matches or outperforms regular BPTT and truncated BPTT in tasks with long-term dependencies.
New bounds show transformers need longer training for length generalization.
problem Understanding when transformers can generalize to longer inputs.
method Analyzing different settings of transformers, providing quantitative bounds.
result Transformers need training data longer than previously thought for length generalization.
Examines international taxation's impact on Georgian businesses.
problem Impact of international taxation on Georgian business entities.
method Analyzes types, features, and approaches of international taxation.
result Provides recommendations to correct current taxation issues.
New tests for ES risk measure introduced to assess forecast quality.
problem Assessing forecast quality for Expected Shortfall (ES) risk measure.
method Developed three ES encompassing test variants using joint loss functions.
result Tests robust to misspecification and effective in finite samples.
Paper predicts user interests from browsing history and event sequences.
problem Capturing subtle user interests and inter-personal influence.
method Deep prediction method based on two RNNs modeling temporal point process and attention mechanism.
result Model outperforms state-of-the-art methods in fine-grained user interest prediction.
Optimized execution model using interbank and internal liquidity.
problem Minimizing market impact in trading.
method Integrates interbank limit and market orders with internal market-making liquidity.
result Reduces market impact and improves execution efficiency.
This study assesses how economic shocks affect the efficiency and robustness of international pesticide trade networks.
problem Economic shocks impact the efficiency and robustness of international pesticide trade networks.
method Simulations were used to quantify efficiency and robustness under different economic shocks. Three strategies were tested: descending, random, and ascending node removal.
result The international pesticide trade networks became more efficient and robust except for clustering coefficient. Import-oriented economies were more vulnerable to shocks.
There are few papers about the international trade of flowers, so it is believed that this paper, with this topic, could be an important contribution to the international scientific community. It is intended to analyze if the international trade flowers tendencies and policies are adapted to the actual world global con…
BERT captures linguistic features in separate semantic and syntactic subspaces.
problem Understanding how transformer models like BERT represent linguistic features internally.
method Qualitative and quantitative investigations of BERT's internal representations.
result Evidence of a fine-grained geometric representation of word senses and syntactic representations.