MAC Net improves natural language question answering with data-driven reasoning.
problem Natural Language Question Answering requires complex reasoning.
method MAC Net architecture separates memory and control for iterative reasoning.
result MAC Net achieves high efficiency and interpretability in NLP tasks.
Slot Attention extracts object-centric representations from images.
problem Learning distributed representations that don't capture natural scene composition.
method Slot Attention module interfaces with CNN outputs to produce task-dependent abstract slots.
result Slot Attention enables generalization to unseen compositions.
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling …
Enhances image captioning with novel context combination methods.
problem Improving machine learning for image captioning with structured learning and meaningful interpretation.
method Combines Feature Distribution Composition (FDC), Multiple Role Representation Crossover (MRRC) attention layers, and language decoder.
result Significantly improved image captioning performance (35.3%) and established new standards.
Simple model outperforms neural networks on language understanding tasks.
problem Neural networks struggle with creating novel expressions from familiar ones.
method Attention-inspired modification of a baseline model, focusing on sequential thinking and acting.
result Simple model achieves good performance on gSCAN tasks, validating the benchmark.
Transformers learn to generalize unseen tasks by composing self-attention layers.
problem How Transformers generalize to unseen, out-of-distribution tasks.
method Examined synthetic examples and pretrained LLMs, focusing on induction heads and latent subspace.
result Transformers can learn hidden rules for unseen tasks by composing self-attention layers, achieving OOD generalization.
Cosmos models scenes using neural encodings and symbolic attributes for compositional generalization.
problem Modeling scenes with high performance on unseen input scenes composed of known visual elements.
method Neurosymbolic grounding with neurosymbolic scene encodings and attention mechanisms.
result Establishes a new state-of-the-art for compositional generalization in world modeling.
CTLNet predicts Shanghai Composite Index with improved accuracy.
problem Predicting Shanghai Composite Index for investors and researchers.
method Proposes CNN-Transformer-LSTM Networks (CTLNet) for multivariate time series forecasting.
result CTLNet outperforms state-of-the-art baselines in Shanghai Composite Index prediction.
Bitcoin's attention is linked to Google Trends data, not general uncertainty.
problem Bitcoin's correlation with Google Trends data was previously misunderstood.
method Analyzed bidirectional relationships between Bitcoin returns and Google Trends attention over six days.
result Information flows from Bitcoin volatility to Google Trends attention, not the other way.
Deep learning model separates syntax and semantics for better language generalization.
problem Standard deep learning methods struggle with systematic generalization in natural language.
method Implemented a Syntactic Attention model that separates syntactic and semantic processing.
result The Syntactic Attention model outperforms standard methods on a compositional generalization task.
Measures compositionality in machine learning representations.
problem Evaluating how compositional structure is reflected in learned representations.
method Measures compositionality by approximating true representation-producing models with composed primitives.
result Characterizes compositional structure in various settings.
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
This paper evaluates how well neural models can solve complex tasks by breaking them into simpler ones.
problem Measuring neural models' ability to solve complex tasks by breaking them into simpler subtasks.
method Characterized axes of compositional generalization, introduced a benchmark suite of tasks, and improved Transformer models' attention mechanisms.
result Modified Transformer models generally perform better than natural baselines in solving complex tasks, but challenges remain.
Proposes a model for multi-agent reinforcement learning with hierarchical graph attention network.
problem Limited transferability of trained policies to new multi-agent tasks.
method Uses hierarchical graph attention network for representation learning and multi-agent actor-critic for policy learning.
result Demonstrates superior performance in mixed cooperative and competitive tasks compared to existing methods.
Bayesian modeling predicts hydroxide ion conductivity in polymer membranes.
problem Quantitative relationship between hydrophilic domain size and hydroxide ion conductivity in polymer membranes is unknown.
method Bayesian sparse modeling applied to copolymer composition data.
result Composition-derived features are identified as critical for predicting hydroxide ion conductivity.
A new transformer model accelerates training with optimization techniques.
problem Training deep neural networks efficiently and effectively.
method Interprets transformer layers as optimization steps, applying Nesterov acceleration.
result The new model outperforms existing models on benchmark datasets.
Transformers learn to solve modular arithmetic tasks by in-context learning and skill composition.
problem Understanding how large language models generalize to unseen tasks in modular arithmetic.
method Pre-training on a set of modular arithmetic tasks and evaluating out-of-distribution performance.
result Transformers require two transformer blocks for out-of-distribution generalization, and deeper models exhibit transient out-of-distribution performance.
RB-Modulation trains free diffusion models without external adapters.
problem Training-free personalization of diffusion models with style and content control.
method Stochastic optimal control with a style descriptor and cross-attention aggregation.
result Precise content and style extraction and control without external adapters.
Enhances graph neural networks by considering feature similarities in node aggregation.
problem Ignoring node feature similarities in traditional graph aggregation schemes.
method Interprets node aggregation as kernel weighting, proposing a framework that considers feature similarities.
result Proposed framework outperforms traditional GCNs in real-world applications.
Hydra boosts efficiency for long-context reasoning in resource-constrained settings.
problem Quadratic complexity of transformers limits long-context reasoning in resource-constrained systems.
method Hydra uses a modular architecture with adaptive routing between sparse global attention, mixture-of-experts, and dual memories.
result Hydra achieves significant throughput and accuracy improvements for long-context reasoning.
Stochastic attention learns to retrieve and generate from memory without training.
problem Learning to retrieve and generate from memory efficiently.
method Langevin dynamics on modern Hopfield energy for stochastic attention.
result Stochastic attention can retrieve and generate from memory without training, matching gold standards.
New method improves RL/IL agents' adaptability to unseen environments.
problem Current RL/IL techniques struggle with generalizing to unseen environments.
method Zero-shot compositional policy learning with multi-modal fusion and attention mechanism.
result Language grounding enhances generalization across varied environments.
This paper presents a Bayesian approach to learning the connectivity structure of a group of neurons from data on configuration frequencies. A major objective of the research is to provide statistical tools for detecting changes in firing patterns with changing stimuli. Our framework is not restricted to the well-under…
Induction of common sense knowledge about prototypical sequences of events has recently received much attention. Instead of inducing this knowledge in the form of graphs, as in much of the previous work, in our method, distributed representations of event realizations are computed based on distributed representations o…
A2I Transformer predicts atom energies from coordinates, avoiding heavy featurization.
problem Efficiently predicting atom energies from molecular coordinates with minimal featurization.
method End-to-end model using self-attention, permutation-equivariant.
result Stable predictions with significantly smaller errors than molecular dynamics simulations.
HRFs adaptively linearize kernels for accurate approximations.
problem Linearizing softmax and Gaussian kernels for machine learning applications.
method Generalizes Bochner's Theorem for kernels, uses random features for compositional kernels.
result Strong theoretical guarantees and unbiased approximation with smaller relative errors.
The study of deep recurrent neural networks (RNNs) and, in particular, of deep Reservoir Computing (RC) is gaining an increasing research attention in the neural networks community. The recently introduced Deep Echo State Network (DeepESN) model opened the way to an extremely efficient approach for designing deep neura…
Transformer-based multi-scale model outperforms traditional methods in solving PDEs on irregular domains.
problem Solving partial differential equations on irregular domains using deep learning.
method Introduces Multi-Scale Attention Transformer (\msat{}) for solving PDEs.
result Achieves state-of-the-art generalization on complex geometry problems with significant speedup.
This work is an analytical and numerical study of the composition of several fractals into one and of the relation between the composite dimension and the dimensions of the component fractals. In the case of composition of standard IFS with segments of equal size, the composite dimension can be expressed as a function …
Paper proposes HTAF for stable training of binary neural networks.
problem Challenges in training binary neural networks with gradient-based optimization.
method HTAF is a smooth approximation to the Heaviside function that enables stable training.
result HTAF enables stable training of various binary neural networks with gradient-based optimization.
SR-GNN models sessions as graphs to predict user actions.
problem Predict user actions based on anonymous sessions.
method SR-GNN models sessions as graphs and uses attention networks.
result SR-GNN outperforms state-of-the-art methods consistently.
A new memory-based fusion layer improves multi-modal deep learning performance.
problem Improving performance of multi-modal deep learning by addressing long-term dependencies.
method Introducing a Memory based Attentive Fusion (MBAF) layer that incorporates both current and long-term dependencies.
result The MBAF layer enhances fusion and improves performance across different modalities and networks.
New geometric approach for analyzing compositional data like gut microbiomes.
problem Analyzing non-negative compositional data with relative values only.
method Reinterpret compositional data as quotient topology of a sphere, using spherical harmonics and reflection group actions.
result Construction of Reproducing Kernel Hilbert Space (RKHS) for compositional data.
This research tackles group fairness in predictive process monitoring by ensuring predictions are independent of sensitive group membership.
problem Predictive models using biased historical data can perpetuate unfair behavior in new cases.
method Investigates independence through metrics like ΔDP and a composite loss function balancing predictive performance and fairness.
result Proposes and validates a composite loss function for training models that balance fairness and performance.
Study on deep neural networks using branching processes and Mehler's formula.
problem Understanding the mathematical role of activation functions in compositional neural networks.
method Connection between compositional kernels and branching processes via Mehler's formula; new random features algorithm.
result Explicit formulas for eigenvalues of compositional kernels quantify complexity.
Uses art composition attributes to guide CycleGAN image translation.
problem Improving image-to-image translation quality.
method Trained ACAN on art composition attributes to influence CycleGAN.
result CycleGAN translations improved with ACAN constraints.
We establish conditions for compositional generalization in machine learning.
problem Achieving compositional generalization in machine learning models.
method We reformulate compositionality as a property of the data-generating process and derive mild conditions on the training distribution and model architecture.
result Our theoretical framework enables compositional generalization under mild conditions.
This paper proves hyperbolicity of virtual knot compositions.
problem Proving hyperbolicity of virtual knot compositions.
method Exploring the composition of hyperbolic virtual knots.
result Strong lower bounds on the volume of compositions.
Develops methods for causal inference in compositional data using instrumental variables.
problem Interpreting summary statistics like diversity indices as causal effects in compositional data.
method Statistical data transformations and regression techniques tailored for compositional data.
result Advantages and limitations of the proposed methods demonstrated on synthetic and real microbiome data.
In classical field theory, the composite fibred manifolds Y -> Z -> X provides the adequate mathematical formulation of gauge models with broken symmetries, e.g., the gauge gravitation theory. This work is devoted to connections on composite fibred manifolds. In particular, we get the horizontal splitting of the vertic…
Efficient neural network optimization reduces costs and improves model performance.
problem High computational costs in optimizing neural networks, especially at scale.
method Introduces self-attentive feed-forward neural units (SAFFU) for efficient optimization.
result Explicit solutions outperform models optimized by backpropagation alone, and further training with backpropagation leads to better optima from smaller data sets.
The p-index improves investment performance for NYSE stocks but not for SSE stocks.
problem Improving investment performance for stocks using the p-index.
method Comparing different p-ratio strategies and empirical efficient frontiers for SSE and NYSE stocks.
result The p-index enhances investment performance for NYSE stocks but not for SSE stocks.
Model predicts composite structures assembly quality with input uncertainty.
problem Accurate prediction of dimensional deviations and residual stress in composite structures assembly.
method Neural Network Gaussian Process considering input uncertainty.
result NNGPIU model outperforms other methods for nonsmooth, nonlinear responses.
New algorithm reduces complexity for optimizing complex machine learning tasks.
problem Optimizing complex machine learning objectives like reinforcement learning and portfolio management.
method Developed SARAH-Compositional algorithm using Stochastic Recursive Gradient Descent.
result Achieved optimal IFO complexity bounds for stochastic compositional optimization.
This paper introduces compositional data analysis for financial ratios, improving industry-level analysis.
problem Statistical issues with standard financial ratios at industry level.
method Compositional data analysis techniques for financial ratios.
result Improved analysis of financial ratios using compositional data methods.
Study challenges neural models in compositional learning tasks.
problem Challenges in neural models for compositional and relational learning.
method Introduced ConceptWorld environment for generating images from compositional concepts, tested various neural architectures.
result Neural models struggle with longer compositional chains and substitutivity tests.
SCL discovers compositional structures in analogical reasoning tasks.
problem Discovering compositional structures in analogical reasoning tasks like Raven's Progressive Matrices.
method Proposes Scattering Compositional Learner (SCL) that composes neural networks in sequence.
result Achieves state-of-the-art performance on RPM datasets with significant improvements.
Composite neural network improves PM2.5 prediction.
problem Improving PM2.5 prediction accuracy.
method Composite neural network framework with pre-trained models.
result Composite neural network outperforms individual models and new components added.