Memory Matching Networks classify DNA sequences for protein binding sites.
problem Manual construction of DNA motifs is difficult due to their complexity.
method Memory Matching Networks (MMN) learn a dynamic memory bank of encoded motifs and match them to new sequences.
result MMN effectively classifies DNA sequences as protein binding or nonbinding sites.
Improved text matching model using deconvolutional networks.
problem Text sequence matching challenges.
method Jointly optimizing generative and discriminative objectives with deconvolutional networks.
result Significantly outperforms sentence-encoding baselines, especially in semi-supervised settings.
Generative model for realistic text generation using feature matching.
problem Applying GAN to text generation due to convergence issues and difficulties with discrete data.
method Employing a long short-term memory network as generator and a convolutional network as discriminator. Matching latent feature distributions via a kernelized discrepancy metric.
result Superior performance in generating realistic-looking sentences compared to standard GAN.
Memory-enhanced model predicts tennis shots based on player history.
problem Predicting shot location and type in tennis.
method Semi-supervised Generative Adversarial Network with neural memory modules.
result The model learns player-specific behavioral patterns from match data.
A new memory system handles non-stationary environments by self-sizing and retaining memories.
problem Non-stationary environments where memories arrive over time and must be distinguished from noise.
method A self-sizing continual associative memory that generalizes Hopfield's network, handling adaptation and novelty.
result The memory system grows to the intrinsic memory demand of the environment and retains memories without forgetting.
DiffusionBlocks trains neural networks by breaking them into independent blocks, reducing memory usage.
problem Memory bottlenecks in end-to-end neural network training.
method Transforming transformer-based networks into independent trainable blocks via a denoising process.
result Independent block-wise training matches end-to-end training performance while reducing memory requirements.
Centroids Matching tackles catastrophic forgetting by matching feature vectors to class centroids.
problem Catastrophic forgetting in neural networks when learning new tasks.
method Centroids Matching operates in the embedding space of neural network features, matching these vectors to class centroids.
result Centroids Matching achieves high accuracy on all tasks without using external memory, even in realistic scenarios.
Deep learning reduces memory and computation for MRF recovery.
problem Heavy storage and computation requirements of dictionary-matching in MRF recovery.
method MRF-Net with dimensionality reduction and manifold clustering.
result MRF-Net saves over 60 times memory and computation.
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of mo…
Bayesian neural networks can be partially stochastic without losing predictive power.
problem The necessity of fully stochastic parameters in Bayesian neural networks.
method Theoretical and empirical investigation of partially stochastic networks compared to fully stochastic ones.
result Expressive predictive distributions require only small amounts of stochasticity, and partially stochastic networks can match or outperform fully stochastic networks.
A new method reduces memory requirements for Graph Transformers by sparsely training a network.
problem Quadratic memory complexity in Graph Transformers limits their scalability to large graphs.
method Spexphormer: trains a narrow network on augmented graph, then uses only active connections in a wider network.
result Spexphormer achieves good performance with drastically reduced memory requirements.
SPARC improves continual learning with minimal memory and computational overhead.
problem Efficient continual learning for deep neural networks.
method Combines task-specific working memories and task-agnostic semantic memory.
result Significantly reduces parameter usage (6% of full-model surrogates) while maintaining performance.
NANSDE-Net models time series with memory using neural ARMA-type noise.
problem Modeling time series with long- or short-memory characteristics.
method Developed NANSDE-Net, a generative model that incorporates Neural Network-kernel ARMA-type noise.
result NANSDE-Net matches or outperforms existing models in reproducing long- and short-memory features of data.
New algorithm improves PPS for multi-object matching.
problem Efficiently synchronize partial permutations for multi-object matching.
method Proposed CEMP-Partial algorithm for partial permutation synchronization (PPS). Uses sparse matrix operations and nonconvex weighted projected power method.
result Proves CEMP-Partial can exactly classify corrupted and clean partial permutations under adversarial corruption.
PARSEC uses a probabilistic approach to reduce memory usage in neural architecture search.
problem Efficiently search over large and complex neural architectures with reduced memory usage.
method Probabilistic sampling to learn a distribution over high-performing architectures, enabling transfer learning.
result Our approach outperforms state-of-the-art methods with significantly less computational cost.
A new algorithm reduces memory usage for deep learning models.
problem Training deep learning models requires significant memory.
method Dynamic Tensor Rematerialization (DTR) is a greedy online algorithm that dynamically plans recomputations.
result DTR achieves comparable performance to optimal static checkpointing with only a small memory budget.
FSR prevents deep nets from forgetting old tasks.
problem Catastrophic forgetting in deep neural networks.
method Logit matching on selected samples from old tasks.
result FSR outperforms previous methods in knowledge retention.
Models predict emotional valence from narratives, matching human raters.
problem Predicting emotional valence from multimodal time-series data.
method Adapted attention-based mechanisms (Transformer, Memory Fusion Network) to emotional narratives.
result Models perform well, matching human raters on emotional valence prediction.
Implicit models can match or exceed explicit models with more test-time compute.
problem Understanding the expressive power and scaling of implicit models.
method Nonparametric analysis of expressive power, mathematical characterization of implicit operators, and test-time scaling experiments.
result Implicit models can progressively express more complex mappings through iteration, matching a richer function class with test-time compute.
Enhances option pricing with fractional order Black-Scholes-Merton model.
problem Improving precision and authenticity of option pricing.
method Integrates fractional order Black-Scholes-Merton with neural networks.
result Improves accuracy in capturing complex diffusion dynamics and memory effects.
New framework analyzes temporal features in state space models.
problem Understanding temporal dependencies in data streams.
method Proposes a framework for rigorous analysis of state representations in ESNs, using temporal feature spaces and kernel machines.
result Phase transition in kernel richness for cycle reservoir topology.
JKO-iFlow uses neural ODEs to improve generative models with reduced memory and training complexity.
problem Efficiently training deep generative models in high dimensions with reduced memory and training complexity.
method JKO scheme inspired neural ODE flow network with adaptive time reparameterization.
result JKO-iFlow achieves competitive performance compared to existing models at reduced computational and memory cost.
Backpropagation-free trunk training improves model performance on various benchmarks.
problem Memory inefficiency and noisy gradient estimates in deep network training.
method Split Forward Gradient (Split-FG) method that splits network into trunk and head, estimating only trunk gradient.
result Split-FG achieves better performance than pure forward-gradient training and backpropagation on various benchmarks.
A new memory-efficient Adam variant reduces second moments when feasible.
problem Memory constraints in training machine learning models.
method Signal-to-Noise Ratio (SNR) analysis to identify dimensions where second moments can be replaced by means.
result Memory-efficient Adam variant (SlimAdam) matches performance and stability of Adam while saving up to 98% of second moments.
A new framework for efficient sequence maps using Bayesian filtering and covariance.
problem Designing efficient recurrent sequence maps from explicit memory assumptions.
method Design-model framework, exact Bayesian filtering, query-dependent readout, linear-Gaussian instantiation.
result Improved robustness and retrieval performance across various benchmarks.
The strength of association between a pair of data vectors is represented by a nonnegative real number, called matching weight. For dimensionality reduction, we consider a linear transformation of data vectors, and define a matching error as the weighted sum of squared distances between transformed vectors with respect…
We analyze generalization in deep learning models using random matrix theory.
problem Understanding the generalization error in deep learning models with random feature representations.
method Applying Random Matrix Theory to derive asymptotic generalization error formulas for various architectures.
result Linear ESNs are equivalent to ridge regression with exponentially time-weighted input covariance, revealing an inductive bias towards recent inputs.
Proposes a method to train deep models with one-element batches.
problem Training deep models with small batches (one element) is challenging.
method Splits the batch into historical and current elements for training.
result Allows training on higher resolution images with one-element batches.
Random weights in GNNs match learned weights in performance.
problem Feature rank collapse in GNNs.
method Replacing learned weights with random weights.
result Random weights achieve comparable performance to learned weights, reducing training time and memory usage.
New method prevents forgetting in neural-linear bandits for high-dimensional data.
problem Catastrophic forgetting in neural-linear bandits with high-dimensional features.
method Limited memory neural-linear bandit with likelihood matching.
result Resilient to catastrophic forgetting and superior performance.
Stochastic attention learns to retrieve and generate from memory without training.
problem Learning to retrieve and generate from memory efficiently.
method Langevin dynamics on modern Hopfield energy for stochastic attention.
result Stochastic attention can retrieve and generate from memory without training, matching gold standards.
New score matching method estimates local intrinsic dimension efficiently.
problem Quantifying the local intrinsic dimension of complex data.
method Denoising score matching loss and equivalent implicit score matching loss.
result Denoising score matching loss is a highly competitive and scalable LID estimator.
ORIGAMI accelerates ML algorithms by splitting compute tasks between in-memory and off-chip accelerators.
problem Memory bandwidth bottleneck in ML processing.
method Heterogeneous in-memory accelerators and off-chip compute platform, pattern-matching for compute patterns, computation-splitting compiler.
result ORIGAMI outperforms state-of-the-art accelerators in performance and energy-efficiency.
ARMIN improves memory efficiency and lightness in neural networks.
problem Complex memory addressing and inefficient memory information exploitation in previous MANNs.
method ARMIN uses hidden state for automatic memory addressing and a novel RNN cell for memory integration.
result ARMIN achieves lower computational overhead and similar performances compared to vanilla LSTM.
Neural network tackles continual learning with neuromodulation and local error signals.
problem Catastrophic forgetting in continuous learning.
method Biologically-inspired neural architecture with local learning and neuromodulation, combined with transfer metalearning.
result Superior performance in continual learning tasks compared to other approaches.
EuSN uses Euler discretization for stable, non-dissipative reservoir computing.
problem Designing stable and efficient reservoir computing models.
method Forward Euler discretization and antisymmetric recurrent matrices.
result EuSN outperforms standard RC models in long-term memory tasks and time-series classification.
A taxonomy classifies memory networks based on their memory organization.
problem Classifying and understanding the expressive power of different memory networks.
method Developed a taxonomy including RNN, LSTM, neural stack, and neural RAM, analyzing their differences and commonality.
result Showed the relative expressive power of memory networks and how they relate to specific tasks.
A new cycleGAN architecture reduces memory and parameter requirements for low-dose CT denoising.
problem Efficient unsupervised low-dose CT denoising with minimal memory and parameter usage.
method Single switchable generator using AdaIN layers for efficient training and inference.
result The proposed method outperforms previous cycleGAN approaches with half the parameters.
LEMN improves memory networks for learning from streaming data.
problem Efficiently processing large data streams in memory-augmented neural networks.
method LEMN uses a RNN-based retention agent to learn and replace less important memory entries based on their importance and historical context.
result LEMN achieves significant improvements over existing methods in learning from streaming data.
AuGMEnT network struggles with long-term memory for hierarchical tasks.
problem Learning and memory in neural networks, especially hierarchical tasks.
method Introduced hybrid AuGMEnT with leaky and non-leaky memory units.
result Hybrid AuGMEnT solves hierarchical and distractor tasks.
LMNs enhance neural networks with memory to adapt online.
problem Online adaptation of neural networks for domain-relevant data.
method LMNs use label-based memory replacement and write to memory only for instances with non-zero loss.
result Significant accuracy gains on various tasks including word-modelling and few-shot learning.
Low-precision quantization improves kernel approximation under memory constraints.
problem Training kernel approximation methods efficiently with limited memory.
method Low-precision quantization of random Fourier features (LP-RFFs).
result LP-RFFs can match the performance of full-precision RFFs and Nyström method with significantly less memory.
Linear Memory Network separates memory and function in RNNs.
problem Complex transduction problems requiring memory and input-output exploitation.
method Conceptual separation between memory and function, using feedforward and autoencoder components.
result Efficient training and competitive performance on polyphonic music datasets.
Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which …
New memory in neural networks mimics computer architectures.
problem Learning algorithms and complex tasks with neural networks.
method Introducing a new memory to store weights for a neural controller, similar to stored-program memory in computers.
result Neural Stored-program Memory enhances neural networks' adaptability and learning capabilities.
Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
The paper aims to develop new combinatorial dimensions for bounded memory learning.
problem Characterize bounded memory learning using combinatorial dimensions.
method Proposes a candidate solution based on the SQ dimension of neighboring distributions and proves upper and lower bounds.
result Characterizes bounded memory learning in a specific parameter regime, matching equivalence between bounded memory and SQ learning.
mGRN improves multivariate time series prediction by managing marginal and joint memories.
problem Extracting dependencies in multivariate sequential data with strong serial and cross-sectional dependencies.
method Developed a novel recurrent network architecture, Memory-Gated Recurrent Networks (mGRN), with gates for marginal and joint memories.
result mGRN consistently outperforms state-of-the-art architectures on various public datasets.