This work tackles online memory selection in continual learning using information theory.
problem Online selection of a representative replay memory from data streams.
method Information-theoretic criteria (surprise, learnability) and Bayesian model for efficient computation.
result InfoRS improves robustness against data imbalance compared to reservoir sampling.
Both the human brain and artificial learning agents operating in real-world or comparably complex environments are faced with the challenge of online model selection. In principle this challenge can be overcome: hierarchical Bayesian inference provides a principled method for model selection and it converges on the sam…
Study finds long memory in some emerging Asian stocks but not in developed markets.
problem Evaluating stock market efficiency in emerging vs developed markets.
method Improved wavelet estimator of long range dependence.
result Emerging Asian markets show more long memory in stock returns than developed markets.
In this paper, we propose a dual memory structure for reinforcement learning algorithms with replay memory. The dual memory consists of a main memory that stores various data and a cache memory that manages the data and trains the reinforcement learning agent efficiently. Experimental results show that the dual memory …
TASO optimizes CNN models for memory-constrained devices.
problem Large CNN models are too resource-intensive for mobile devices.
method Integer Linear Programming (ILP) for selecting operations and workspace allocation.
result 8x speedup and 2.2x reduction in memory usage compared to greedy algorithms.
This study examines memory effects in S&P500 market correlations using Langevin models.
problem The neglect of memory effects in market correlations for optimal portfolio selection.
method Fit a generalised Langevin equation (GLE) to S&P500 market correlation data.
result Memory effects in market correlations significantly improve forecasting accuracy and suggest a hidden slow time scale.
New method reduces memory usage for high-dimensional variable selection.
problem Scalability issues in high-dimensional variable selection, especially in genomics.
method Adaptive sampling of null features to eliminate dummy matrix materialization.
result Reduces memory and runtime by several orders of magnitude while preserving FDR control.
We propose a new data-driven method to select the optimal number of relevant components in Principal Component Analysis (PCA). This new method applies to correlation matrices whose time autocorrelation function decays more slowly than an exponential, giving rise to long memory effects. In comparison with other availabl…
Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance. Unlike solutions that require physically upgrade GPUs, the G…
A new method selects features efficiently for high-dimensional data.
problem High computational costs and memory requirements in high-dimensional data.
method QuickSelection uses the strength of neurons in sparse autoencoders to select features.
result QuickSelection achieves the best trade-off of accuracy, speed, and memory usage.
Dealing with datasets of very high dimension is a major challenge in machine learning. In this paper, we consider the problem of feature selection in applications where the memory is not large enough to contain all features. In this setting, we propose a novel tree-based feature selection approach that builds a sequenc…
Feature selection is an important challenge in machine learning. It plays a crucial role in the explainability of machine-driven decisions that are rapidly permeating throughout modern society. Unfortunately, the explosion in the size and dimensionality of real-world datasets poses a severe challenge to standard featur…
Paper proposes a new method to select memory data for online class-incremental learning.
problem Selecting which buffered images to replay for online class-incremental learning.
method Adversarial Shapley value scoring method to preserve latent decision boundaries.
result Proposed ASER method provides competitive or improved performance compared to state-of-the-art methods.
Large-scale Hierarchical Classification (HC) involves datasets consisting of thousands of classes and millions of training instances with high-dimensional features posing several big data challenges. Feature selection that aims to select the subset of discriminant features is an effective strategy to deal with large-sc…
New theorems show agents need specific internal structures to perform well under uncertainty.
problem How do agents need to be structured to perform well under uncertainty?
method Proved selection theorems showing strong task performance forces specific internal structures.
result Strong task performance forces world models, belief-like memory, and persistent regime-tracking variables.
Recomputation algorithms collectively refer to a family of methods that aims to reduce the memory consumption of the backpropagation by selectively discarding the intermediate results of the forward propagation and recomputing the discarded results as needed. In this paper, we will propose a novel and efficient recompu…
BAM integrates new data while selectively remembering past observations.
problem Slow adaptation and convergence to incorrect parameter values in non-stationary environments.
method Bayes' theorem with adaptive memory selection.
result BAM generalizes and demonstrates continuous adaptation in changing environments.
Benchmark evaluates LLM trading agents by masking identifiers to prevent memory leaks.
problem Evaluate LLM trading agents without relying on market memory or noise.
method Data-side masking protocol, Barra-style performance attribution framework.
result LLM agents' returns are largely explained by market and style exposure, not stock selection.
Probabilistic Quantum Memory (PQM) is a data structure that computes the distance from a binary input to all binary patterns stored in superposition on the memory. This data structure allows the development of heuristics to speed up artificial neural networks architecture selection. In this work, we propose an improved…
New method achieves superlinear convergence rate with limited memory.
problem Achieving superlinear convergence rate in quasi-Newton methods with limited memory.
method Limited-memory Greedy BFGS (LG-BFGS) method with displacement aggregation and basis vector selection.
result Explicit non-asymptotic superlinear convergence rate demonstrated.
HabitatAgent offers a multi-agent system for transparent housing consultation.
problem Opaque reasoning and brittle multi-constraint handling in housing recommendation systems.
method HabitatAgent is a multi-agent architecture with specialized roles for memory, retrieval, generation, and validation.
result HabitatAgent achieves 95% accuracy in real user consultation scenarios, significantly outperforming a strong baseline.
We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the prev…
New method for multi-modal depth prediction challenges.
problem Challenges in applying unimodal coreset selection to multi-modal data.
method Adapted state-of-the-art coreset selection technique for multimodal data.
result Challenges in extending unimodal algorithms to multi-modal scenarios.
SGQuant reduces GNN memory usage without significant accuracy loss.
problem High memory consumption in GNNs limits their applicability on memory-constrained devices.
method Proposes a specialized GNN quantization scheme (SGQuant) with a quantization algorithm, fine-tuning scheme, and multi-granularity strategy.
result SGQuant reduces GNN memory footprint from 4.25x to 31.9x with minimal accuracy loss.
Sequential learning of tasks using gradient descent leads to an unremitting decline in the accuracy of tasks for which training data is no longer available, termed catastrophic forgetting. Generative models have been explored as a means to approximate the distribution of old tasks and bypass storage of real data. Here …
SAGE improves memory efficiency by selectively adding, merging, or ignoring new facts.
problem Efficiently managing new facts in agentic LLMs to avoid costly write-time reasoning.
method SAGE uses a von Mises-Fisher-based density estimator to score and route candidate facts.
result SAGE achieves the best average token-F1 on LoCoMo and reduces add-phase API cost by 3.4x on GPT-4o-mini.
Study characterizes memory capacity of quantum reservoirs using transmon qubits.
problem Understanding the memory capacity of quantum reservoirs built with transmon qubits.
method Characterized memory capacity of quantum reservoirs using transmon qubits from IBM, focusing on NMSE and topology complexity.
result Found a peak in memory capacity for configurations with n-1 self-loops, suggesting optimal design for forecasting tasks.
We show that fMRI analysis using machine learning tools are sufficient to distinguish valence (i.e., positive or negative) of freely retrieved autobiographical memories in a cross-participant setting. Our methodology uses feature selection (ReliefF) in combination with boosting methods, both applied directly to data re…
Novel method learns memory kernels in Langevin equations.
problem Estimating memory kernels in Langevin equations.
method Regularized Prony method for correlation functions, followed by regression over Sobolev norm-based loss function with RKHS regularization.
result Method outperforms other regression estimators in exponentially weighted L^2 space.
Paper proposes efficient optimizers for large language models with fast convergence and low memory usage.
problem Designing efficient optimizers for large language models with low-memory requirements and fast convergence.
method Structured Fisher information matrix approximation and low-rank extension framework.
result New optimizers (RACS and Alice) achieve better convergence and lower memory usage than existing methods.
We propose a general method to obtain approximation of the first passage time distribution for the birth-death processes. We rely on the general properties of birth-death processes, Keilson's theorem and the concept of Riemann sum to obtain closed-form expressions. We apply the method to the three selected birth-death …
The study proves necessary conditions for robust decision-making in uncertain environments.
problem Conditions for robust decision-making in uncertain environments.
method Quantitative selection theorems and binary betting decisions.
result World models, belief-like memory, and persistent variables are necessary for strong task performance.
The Unlearning algorithm improves neural network performance in memory tasks.
problem Improving neural network performance in memory tasks.
method Simplified Unlearning algorithm, structured training data, and new regularization technique for Boltzmann Machines.
result The Unlearning rule outperforms traditional Boltzmann-Machine learning in learning hidden probability distributions.
Research in automatic affect recognition has seldom addressed the issue of computational resource utilization. With the advent of ambient intelligence technology which employs a variety of low-power, resource-constrained devices, this issue is increasingly gaining interest. This is especially the case in the context of…
Algorithm improves RL model selection for repeated games with utility maximization.
problem Optimal policy learning in repeated games with unknown opponent strategy.
method Proposes MRBEAR for average reward RL, applying to utility maximization in repeated games.
result Regret bound shows linear dependence on number of model classes in average reward RL.
Recent empirical results on long-term dependency tasks have shown that neural networks augmented with an external memory can learn the long-term dependency tasks more easily and achieve better generalization than vanilla recurrent neural networks (RNN). We suggest that memory augmented neural networks can reduce the ef…
Safe screening rule improves Group SLOPE efficiency.
problem Efficiently selecting groups of predictors in high-dimensional sparse learning.
method Safe screening rule for Group SLOPE, addressing block non-separable group effects.
result Significant computational efficiency gains without sacrificing accuracy.
Improved deep learning model deployment on tiny MCUs with mixed-precision quantization.
problem Memory limitations prevent accurate deployment of DNN models on tiny MCUs.
method Automated mixed-precision quantization using Reinforcement Learning for MCU constraints.
result Mixed-precision models achieve high accuracy with uniform quantization policies.
Study reduces memory needs for active learning with enriched queries.
problem Expensive labeling costs in active learning.
method Introduces bounded memory active learning through enriched queries, introduces lossless sample compression.
result Can learn classifiers with bounded memory and query optimality.
AlphaGrad optimizes memory usage in RL algorithms by normalizing gradients.
problem Memory overhead and hyperparameter complexity in adaptive optimizers.
method Tensor-wise L2 normalization followed by a smooth hyperbolic tangent transformation controlled by a single parameter.
result AlphaGrad provides enhanced training stability and competitive performance in various RL algorithms.
Hydra boosts efficiency for long-context reasoning in resource-constrained settings.
problem Quadratic complexity of transformers limits long-context reasoning in resource-constrained systems.
method Hydra uses a modular architecture with adaptive routing between sparse global attention, mixture-of-experts, and dual memories.
result Hydra achieves significant throughput and accuracy improvements for long-context reasoning.
Unified statistical framework for LSTM model selection.
problem Model selection and hyperparameter tuning in LSTM networks is heuristic and computationally expensive.
method Proposes a statistical framework extending classical model selection ideas to LSTM networks.
result Improved performance of the proposed framework demonstrated on biomedical data.
Paper proposes new method for time series confidence intervals using LSTM.
problem Constructing accurate confidence intervals for multivariate time series.
method Uses Long Short Term Memory Network (LSTM) and novel block bootstrap techniques.
result Demonstrates improved accuracy in constructing confidence intervals.
We study finite sample properties of estimators of power-law cross-correlations -- detrended cross-correlation analysis (DCCA), height cross-correlation analysis (HXA) and detrending moving-average cross-correlation analysis (DMCA) -- with a special focus on short-term memory bias as well as power-law coherency. Presen…
Efficiently processes high res images by selecting relevant patches.
problem High memory and compute requirements for processing large images.
method Differentiable Top-K operator to select relevant patches.
result End-to-end trainable model using backpropagation.
Spectrum selectively trains LLMs based on SNR to save resources.
problem Efficiently training large language models with limited resources.
method Targeting layer modules based on SNR for selective training.
result Spectrum achieves similar performance to full fine-tuning but with reduced VRAM usage.
Paper presents methods to create stock price confidence intervals using LSTM models.
problem Creating accurate confidence intervals for LSTM-estimated stock prices.
method Three bootstrap methods for dependent data, optimal block length selection, and benchmark comparison.
result Illustrated through stock price data, different bootstrap strategies provide varying confidence intervals.
We present apricot, an open source Python package for selecting representative subsets from large data sets using submodular optimization. The package implements an efficient greedy selection algorithm that offers strong theoretical guarantees on the quality of the selected set. Two submodular set functions are impleme…