Memory split advantage: thinner networks outperform a single wide network.
problem Optimizing deep learning models with limited memory.
method Investigated training a single wide network vs. an ensemble of thinner networks with the same total number of parameters.
result An ensemble of several thinner networks outperforms a single wide network for large memory budgets.
New memory models improve LSSVM's generalization and reduce time cost.
problem Improving LSSVM's generalization performance and reducing training time.
method Introducing MIMM and WIMM models that modify LSSVM's memory influence mechanism.
result MIMM and WIMM models outperform LSSVM in generalization and time efficiency.
In distributed reinforcement learning, it is common to exchange the experience memory of each agent and thereby collectively train their local models. The experience memory, however, contains all the preceding state observations and their corresponding policies of the host agent, which may violate the privacy of the ag…
Product Kanerva Machines dynamically combine smaller models for better memory organization.
problem Limited organization in the Kanerva Machine.
method Introducing Product Kanerva Machines that dynamically combine multiple smaller Kanerva Machines.
result Product Kanerva Machines can discover spatial tunings that approximately factorize simple images by object.
Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing pop…
We propose a new architecture called Memory-Augmented Encoder-Solver (MAES) that enables transfer learning to solve complex working memory tasks adapted from cognitive psychology. It uses dual recurrent neural network controllers, inside the encoder and solver, respectively, that interface with a shared memory module a…
LSTMs improve bond yield forecasting with unique signals.
problem Improving bond yield forecasting accuracy.
method Long short-term memory (LSTM) networks with sequence-to-sequence architectures and LSTM-LagLasso methodology.
result Univariate LSTM models with additional memory can achieve similar results as multivariate MLP models using exogenous information.
SliceOut speeds up deep learning training without sacrificing accuracy.
problem Frequent model re-training and large model training workloads in deep learning.
method SliceOut uses dropout-inspired scheme to drop contiguous sets of units at random, leveraging GPU memory layout.
result 10-40% speedups and memory reduction with minimal accuracy loss.
Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize t…
Novel method learns memory kernels in Langevin equations.
problem Estimating memory kernels in Langevin equations.
method Regularized Prony method for correlation functions, followed by regression over Sobolev norm-based loss function with RKHS regularization.
result Method outperforms other regression estimators in exponentially weighted L^2 space.
Principal components analysis (PCA) is a fundamental algorithm in data analysis. Its memory-restricted online versions are useful in many modern applications, where the data are too large to fit in memory, or when data arrive as a stream of items. In this paper, we propose ROIPCA and fROIPCA, two online PCA algorithms …
A new method reduces memory usage for deep neural networks ensembles.
problem Reducing memory footprint of ensemble methods for deep learning.
method Extract multiple sub-networks from a single neural network through end-to-end optimization.
result Achieves higher or comparable accuracy with significantly less memory usage.
Investor optimizes investment timing with future knowledge, overcoming transaction costs.
problem Optimal investment timing with future peeking, constrained by transaction costs.
method Solves control problem with infinite-dimensional memory using Gaussian Volterra integral equations.
result Explicit solution to optimal investment problem in Bachelier setting.
Long memory and volatility clustering are two stylized facts frequently related to financial markets. Traditionally, these phenomena have been studied based on conditionally heteroscedastic models like ARCH, GARCH, IGARCH and FIGARCH, inter alia. One advantage of these models is their ability to capture nonlinear dynam…
AlphaGrad optimizes memory usage in RL algorithms by normalizing gradients.
problem Memory overhead and hyperparameter complexity in adaptive optimizers.
method Tensor-wise L2 normalization followed by a smooth hyperbolic tangent transformation controlled by a single parameter.
result AlphaGrad provides enhanced training stability and competitive performance in various RL algorithms.
Sparse codes improve optimal control tasks with correlated inputs.
problem Optimal control tasks with correlated feature inputs.
method Used a sparse code to represent natural images in an optimal control task solved with neuro-dynamic programming.
result An over-complete sparse code increases memory capacity and learning speed beyond a complete code.
This work investigates power laws in deep neural network ensembles and predicts their performance.
problem Understanding the performance of deep neural network ensembles and their optimal structure.
method Investigated the behavior of negative log-likelihood (CNLL) of a deep ensemble as a function of ensemble size and member network size, identifying power law dependencies.
result One large network may perform worse than an ensemble of several medium-size networks, known as a memory split.
In neural architecture search (NAS), the space of neural network architectures is automatically explored to maximize predictive accuracy for a given task. Despite the success of recent approaches, most existing methods cannot be directly applied to large scale problems because of their prohibitive computational complex…
Inspired by dynamic programming, we propose Stochastic Virtual Gradient Descent (SVGD) algorithm where the Virtual Gradient is defined by computational graph and automatic differentiation. The method is computationally efficient and has little memory requirements. We also analyze the theoretical convergence properties …
With the abundance of data in recent years, interesting challenges are posed in the area of recommender systems. Producing high quality recommendations with scalability and performance is the need of the hour. Singular Value Decomposition(SVD) based recommendation algorithms have been leveraged to produce better result…
In this paper, we consider recommender systems with side information in the form of graphs. Existing collaborative filtering algorithms mainly utilize only immediate neighborhood information and have a hard time taking advantage of deeper neighborhoods beyond 1-2 hops. The main caveat of exploiting deeper graph informa…
Study identifies three quantization regimes for ReLU networks.
problem Approximation of Lipschitz functions by ReLU networks with finite-precision weights.
method Established through nonasymptotic tight lower and upper bounds on minimax approximation error.
result Memory-optimality achieved in proper quantization regime for deep networks.
Paper improves GNN inference speed and memory usage.
problem Scalability issues in GNNs during inference.
method Graph coarsening techniques for faster inference.
result Significant reduction in inference time and memory usage.
BAM integrates new data while selectively remembering past observations.
problem Slow adaptation and convergence to incorrect parameter values in non-stationary environments.
method Bayes' theorem with adaptive memory selection.
result BAM generalizes and demonstrates continuous adaptation in changing environments.
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
RSmote improves PINNs accuracy with less memory usage.
problem Imbalanced learning in Physics-Informed Neural Networks (PINNs).
method Residual-based Smote (RSmote) for local adaptive sampling.
result RSmote achieves or exceeds accuracy of state-of-the-art methods while reducing memory usage.
Global convergence of an online (stochastic) limited memory version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established. Lower and upper bounds on the Hessian eigenvalues of the sample …
GPG improves RL from feedback with less memory and compute.
problem Improving reinforcement learning from human feedback.
method Group-based Monte Carlo advantage estimator replacing learned value function.
result GPG matches or outperforms PPO on standard benchmarks.
Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also …
Different neural network (NN) architectures have different advantages. Convolutional neural networks (CNNs) achieved enormous success in computer vision, while recurrent neural networks (RNNs) gained popularity in speech recognition. It is not known which type of NN architecture is the best fit for classification of co…
Deep RNNs excel at capturing long-term dependencies in sequential data.
problem Lack of a formal measure for RNNs' long-term memory capacity.
method Introduced a measure called Start-End separation rank to quantify RNNs' ability to model long-term dependencies.
result Deep RNNs support Start-End separation ranks that are combinatorially higher than shallow ones.
Quantum-assisted VAE improves similarity search in high-dimensional datasets.
problem Finding fast and memory-efficient similarity search in high-dimensional data.
method Construct a space-efficient search index based on the latent space of a Quantum-assisted Variational Autoencoder (QVAE).
result Real-world speedups and memory-efficient scaling to half a billion data points.
New model captures long-range patterns in sequences efficiently.
problem Efficiently capturing long-range patterns in sequential data.
method Inspired by wavelet multiresolution analysis, introduces MultiresLayer with multiresolution convolution.
result State-of-the-art performance on sequence classification and autoregressive density estimation tasks.
HFformer outperforms LSTM in high-frequency trading with multiple signals.
problem Improving high-frequency trading performance using deep learning models.
method Introducing HFformer, a hybrid Transformer model for time series forecasting.
result HFformer achieves higher cumulative PnL than LSTM in backtesting.
A method for learning complex functions from data with reduced memory usage.
problem Learning highly nonlinear, multivariate functions from examples.
method Transforming function learning into tensor reconstruction, incrementally building tensors from rank-one terms.
result Efficient gradient-based algorithm with linear time complexity in sample size and tensor dimensions.
We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is…
The paper tackles scalability issues in Graph Representation Learning.
problem Prohibitive time and memory complexities in Graph Representation Learning.
method Leveraging the K-Core Decomposition property of Graphs to reduce time and memory consumption.
result Proposed techniques significantly reduce computational resources without compromising embedding quality.
Muon spectral optimizer outperforms SGD in associative memory tasks.
problem Understanding the advantage of spectral optimizers in learning associative memory.
method Linear associative memory problem, Gaussian inputs and outputs, power law frequency distribution, thresholded gradient approximation.
result Muon significantly outperforms SGD in storage capacity and recovery rates.
We extend the randomized singular value decomposition (SVD) algorithm \citep{Halko2011finding} to estimate the SVD of a shifted data matrix without explicitly constructing the matrix in the memory. With no loss in the accuracy of the original algorithm, the extended algorithm provides for a more efficient way of matrix…
We introduce interacting particle Markov chain Monte Carlo (iPMCMC), a PMCMC method based on an interacting pool of standard and conditional sequential Monte Carlo samplers. Like related methods, iPMCMC is a Markov chain Monte Carlo sampler on an extended space. We present empirical results that show significant improv…
Adaptive synchronization improves deep reinforcement learning performance.
problem Fixed step size synchronization can cause loss of properly learned networks.
method Adaptive synchronization based on recent agent behavior.
result Adaptive synchronization leads to better performance in games.
A new method for estimating SW from streaming data.
problem Estimating Wasserstein distance from sample streams.
method Introducing a streaming estimator of the 1DW and applying it to all projections.
result Stream-SW achieves more accurate approximation of SW than random subsampling.
PES method reduces bias in gradient estimation for unrolled graphs.
problem High variance and bias in gradient estimation for unrolled computation graphs.
method Divide graph into unrolls, apply ES update, accumulate correction terms.
result PES provides unbiased, low-variance gradient estimates.
We show that a simple modification of the 1-nearest neighbor classifier yields a strongly Bayes consistent learner. Prior to this work, the only strongly Bayes consistent proximity-based method was the k-nearest neighbor classifier, for k growing appropriately with sample size. We will argue that a margin-regularized 1…
Quantum networks offer exponential communication savings for large machine learning models.
problem Training and inference of large models require efficient communication.
method Quantum encoding and gradient descent for distributed computation.
result Exponential reduction in communication for gradient descent on quantum networks.
FF layers in transformers are nearly as interpretable as sparse autoencoders.
problem Comparing interpretability of feature vectors in FF layers vs. sparse autoencoders.
method Revisited interpretability of FF layers as key-value memories using modern benchmarks.
result FF and SAE feature vectors are similarly interpretable, but FFs can be better in some aspects.
HAMN combines CF models to improve drug repositioning.
problem Efficient drug repositioning with cold start problem.
method Hybrid Attentional Memory Network (HAMN) integrating memory and attention mechanisms.
result HAMN outperforms other models in drug repositioning tasks.
Neural RDEs extend CDEs to irregular time series.
problem Modeling long irregular time series efficiently.
method Representing time series through log-signature and solving RDEs.
result Significant training speed-ups and improved model performance.