In this paper, we show that Generative Adversarial Networks (GANs) suffer from catastrophic forgetting even when they are trained to approximate a single target distribution. We show that GAN training is a continual learning problem in which the sequence of changing model distributions is the sequence of tasks to the d…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “catastrophic forgetting prevention”
Synthetic data helps prevent forgetting when learning sequentially.
problem Catastrophic forgetting in neural networks.
method Generate synthetic data via two-step optimisation process using meta-gradients.
result Training on synthetic data prevents forgetting when learning sequentially.
Overparameterization helps prevent forgetting in sequential learning tasks.
problem Catastrophic forgetting in continual learning systems.
method Analytical study of gradient descent with linear regression model.
result Overparameterization can mitigate forgetting in a two-task setting.
New method prevents forgetting in continual learning tasks.
problem Catastrophic forgetting in continual learning.
method Generative regularization integrated into Bayesian inference framework.
result Empirically outperforms state-of-the-art methods on various datasets.
Paper proposes ensemble methods to prevent forgetting in neural networks.
problem Catastrophic forgetting in retraining neural networks.
method Gradient boosting and meta-learning approaches.
result Prevents forgetting in pre-trained neural network models.
Proposes a new method to prevent forgetting in neural networks.
problem Catastrophic forgetting in neural networks.
method Total absolute signal passing through each connection.
result Simpler and closer to brain processes.
Paper tackles catastrophic forgetting in sequential learning.
problem Catastrophic forgetting in sequential learning.
method Regularizes training with sketches of Jacobian matrix of past data.
result Proves overcoming catastrophic forgetting for linear and wide neural networks.
Improves neural networks' ability to learn new tasks without forgetting earlier ones.
problem Preventing catastrophic forgetting in neural networks.
method Introducing a second discriminator in the GAN to generate important features for task retention.
result Significant reduction in catastrophic forgetting compared to standard methods.
EWC helps prevent forgetting in neural networks by adjusting weights dynamically.
problem Preventing forgetting in neural networks during training.
method EWC adjusts weights dynamically to prevent forgetting.
result EWC effectively prevents catastrophic forgetting in neural networks.
AI learns to learn sequentially without forgetting.
problem Preventing catastrophic forgetting in machine learning models.
method Meta-learning a neuromodulatory activation-gating function to control selective activation in deep neural networks.
result State-of-the-art continual learning performance with 600 classes (9,000 updates).
Dynamic memory prevents forgetting in continuous learning of medical images.
problem Catastrophic forgetting in machine learning models over time due to domain shifts.
method Dynamic memory to store and replay diverse training data subsets.
result Dynamic memory mitigates forgetting without knowing when shifts occur.
Residual Continual Learning prevents forgetting in sequential tasks.
problem Preventing catastrophic forgetting in sequential learning of multiple tasks.
method ResCL reparameterizes network parameters by combining original and fine-tuned networks, keeping network size constant.
result ResCL achieves state-of-the-art performance in various continual learning scenarios.
Deep neural networks are known to suffer the catastrophic forgetting problem, where they tend to forget the knowledge from the previous tasks when sequentially learning new tasks. Such failure hinders the application of deep learning based vision system in continual learning settings. In this work, we present a simple …
A new method prevents forgetting during knowledge transfer.
problem Catastrophic forgetting in transfer learning.
method Transfer without Forgetting (TwF) using a fixed pretrained network.
result TwF outperforms other CL methods by 4.81% in Class-Incremental accuracy.
ER-GNN uses experience replay to prevent GNNs from forgetting previous tasks.
problem Catastrophic forgetting in GNNs when learning multiple tasks sequentially.
method Experience Replay framework to store and replay knowledge from previous tasks.
result ER-GNN effectively mitigates catastrophic forgetting in GNNs.
EVCL combines VCL and EWC to prevent forgetting new tasks.
problem Preventing catastrophic forgetting in continual learning.
method Hybrid model integrating VCL and EWC.
result Consistently outperforms baselines in learning new tasks.
Sparse routing networks with co-training prevent catastrophic forgetting in continual learning.
problem Catastrophic forgetting in neural networks trained on a sequence of tasks.
method Sparse routing networks with co-training to minimize interference between dissimilar tasks.
result Sparse routing networks with co-training outperform densely connected networks on benchmarks.
Continual learning is the ability to sequentially learn over time by accommodating knowledge while retaining previously learned experiences. Neural networks can learn multiple tasks when trained on them jointly, but cannot maintain performance on previously learned tasks when tasks are presented one at a time. This pro…
Study examines how training regime affects neural networks' forgetting.
problem Catastrophic forgetting in neural networks when learning multiple tasks sequentially.
method Analyzes the impact of different training regimes (learning rate, batch size, regularization) on forgetting.
result Training regimes that widen tasks' local minima help prevent catastrophic forgetting.
New method prevents deep learning forgetting past by remembering key examples.
problem Catastrophic forgetting in continual learning.
method Functional regularisation using Gaussian Process formulation.
result Achieves state-of-the-art performance on benchmarks.
Catastrophic forgetting is the notorious vulnerability of neural networks to the change of the data distribution while learning. This phenomenon has long been considered a major obstacle for allowing the use of learning agents in realistic continual learning settings. A large body of continual learning research assumes…
Paper analyzes convergence of continual learning with adaptive methods.
problem Preventing catastrophic forgetting in sequential learning tasks.
method Adaptive method for nonconvex continual learning (NCCL) adjusts step sizes of previous and current tasks.
result Proposed adaptive method achieves same convergence rate as SGD when catastrophic forgetting is suppressed.
LaLoRA prevents forgetting in LoRA fine-tuning.
problem Catastrophic forgetting in fine-tuned models.
method LaLoRA applies Laplace approximation to LoRA weights for regularization.
result Improved learning-forgetting trade-off with controllable regularization strength.
Adam optimizer leads to more forgetting in neural networks.
problem Understanding and quantifying catastrophic forgetting in neural networks.
method Comparative analysis of various optimization algorithms and metrics in different learning scenarios.
result Adam optimizer causes more forgetting compared to classical algorithms like SGD.
Paper introduces Auto DeepVis to explain catastrophic forgetting in continual learning.
problem Catastrophic forgetting in continual learning of deep neural networks.
method Auto DeepVis and critical freezing techniques to address catastrophic forgetting.
result Critical freezing outperforms other methods on both past and future tasks.
Bayesian online meta-learning framework tackles catastrophic forgetting in few-shot classification.
problem Catastrophic forgetting in few-shot classification problems.
method Bayesian online learning, meta-learning, Laplace approximation, variational inference.
result Framework effectively achieves goal of overcoming catastrophic forgetting in few-shot classification.
VAR-GPs solve continual learning by updating posteriors sequentially.
problem Catastrophic forgetting in sequential learning tasks.
method Sparse inducing point approximations and auto-regressive variational distribution.
result VAR-GPs prevent catastrophic forgetting and outperform baselines.
In supervised machine learning, an agent is typically trained once and then deployed. While this works well for static settings, robots often operate in changing environments and must quickly learn new things from data streams. In this paradigm, known as streaming learning, a learner is trained online, in a single pass…
Despite all the success that deep neural networks have seen in classifying certain datasets, the challenge of finding optimal solutions that generalize still remains. In this paper, we propose the Boundary Optimizing Network (BON), a new approach to generalization for deep neural networks when used for supervised learn…
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
This paper describes some of the possibilities of artificial neural networks that open up after solving the problem of catastrophic forgetting. A simple model and reinforcement learning applications of existing methods are also proposed.
This work tackles catastrophic forgetting in neural networks by mimicking brain's metaplasticity.
problem Catastrophic forgetting in neural networks, where new tasks erase previously learned ones.
method Interpreting binarized neural networks as metaplastic systems, adjusting their training technique.
result Training technique reduces catastrophic forgetting without needing previously presented data.
Catastrophic forgetting of connectionist neural networks is caused by the global sharing of parameters among all training examples. In this study, we analyze parameter sharing under the conditional computation framework where the parameters of a neural network are conditioned on each input example. At one extreme, if e…
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
Artificial neural networks (ANNs) suffer from catastrophic forgetting when trained on a sequence of tasks. While this phenomenon was studied in the past, there is only very limited recent research on this phenomenon. We propose a method for determining the contribution of individual parameters in an ANN to catastrophic…
Adversarial Feature Alignment: Avoid Catastrophic Forgetting in Incremental Task Lifelong Learningcs.LG
Human beings are able to master a variety of knowledge and skills with ongoing learning. By contrast, dramatic performance degradation is observed when new tasks are added to an existing neural network model. This phenomenon, termed as \emph{Catastrophic Forgetting}, is one of the major roadblocks that prevent deep neu…
SpaRCe optimizes reservoir computing by learning neuron thresholds to improve performance and prevent forgetting.
problem Improving performance and preventing forgetting in reservoir computing networks.
method Integrates neuron-specific learnable thresholds to optimize sparsity without altering dynamics, learning read-out weights and thresholds via gradient rule.
result Threshold learning improves performance and alleviates catastrophic forgetting.
Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distrib…
Continual learning is hard for AI, leading to forgetting old knowledge.
problem Catastrophic forgetting in AI when learning new data.
method Review of continual learning in deep learning.
result Challenges and insights in continual learning.
A new method combines generative models and regularization to prevent forgetting in continual learning.
problem Catastrophic forgetting in neural networks when learning new tasks.
method Uses a normalizing flow as a generative model to keep past data embeddings and regularize them.
result Performs favorably compared to existing methods with constant memory overhead.
Most artificial intelligence models have limiting ability to solve new tasks faster, without forgetting previously acquired knowledge. The recently emerging paradigm of continual learning aims to solve this issue, in which the model learns various tasks in a sequential fashion. In this work, a novel approach for contin…
Though neural networks have achieved much progress in various applications, it is still highly challenging for them to learn from a continuous stream of tasks without forgetting. Continual learning, a new learning paradigm, aims to solve this issue. In this work, we propose a new model for continual learning, called Ba…
SPACE algorithm prevents forgetting in neural networks by partitioning learned knowledge.
problem Catastrophic Forgetting in neural networks when learning new tasks.
method Partitioned learning space into Core and Residual spaces, analyzing Residual for redundancy and adding necessary dimensions to Core.
result Comparable accuracy to state-of-the-art methods while overcoming catastrophic forgetting.
New method prevents deep learning models from forgetting past tasks.
problem Catastrophic forgetting in continual learning.
method Direction-constrained optimization (DCO) with autoencoders.
result Model performance is improved without forgetting past tasks.
Catastrophic forgetting/interference is a critical problem for lifelong learning machines, which impedes the agents from maintaining their previously learned knowledge while learning new tasks. Neural networks, in particular, suffer plenty from the catastrophic forgetting phenomenon. Recently there has been several eff…
OGD proves robustness to Catastrophic Forgetting in Continual Learning.
problem Catastrophic Forgetting in Continual Learning with deep neural networks.
method Theoretical framework based on Neural Tangent Kernel for OGD.
result First generalization bound for SGD and OGD in Continual Learning.
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.