Adam optimizer leads to more forgetting in neural networks.
problem Understanding and quantifying catastrophic forgetting in neural networks.
method Comparative analysis of various optimization algorithms and metrics in different learning scenarios.
result Adam optimizer causes more forgetting compared to classical algorithms like SGD.
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
New theory explains forgetting in learning algorithms.
problem Tendency of learning algorithms to forget past knowledge.
method Proposes a self-consistency theory of forgetting as a loss of predictive information.
result Exact Bayesian inference allows for adaptation without forgetting.
The paper proposes selective forgetting for deep neural networks at a finer level than samples.
problem Selective forgetting of deep neural networks to handle outliers, poisoned data, or sensitive information.
method Formulated selective forgetting at a finer level than samples, introduced as an optimization problem on three criteria.
result Experimental results show the model can forget specific information for classification, improving accuracy in specific cases.
The paper analyzes how forgetting in LLMs is linked to simple task-upstream example associations.
problem Forgetting of upstream knowledge in fine-tuned LLMs.
method Empirical analysis of forgotten examples in N upstream examples after M new tasks, using low-rank matrix approximation. result Forgetting can be predicted efficiently using matrix completion over empirical associations.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
Bayesian online meta-learning framework tackles catastrophic forgetting in few-shot classification.
problem Catastrophic forgetting in few-shot classification problems.
method Bayesian online learning, meta-learning, Laplace approximation, variational inference.
result Framework effectively achieves goal of overcoming catastrophic forgetting in few-shot classification.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
Paper tackles catastrophic forgetting in sequential learning.
problem Catastrophic forgetting in sequential learning.
method Regularizes training with sketches of Jacobian matrix of past data.
result Proves overcoming catastrophic forgetting for linear and wide neural networks.
This work tackles catastrophic forgetting in neural networks by mimicking brain's metaplasticity.
problem Catastrophic forgetting in neural networks, where new tasks erase previously learned ones.
method Interpreting binarized neural networks as metaplastic systems, adjusting their training technique.
result Training technique reduces catastrophic forgetting without needing previously presented data.
Overparameterization helps prevent forgetting in sequential learning tasks.
problem Catastrophic forgetting in continual learning systems.
method Analytical study of gradient descent with linear regression model.
result Overparameterization can mitigate forgetting in a two-task setting.
Interpreting the behaviors of Deep Neural Networks (usually considered as a black box) is critical especially when they are now being widely adopted over diverse aspects of human life. Taking the advancements from Explainable Artificial Intelligent, this paper proposes a novel technique called Auto DeepVis to dissect c…
Continual learning is hard for AI, leading to forgetting old knowledge.
problem Catastrophic forgetting in AI when learning new data.
method Review of continual learning in deep learning.
result Challenges and insights in continual learning.
Study examines how training regime affects neural networks' forgetting.
problem Catastrophic forgetting in neural networks when learning multiple tasks sequentially.
method Analyzes the impact of different training regimes (learning rate, batch size, regularization) on forgetting.
result Training regimes that widen tasks' local minima help prevent catastrophic forgetting.
Hybrid framework prevents forgetting in continual learning.
problem Avoiding forgetting in learning new tasks without forgetting old ones.
method Hybrid continual learning framework combining architecture growth and experience replay.
result Hybrid approach effectively avoids forgetting across multiple tasks.
Adaptive model learns from time series data with changing distributions.
problem Predicting time series data under distribution shift.
method Formulates distribution shift as weighted empirical risk minimization. Uses a gradient-based learning method for a forgetting mechanism.
result Proposes an efficient method for adaptive time series prediction.
Study on neural networks forgetting in continual learning.
problem Forgetting in neural networks during continual learning.
method Gradient descent analysis on XOR-cluster datasets.
result Explicit bounds on forgetting rate and generalization gap.
In this paper, we show that Generative Adversarial Networks (GANs) suffer from catastrophic forgetting even when they are trained to approximate a single target distribution. We show that GAN training is a continual learning problem in which the sequence of changing model distributions is the sequence of tasks to the d…
Synthetic data helps prevent forgetting when learning sequentially.
problem Catastrophic forgetting in neural networks.
method Generate synthetic data via two-step optimisation process using meta-gradients.
result Training on synthetic data prevents forgetting when learning sequentially.
A new method prevents forgetting during knowledge transfer.
problem Catastrophic forgetting in transfer learning.
method Transfer without Forgetting (TwF) using a fixed pretrained network.
result TwF outperforms other CL methods by 4.81% in Class-Incremental accuracy.
Holomorphic maps between moduli spaces are shown to be forgetful for large g.
problem Characterizing holomorphic maps between moduli spaces.
method Proving that only forgetful maps are non-constant for large g.
result Forgetful maps are the only non-constant holomorphic maps between moduli spaces for g≥4. Proposes a new method to prevent forgetting in neural networks.
problem Catastrophic forgetting in neural networks.
method Total absolute signal passing through each connection.
result Simpler and closer to brain processes.
Dynamic memory prevents forgetting in continuous learning of medical images.
problem Catastrophic forgetting in machine learning models over time due to domain shifts.
method Dynamic memory to store and replay diverse training data subsets.
result Dynamic memory mitigates forgetting without knowing when shifts occur.
Paper proposes a new pipeline for few-shot classification using forget-update module and channel vector sequence.
problem Few-shot classification with limited support samples.
method Channel vector sequence construction module and forget-update module.
result Pipeline achieves state-of-the-art results on various datasets.
Artificial neural networks (ANNs) suffer from catastrophic forgetting when trained on a sequence of tasks. While this phenomenon was studied in the past, there is only very limited recent research on this phenomenon. We propose a method for determining the contribution of individual parameters in an ANN to catastrophic…
AI learns to learn sequentially without forgetting.
problem Preventing catastrophic forgetting in machine learning models.
method Meta-learning a neuromodulatory activation-gating function to control selective activation in deep neural networks.
result State-of-the-art continual learning performance with 600 classes (9,000 updates).
Dropout helps a stable network learn new tasks without forgetting old ones.
problem Catastrophic forgetting in neural networks when learning multiple tasks.
method Investigate the relationship between dropout and stability in neural networks, showing dropout acts as an implicit gating mechanism.
result Dropout stabilizes a network's learning, allowing it to learn new tasks without forgetting old ones.
Study on reducing forgetting in neural networks using compression theory.
problem Catastrophic forgetting in neural networks.
method Defined forgetting as increased description lengths, compared variational posterior approaches to prequential coding methods.
result Proposed a new continual learning method combining ML plug-in and Bayesian mixture codes.
We study forgetful maps between Deligne-Mostow moduli spaces of weighted points on P^1, and classify the forgetful maps that extend to a map of orbifolds between the stable completions. The cases where this happens include the Livné fibrations and the Mostow/Toledo maps between complex hyperbolic surfaces. They also in…
We quantify forgetting in post-training models, distinguishing mass and drift.
problem Understanding and preventing forgetting in post-training generative models.
method Developed theoretical results under a two-mode mixture abstraction, formalizing mass and drift forgetting.
result Forgetting can be precisely quantified based on divergence direction, geometric overlap, and training regime.
New insights into neural network forgetting reveal a trade-off between node activation and re-use.
problem Challenges in maintaining performance on old tasks while learning new ones.
method Theoretical analysis of synthetic and real data setups, focusing on node activation vs re-use.
result Worst forgetting occurs in an intermediate similarity regime between learned tasks.
Unified study of stateful replay for streaming learning, reducing forgetting by 2-3x.
problem Catastrophic forgetting in streaming generative and predictive learning.
method Unified analysis of stateful replay for autoencoding, forecasting, and classification tasks.
result Stateful replay reduces average forgetting by a factor of 2-3 on heterogeneous multi-task streams.
Model forgets examples; this research predicts which ones to replay.
problem Language models forget examples during updates, leading to errors.
method Train forecasting models to predict which examples will be forgotten.
result Forecasting models can reduce forgetting of upstream pretraining examples.
Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models "forget" how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to…
GANs involve training two networks in an adversarial game, where each network's task depends on its adversary. Recently, several works have framed GAN training as an online or continual learning problem. We focus on the discriminator, which must perform classification under an (adversarially) shifting data distribution…
This paper describes some of the possibilities of artificial neural networks that open up after solving the problem of catastrophic forgetting. A simple model and reinforcement learning applications of existing methods are also proposed.
Hybrid model avoids forgetting across tasks and classes.
problem Challenges of continual learning in modern deep learning.
method Hybrid generative-discriminative approach using normalizing flows.
result Strong performance on various continual learning benchmarks.
We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn in…
EVCL combines VCL and EWC to prevent forgetting new tasks.
problem Preventing catastrophic forgetting in continual learning.
method Hybrid model integrating VCL and EWC.
result Consistently outperforms baselines in learning new tasks.
Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occur…
LaLoRA prevents forgetting in LoRA fine-tuning.
problem Catastrophic forgetting in fine-tuned models.
method LaLoRA applies Laplace approximation to LoRA weights for regularization.
result Improved learning-forgetting trade-off with controllable regularization strength.
EWC helps prevent forgetting in neural networks by adjusting weights dynamically.
problem Preventing forgetting in neural networks during training.
method EWC adjusts weights dynamically to prevent forgetting.
result EWC effectively prevents catastrophic forgetting in neural networks.
Bayesian inference forgetting framework removes influence of single data points.
problem Enforcement of the right to be forgotten in machine learning causes high costs for companies.
method Develops forgetting algorithms for variational and Markov chain Monte Carlo in Bayesian inference.
result Proves removal of influence of single datums on learned models with guaranteed generalizability.
Pseudo-rehearsal allows neural networks to learn a sequence of tasks without forgetting how to perform in earlier tasks. Preventing forgetting is achieved by introducing a generative network which can produce data from previously seen tasks so that it can be rehearsed along side learning the new task. This has been fou…
The brain optimizes memory by forgetting what's predictable, improving generalization.
problem Memory consolidation struggles with representational drift, semanticisation, and offline replay.
method Proposes predictive forgetting as a mechanism to optimize generalization by reducing complexity.
result Predictive forgetting improves information-theoretic generalization bounds on stored representations.
In recent years, deep neural networks have found success in replicating human-level cognitive skills, yet they suffer from several major obstacles. One significant limitation is the inability to learn new tasks without forgetting previously learned tasks, a shortcoming known as catastrophic forgetting. In this research…
Catastrophic forgetting/interference is a critical problem for lifelong learning machines, which impedes the agents from maintaining their previously learned knowledge while learning new tasks. Neural networks, in particular, suffer plenty from the catastrophic forgetting phenomenon. Recently there has been several eff…