The paper analyzes how forgetting in LLMs is linked to simple task-upstream example associations.
problem Forgetting of upstream knowledge in fine-tuned LLMs.
method Empirical analysis of forgotten examples in N upstream examples after M new tasks, using low-rank matrix approximation. result Forgetting can be predicted efficiently using matrix completion over empirical associations.
Study finds some examples are forgotten during neural network learning.
problem Understanding forgetting during neural network training.
method Defined forgetting event and analyzed across multiple data sets.
result High frequency of certain examples being forgotten, some not.
Model forgets examples; this research predicts which ones to replay.
problem Language models forget examples during updates, leading to errors.
method Train forecasting models to predict which examples will be forgotten.
result Forecasting models can reduce forgetting of upstream pretraining examples.
A study on preventing catastrophic forgetting in neural networks using conditional computation.
problem Catastrophic forgetting in connectionist neural networks.
method Conditional computation framework where parameters are conditioned on each input example.
result Proposed conditional rehearsal to prevent forgetting of previously learned examples.
New method prevents deep learning forgetting past by remembering key examples.
problem Catastrophic forgetting in continual learning.
method Functional regularisation using Gaussian Process formulation.
result Achieves state-of-the-art performance on benchmarks.
Paper tackles catastrophic forgetting in sequential learning.
problem Catastrophic forgetting in sequential learning.
method Regularizes training with sketches of Jacobian matrix of past data.
result Proves overcoming catastrophic forgetting for linear and wide neural networks.
Privacy-preserving continual learning for hospitals.
problem Forgetting old data when updating neural networks for privacy.
method Differentially private variational inference and generative models.
result Estimates likelihood of old data given current model.
Paper proposes mnemonics framework for MCIL without forgetting.
problem Learning new concepts without forgetting previous ones in MCIL.
method Automatic framework using bilevel optimizations for parameterizing and optimizing exemplars.
result Mnemonics exemplars surpass state-of-the-art performance.
Dynamic memory prevents forgetting in continuous learning of medical images.
problem Catastrophic forgetting in machine learning models over time due to domain shifts.
method Dynamic memory to store and replay diverse training data subsets.
result Dynamic memory mitigates forgetting without knowing when shifts occur.
GMED edits stored examples to improve continual learning.
problem Catastrophic forgetting in task-free continual learning.
method Gradient-based memory editing of stored examples.
result GMED-edited examples help prevent forgetting.
New method reduces forgetting in neural networks by focusing on representation rather than output accuracy.
problem Catastrophic forgetting in neural networks during sequential learning.
method Developed a novel meta-learning algorithm to reduce representational forgetting, improving representations on new tasks without losing old task representations.
result Meta-learner produces weight updates that maintain old task representations while improving new task learning.
ExStream improves memory efficiency for streaming learning.
problem Catastrophic forgetting in streaming learning.
method Memory-efficient experience replay using ExStream algorithm.
result ExStream performs well with less memory and computation.
Neural nets learn and forget tasks sequentially, showing promising scalability.
problem Learning and forgetting of multiple visual tasks in a sequential setting.
method Simulated sequential learning of ten related visual tasks.
result Neural nets show forward facilitation and backward interference, which are key phenomena.
Model learns continuously from text without forgetting.
problem Catastrophic forgetting in lifelong language learning.
method Episodic memory with sparse experience replay and local adaptation.
result Model can continuously learn from new datasets.
L2F learns to forget, improving few-shot learning performance.
problem Few-shot learning challenges in adapting to unseen tasks.
method Task-and-layer-wise attenuation on compromised initialization to reduce prior knowledge influence.
result Faster adaptation and improved performance demonstrated experimentally.
Adam optimizer leads to more forgetting in neural networks.
problem Understanding and quantifying catastrophic forgetting in neural networks.
method Comparative analysis of various optimization algorithms and metrics in different learning scenarios.
result Adam optimizer causes more forgetting compared to classical algorithms like SGD.
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
New theory explains forgetting in learning algorithms.
problem Tendency of learning algorithms to forget past knowledge.
method Proposes a self-consistency theory of forgetting as a loss of predictive information.
result Exact Bayesian inference allows for adaptation without forgetting.
The paper proposes selective forgetting for deep neural networks at a finer level than samples.
problem Selective forgetting of deep neural networks to handle outliers, poisoned data, or sensitive information.
method Formulated selective forgetting at a finer level than samples, introduced as an optimization problem on three criteria.
result Experimental results show the model can forget specific information for classification, improving accuracy in specific cases.
This research debiases machine unlearning by using counterfactual examples.
problem Machine unlearning processes can be biased, leading to inaccurate results.
method Intervention-based approach using counterfactual examples to mitigate biases.
result The method outperforms existing baselines on evaluation metrics.
Our method upweights easy samples to mitigate forgetting in fine-tuning.
problem Catastrophic forgetting in fine-tuning pre-trained models.
method Sample weighting based on pre-trained model's losses.
result Our method reduces forgetting by up to 0.8% on MetaMathQA while preserving more accuracy on pre-training datasets.
Study helps identify which ANN parameters cause forgetting.
problem Catastrophic forgetting in neural networks.
method Determines individual parameter contributions to forgetting.
result Identifies specific ANN parameters causing forgetting.
Paper introduces Auto DeepVis to explain catastrophic forgetting in continual learning.
problem Catastrophic forgetting in continual learning of deep neural networks.
method Auto DeepVis and critical freezing techniques to address catastrophic forgetting.
result Critical freezing outperforms other methods on both past and future tasks.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
New framework prevents forgetting in sequential learning tasks.
problem Catastrophic forgetting in Convolutional Neural Networks.
method Conditional channel gating modules and task classifier.
result Consistent improvement over existing methods in continual learning.
NADINE builds MLPs from streaming data, overcoming forgetting issues.
problem Building deep neural networks from streaming data efficiently and avoiding forgetting.
method NADINE uses a fully open MLP structure that dynamically evolves its depth and width online, resolving catastrophic forgetting through soft-forgetting and adaptive memory.
result NADINE outperforms existing methods in nine data stream classification and regression problems.
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
Bayesian online meta-learning framework tackles catastrophic forgetting in few-shot classification.
problem Catastrophic forgetting in few-shot classification problems.
method Bayesian online learning, meta-learning, Laplace approximation, variational inference.
result Framework effectively achieves goal of overcoming catastrophic forgetting in few-shot classification.
New approach for deep neural networks to learn invariance through adversarial forgetting.
problem Learning invariance for deep neural networks in the presence of nuisance and bias factors.
method Adversarial forgetting mechanism to induce amnesia to unwanted data factors.
result State-of-the-art performance in learning invariance across various datasets and tasks.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
Self-supervised GAN prevents forgetting in sequential tasks.
problem Discriminator forgetting in GANs leads to training instability.
method Add self-supervision to the discriminator to maintain useful representations.
result Self-supervised GAN outperforms regular GANs in learning better representations.
This work tackles catastrophic forgetting in neural networks by mimicking brain's metaplasticity.
problem Catastrophic forgetting in neural networks, where new tasks erase previously learned ones.
method Interpreting binarized neural networks as metaplastic systems, adjusting their training technique.
result Training technique reduces catastrophic forgetting without needing previously presented data.
Overparameterization helps prevent forgetting in sequential learning tasks.
problem Catastrophic forgetting in continual learning systems.
method Analytical study of gradient descent with linear regression model.
result Overparameterization can mitigate forgetting in a two-task setting.
Continual learning is hard for AI, leading to forgetting old knowledge.
problem Catastrophic forgetting in AI when learning new data.
method Review of continual learning in deep learning.
result Challenges and insights in continual learning.
Generative replay improves continual learning by using generated data as negative examples.
problem Catastrophic forgetting in continual learning.
method Using generative models to provide negative examples for new classes.
result Generative replay can improve learning new classes even when existing approaches fail.
Study examines how training regime affects neural networks' forgetting.
problem Catastrophic forgetting in neural networks when learning multiple tasks sequentially.
method Analyzes the impact of different training regimes (learning rate, batch size, regularization) on forgetting.
result Training regimes that widen tasks' local minima help prevent catastrophic forgetting.
A simple method to prevent forgetting in neural networks.
problem Catastrophic forgetting in neural networks.
method Weight friction, inspired by neurology and physics, modifies the gradient descent update rule.
result Weight friction enables continual learning with comparable performance and efficiency.
This work tackles continual learning with semi-supervised data, showing that even with minimal labeled data, performance can match full-supervised methods.
problem Training deep networks on a stream of tasks without forgetting, especially when labeled data is scarce.
method Designing a novel CSSL method that leverages metric learning and consistency regularization to learn from both labeled and unlabeled data.
result Our method outperforms state-of-the-art methods trained with full supervision, achieving comparable performance with only 25% labeled data.
Proposes a method to prevent neural networks from forgetting learned tasks.
problem Catastrophic forgetting in neural networks.
method Attention-based selective plasticity of synapses inspired by the cholinergic neuromodulatory system.
result Competitive performance on benchmark tasks compared to state-of-the-art methods.
Hybrid framework prevents forgetting in continual learning.
problem Avoiding forgetting in learning new tasks without forgetting old ones.
method Hybrid continual learning framework combining architecture growth and experience replay.
result Hybrid approach effectively avoids forgetting across multiple tasks.
Framework uses experience replay to prevent deep networks from forgetting past tasks.
problem Deep networks forget past tasks after learning new ones in sequential multitask learning.
method Generative model that couples current task with past learned tasks through a discriminative embedding space.
result Framework learns a shared abstract distribution across all tasks, preventing catastrophic forgetting.
Adaptive model learns from time series data with changing distributions.
problem Predicting time series data under distribution shift.
method Formulates distribution shift as weighted empirical risk minimization. Uses a gradient-based learning method for a forgetting mechanism.
result Proposes an efficient method for adaptive time series prediction.
Study on neural networks forgetting in continual learning.
problem Forgetting in neural networks during continual learning.
method Gradient descent analysis on XOR-cluster datasets.
result Explicit bounds on forgetting rate and generalization gap.
In this paper, we show that Generative Adversarial Networks (GANs) suffer from catastrophic forgetting even when they are trained to approximate a single target distribution. We show that GAN training is a continual learning problem in which the sequence of changing model distributions is the sequence of tasks to the d…
Improves neural networks' ability to learn new tasks without forgetting earlier ones.
problem Preventing catastrophic forgetting in neural networks.
method Introducing a second discriminator in the GAN to generate important features for task retention.
result Significant reduction in catastrophic forgetting compared to standard methods.
Synthetic data helps prevent forgetting when learning sequentially.
problem Catastrophic forgetting in neural networks.
method Generate synthetic data via two-step optimisation process using meta-gradients.
result Training on synthetic data prevents forgetting when learning sequentially.
FSR prevents deep nets from forgetting old tasks.
problem Catastrophic forgetting in deep neural networks.
method Logit matching on selected samples from old tasks.
result FSR outperforms previous methods in knowledge retention.
A new method prevents forgetting during knowledge transfer.
problem Catastrophic forgetting in transfer learning.
method Transfer without Forgetting (TwF) using a fixed pretrained network.
result TwF outperforms other CL methods by 4.81% in Class-Incremental accuracy.