Synthetic data helps prevent forgetting when learning sequentially.
problem Catastrophic forgetting in neural networks.
method Generate synthetic data via two-step optimisation process using meta-gradients.
result Training on synthetic data prevents forgetting when learning sequentially.
Unified study of stateful replay for streaming learning, reducing forgetting by 2-3x.
problem Catastrophic forgetting in streaming generative and predictive learning.
method Unified analysis of stateful replay for autoencoding, forecasting, and classification tasks.
result Stateful replay reduces average forgetting by a factor of 2-3 on heterogeneous multi-task streams.
Improves neural networks' ability to learn new tasks without forgetting earlier ones.
problem Preventing catastrophic forgetting in neural networks.
method Introducing a second discriminator in the GAN to generate important features for task retention.
result Significant reduction in catastrophic forgetting compared to standard methods.
New method reduces forgetting in neural networks by focusing on representation rather than output accuracy.
problem Catastrophic forgetting in neural networks during sequential learning.
method Developed a novel meta-learning algorithm to reduce representational forgetting, improving representations on new tasks without losing old task representations.
result Meta-learner produces weight updates that maintain old task representations while improving new task learning.
Proposes a CL technique to improve accuracy and reduce forgetting.
problem Sequential task learners struggle with forgetting information from previous tasks.
method Extracts modular parts of neural networks and estimates task relatedness.
result Remarkable performance gain in robustness to forgetting for EWC and GEM methods.
The paper analyzes how forgetting in LLMs is linked to simple task-upstream example associations.
problem Forgetting of upstream knowledge in fine-tuned LLMs.
method Empirical analysis of forgotten examples in N upstream examples after M new tasks, using low-rank matrix approximation. result Forgetting can be predicted efficiently using matrix completion over empirical associations.
This work tackles catastrophic forgetting in neural networks by mimicking brain's metaplasticity.
problem Catastrophic forgetting in neural networks, where new tasks erase previously learned ones.
method Interpreting binarized neural networks as metaplastic systems, adjusting their training technique.
result Training technique reduces catastrophic forgetting without needing previously presented data.
Dynamic information balancing reduces catastrophic forgetting in modular neural networks.
problem Catastrophic forgetting in neural networks when learning multiple tasks.
method Dynamic Information Balancing (DIB) using reinforcement learning to adaptively route inputs based on module information load.
result DIB combined with EWC regularization outperforms models with similar capacity and EWC regularization.
Reversible RNNs reduce memory usage in training without sacrificing performance.
problem Memory-intensive training of RNNs limits model flexibility.
method Developed a scheme for perfect reversible RNNs with forgetting, reducing memory by 10-15x.
result Achieved comparable performance to traditional models with reduced activation memory cost.
Study on reducing forgetting in neural networks using compression theory.
problem Catastrophic forgetting in neural networks.
method Defined forgetting as increased description lengths, compared variational posterior approaches to prequential coding methods.
result Proposed a new continual learning method combining ML plug-in and Bayesian mixture codes.
Proposes MEDIC to improve incremental learning by preventing forgetting and intransigence.
problem Challenges of forgetting old knowledge and intransigence on new knowledge in incremental learning.
method Maximum Entropy Regularizer (MER) and DropOut Sampling (DOS) to penalize uncertain knowledge and reduce class imbalance.
result Proposed method 'MEDIC' outperforms state-of-the-art algorithms in accuracy, forgetting, and intransigence.
Model forgets examples; this research predicts which ones to replay.
problem Language models forget examples during updates, leading to errors.
method Train forecasting models to predict which examples will be forgotten.
result Forecasting models can reduce forgetting of upstream pretraining examples.
The brain optimizes memory by forgetting what's predictable, improving generalization.
problem Memory consolidation struggles with representational drift, semanticisation, and offline replay.
method Proposes predictive forgetting as a mechanism to optimize generalization by reducing complexity.
result Predictive forgetting improves information-theoretic generalization bounds on stored representations.
ANTIDOTE reduces noisy labels influence during learning.
problem Learning with noisy labels.
method Information-divergence neighborhood relaxation and adversarial training.
result ANTIDOTE outperforms standard cross-entropy loss in noisy label settings.
Autoencoder reduces memory overhead in incremental class learning.
problem Catastrophic forgetting in continual learning.
method Uses autoencoder to extract prototypes and store mean prototypes per class.
result Comparable performance to state-of-the-art with lower memory cost.
New sensor placement affects robot controller learnability.
problem Catastrophic forgetting in neural controllers for robots.
method Demonstrated how sensor placement alters loss function manifolds.
result Sensor placement can reduce or induce catastrophic forgetting.
A study on preventing catastrophic forgetting in neural networks using conditional computation.
problem Catastrophic forgetting in connectionist neural networks.
method Conditional computation framework where parameters are conditioned on each input example.
result Proposed conditional rehearsal to prevent forgetting of previously learned examples.
SIGUA prevents overfitting in noisy data by forgetting unwanted memorization.
problem Overfitting in deep learning models with noisy labels.
method SIGUA combines gradient descent on good data and learning-rate-reduced gradient ascent on bad data.
result SIGUA improves the performance of two base learning methods significantly.
The paper tackles CF in CL by analyzing NTK overlap matrix and proposing OGD.
problem Catastrophic Forgetting in continual learning.
method Analysis of NTK overlap matrix, OGD with PCA.
result Proposes OGD to mitigate CF, supported by experiments.
New method improves ABI for sequential data, reducing forgetting and improving accuracy.
problem Performance degradation of ABI under model misspecification and distribution shifts.
method Decouples simulation-based pre-training from unsupervised SC fine-tuning, using memory buffer and elastic weight consolidation.
result Significant mitigation of forgetting and improved posterior estimates compared to standard simulation-based training.
Experience replay helps neural networks learn new tasks without forgetting old knowledge.
problem Catastrophic forgetting in neural networks trained on non-stationary data.
method Experience replay buffers with a mixture of on- and off-policy learning.
result Experience replay can learn new tasks quickly and reduce catastrophic forgetting.
Paper proposes AdaBoost-assisted ELM for efficient online sequential classification.
problem Efficient online sequential classification with improved accuracy and stability.
method Utilizes AdaBoost for cost-sensitive learning and forgetting mechanism for stability.
result Achieves 94.41% accuracy on MNIST dataset with reduced standard deviation.
ResRep prunes CNNs without losing accuracy by separating remembering and forgetting.
problem Pruning CNNs to reduce FLOPs without sacrificing accuracy.
method Decoupling remembering and forgetting in CNNs, using SGD for remembering and a novel update rule for forgetting.
result Achieved lossless pruning with high compression ratio (76.15% accuracy on ImageNet with 45% FLOPs reduction).
Model learns continuously from text without forgetting.
problem Catastrophic forgetting in lifelong language learning.
method Episodic memory with sparse experience replay and local adaptation.
result Model can continuously learn from new datasets.
Catastrophic forgetting occurs when a neural network loses the information learned in a previous task after training on subsequent tasks. This problem remains a hurdle for artificial intelligence systems with sequential learning capabilities. In this paper, we propose a task-based hard attention mechanism that preserve…
L2F learns to forget, improving few-shot learning performance.
problem Few-shot learning challenges in adapting to unseen tasks.
method Task-and-layer-wise attenuation on compromised initialization to reduce prior knowledge influence.
result Faster adaptation and improved performance demonstrated experimentally.
While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many t…
Paper tackles catastrophic forgetting in neural networks with an adversarial feature alignment method.
problem Dramatic performance degradation when new tasks are added to an existing neural network model.
method Inspired by human learning, the paper proposes an adversarial feature alignment method to decompose complex tasks into easier goals.
result The proposed method outperforms state-of-the-art methods in both accuracies on new tasks and performance preservation on old tasks.
Study shows optimal model performance at critical level of feature learning.
problem Catastrophic forgetting in neural networks, especially in non-stationary environments.
method Systematic study on model scale and feature learning, using dynamical mean field theory.
result Optimal performance achieved at a critical level of feature learning, dependent on task non-stationarity and model scale.
The paper analyzes the efficiency of unlearning methods and establishes bounds for minimax computation times.
problem Efficiency of removing specific data points from a trained model without full retraining.
method Analysis of unlearning methods and establishment of upper and lower bounds on computation times.
result A phase diagram for the unlearning complexity ratio, revealing three regimes of feasibility.
Paper tackles forgetting in neural networks, proposing solutions.
problem Catastrophic forgetting in artificial neural networks.
method Simple model and reinforcement learning applications.
result Proposes solutions to prevent forgetting in neural networks.
A model retains learned knowledge for longer by adding a plastic component to neural networks.
problem Catastrophic forgetting in neural networks when learning new tasks.
method Differentiable Hebbian Consolidation model with a DHP Softmax layer.
result Reduces forgetting in benchmarks like Permuted MNIST and Vision Datasets Mixture.
Developments in deep generative models have allowed for tractable learning of high-dimensional data distributions. While the employed learning procedures typically assume that training data is drawn i.i.d. from the distribution of interest, it may be desirable to model distinct distributions which are observed sequenti…
A novel approach stores encoded images as centroids and covariance matrices to improve classification accuracy with less memory.
problem Catastrophic forgetting and memory limitations in continual learning.
method Trains autoencoders with Neural Style Transfer to encode images, replay encoded episodes to avoid forgetting, and use centroids and covariance matrices for pseudo-images when memory is full.
result Increases classification accuracy by 13-17% over state-of-the-art methods on benchmark datasets, while requiring 78% less storage space.
Adam optimizer leads to more forgetting in neural networks.
problem Understanding and quantifying catastrophic forgetting in neural networks.
method Comparative analysis of various optimization algorithms and metrics in different learning scenarios.
result Adam optimizer causes more forgetting compared to classical algorithms like SGD.
A new method helps deep learning systems adapt to changing conditions.
problem Deep learning systems struggle with environmental drifts and long healing cycles.
method Intentional forgetting integrated into continual learning to overcome issues.
result Dr. DRL reduces healing time and fine-tuning episodes by 18.74% and 17.72% respectively.
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
New theory explains forgetting in learning algorithms.
problem Tendency of learning algorithms to forget past knowledge.
method Proposes a self-consistency theory of forgetting as a loss of predictive information.
result Exact Bayesian inference allows for adaptation without forgetting.
The paper proposes selective forgetting for deep neural networks at a finer level than samples.
problem Selective forgetting of deep neural networks to handle outliers, poisoned data, or sensitive information.
method Formulated selective forgetting at a finer level than samples, introduced as an optimization problem on three criteria.
result Experimental results show the model can forget specific information for classification, improving accuracy in specific cases.
New method prevents forgetting in LLMs by dynamically identifying task-specific subspaces.
problem Catastrophic forgetting in continual learning of LLMs.
method Adaptive Singular Value Decomposition (SVD) for constrained full fine-tuning.
result Achieves state-of-the-art results in continual learning benchmarks.
Study helps identify which ANN parameters cause forgetting.
problem Catastrophic forgetting in neural networks.
method Determines individual parameter contributions to forgetting.
result Identifies specific ANN parameters causing forgetting.
LoRA fine-tuning creates intruder dimensions that can cause forgetting, and a new law predicts when this happens.
problem Predicting when LoRA fine-tuning creates intruder dimensions that can cause catastrophic forgetting.
method Derived a per-layer critical update strength s∗ and an exact secular-equation characterization of the updated spectrum. result The law localizes the empirical threshold within a factor of two on 82% of layers and separates intruder-bearing from intruder-free layers at deployment.
A new method reduces memory usage for deep neural networks ensembles.
problem Reducing memory footprint of ensemble methods for deep learning.
method Extract multiple sub-networks from a single neural network through end-to-end optimization.
result Achieves higher or comparable accuracy with significantly less memory usage.
Paper introduces Auto DeepVis to explain catastrophic forgetting in continual learning.
problem Catastrophic forgetting in continual learning of deep neural networks.
method Auto DeepVis and critical freezing techniques to address catastrophic forgetting.
result Critical freezing outperforms other methods on both past and future tasks.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
The paper examines GANs' forgetting and mode collapse, showing how they relate and impact training.
problem Catastrophic forgetting and mode collapse in GANs during continual learning.
method Investigates the continual learning nature of GANs, analyzing discriminator's output landscapes and convergence.
result Catastrophic forgetting and mode collapse are interrelated and prevent GANs from converging.
Bayesian online meta-learning framework tackles catastrophic forgetting in few-shot classification.
problem Catastrophic forgetting in few-shot classification problems.
method Bayesian online learning, meta-learning, Laplace approximation, variational inference.
result Framework effectively achieves goal of overcoming catastrophic forgetting in few-shot classification.