A new algorithm for efficiently removing specific classes from a model without retraining.
problem Class forgetting in machine learning models.
method Estimating retain and forget spaces using SVD, removing shared information, and updating weights.
result Achieved up to 1.38% accuracy improvement on ImageNet dataset with minimal samples.
Hybrid model avoids forgetting across tasks and classes.
problem Challenges of continual learning in modern deep learning.
method Hybrid generative-discriminative approach using normalizing flows.
result Strong performance on various continual learning benchmarks.
A new method prevents forgetting during knowledge transfer.
problem Catastrophic forgetting in transfer learning.
method Transfer without Forgetting (TwF) using a fixed pretrained network.
result TwF outperforms other CL methods by 4.81% in Class-Incremental accuracy.
Hybrid framework prevents forgetting in continual learning.
problem Avoiding forgetting in learning new tasks without forgetting old ones.
method Hybrid continual learning framework combining architecture growth and experience replay.
result Hybrid approach effectively avoids forgetting across multiple tasks.
Paper tackles few-shot class-incremental learning with a neural gas network.
problem Incrementally learn new classes from very few labelled samples without forgetting old classes.
method Proposes TOPIC framework using a neural gas network to preserve class topology and adapt to new samples.
result Significantly outperforms other methods on CIFAR100, miniImageNet, and CUB200 datasets.
Synthetic data helps prevent forgetting when learning sequentially.
problem Catastrophic forgetting in neural networks.
method Generate synthetic data via two-step optimisation process using meta-gradients.
result Training on synthetic data prevents forgetting when learning sequentially.
Incremental learning suffers from two challenging problems; forgetting of old knowledge and intransigence on learning new knowledge. Prediction by the model incrementally learned with a subset of the dataset are thus uncertain and the uncertainty accumulates through the tasks by knowledge transfer. To prevent overfitti…
TKIL improves class-balanced performance in incremental learning.
problem Catastrophic forgetting in sequential learning tasks.
method Introduces Tangent Kernel for Incremental Learning (TKIL) based on Neural Tangent Kernel (NTK).
result TKIL achieves better overall accuracy and variance across classes.
CILF learns adaptive embeddings for class-incremental learning with novel class detection and model update.
problem Handling unknown classes and model update in streaming data with new classes.
method CILF uses decoupled prototype based loss for intra-class and inter-class structure improvement, and a learnable curriculum clustering operator for adaptive embedding.
result CILF effectively detects multiple novel classes and mitigates embedding confusion, while updating the model without catastrophic forgetting.
A plain well-trained deep learning model often does not have the ability to learn new knowledge without forgetting the previously learned knowledge, which is known as catastrophic forgetting. Here we propose a novel method, SupportNet, to efficiently and effectively solve the catastrophic forgetting problem in the clas…
Paper proposes a new pipeline for few-shot classification using forget-update module and channel vector sequence.
problem Few-shot classification with limited support samples.
method Channel vector sequence construction module and forget-update module.
result Pipeline achieves state-of-the-art results on various datasets.
Paper tackles graph class-incremental learning with task profiling and prompting.
problem Challenges in separating classes from different tasks in graph CIL.
method Laplacian smoothing-based task profiling and graph prompting approach.
result 100% task ID prediction accuracy and significant performance improvement.
Paper tackles class-incremental learning by combining self-supervised learning to mitigate prior information loss.
problem Catastrophic forgetting and prior information loss in class-incremental learning.
method Combining self-supervised learning with class-incremental learning to mitigate prior information loss.
result Proposed method outperforms state-of-the-art methods.
Meta-learning approach improves object detection on new classes.
problem Deterioration of object detection performance on old classes in incremental settings.
method Meta-learning to reshape model gradients for optimal task adaptation.
result Meta-learning approach outperforms state-of-the-art methods in incremental object detection.
Efficiently preserves old class knowledge in memory-limited settings.
problem Catastrophic forgetting in class-incremental learning.
method Memory-efficient exemplar preserving scheme and domain-compatible feature extractors.
result Low-fidelity exemplar samples can replace high-fidelity ones with less memory cost.
Continual lifelong learning requires an agent or model to learn many sequentially ordered tasks, building on previous knowledge without catastrophically forgetting it. Much work has gone towards preventing the default tendency of machine learning models to catastrophically forget, yet virtually all such work involves m…
Adam optimizer leads to more forgetting in neural networks.
problem Understanding and quantifying catastrophic forgetting in neural networks.
method Comparative analysis of various optimization algorithms and metrics in different learning scenarios.
result Adam optimizer causes more forgetting compared to classical algorithms like SGD.
D-CBRS manages memory for continual learning by accounting for intra-class diversity.
problem Forgetting in continual learning, especially with class-imbalanced data.
method D-CBRS introduces a novel approach to store instances in memory, considering intra-class diversity.
result D-CBRS outperforms existing methods on data sets with intra-class diversity.
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
New theory explains forgetting in learning algorithms.
problem Tendency of learning algorithms to forget past knowledge.
method Proposes a self-consistency theory of forgetting as a loss of predictive information.
result Exact Bayesian inference allows for adaptation without forgetting.
Incremental class learning, a scenario in continual learning context where classes and their training data are sequentially and disjointedly observed, challenges a problem widely known as catastrophic forgetting. In this work, we propose a novel incremental class learning method that can significantly reduce memory ove…
The paper proposes selective forgetting for deep neural networks at a finer level than samples.
problem Selective forgetting of deep neural networks to handle outliers, poisoned data, or sensitive information.
method Formulated selective forgetting at a finer level than samples, introduced as an optimization problem on three criteria.
result Experimental results show the model can forget specific information for classification, improving accuracy in specific cases.
The main purpose of incremental learning is to learn new knowledge while not forgetting the knowledge which have been learned before. At present, the main challenge in this area is the catastrophe forgetting, namely the network will lose their performance in the old tasks after training for new tasks. In this paper, we…
Class incremental learning refers to a special multi-class classification task, in which the number of classes is not fixed but is increasing with the continual arrival of new data. Existing researches mainly focused on solving catastrophic forgetting problem in class incremental learning. To this end, however, these m…
We study the class of holomorphic and isometric submersions between finite-type Teichmüller spaces. We prove that, with potential exceptions coming from low-genus phenomena, any such map is a forgetful map Tg,n→Tg,m obtained by filling in punctures. This generalizes a classical r…
Paper tackles class-incremental time series classification with dual-stream feature extraction.
problem Class-incremental continual learning for multivariate time series data.
method Dual-stream feature extraction pipeline combining deep temporal embedding features and statistical features.
result Competitive average accuracy across multiple datasets with low forgetting rates.
The paper analyzes how forgetting in LLMs is linked to simple task-upstream example associations.
problem Forgetting of upstream knowledge in fine-tuned LLMs.
method Empirical analysis of forgotten examples in N upstream examples after M new tasks, using low-rank matrix approximation. result Forgetting can be predicted efficiently using matrix completion over empirical associations.
SpaceNet improves continual learning by intelligently compressing neural connections.
problem Catastrophic forgetting in sequential learning tasks.
method SpaceNet trains sparse deep neural networks adaptively, compressing task-specific connections.
result SpaceNet outperforms existing methods in class incremental learning scenarios.
Lifelong learning with deep neural networks is well-known to suffer from catastrophic forgetting: the performance on previous tasks drastically degrades when learning a new task. To alleviate this effect, we propose to leverage a large stream of unlabeled data easily obtainable in the wild. In particular, we design a n…
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
This paper tackles catastrophic forgetting in neural networks by providing a unified framework for regularization-based continual learning.
problem Catastrophic forgetting in neural networks trained sequentially on multiple tasks.
method Formulates regularization-based continual learning as a second-order Taylor approximation of the loss function, leading to a unified framework.
result Theoretical results indicate the importance of accurate approximation of the Hessian matrix for optimization and generalization.
The study classifies normal subgroups of mapping class groups of surfaces with Cantor subsets.
problem Understanding the structure of normal subgroups in mapping class groups of surfaces with specific subsets.
method Proves two structure theorems: purity and inertia, characterizing normal subgroups.
result Characterizes finite-type normal subgroups of mapping class groups of surfaces with Cantor subsets.
Bayesian online meta-learning framework tackles catastrophic forgetting in few-shot classification.
problem Catastrophic forgetting in few-shot classification problems.
method Bayesian online learning, meta-learning, Laplace approximation, variational inference.
result Framework effectively achieves goal of overcoming catastrophic forgetting in few-shot classification.
Multi-Class Incremental Learning (MCIL) aims to learn new concepts by incrementally updating a model trained on previous concepts. However, there is an inherent trade-off to effectively learning new concepts without catastrophic forgetting of previous ones. To alleviate this issue, it has been proposed to keep around a…
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
Paper tackles catastrophic forgetting in sequential learning.
problem Catastrophic forgetting in sequential learning.
method Regularizes training with sketches of Jacobian matrix of past data.
result Proves overcoming catastrophic forgetting for linear and wide neural networks.
This work tackles catastrophic forgetting in neural networks by mimicking brain's metaplasticity.
problem Catastrophic forgetting in neural networks, where new tasks erase previously learned ones.
method Interpreting binarized neural networks as metaplastic systems, adjusting their training technique.
result Training technique reduces catastrophic forgetting without needing previously presented data.
ANTIDOTE reduces noisy labels influence during learning.
problem Learning with noisy labels.
method Information-divergence neighborhood relaxation and adversarial training.
result ANTIDOTE outperforms standard cross-entropy loss in noisy label settings.
Overparameterization helps prevent forgetting in sequential learning tasks.
problem Catastrophic forgetting in continual learning systems.
method Analytical study of gradient descent with linear regression model.
result Overparameterization can mitigate forgetting in a two-task setting.
A model retains learned knowledge for longer by adding a plastic component to neural networks.
problem Catastrophic forgetting in neural networks when learning new tasks.
method Differentiable Hebbian Consolidation model with a DHP Softmax layer.
result Reduces forgetting in benchmarks like Permuted MNIST and Vision Datasets Mixture.
Interpreting the behaviors of Deep Neural Networks (usually considered as a black box) is critical especially when they are now being widely adopted over diverse aspects of human life. Taking the advancements from Explainable Artificial Intelligent, this paper proposes a novel technique called Auto DeepVis to dissect c…
Continual learning is hard for AI, leading to forgetting old knowledge.
problem Catastrophic forgetting in AI when learning new data.
method Review of continual learning in deep learning.
result Challenges and insights in continual learning.
Study examines how training regime affects neural networks' forgetting.
problem Catastrophic forgetting in neural networks when learning multiple tasks sequentially.
method Analyzes the impact of different training regimes (learning rate, batch size, regularization) on forgetting.
result Training regimes that widen tasks' local minima help prevent catastrophic forgetting.
Adaptive model learns from time series data with changing distributions.
problem Predicting time series data under distribution shift.
method Formulates distribution shift as weighted empirical risk minimization. Uses a gradient-based learning method for a forgetting mechanism.
result Proposes an efficient method for adaptive time series prediction.
DPTA improves CIL by adapting PTMs with dual prototypes.
problem Catastrophic forgetting in incremental learning with pre-trained models.
method Dual-Prototype Network with Task-wise Adaptation (DPTA).
result DPTA consistently outperforms recent methods by 1\%-5\% on multiple benchmarks.
Study on neural networks forgetting in continual learning.
problem Forgetting in neural networks during continual learning.
method Gradient descent analysis on XOR-cluster datasets.
result Explicit bounds on forgetting rate and generalization gap.
A new method improves continual learning by replaying pseudo data and using orthogonal weight modification.
problem Catastrophic forgetting in class incremental learning.
method Generative replay with orthogonal weight modification.
result Our method achieves substantial improvement over conventional generative replay and OWM.