Multi-model forgetting occurs when training multiple deep networks sequentially, leading to performance degradation of previously trained models.
problem Performance degradation of previously trained models when sequentially training multiple deep networks with shared parameters.
method Introduce a weight plasticity loss that regularizes the learning of shared parameters based on their importance for previous models.
result Weight plasticity loss effectively preserves the performance of previously trained models during sequential training and neural architecture search.
Multi-model FL improves performance without sharing data.
problem Training multiple models in a federated setting without data sharing.
method Proposed two variants of FedAvg for multi-model FL with provable convergence guarantees.
result Multi-model FL can have better performance than training each model separately.
Paper improves gender detection on social media using deep learning.
problem Traditional classifiers struggle with social media data volume.
method Ensemble deep learning with multi-model architectures.
result Improved gender detection accuracy on social media posts.
GANs improve stochastic dynamics prediction by selecting randomly between models.
problem Accurately generating stochastic dynamics from multi-model GANs.
method Applying GANs to a lattice stochastic process, adding noise, and selecting randomly between models.
result Multi-model approach leads to significant improvement in accuracy of stochastic dynamics prediction.
CoSMIC extends flow-based SVI to transdimensional problems.
problem Bayesian structure learning and model selection with multi-model parameter spaces.
method Normalizing flows with a combined stochastic variational transdimensional inference approach.
result Improved performance on high-cardinality model spaces.
PRETZEL optimizes machine learning prediction serving systems for better performance.
problem Low latency, high throughput, and graceful performance degradation under heavy load in prediction serving systems.
method Introducing a novel white box architecture enabling both end-to-end and multi-model optimizations.
result Average 5.5x reduction in 99th percentile latency, 25x reduction in memory footprint, and 4.7x increase in throughput compared to state-of-the-art approaches.
This paper improves model fusion by training-time neuron alignment, reducing barriers in multi-model fusion.
problem Diverse neuron permutations across different settings hinder model fusion performances.
method Training-time neuron alignment using fixed neuron anchors to reduce training-time permutations.
result Training-time neuron alignment improves fusion of pretrained models and federated learning performances.
Adam optimizer leads to more forgetting in neural networks.
problem Understanding and quantifying catastrophic forgetting in neural networks.
method Comparative analysis of various optimization algorithms and metrics in different learning scenarios.
result Adam optimizer causes more forgetting compared to classical algorithms like SGD.
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
New theory explains forgetting in learning algorithms.
problem Tendency of learning algorithms to forget past knowledge.
method Proposes a self-consistency theory of forgetting as a loss of predictive information.
result Exact Bayesian inference allows for adaptation without forgetting.
The paper proposes selective forgetting for deep neural networks at a finer level than samples.
problem Selective forgetting of deep neural networks to handle outliers, poisoned data, or sensitive information.
method Formulated selective forgetting at a finer level than samples, introduced as an optimization problem on three criteria.
result Experimental results show the model can forget specific information for classification, improving accuracy in specific cases.
NetFuse merges different DNN models with varying weights for faster inference.
problem Inference speed of DNN models with different weights cannot be improved using existing techniques.
method NetFuse merges models with the same architecture but different weights and inputs, replacing operations with more general ones.
result NetFuse can speed up DNN inference time up to 3.6x on a NVIDIA V100 GPU.
With the increasing penetration of solar power into power systems, forecasting becomes critical in power system operations. In this paper, an hourly-similarity (HS) based method is developed for 1-hour-ahead (1HA) global horizontal irradiance (GHI) forecasting. This developed method utilizes diurnal patterns, statistic…
Study helps identify which ANN parameters cause forgetting.
problem Catastrophic forgetting in neural networks.
method Determines individual parameter contributions to forgetting.
result Identifies specific ANN parameters causing forgetting.
Paper introduces Auto DeepVis to explain catastrophic forgetting in continual learning.
problem Catastrophic forgetting in continual learning of deep neural networks.
method Auto DeepVis and critical freezing techniques to address catastrophic forgetting.
result Critical freezing outperforms other methods on both past and future tasks.
The paper analyzes how forgetting in LLMs is linked to simple task-upstream example associations.
problem Forgetting of upstream knowledge in fine-tuned LLMs.
method Empirical analysis of forgotten examples in N upstream examples after M new tasks, using low-rank matrix approximation. result Forgetting can be predicted efficiently using matrix completion over empirical associations.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
Bayesian online meta-learning framework tackles catastrophic forgetting in few-shot classification.
problem Catastrophic forgetting in few-shot classification problems.
method Bayesian online learning, meta-learning, Laplace approximation, variational inference.
result Framework effectively achieves goal of overcoming catastrophic forgetting in few-shot classification.
New approach for deep neural networks to learn invariance through adversarial forgetting.
problem Learning invariance for deep neural networks in the presence of nuisance and bias factors.
method Adversarial forgetting mechanism to induce amnesia to unwanted data factors.
result State-of-the-art performance in learning invariance across various datasets and tasks.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
Self-supervised GAN prevents forgetting in sequential tasks.
problem Discriminator forgetting in GANs leads to training instability.
method Add self-supervision to the discriminator to maintain useful representations.
result Self-supervised GAN outperforms regular GANs in learning better representations.
Paper tackles catastrophic forgetting in sequential learning.
problem Catastrophic forgetting in sequential learning.
method Regularizes training with sketches of Jacobian matrix of past data.
result Proves overcoming catastrophic forgetting for linear and wide neural networks.
This work tackles catastrophic forgetting in neural networks by mimicking brain's metaplasticity.
problem Catastrophic forgetting in neural networks, where new tasks erase previously learned ones.
method Interpreting binarized neural networks as metaplastic systems, adjusting their training technique.
result Training technique reduces catastrophic forgetting without needing previously presented data.
Overparameterization helps prevent forgetting in sequential learning tasks.
problem Catastrophic forgetting in continual learning systems.
method Analytical study of gradient descent with linear regression model.
result Overparameterization can mitigate forgetting in a two-task setting.
Study finds some examples are forgotten during neural network learning.
problem Understanding forgetting during neural network training.
method Defined forgetting event and analyzed across multiple data sets.
result High frequency of certain examples being forgotten, some not.
Continual learning is hard for AI, leading to forgetting old knowledge.
problem Catastrophic forgetting in AI when learning new data.
method Review of continual learning in deep learning.
result Challenges and insights in continual learning.
Study examines how training regime affects neural networks' forgetting.
problem Catastrophic forgetting in neural networks when learning multiple tasks sequentially.
method Analyzes the impact of different training regimes (learning rate, batch size, regularization) on forgetting.
result Training regimes that widen tasks' local minima help prevent catastrophic forgetting.
Solar forecasting accuracy is affected by weather conditions, and weather awareness forecasting models are expected to improve the performance. However, it may not be available and reliable to classify different forecasting tasks by using only meteorological weather categorization. In this paper, an unsupervised cluste…
A simple method to prevent forgetting in neural networks.
problem Catastrophic forgetting in neural networks.
method Weight friction, inspired by neurology and physics, modifies the gradient descent update rule.
result Weight friction enables continual learning with comparable performance and efficiency.
Proposes a method to prevent neural networks from forgetting learned tasks.
problem Catastrophic forgetting in neural networks.
method Attention-based selective plasticity of synapses inspired by the cholinergic neuromodulatory system.
result Competitive performance on benchmark tasks compared to state-of-the-art methods.
Hybrid framework prevents forgetting in continual learning.
problem Avoiding forgetting in learning new tasks without forgetting old ones.
method Hybrid continual learning framework combining architecture growth and experience replay.
result Hybrid approach effectively avoids forgetting across multiple tasks.
Framework uses experience replay to prevent deep networks from forgetting past tasks.
problem Deep networks forget past tasks after learning new ones in sequential multitask learning.
method Generative model that couples current task with past learned tasks through a discriminative embedding space.
result Framework learns a shared abstract distribution across all tasks, preventing catastrophic forgetting.
Adaptive model learns from time series data with changing distributions.
problem Predicting time series data under distribution shift.
method Formulates distribution shift as weighted empirical risk minimization. Uses a gradient-based learning method for a forgetting mechanism.
result Proposes an efficient method for adaptive time series prediction.
Study on neural networks forgetting in continual learning.
problem Forgetting in neural networks during continual learning.
method Gradient descent analysis on XOR-cluster datasets.
result Explicit bounds on forgetting rate and generalization gap.
In this paper, we show that Generative Adversarial Networks (GANs) suffer from catastrophic forgetting even when they are trained to approximate a single target distribution. We show that GAN training is a continual learning problem in which the sequence of changing model distributions is the sequence of tasks to the d…
Improves neural networks' ability to learn new tasks without forgetting earlier ones.
problem Preventing catastrophic forgetting in neural networks.
method Introducing a second discriminator in the GAN to generate important features for task retention.
result Significant reduction in catastrophic forgetting compared to standard methods.
Synthetic data helps prevent forgetting when learning sequentially.
problem Catastrophic forgetting in neural networks.
method Generate synthetic data via two-step optimisation process using meta-gradients.
result Training on synthetic data prevents forgetting when learning sequentially.
FSR prevents deep nets from forgetting old tasks.
problem Catastrophic forgetting in deep neural networks.
method Logit matching on selected samples from old tasks.
result FSR outperforms previous methods in knowledge retention.
A new method prevents forgetting during knowledge transfer.
problem Catastrophic forgetting in transfer learning.
method Transfer without Forgetting (TwF) using a fixed pretrained network.
result TwF outperforms other CL methods by 4.81% in Class-Incremental accuracy.
New method prevents forgetting in continual learning tasks.
problem Catastrophic forgetting in continual learning.
method Generative regularization integrated into Bayesian inference framework.
result Empirically outperforms state-of-the-art methods on various datasets.
NN-GPR improves climate model predictions by preserving fine-scale spatial information.
problem Dilution of fine-scale spatial information and bias in model averaging.
method Gaussian process regression with an infinitely wide deep neural network.
result NN-GPR produces more accurate and detailed climate projections.
Holomorphic maps between moduli spaces are shown to be forgetful for large g.
problem Characterizing holomorphic maps between moduli spaces.
method Proving that only forgetful maps are non-constant for large g.
result Forgetful maps are the only non-constant holomorphic maps between moduli spaces for g≥4. Proposes a new method to prevent forgetting in neural networks.
problem Catastrophic forgetting in neural networks.
method Total absolute signal passing through each connection.
result Simpler and closer to brain processes.
Dynamic memory prevents forgetting in continuous learning of medical images.
problem Catastrophic forgetting in machine learning models over time due to domain shifts.
method Dynamic memory to store and replay diverse training data subsets.
result Dynamic memory mitigates forgetting without knowing when shifts occur.
Paper proposes a new pipeline for few-shot classification using forget-update module and channel vector sequence.
problem Few-shot classification with limited support samples.
method Channel vector sequence construction module and forget-update module.
result Pipeline achieves state-of-the-art results on various datasets.
New method prevents forgetting in neural-linear bandits for high-dimensional data.
problem Catastrophic forgetting in neural-linear bandits with high-dimensional features.
method Limited memory neural-linear bandit with likelihood matching.
result Resilient to catastrophic forgetting and superior performance.
New neural network avoids forgetting old knowledge as new data comes in.
problem Catastrophic forgetting in lifelong learning systems.
method Sequential Neural Coding Network, biologically-plausible synapse adaptation, task-executive control mimicry.
result Significantly less forgetting compared to standard neural models in experiments.