EVCL combines VCL and EWC to prevent forgetting new tasks.
problem Preventing catastrophic forgetting in continual learning.
method Hybrid model integrating VCL and EWC.
result Consistently outperforms baselines in learning new tasks.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Elastic Weight Consolidation”
EWC helps prevent forgetting in neural networks by adjusting weights dynamically.
problem Preventing forgetting in neural networks during training.
method EWC adjusts weights dynamically to prevent forgetting.
result EWC effectively prevents catastrophic forgetting in neural networks.
Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel algorithm designed to safeguard against catastrophic forgetting in neural networks. EWC can be seen as an approximation to Laplace propagation (Eskin et al, 2004), and this view is consistent with the motivation given by Kirkpatrick et al (2017). In…
Blog post discusses various implementations of Fisher Information for EWC in continual learning.
problem Improving Elastic Weight Consolidation (EWC) results by optimizing Fisher Information computation.
method Empirically compares different implementations of Fisher Information for EWC.
result Many reported EWC results can be improved by changing Fisher Information computation methods.
Proposes a CL technique to improve accuracy and reduce forgetting.
problem Sequential task learners struggle with forgetting information from previous tasks.
method Extracts modular parts of neural networks and estimates task relatedness.
result Remarkable performance gain in robustness to forgetting for EWC and GEM methods.
Paper benchmarks CF mitigation in federated time series forecasting.
problem Catastrophic forgetting in federated learning for time series forecasting.
method Comprehensive evaluation of CF mitigation strategies in federated time series forecasting.
result Introduction of a new benchmark for CF in time series federated learning.
This research proposes a CL model for RNNs to handle sequential data without forgetting.
problem Learning in dynamic environments without forgetting previous knowledge for sequential data.
method A Recurrent Neural Network (RNN) model with Elastic Weight Consolidation (EWC) for CL.
result The proposed model outperforms EWC and RNNs on CL benchmarks for sequential data.
New method improves ABI for sequential data, reducing forgetting and improving accuracy.
problem Performance degradation of ABI under model misspecification and distribution shifts.
method Decouples simulation-based pre-training from unsupervised SC fine-tuning, using memory buffer and elastic weight consolidation.
result Significant mitigation of forgetting and improved posterior estimates compared to standard simulation-based training.
Modified PCA algorithm with continual learning preserves features of previous modes for multimode process monitoring.
problem Catastrophic forgetting of previous modes in monitoring models for successive modes.
method Modified PCA algorithm with elastic weight consolidation (EWC) to preserve features of previous modes.
result PCA-EWC algorithm effectively monitors multimode processes without performance decrease.
We present a new replay-based method of continual classification learning that we term "conditional replay" which generates samples and labels together by sampling from a distribution conditioned on the class. We compare conditional replay to another replay-based continual learning paradigm (which we term "marginal rep…
Distributed machine learning training is one of the most common and important workloads running on data centers today, but it is rarely executed alone. Instead, to reduce costs, computing resources are consolidated and shared by different applications. In this scenario, elasticity and proper load balancing are vital to…
Sequential learning of multiple tasks in artificial neural networks using gradient descent leads to catastrophic forgetting, whereby previously learned knowledge is erased during learning of new, disjoint knowledge. Here, we propose a new approach to sequential learning which leverages the recent discovery of adversari…
Collecting the large datasets needed to train deep neural networks can be very difficult, particularly for the many applications for which sharing and pooling data is complicated by practical, ethical, or legal concerns. However, it may be the case that derivative datasets or predictive models developed within individu…
This paper presents a new family of backpropagation-free neural architectures, Gated Linear Networks (GLNs). What distinguishes GLNs from contemporary neural networks is the distributed and local nature of their credit assignment mechanism; each neuron directly predicts the target, forgoing the ability to learn feature…
Lifelong learning is a very important step toward realizing robust autonomous artificial agents. Neural networks are the main engine of deep learning, which is the current state-of-the-art technique in formulating adaptive artificial intelligent systems. However, neural networks suffer from catastrophic forgetting when…
Study evaluates CL methods in RNNs, highlighting differences from feedforward networks.
problem Preventing catastrophic forgetting in RNNs processing sequential data.
method Comprehensive evaluation of CL methods, including elastic weight consolidation and hypernetworks.
result Weight-importance methods perform similarly regardless of sequence length but require more stability for high working memory demands.
Model for material elasticity and plasticity using networks.
problem Understanding the elasticity and plasticity of materials.
method Developed a mathematical model based on networks, defining tension tensor for periodic graphs.
result The model explains elasticity and plasticity through local moves on graphs.
Standard artificial neural networks suffer from the well-known issue of catastrophic forgetting, making continual or lifelong learning difficult for machine learning. In recent years, numerous methods have been proposed for continual learning, but due to differences in evaluation protocols it is difficult to directly c…
It is well known that the out-of-sample performance of Markowitz's mean-variance portfolio criterion can be negatively affected by estimation errors in the mean and covariance. In this paper we address the problem by regularizing the mean-variance objective function with a weighted elastic net penalty. We show that the…
Improved continual learning method using variational inference and FiLM layers.
problem Training models on new tasks and datasets in an online fashion.
method Generalized Variational Continual Learning (GVCL) with likelihood-tempering and FiLM layers.
result GVCL outperforms existing baselines in both small and large datasets, providing better calibration.
Proposes fwelnet to improve prediction using feature information.
problem Improving prediction accuracy in supervised learning settings.
method fwelnet uses feature-weighted elastic net to adapt feature penalties.
result fwelnet outperforms lasso in test MSE and feature selection.
Sequential learning of tasks using gradient descent leads to an unremitting decline in the accuracy of tasks for which training data is no longer available, termed catastrophic forgetting. Generative models have been explored as a means to approximate the distribution of old tasks and bypass storage of real data. Here …
Paper tackles catastrophic forgetting in sequential learning.
problem Catastrophic forgetting in sequential learning.
method Regularizes training with sketches of Jacobian matrix of past data.
result Proves overcoming catastrophic forgetting for linear and wide neural networks.
This paper tackles catastrophic forgetting in neural networks by providing a unified framework for regularization-based continual learning.
problem Catastrophic forgetting in neural networks trained sequentially on multiple tasks.
method Formulates regularization-based continual learning as a second-order Taylor approximation of the loss function, leading to a unified framework.
result Theoretical results indicate the importance of accurate approximation of the Hessian matrix for optimization and generalization.
PIE-PINN estimates elastic properties from noisy, low-res displacement data.
problem Estimating heterogeneous elastic properties from low-resolution, noisy data.
method Probabilistic Physics-Informed Neural Network (PIE-PINN) framework combining B-spline and hierarchical scale model.
result Robust estimation of Young's modulus and Poisson's ratio from noisy, low-resolution displacement data.
Multiple kernel learning (MKL), structured sparsity, and multi-task learning have recently received considerable attention. In this paper, we show how different MKL algorithms can be understood as applications of either regularization on the kernel weights or block-norm-based regularization, which is more common in str…
A major obstacle to developing artificial intelligence applications capable of true lifelong learning is that artificial neural networks quickly or catastrophically forget previously learned tasks when trained on a new one. Numerous methods for alleviating catastrophic forgetting are currently being proposed, but diffe…
In this paper we investigate two hypothesis regarding the use of deep reinforcement learning in multiple tasks. The first hypothesis is driven by the question of whether a deep reinforcement learning algorithm, trained on two similar tasks, is able to outperform two single-task, individually trained algorithms, by more…
Paper formalizes continual semi-supervised anomaly detection, showing promising results.
problem Formalizing continual semi-supervised anomaly detection in real-world conditions.
method Baseline model of variational autoencoder (VAE) with deep generative replay and outlier rejection.
result Outlier rejection shows promising results, often surpassing baseline methods.
Study on local elasticity in neural network training, improving detection of class-specific changes.
problem Improving the detection of class-specific changes in neural network training.
method Comprehensive study of local elasticity, proposing a new definition to address limitations.
result New definition of local elasticity more sharply detects class-specific changes in neural network training.
Unified understanding of three continual learning regularisation methods.
problem Maintaining knowledge of earlier tasks without re-accessing them.
method Three regularisation approaches: Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI), and Memory Aware Synapses (MAS).
result EWC, SI, and MAS are linked to the same theoretical quantity, the square root of the Fisher Information.
This papers introduces an algorithm for the solution of multiple kernel learning (MKL) problems with elastic-net constraints on the kernel weights. The algorithm compares very favourably in terms of time and space complexity to existing approaches and can be implemented with simple code that does not rely on external l…
Unified framework for adaptive learning systems using consolidation and expansion operations.
problem Managing the balance between consolidating known knowledge and expanding into new evidence in adaptive learning systems.
method Introduces Consolidation-Expansion Operator Mechanics (OpMech) with the order-gap metric to control the balance.
result The order-gap signal provides real-time control and termination guarantees for adaptive learning systems.
New SVM model balances sparsity and robustness in noisy data.
problem Noise sensitivity and lack of sparsity in traditional SVM models.
method Combines elastic net loss with robust loss framework, integrates with SVM, uses half-quadratic algorithm.
result Proves sparsity and robustness, outperforms traditional SVMs in noisy environments.
The brain optimizes memory by forgetting what's predictable, improving generalization.
problem Memory consolidation struggles with representational drift, semanticisation, and offline replay.
method Proposes predictive forgetting as a mechanism to optimize generalization by reducing complexity.
result Predictive forgetting improves information-theoretic generalization bounds on stored representations.
Proposes a framework for semi-supervised continual learning from sequentially arriving data.
problem Learning from data with changing task distribution over time, especially in domains with a mix of labeled and unlabeled data.
method Meta-Consolidation for Continual Semi-Supervised Learning (MCSSL) framework with a hypernetwork and semi-supervised auxiliary classifier.
result Significant improvements in continual semi-supervised learning setting.
We empirically investigate the best trade-off between sparse and uniformly-weighted multiple kernel learning (MKL) using the elastic-net regularization on real and simulated datasets. We find that the best trade-off parameter depends not only on the sparsity of the true kernel-weight spectrum but also on the linear dep…
Proposes novel wSVMs for sparse learning and accurate probability estimation.
problem Sparse features with redundant noise limit the performance of existing wSVMs.
method Develops -norm and elastic net regularized wSVMs for automatic variable selection and probability estimation.
result Elastic net regularized wSVMs achieve superior performance in variable selection and probability estimation.
Technological progress is leading to proliferation and diversification of trading venues, thus increasing the relevance of the long-standing question of market fragmentation versus consolidation. To address this issue quantitatively, we analyse systems of adaptive traders that choose where to trade based on their previ…
In information retrieval (IR) and related tasks, term weighting approaches typically consider the frequency of the term in the document and in the collection in order to compute a score reflecting the importance of the term for the document. In tasks characterized by the presence of training data (such as text classifi…
Algorithm improves model performance on shifted concepts without retraining.
problem Improving model performance on shifted concepts with limited source data.
method Model consolidation of intermediate internal distributions after adaptation.
result Effective improvement in model performance on shifted concepts.
Extended Linear Regression: A Kalman Filter Approach for Minimizing Loss via Area Under the Curvecs.LG
Enhances linear regression with Kalman filter for loss minimization.
problem Minimizing loss in linear regression models.
method Integrates Kalman filter and SGD for optimal weight updates.
result Develops optimal linear regression equation with minimum area under curve.
We study a sparse negative binomial regression (NBR) for count data by showing the non-asymptotic advantages of using the elastic-net estimator. Two types of oracle inequalities are derived for the NBR's elastic-net estimates by using the Compatibility Factor Condition and the Stabil Condition. The second type of oracl…
Artificial neural networks face the well-known problem of catastrophic forgetting. What's worse, the degradation of previously learned skills becomes more severe as the task sequence increases, known as the long-term catastrophic forgetting. It is due to two facts: first, as the model learns more tasks, the intersectio…
RENT selects stable features for robust model interpretation.
problem Stability in feature selection for robust model interpretation.
method RENT uses an ensemble of elastic net models trained on distinct subsets of data.
result RENT selects features with high stability improving model robustness.
Analyzes properties of stiffness tensors for elastic wave imaging.
problem Characterizing stiffness tensor fields for elastic wave imaging.
method Finsler-geometric methods applied to anisotropic stiffness tensor fields.
result Conditions for Finsler-geometric methods to be applicable.
A model retains learned knowledge for longer by adding a plastic component to neural networks.
problem Catastrophic forgetting in neural networks when learning new tasks.
method Differentiable Hebbian Consolidation model with a DHP Softmax layer.
result Reduces forgetting in benchmarks like Permuted MNIST and Vision Datasets Mixture.
Maximizes stock portfolio predictability using machine learning.
problem Improving stock portfolio performance through predictive modeling.
method Optimal constrained weights in the MPP constructed using Elastic Net, Random Forest, and Support Vector Regression models.
result MPP portfolios can outperform or underperform the index based on the time period.