Study builds a Japanese financial-specific LLM through continual pre-training.
problem Lack of domain-specific Japanese financial LLMs.
method Continual pre-training on Japanese financial-focused datasets using a base Japanese LLM.
result Tuned model outperforms original model on Japanese financial benchmarks.
Sparse routing networks with co-training prevent catastrophic forgetting in continual learning.
problem Catastrophic forgetting in neural networks trained on a sequence of tasks.
method Sparse routing networks with co-training to minimize interference between dissimilar tasks.
result Sparse routing networks with co-training outperform densely connected networks on benchmarks.
Continuous semi-implicit models enable faster training and better performance in generative modeling.
problem Slow convergence in hierarchical semi-implicit models during training.
method CoSIM, a continuous semi-implicit model that incorporates a continuous transition kernel for efficient training.
result CoSIM achieves superior performance on image generation tasks compared to existing methods.
Improved continuous-time consistency models for large-scale image generation.
problem Training instability and discretization errors in existing diffusion models.
method Unified theoretical framework, improved diffusion process, and network architecture.
result Trained continuous-time CMs at 1.5B parameters, achieving state-of-the-art FID scores.
Improves continual learning with theoretical guarantees and a new algorithm.
problem Learning incremental tasks with dynamic data distributions.
method Contrastive and distillation losses with theoretical performance guarantees.
result Theoretical performance bounds and improved continual learning performance.
Unified continuous diffusion model outperforms discrete alternatives in scalability and quality.
problem Continuous diffusion models were perceived as less scalable than discrete models.
method Reconstructed Plaid model and compared it with modern discrete DLMs, optimizing noise schedule and embeddings via likelihood.
result Unified continuous diffusion model (RePlaid) outperforms discrete models in compute efficiency and quality.
Modyn automates continuous ML model training on growing datasets.
problem Continuous model retraining is costly and impractical.
method Data-centric ML platform with policies for continuous training.
result Modyn enables high throughput training with sample-level data selection.
Continuous time framework for discrete data denoising models.
problem Efficient training and sampling for discrete data denoising models.
method Formulated as Continuous Time Markov Chains (CTMCs), efficient training using continuous time ELBO, high-dimensional CTMC simulation, novel theoretical error bound.
result Continuous time treatment enables novel theoretical error bound between generated and true data distributions.
Catastrophic forgetting is the notorious vulnerability of neural networks to the change of the data distribution while learning. This phenomenon has long been considered a major obstacle for allowing the use of learning agents in realistic continual learning settings. A large body of continual learning research assumes…
Improved tabular models learn better from real-world data.
problem Tabular models perform poorly on real-world datasets when trained only on synthetic data.
method Continued pre-training on a curated set of real-world datasets.
result Real-TabPFN achieves superior predictive accuracy on 29 datasets.
EBMs improve continual learning without external memory or regularization.
problem Improving continual learning without external memory or regularization.
method Energy-Based Models with contrastive divergence training objective.
result EBMs outperform baseline methods on various benchmarks.
RSAC improves lightweight continuous learning efficiency.
problem Excessive training time and memory usage in continuous learning.
method Regularized subspace approximation classifier with feature reduction and regularization.
result RSAC achieves more efficient continuous learning than prior methods.
ContinuousNet generalizes ResNets to continuous dynamical systems.
problem ResNets fail to be meaningful dynamical integrators.
method Embedding continuous dynamical systems into higher-order numerical integration schemes (Runge Kutta).
result ContinuousNet exhibits invariance to discrete time step sizes and numerical integration schemes.
Bilevel Continual Learning improves continual learning by transferring knowledge effectively.
problem Catastrophic forgetting and poor generalization in continual learning.
method Bilevel optimization and dual memory management strategies.
result BCL achieves effective knowledge transfer and alleviates catastrophic forgetting.
Generative models learn distributions of continuous functions.
problem Training generative models on discretized grids limits model size and data type.
method Parameterize data points by continuous functions, learn distributions over these functions.
result Models can learn rich distributions of functions independently of data type and resolution.
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constan…
Continual learning consists in incrementally training a model on a sequence of datasets and testing on the union of all datasets. In this paper, we examine continual learning for the problem of sound classification, in which we wish to refine already trained models to learn new sound classes. In practice one does not w…
VAIOM models financial returns using continuous input and categorical output.
problem Modeling continuous, noisy, and heterogeneous financial data.
method VAIOM is a decoder-only Transformer that separates input representation from output likelihood.
result VAIOM models outperform fixed single-bar LightGBM baseline in both Test halves.
This paper improves continuous adversarial training for LLMs using in-context learning theory.
problem Efficiently defending large language models (LLMs) against jailbreak attacks.
method The paper presents a theoretical analysis of continuous adversarial training (CAT) for LLMs based on in-context learning (ICL) theory, proving a robust generalization bound and proposing an improved regularization term.
result The robust generalization bound explains why CAT can defend against jailbreak prompts and shows that LLM robustness is related to embedding matrix singular values.
In most machine learning algorithms, training data is assumed to be independent and identically distributed (iid). When it is not the case, the algorithm's performances are challenged, leading to the famous phenomenon of catastrophic forgetting. Algorithms dealing with it are gathered in the Continual Learning research…
Study on neural networks' performance in sequential task learning.
problem Understanding the performance of neural networks in sequential task learning.
method Theoretical analysis of generalization performance in continual learning using statistical mechanical analysis of kernel ridge-less regression.
result Characteristic transitions from positive to negative transfer observed in neural networks.
Faster ZSL with continual learning and self-gating.
problem Generalizing models to unseen categories and handling sequential data.
method Meta-continual zero-shot learning (MCZSL) with self-gating and scaled class normalization.
result Outperforms state-of-the-art results with faster training (>100imes). Proposes a new method for continual learning in neural networks.
problem Challenges in applying sequential Bayesian inference to neural networks.
method Sequential function-space variational inference.
result Neural networks trained with the proposed method achieve better predictive accuracy.
EWFM trains continuous flows with only energy evaluations, improving sample quality with fewer computations.
problem Efficiently sampling from complex, high-dimensional Boltzmann distributions using only energy evaluations.
method Energy-Weighted Flow Matching (EWFM) using importance sampling and iterative/annealed training.
result Improved sample quality with up to 3 orders of magnitude fewer energy evaluations compared to existing methods.
Study Transformer layers under cross-entropy training using mean field control.
problem Understanding the behavior of Transformer layers in cross-entropy training.
method Continuous-depth mean field control analysis, treating depth as time and layer parameters as controls.
result Derivation of a Pontryagin condition for the limiting population problem, involving the softmax residual.
Paper proposes a new strategy to improve initial performance of federated models.
problem Weight divergence in Federated Averaging (FedAvg) leads to poor initial performance in federated models.
method Local continual training with importance weights evaluated on a proxy dataset.
result The method significantly improves the initial performance of federated models with minimal extra communication costs.
New method trains normalizing flows using entropy-regularized transport.
problem Training continuous normalizing flows efficiently.
method Formulates flows as gradients of scalar potentials, training only these potentials.
result Trains normalizing flows without explicit flow computation during training.
Study on neural networks forgetting in continual learning.
problem Forgetting in neural networks during continual learning.
method Gradient descent analysis on XOR-cluster datasets.
result Explicit bounds on forgetting rate and generalization gap.
The study analyzes implicit biases in neural networks using backward error analysis.
problem Analyzing implicit biases in multitask and continual learning settings.
method Backward error analysis to compute implicit training biases, deriving modified losses with three terms.
result The conflict term, measuring gradient alignment, is a new quantity in continual learning.
We model how Lipschitz continuity changes during neural network training.
problem Understanding how Lipschitz continuity evolves during training.
method We use a system of stochastic differential equations to capture the dynamics of Lipschitz continuity under SGD.
result We identify three factors driving the evolution of Lipschitz continuity: gradient flow projection, gradient noise, and Hessian projection.
Study explores efficient continual learning of diffusion models.
problem Efficiently reusing computation in training diffusion models as data evolves.
method Benchmarked and analyzed CL methods for Denoising Diffusion Probabilistic Models (DDPMs).
result Experience replay with reduced rehearsal coefficient shows strong performance.
In order to mimic the human ability of continual acquisition and transfer of knowledge across various tasks, a learning system needs the capability for continual learning, effectively utilizing the previously acquired skills. As such, the key challenge is to transfer and generalize the knowledge learned from one task t…
This research explores principles of Lipschitz continuity in neural networks for robustness and generalization.
problem Ensuring robustness and generalization in neural networks, especially to small input perturbations and out-of-distribution data.
method Two complementary perspectives: internal (training dynamics) and external (frequency signal propagation).
result Advances in understanding the principles of Lipschitz continuity in neural networks.
Paper tackles continual learning with Transformers, maintaining performance and speed.
problem Catastrophic forgetting in continual learning with limited resources.
method Incremental training with pre-trained Transformers and Adapters.
result Maintains good predictive performance and inference speed without retraining.
Lipschitz continuity recently becomes popular in generative adversarial networks (GANs). It was observed that the Lipschitz regularized discriminator leads to improved training stability and sample quality. The mainstream implementations of Lipschitz continuity include gradient penalty and spectral normalization. In th…
Study on continuous sequence classification with distribution uncertainty.
problem Classifying continuous sequences with varying distribution uncertainty.
method Proposes distribution-free tests for three test designs: fixed-length, sequential, and two-phase tests.
result Error probabilities decay exponentially fast for all test designs.
This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks. The framework can successfully train both deep discriminative models and deep generative models in compl…
Training a neural network model can be a lifelong learning process and is a computationally intensive one. A severe adverse effect that may occur in deep neural network models is that they can suffer from catastrophic forgetting during retraining on new data. To avoid such disruptions in the continuous learning, one ap…
Neural SDEs model continuous sequences using neural networks.
problem Modeling continuous-time dynamics in sequence data.
method Interprets time-series as samples from a continuous dynamical system, parameterized by Neural SDE.
result Demonstrates superior performance in diverse sequence modeling tasks.
A new method for continual learning in GANs learns new modes with limited data.
problem Learning new target modes with limited samples while preserving previously learned ones.
method Mode-affinity score for generative modeling, generator replay, and weighted label generation.
result Gains over state-of-the-art methods, even with fewer training samples.
New HPO frameworks needed for continual learning.
problem No standard HPO for continual learning.
method Comparative study of HPO frameworks.
result No HPO framework consistently outperforms others.
Model proposes neural network for continuous time dynamics with inductive biases.
problem Training neural networks for small datasets with nonlinear dynamics.
method Inductive biases on decay rates and frequencies using Koopman operator theory.
result Higher forecasting performance with single short training sequence.
ARCADe detects anomalies in a sequence of tasks with limited data.
problem Learning a sequence of anomaly detection tasks with only normal class examples.
method Formulated as a meta-learning problem, ARCADe addresses catastrophic forgetting and overfitting.
result ARCADe outperforms baselines on three datasets.
This paper tackles continuous covariate shift by adaptively training predictors.
problem Continuous covariate shift where input distributions change over time.
method Online density ratio estimation method to adaptively train predictors.
result Excess risk guarantee for the predictor through dynamic regret bound.
Proposes a value-based method for continuous control without an actor.
problem Computational infeasibility of evaluating Q-values in continuous action spaces.
method Structurally maximizable Q-functions, actor-free approach.
result Performance and sample efficiency comparable to actor-critic methods.
One of the challenges in display advertising is that the distribution of features and click through rate (CTR) can exhibit large shifts over time due to seasonality, changes to ad campaigns and other factors. The predominant strategy to keep up with these shifts is to train predictive models continuously, on fresh data…
New adversarial training method improves robustness of power system controllers.
problem Designing robust controllers for complex cyber-physical power systems.
method Adversarial training approach with fixed opponent policy.
result Adversarial trained controllers show useful preventive behaviors in the N-1 problem.
Estimates individual treatment effects using gradient interpolation and kernel smoothing.
problem Estimating individualized continuous treatment effects in observational data.
method Augment training data with independently sampled treatments and inferred counterfactual outcomes using gradient interpolation and kernel smoothing.
result Our method outperforms state-of-the-art methods on counterfactual estimation error.