Adapters add few trainable parameters per task, improving NLP performance.
problem Parameter inefficiency in fine-tuning large pre-trained models for multiple downstream tasks.
method Adapter modules that add only a few trainable parameters per task, allowing for high parameter sharing and task extensibility.
result Adapters achieve near state-of-the-art performance with minimal additional parameters.
We study the performance of the adaptive construction scheme for a Bayesian inference on the Quadratic GARCH model which introduces the asymmetry in time series dynamics. In the adaptive construction scheme a proposal density in the Metropolis-Hastings algorithm is constructed adaptively by changing the parameters of t…
A probabilistic framework for online test-time adaptation
problem Adapting models to new data under distributional shift
method State-space modelling architecture
result Characterizing parameter learning, time evolution, prior tuning, and prediction
CAVIA improves meta-learning by adapting context parameters at test time.
problem Meta-learning overfits and requires complex parallelisation.
method CAVIA splits model into context and shared parameters, updating only the former.
result CAVIA outperforms MAML across regression, classification, and reinforcement learning.
Adapts MBDOE for real-time parameter estimation in complex systems.
problem Costly posterior inference and design optimization in nonlinear systems.
method Combines DAD with differentiable mechanistic models for real-time parameter estimation.
result Demonstrated on four systems, including a DC motor.
ADAPT improves robustness of Vision Transformers without full model fine-tuning.
problem Vulnerability of Vision Transformers to adversarial attacks.
method Parameter-efficient prompt tuning with ADAPT framework for adaptive adversarial training.
result ADAPT achieves robust accuracy of ~40% w.r.t. SOTA methods using only ~1% of the parameters.
LoRA enhances model adaptability without increasing parameters.
problem Theoretical understanding of LoRA's adaptability.
method Theoretical analysis of LoRA's expressive power.
result LoRA can adapt any model to a smaller target model with a specific rank.
Method adapts frozen models for few-shot tasks without training.
problem Deployment constraints limit model updates, necessitating new adaptation methods.
method Exponential tilting of latent distribution for inference.
result Method outperforms parameter-update methods across benchmarks.
Optimizing full likelihoods adapts loss scales and shapes for robust modeling.
problem Rigid loss functions limit model adaptability and robustness.
method Optimize full likelihoods with adjustable parameters.
result Adaptive tuning of loss scales and shapes improves model robustness.
We derive new theoretical results on the properties of the adaptive least absolute shrinkage and selection operator (adaptive lasso) for time series regression models. In particular, we investigate the question of how to conduct finite sample inference on the parameters given an adaptive lasso model for some fixed valu…
BOFFIN TTS optimizes hyper-parameters for new speaker adaptation.
problem Fine-tuning a pre-trained TTS model for a new speaker with limited data.
method Bayesian optimization to efficiently find optimal hyper-parameters.
result Average 30% improvement in speaker similarity over standard techniques.
Adaptive multi-domain learning reduces parameter count for efficient deep learning.
problem Different domains have varying complexity, leading to inefficient model training.
method Proposes adaptive parameterization to reduce model complexity without sacrificing performance.
result Efficient multi-domain learning solutions with far fewer parameters.
Unified view of context-adaptive inference across statistics, meta-learning, and foundation models.
problem Adapting predictive systems to specific situations.
method Unified framework for context-adaptive inference, bridging statistics, meta-learning, and foundation models.
result Explicit and implicit approaches are mathematically equivalent under certain conditions.
Framework transfers knowledge across multiple target domains without shared categories.
problem Learning unlabeled target domains without shared categories.
method Model parameter adaptation (PA-1SmT) to transfer knowledge through a common model parameter dictionary.
result Framework demonstrates superiority on three domain adaptation benchmark datasets.
ATSDLN adapts to time series data for anomaly detection.
problem Challenges in selecting and optimizing anomaly detectors for time series data.
method Adaptive Time Series Detector Learning Network (ATSDLN) that selects and optimizes detectors and parameters.
result ATSDLN outperforms other methods in anomaly detection across various datasets.
Bayesian adaptive designs can be biased by active learning, especially with misspecified models.
problem Active learning bias in Bayesian adaptive experimental designs.
method Analysis of linear and preference learning models, empirical testing.
result Model misspecification and noise influence active learning bias in Bayesian designs.
New method reduces memory usage for adaptive optimization.
problem Significant memory overhead in adaptive optimizers like Adagrad and Adam.
method Memory-efficient adaptive optimization method that retains per-parameter adaptivity.
result Significantly larger models and batch sizes possible with up to 2-fold speedups.
Paper proposes adaptive parameter selection for KGD algorithms.
problem Improving parameter selection for kernel-based gradient descent.
method Integrates bias-variance analysis with splitting method, introduces empirical effective dimension.
result Adaptive parameter selection strategy achieves optimal generalization error bound.
Adaptive networks improve model robustness through conditional normalization.
problem Limited robustness of adversarial-trained networks due to network capacity and training samples.
method Proposes a conditional normalization module to adapt networks during adversarial training.
result Adaptive networks outperform both clean validation accuracy and robustness compared to non-adaptive counterparts.
We perform Markov chain Monte Carlo simulations for a Bayesian inference of the GJR-GARCH model which is one of asymmetric GARCH models. The adaptive construction scheme is used for the construction of the proposal density in the Metropolis-Hastings algorithm and the parameters of the proposal density are determined ad…
ARU adapts deep forecasting models in streaming data with efficient updates.
problem Adapting deep globally trained models for streaming data efficiently.
method ARU combines deep global models with closed-form linear models for per-series adaptation.
result ARU outperforms local adaptation methods on various datasets.
We consider the scenario where the parameters of a probabilistic model are expected to vary over time. We construct a novel prior distribution that promotes sparsity and adapts the strength of correlation between parameters at successive timesteps, based on the data. We derive approximate variational inference procedur…
This paper presents the construction of a particle filter, which incorporates elements inspired by genetic algorithms, in order to achieve accelerated adaptation of the estimated posterior distribution to changes in model parameters. Specifically, the filter is designed for the situation where the subsequent data in on…
New method uses fewer parameters to match state-of-the-art performance on multiple natural language tasks.
problem Efficiently adapting BERT for multiple tasks with fewer parameters.
method PALs (projected attention layers) for shared BERT model with task-specific parameters.
result Matches state-of-the-art performance on GLUE benchmark with 7 times fewer parameters.
Improved SGD with AdaGrad stepsizes adapts to unknown parameters and unbounded gradients.
problem Adaptive optimization with unknown parameters and unbounded gradients.
method Stochastic Gradient Descent with AdaGrad stepsizes, without assuming problem parameters or strong global Lipschitz conditions.
result Sharp rates of convergence in both low-noise and high-noise regimes, supporting an affine variance noise model.
ADML combines debiased learning with data-driven model selection for efficient inference.
problem Debiased machine learning estimators can be unstable and biased in nonparametric models.
method Data-driven model selection techniques combined with debiased machine learning.
result ADML estimators yield superefficient inference for pathwise differentiable parameters.
LaLoRA prevents forgetting in LoRA fine-tuning.
problem Catastrophic forgetting in fine-tuned models.
method LaLoRA applies Laplace approximation to LoRA weights for regularization.
result Improved learning-forgetting trade-off with controllable regularization strength.
A method connects KDE to sparse mixture models with adaptive regularization.
problem Estimating Gaussian mixture models from sparse data.
method Generalized expectation-maximization method with adaptive regularization.
result Sparse mixture models retain details from adaptive KDE.
New analysis shows surprising results on adaptation speed of causal models.
problem Investigate the adaptation speed of causal models under interventions.
method Use convergence rates from stochastic optimization to measure adaptation speed.
result Surprising findings: anticausal model can be faster than causal model under certain conditions.
Method aggregates models with different hyper-parameters to adapt to target domain.
problem Choosing hyper-parameters for unsupervised domain adaptation.
method Linear aggregation of models with different hyper-parameters using weighted least squares for vector-valued functions.
result The target error is asymptotically not worse than twice the error of the optimal aggregation.
PriorGuide adapts diffusion models to new priors at test time.
problem Limited applicability of prior distributions in diffusion-based inference.
method PriorGuide uses a guidance approximation to adapt diffusion models to new priors at test time.
result Enhances the versatility of pre-trained inference models by allowing flexible adaptation to new priors.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
Federated learning improves with adaptive hyper-parameters and representation matching.
problem Heterogeneous client data leads to divergent local models in federated learning.
method Representation matching and adaptive hyper-parameters.
result Significant performance and robustness improvements in federated learning.
KOMET identifies Koopman operators from model parameter trajectories to adapt to evolving data distributions.
problem Adaptation of parametric models to non-stationary environments.
method Data-driven framework using Koopman operator identification and Extended Dynamic Mode Decomposition (EDMD).
result KOMET achieves high autonomous-rollout accuracies of 0.981 to 1.000 over 100 time steps on various drifting datasets.
GeLoRA optimizes LoRA fine-tuning by dynamically adjusting ranks based on intrinsic dimensionality.
problem Efficient fine-tuning of large language models with limited computational resources.
method GeLoRA computes intrinsic dimensionality to adaptively select LoRA ranks, balancing expressivity and efficiency.
result GeLoRA consistently outperforms recent baselines within the same parameter budget on multiple tasks.
Adaptive conformal inference without data exchangeability assumptions.
problem Real-world scenarios often violate the data exchangeability assumption for conformal prediction.
method Parameter-free online convex optimization for adaptive conformal inference.
result Controls long-term miscoverage frequency at a nominal level empirically.
NeAda solves nonconvex minimax optimization by balancing primal and dual variables adaptively.
problem Nonconvex minimax optimization challenges with parameter-agnostic adaptive algorithms.
method Nested Adaptive (NeAda) framework with inner and outer loops for primal and dual variables.
result Achieves near-optimal convergence rates for nonconvex-strongly-concave problems.
Adaptive weight-decay improves deep neural network performance.
problem Overfitting in deep neural networks.
method AdaDecay adjusts weight-decay adaptively based on gradient norms within each layer.
result AdaDecay improves generalization and accuracy across various datasets and models.
Adaptive tuning of portfolio selection parameters improves performance in volatile markets.
problem Improving online portfolio selection in volatile financial markets.
method Modeling parameter space with Gaussian process prior and using adaptive Bayesian optimization for automatic configuration.
result Oracle-based adaptive configuration enhances performance of online portfolio selection algorithms.
AdaCat improves density estimation and planning in autoregressive models.
problem Efficiently modeling sharp density changes in continuous data.
method Adaptive Categorical Discretization (AdaCat) for autoregressive models.
result Improves density estimation and planning in various data types.
This paper develops a matrix-variate adaptive Markov chain Monte Carlo (MCMC) methodology for Bayesian Cointegrated Vector Auto Regressions (CVAR). We replace the popular approach to sampling Bayesian CVAR models, involving griddy Gibbs, with an automated efficient alternative, based on the Adaptive Metropolis algorith…
Bayesian approach improves speech recognition with limited speaker data.
problem Reduces mismatch between training and evaluation data due to speaker differences.
method Bayesian learning for DNN adaptation models with limited speaker data.
result Bayesian adaptation consistently outperforms deterministic methods, reducing word error rates up to 1.4%.
This paper proposes a new method to adapt ROMs for new parameter settings.
problem ROMs lack robustness when applied to new parameter settings.
method Regression trees on Grassmann Manifold to learn the mapping between parameters and POD bases.
result The proposed method is capable of establishing the mapping between parameters and POD bases, thus adapting ROMs for new parameters.
A new method transfers parameters in ELM networks using projective model.
problem Parameter transfer in extreme learning machine networks.
method Projective model to bridge source and target model parameters, L2,1-norm penalty for joint feature selection and parameter transfer.
result Significantly outperforms non-transfer ELM networks and other methods.
Paper explores BERT's efficiency on SQuAD2.0, freezing layers and using adapters.
problem Improving BERT's efficiency for SQuAD2.0 while maintaining performance.
method Freezing transformer layers, using adapters, and context-aware convolutional filters.
result Context-aware convolutional filters do not improve practical efficiency.
GIST adapts HMC by tuning parameters based on position and momentum.
problem Locally adaptive sampling in Hamiltonian Monte Carlo.
method GIST uses Gibbs sampling to adaptively tune HMC parameters.
result GIST improves sampling efficiency for high-dimensional models.
Paper analyzes adaptive Lasso for high-dimensional diffusion processes, improving support recovery and bias.
problem Support recovery for high-dimensional diffusion processes under sparsity constraints.
method Adaptive Lasso estimator for d-dimensional ergodic diffusion process, focusing on linear models.
result Adaptive Lasso achieves support recovery and asymptotic normality for drift parameter under certain conditions.
New method improves neural network robustness to adversarial attacks.
problem Improving adversarial robustness of neural networks.
method Inspired by adaptive control theory, the approach uses persistency of excitation to constrain gradient descent updates.
result Networks trained with the PoE-motivated learning rate schedule are significantly more robust to adversarial attacks.