Adaptive networks improve model robustness through conditional normalization.
problem Limited robustness of adversarial-trained networks due to network capacity and training samples.
method Proposes a conditional normalization module to adapt networks during adversarial training.
result Adaptive networks outperform both clean validation accuracy and robustness compared to non-adaptive counterparts.
Enhances physics-informed neural networks with adaptive sampling and weighting.
problem Challenges in training physics-informed neural networks on complex problems.
method Hybrid adaptive sampling and weighting method.
result Consistently improves prediction accuracy and training efficiency.
Self-training improves gradual domain adaptation with unlabeled data.
problem Improving machine learning models' adaptability to gradually shifting data distributions.
method Proved upper bounds on self-training error, highlighted the importance of regularization and label sharpening, and demonstrated algorithmic insights.
result Self-training works well for gradual shifts, especially with small Wasserstein-infinity distance.
learn2mix trains neural nets faster by adjusting class proportions dynamically.
problem Training neural nets efficiently with limited resources and imbalanced classes.
method Adaptive class proportion adjustment during training.
result Neural nets trained with learn2mix converge faster than static methods.
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
problem The generalization gap between large-batch and small-batch training in deep learning.
method AdAdaGrad introduces adaptive batch size strategies derived from adaptive sampling methods.
result AdAdaGradNorm converges to a first-order stationary point with a rate of O(1/K) in K iterations.
Self-adaptive training improves deep learning robustness.
problem Improving deep learning performance on corrupted data.
method Dynamic correction of problematic labels using model predictions.
result Self-adaptive training significantly improves generalization over ERM under various levels of noise.
Adaptive kernel density estimation improves accuracy in high dimensions.
problem Challenges in high-dimensional density estimation with traditional methods.
method Pre-training a neural network to recommend location-adaptive kernels.
result Effective density estimation in high dimensions with improved accuracy.
Paper proposes an adaptive gradient method for federated learning.
problem Communication efficiency in federated learning.
method Adaptive gradient method for federated learning.
result Guaranteed convergence and communication efficiency in federated learning.
Proposes blockwise adaptive stepsize for faster training and better generalization in deep learning.
problem Widespread use of coordinate-wise adaptive methods like RMSprop and Adam leads to worse generalization than SGD.
method Splits network parameters into blocks and uses a blockwise adaptive stepsize, balancing adaptivity and generalization.
result Blockwise adaptive gradient descent converges faster and has lower generalization error than coordinate-wise adaptive methods.
Unified framework for adaptive connection sampling in GNNs improves performance and robustness.
problem Over-smoothing and over-fitting in deep GNNs.
method Adaptive connection sampling trained jointly with GNN model parameters.
result Adaptive connection sampling mathematically equivalent to Bayesian GNNs approximation.
ARU adapts deep forecasting models in streaming data with efficient updates.
problem Adapting deep globally trained models for streaming data efficiently.
method ARU combines deep global models with closed-form linear models for per-series adaptation.
result ARU outperforms local adaptation methods on various datasets.
Paper investigates robustness to interference as a new training signal for meta-learning.
problem Improving incremental learning through robust representations.
method Directly minimizing catastrophic interference as a training signal.
result Representations learned to minimize interference lead to better incremental learning.
Transfer learning improves ASR on limited resources.
problem End-to-end ASR training requires massive data and compute.
method Model adaptation using transfer learning.
result ASR models trained on limited resources achieve similar accuracy.
New algorithm closes the generalization gap of adaptive gradient methods in deep neural networks.
problem Generalization gap of adaptive gradient methods in training deep neural networks.
method Designing a new algorithm called Partially adaptive momentum estimation method (PAM) that unifies Adam/Amsgrad with SGD.
result Our proposed algorithm maintains fast convergence rates as Adam/Amsgrad while generalizing as well as SGD.
Aligns uncertainty predictions for domain adaptation using pre-trained deep networks.
problem Domain adaptation with unlabelled target data.
method Adversarial learning to align uncertainty predictions between source and target domains.
result Class prediction uncertainty on target domain matches source domain.
Prodigy estimates learning rate without tuning, improving convergence.
problem Estimating optimal learning rate in adaptive methods.
method Prodigy modifies D-Adaptation to estimate distance to solution D D D . result Prodigy improves convergence rate by a factor of O ( log ( D / d 0 ) ) O(\sqrt{\log(D/d_0)}) O ( log ( D / d 0 ) ) . FPGA-based multi-layer equalizer adapts to changing channels.
problem Real-time adaptation to time-varying channel impairments.
method Multi-layer machine learning on FPGA with on-chip gradient backpropagation training.
result Real-time adaptation to changing channel conditions achieved.
CrossTrainer adapts models across diverse datasets using loss reweighting.
problem Training models with data from varying sources or of varying quality.
method Loss reweighting with optimizations for hyperparameter tuning.
result High model accuracy across various datasets with improved training time.
Local AdaAlter reduces communication in SGD with adaptive learning rates.
problem Communication overhead in distributed training.
method Novel SGD variant with adaptive learning rates and reduced communication.
result Empirically reduces communication overhead by up to 30%.
Adaptive batch size schedules improve language model training efficiency and generalization.
problem Dilemma of choosing batch sizes in large-scale model training.
method General-purpose adaptive batch size schedules compatible with data and model parallelism.
result Adaptive batch size schedules outperform constant batch sizes and heuristic warmup schedules.
Adaptive methods optimize machine learning training faster.
problem Non-adaptive stochastic optimization requires tuning for each application.
method Develop adaptive stochastic optimization methods.
result Adaptive methods offer computational savings for large-scale systems.
New model-based methods adapt pre-trained policies to unseen environments efficiently.
problem High sample complexity in reinforcement learning limits practical applications.
method Combines online learning and adaptive control to adapt policies in unseen environments.
result Proves policies can quickly recover trajectories from source to target environments.
Improves deep neural network training and accuracy with adaptive basis approach.
problem Gap between theoretical and practical performance of deep neural networks.
method Adaptive basis viewpoint, novel initializations, hybrid optimizer.
result Dramatic increases in accuracy and convergence rate for various DNN applications.
Generative adversarial network improves spectrum sensing accuracy.
problem Lack of sufficient and adaptable training data for spectrum sensing.
method Generative adversarial network (GAN) for data augmentation and adaptation.
result Training data augmentation significantly increases classifier accuracy.
ADASS selects adaptive subsets for SGD training acceleration.
problem Fixed sample size in SGD limits training efficiency.
method ADASS selects adaptive subsets based on Lipschitz constants.
result ADASS achieves comparable accuracy with full training set.
Supporting evidence for adaptive feature program across diverse models.
problem Analyzing feature learning in neural networks.
method Over-parameterized sequence models and feature error measure (FEM).
result FEM is decreasing during training of adaptive feature models.
Adaptive loss scaling speeds up and improves deep learning training.
problem Numerical underflow in mixed precision training.
method Adaptive loss scaling that automatically computes layer-wise loss scale values during training.
result Adaptive loss scaling leads to shorter convergence time and improved accuracy.
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
Paper tackles domain adaptation in object detection, improving accuracy.
problem Real-world object detection faces domain shift issues.
method Formulates domain adaptation as noisy label training; uses noisy bounding boxes from source domain.
result Significantly improves object detection accuracy on various domain adaptation scenarios.
AdaScale SGD adapts learning rates for large-batch training efficiently.
problem Adapting learning rates for large-batch training to balance speed-ups and model quality.
method Adaptive learning rate adaptation based on gradient variance.
result AdaScale achieves reliable speed-ups for a wide range of batch sizes without degrading model quality.
The paper improves generalization bounds for domain adaptation.
problem Improving generalization bounds for domain adaptation under practical conditions.
method Derives generalization bounds for domain adaptation based on finitely many moments and smoothness conditions.
result Obtains generalization bounds for domain adaptation.
Adaptive methods often find worse generalization than SGD in overparameterized problems.
problem The performance of adaptive methods in overparameterized problems.
method Adaptive methods (AdaGrad, RMSProp, Adam) compared to gradient descent (GD) and stochastic gradient descent (SGD).
result Adaptive methods often generalize worse than SGD, even when they have better training performance.
New method AdaMod stabilizes deep neural network training by limiting adaptive learning rates.
problem Adaptive learning rates can produce extremely large values at the start of training, hindering learning.
method AdaMod uses adaptive and momental upper bounds to restrict learning rates dynamically.
result AdaMod eliminates large learning rates and improves training on complex networks.
A probabilistic framework for online test-time adaptation
problem Adapting models to new data under distributional shift
method State-space modelling architecture
result Characterizing parameter learning, time evolution, prior tuning, and prediction
This paper explains why Adam generalizes worse than SGD by analyzing its components.
problem Understanding why Adam generalizes worse than Stochastic Gradient Descent (SGD).
method Diffusion theoretical framework to disentangle the effects of Adaptive Learning Rate and Momentum.
result Adaptive Learning Rate helps escape saddle points but not select flat minima, while Momentum provides a drift effect to help pass through saddle points.
Discriminative active learning reduces data annotation costs for domain adaptation.
problem Conditional shift problem hinders domain adaptation between related but different domains.
method Three-stage active adversarial training: invariant feature space learning, uncertainty and diversity criteria, re-training with queried labels.
result Empirical comparisons show the proposed approach is more effective than existing methods.
Conformal Bayes under label shift: post-hoc calibration vs. in-training adaptation
problem Bayesian prediction sets under label shift
method Post-hoc calibration vs. In-training adaptation
result Both strategies achieve valid coverage equally in an unbiased training regime
Domain adaptation reduces prosthetic training time for amputees.
problem Reducing training time for non-invasive myoelectric prostheses.
method Evaluation of domain adaptation algorithms on amputee and intact subjects data.
result Previous experience from other subjects reduces training time by about an order of magnitude.
AdaBatch dynamically adjusts batch size during training for deep learning models.
problem Choosing optimal batch size for deep neural networks.
method Adaptive batch size adjustment during training.
result Adaptive batch sizes improve performance by up to 6.25x on 4 GPUs with minimal accuracy loss.
New method adapts without backprop, faster and better.
problem Efficient domain adaptation without source data.
method Computing class prototypes from pre-trained model.
result Significant accuracy improvements over pre-trained model.
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
problem Communication overhead in distributed machine learning systems.
method Adaptive gradient method with gradient compression, scalable system BytePS-Compress.
result Convergence rate of O ( 1 / T ) \mathcal{O}(1/\sqrt{T}) O ( 1/ T ) for non-convex problems. Domain adaptation is the supervised learning setting in which the training and test data are sampled from different distributions: training data is sampled from a source domain, whilst test data is sampled from a target domain. This paper proposes and studies an approach, called feature-level domain adaptation (FLDA), …
Improves adversarial training generalization with domain adaptation.
problem Weak generalization of adversarial training due to lack of representative adversarial samples.
method Adversarial Training with Domain Adaptation (ATDA) method.
result ATDA greatly improves adversarial training generalization and model smoothness.
Study on sparsity in CNNs trained with adaptive methods.
problem Understanding and optimizing sparsity in CNNs trained with adaptive methods.
method Experimental study of filter level sparsity in CNNs with BN and ReLU, using adaptive gradient descent and L2 regularization.
result Implicit sparsity can improve CNN performance and speedup without modifications.
Bayesian method improves SOM training for dynamic data.
problem Training Self Organizing Maps (SOM) for non-stationary data.
method Variational Bayesian approach with adaptive neighborhood radius.
result Method outperforms other adaptive methods in high dimensions.
This thesis studies domain adaptation under minimal distribution similarity assumptions using moments.
problem Learning from samples with distributions different from training samples.
method Uses minimal similarity assumptions modeled by moments.
result Establishes learning bounds and algorithms for domain adaptation.
Adaptive Gaussian process models for efficient Bayesian inference.
problem Expensive forward models in Bayesian inference.
method Fully Bayesian approach with adaptive training designs maximizing expected improvement.
result Adaptive designs lead to more accurate posterior estimation at lower cost.
Improves target data performance of adaptive classifiers.
problem Adaptive classifiers perform poorly on target data due to distribution shift.
method Constructs robust discriminant analysis estimators.
result Robust discriminant analysis outperforms non-adaptive classifiers on target data.