WASH trains ensembles with shuffled weights to improve accuracy and reduce communication.
problem Training ensembles for weight averaging leads to models converging to different loss basins.
method WASH randomly shuffles a small percentage of weights during training to keep models within the same basin.
result WASH achieves state-of-the-art image classification accuracy with lower communication costs.
Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.
problem Improving convergence of SGD in various settings.
method Develops a general weighted averaging scheme for SGD and establishes asymptotic normality.
result Establishes asymptotic normality and optimality of weighted averaged SGD solutions.
A new method for averaging model predictions using minimum divergence.
problem Improving model averaging methods, especially in small samples.
method Minimum divergence framework for model weight calculation.
result Empirically outperforms standard model averaging methods.
Paper introduces stability in model averaging and proposes a L2-penalty method.
problem Theoretical properties of model averaging from stability perspective.
method Introduces stability, defines asymptotic empirical risk minimizer, and proposes L2-penalty model averaging method.
result Proposed L2-penalty method ensures stability and consistency under reasonable conditions.
This study optimizes model averaging for personalized collaborative learning.
problem Differences in data or objectives between nodes in federated learning.
method Weighted averaging between local and global models for scalar mean estimation.
result There is always some positive model averaging that reduces expected squared error.
BayesBlend blends multiple models' predictions for better insurance loss predictions.
problem Improving insurance loss predictions by combining multiple models.
method Pseudo-Bayesian model averaging, stacking, and hierarchical stacking.
result BayesBlend provides a user-friendly way to blend model predictions and estimate weights.
In mixture model-based clustering applications, it is common to fit several models from a family and report clustering results from only the `best' one. In such circumstances, selection of this best model is achieved using a model selection criterion, most often the Bayesian information criterion. Rather than throw awa…
The study introduces anytime learning schedules for large language models without fixed horizons.
problem Training large language models without knowing the total training horizon.
method Theoretical analysis and weight averaging to create anytime learning schedules.
result Theoretical and empirical evidence shows that weight averaging with simple step sizes can achieve comparable final loss to well-tuned cosine schedules.
We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly and then refines it by averaging the weights of multiple models computed independently and in parallel. The resulting models generalize equ…
Averaging recent model checkpoints speeds up training time.
problem Training large vision or language models is time-consuming.
method Average the weights of the k latest checkpoints.
result Speeds up training by dozens of epochs, saving up to 68 GPU hours.
Designing a deep neural network (DNN) with good generalization capability is a complex process especially when the weights are severely quantized. Model averaging is a promising approach for achieving the good generalization capability of DNNs, especially when the loss surface for training contains many sharp minima. W…
Unified analysis of finite weight averaging methods in deep learning.
problem Theoretical explanation of finite weight averaging methods in deep learning.
method Generalized SGD and LAWA, establishing convergence and generalization bounds.
result FWA achieves faster convergence and better generalization compared to SGD.
This paper analyzes SGD with increasingly weighted averaging for optimization and generalization.
problem Improving optimization and generalization for non-strongly convex objectives.
method Comprehensive analysis of increasingly weighted averaging schemes for convex, strongly convex, and non-convex objectives.
result The weight α affects both optimization and generalization errors, revealing a trade-off. Optimal model averaging for conditional generative models improves performance across various data types.
problem Multiple plausible generators for conditional distributions can vary in performance.
method Sample-based maximum mean discrepancy, static model averaging, and mixture-of-experts model averaging.
result MoEMA improves over competing baselines across various data types.
clusterBMA combines clustering results from multiple models using Bayesian model averaging.
problem Uncertainty in model selection for clustering.
method Bayesian model averaging to combine results from multiple clustering algorithms.
result ClusterBMA offers probabilistic cluster allocations and quantifies model-based uncertainty.
Bayesian stacking improves model performance with varying model weights.
problem Improving model predictions with heterogeneous input performance.
method Bayesian hierarchical stacking with varying model weights inferred via Bayesian inference.
result Hierarchical stacking yields better predictions than linear averaging.
New moving average adapts weight dynamically based on polynomial and wavefunction.
problem Lagging traditional moving averages in adjusting to changes in data.
method Develops a moving average with weight as a polynomial of a wavefunction from an eigenproblem.
result Immediate 'switch' without lag, adapting to changes in data.
Proposes a non-convex optimization method for a parsimonious weighted naive Bayes classifier.
problem Improving naïve Bayes classifier performance with a large number of input variables.
method Sparse regularization of model log-likelihood for direct estimation of variable weights.
result Optimization-based weighted naïve Bayes classifiers achieve equivalent performance to averaging-based classifiers.
Bayesian model averaging improves causal effect estimation by averaging over multiple models.
problem Estimating causal effects under linear Structural Causal Models (SCMs).
method Bayesian model averaging using Gaussian scale mixture distributions for computational efficiency.
result Bayesian model averaging is optimal for causal effect estimation.
This paper solves aggregation of Pareto optimal models by using Bayesian priors and weighted averaging.
problem How to rationally aggregate Pareto optimal models while preserving Pareto efficiency.
method Four logical steps: 1) Bayesian models, 2) Prior as preference ranking, 3) Consistent aggregation, 4) Weighted average of priors.
result All rational/consistent aggregation rules follow a generalized hierarchical Bayesian model.
This paper studies robust estimation methods in high dimensions, comparing model-averaged and composite quantile estimators.
problem Understanding robustness in high-dimensional regularized estimation.
method Optimal weights are determined by minimizing the asymptotic mean squared error, incorporating regularization effects without perfect selection.
result Model-averaged and composite quantile estimators often outperform least-squares methods in prediction quality.
Distributed statistical learning problems arise commonly when dealing with large datasets. In this setup, datasets are partitioned over machines, which compute locally, and communicate short messages. Communication is often the bottleneck. In this paper, we study one-step and iterative weighted parameter averaging in s…
Optimal weight windows are found by projecting the origin onto a convex polytope.
problem Finding the best weight windows for a weighted moving average smoother.
method Formulated as a quadratic program and projection onto a convex polytope.
result Optimal weight windows are symmetrical and decrease in weight away from the center.
Federated learning technique improves convergence speed with communication delays.
problem Communication delays between edge nodes and aggregator in federated learning.
method Developed FedDelAvg, a technique that generalizes federated averaging to incorporate a weighting between current local model and delayed global model.
result FedDelAvg achieves a significant improvement in convergence speed, especially when optimizing the weighting scheme to account for delays.
In causal inference, a variety of causal effect estimands have been studied, including the sample, uncensored, target, conditional, optimal subpopulation, and optimal weighted average treatment effects. Ad-hoc methods have been developed for each estimand based on inverse probability weighting (IPW) and on outcome regr…
Improved averaging method for noisy observations converges strongly.
problem Noisy observations from random dynamical systems require stable estimates.
method Introduced p-EMA, a modified exponential moving average with subharmonic weight decay. result Stochastic convergence guarantees for p-EMA under mild assumptions. Improved neural networks by averaging late-stage weights.
problem Improving the performance of neural networks.
method Ensemble late-stage weights and average them.
result Augmenting standard models with late-phase weights improves generalization.
Simplified analysis of SGD for linear regression with weight averaging.
problem Understanding SGD optimization in linear regression models.
method Simplified analysis using linear algebra tools, bypassing complex operator manipulations.
result Recovery of bias and variance bounds for SGD in linear regression.
New class of heavy-tailed distributions shows weighted averages dominate individual variables.
problem Understanding and comparing risks in heavy-tailed distributions.
method Introducing a new class of heavy-tailed distributions and proving stochastic dominance relations.
result Weighted averages of random variables in this class are stochastically larger than individual variables.
We propose methods for distributed graph-based multi-task learning that are based on weighted averaging of messages from other machines. Uniform averaging or diminishing stepsize in these methods would yield consensus (single task) learning. We show how simply skewing the averaging weights or controlling the stepsize a…
The study analyzes when Bayesian averaging over decision trees is reliable.
problem When do Bayesian model averaging weights over decision trees provide reliable information?
method Closed-form solution for Bayesian decision trees with Catalan-exponential priors.
result Established a complete non-asymptotic theory of rational commitment thresholds.
We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this …
Revises Bayesian model averaging for foundation models.
problem Ensemble pre-trained and lightly-finetuned foundation models for improved classification performance.
method Introduces trainable linear classifiers and computationally cheaper model averaging scheme (OMA).
result Ensembled models can better predict on various datasets.
Sharp analysis of out-of-distribution error in overparameterized models with importance weights.
problem Understanding and quantifying the degradation of performance in overparameterized models when faced with underrepresented data.
method Sharp analysis of an overparameterized Gaussian mixture model with spurious features and cost-sensitive interpolating solutions incorporating importance weights.
result Characterization of a novel tradeoff between worst-case robustness and average accuracy as a function of importance weight magnitude.
CDST improves ensemble prediction by adjusting model weights based on covariates.
problem Improving ensemble prediction accuracy in complex scenarios.
method Covariate-dependent stacking (CDST) with flexible model weights estimated via cross-validation.
result CDST consistently outperforms conventional model averaging methods in complex datasets.
New algorithms bound graph structure sampling and learning high-dimensional graphical models.
problem Learning high-dimensional graphical models and efficient graph structure sampling.
method Online learning framework with exponentially weighted average (EWA) or randomized weighted majority (RWM) forecasters using log loss function.
result New sample complexity bounds and efficient algorithms for learning Bayes nets, including trees and chordal skeletons.
Optimal weighted random forests improve prediction accuracy.
problem Unequal prediction performance among random forest trees.
method Proposes 1-step and 2-step optimal weighting algorithms.
result Asymptotically optimal in terms of squared loss and risk.
In this paper, we focus on quantifying model stability as a function of random seed by investigating the effects of the induced randomness on model performance and the robustness of the model in general. We specifically perform a controlled study on the effect of random seeds on the behaviour of attention, gradient-bas…
Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedure…
VISA improves inference efficiency for complex models.
problem Efficient approximate inference in computationally intensive models.
method Sequential sample-average approximations within a trust region.
result VISA achieves comparable accuracy with computational savings.
Kernel balancing weights are generalized as KRRR, providing better confidence intervals for treatment effects.
problem Lack of generalization error, correct feature specification, and limited to average effects.
method Interpreting kernel balancing weights as KRRR, relaxing feature specification, and extending Gaussian approximation.
result KRRR provides strong generalization properties and justifies confidence sets for causal functions.
A new method forecasts financial tail risks by combining and weighting quantiles.
problem Reducing uncertainty in financial tail risk forecasting.
method Two-step procedure: quantile combination followed by ES computation.
result The proposed framework outperforms individual models and simple approaches.
Enhances trading signals using image analysis and weighted moving averages.
problem Improving price trend trading strategies in financial markets.
method Image-induced importance weights applied to weighted moving averages of trading signals.
result Significant enhancement of price trend trading signals with improved portfolio selection.
Proposes SWA for adversarial training to improve model robustness.
problem Overfitting in adversarial training due to insufficient sample complexity.
method Adversarial training with Stochastic Weight Average (SWA).
result Improves model robustness on CIFAR-10, CIFAR-100, and SVHN datasets.
New method SF-AdamW trains large models without decay phases or memory overhead.
problem Inadequate fixed compute budgets for large-scale training.
method Schedule-Free (SF) method revisited and refined.
result SF-AdamW effectively navigates loss landscape without decay phases or memory overhead.
New averaging technique speeds up Newton method convergence.
problem Superlinear convergence of stochastic Newton methods with noisy Hessians.
method Hessian averaging to reduce noise and maintain superlinear convergence.
result Hessian averaging achieves superlinear convergence with a non-asymptotic rate.
A new method for exponentially weighted moving models using approximations.
problem Efficiently updating moving averages for time series data.
method Approximates EWMM using a fixed window and quadratic term, solving non-growing problems.
result Approximation produces estimates similar to exact EWMM.
IA-BMA adapts model weights to inputs for better predictions.
problem Predicting with multiple models in heterogeneous settings.
method Input adaptive Bayesian Model Averaging (IA-BMA) with an input adaptive prior and amortized variational inference.
result IA-BMA consistently delivers more accurate and better-calibrated predictions.