Paper introduces stability in model averaging and proposes a L2-penalty method.
problem Theoretical properties of model averaging from stability perspective.
method Introduces stability, defines asymptotic empirical risk minimizer, and proposes L2-penalty model averaging method.
result Proposed L2-penalty method ensures stability and consistency under reasonable conditions.
A new averaging scheme improves stochastic gradient methods.
problem Improving the performance of stochastic gradient descent.
method Proposes a weighted averaging scheme with decaying weights.
result The method has a regularizing effect equivalent to ridge regression.
Increasing iterate averaging improves convergence rates for saddle-point problems.
problem Solving saddle-point problems efficiently.
method Increasing iterate averaging schemes applied to various first-order methods.
result Increasing iterate averaging preserves the O(1/T) convergence rate with no additional assumptions or overhead. WASH trains ensembles with shuffled weights to improve accuracy and reduce communication.
problem Training ensembles for weight averaging leads to models converging to different loss basins.
method WASH randomly shuffles a small percentage of weights during training to keep models within the same basin.
result WASH achieves state-of-the-art image classification accuracy with lower communication costs.
A new method for averaging model predictions using minimum divergence.
problem Improving model averaging methods, especially in small samples.
method Minimum divergence framework for model weight calculation.
result Empirically outperforms standard model averaging methods.
This work proposes a model averaging method for SVM that avoids redundant covariates and achieves asymptotic optimality.
problem Redundant covariates impair SVM performance in high-dimensional settings.
method Frequentist model averaging procedure for SVM using cross-validation to select optimal weights.
result The proposed method achieves asymptotic optimality in SVM model averaging.
Study develops method for estimating causal effects in continuous variables.
problem Lack of methods for estimating causal effects in continuous variables.
method Develops a method independent of data generating models for continuous variable interventions.
result Preserves identifiability of data and applies to any generating models.
Nested model averaging improves high-dimensional linear regression performance.
problem High-dimensional linear regression with predictor ordering impact.
method Combining model averaging with regularized estimators on the solution path.
result Nested model averaging with lasso and SLOPE outperforms competing methods.
Proposes a novel neural network method to estimate average treatment effect.
problem Bias in estimating average treatment effect due to confounding and instrumental variables.
method Self-balancing neural network (Sbnet) that estimates pseudo propensity scores and average treatment effect in one step.
result Proposed method outperforms state-of-the-art methods in simulations and real-world datasets.
LASSO-PCA combines LASSO and PCA for automated forecast averaging.
problem Automating the selection of forecast averaging methods and tuning parameters.
method LASSO estimation combined with PCA, using information criteria for parameter selection.
result LASSO-PCA outperforms other methods in forecast error reduction.
Deterministic method for certifying neural network robustness.
problem Certifying neural network robustness against adversarial attacks.
method Equivalence between training and Gaussian averaging for robustness certification.
result Comparable certified accuracy and robustness to stochastic methods but with single model evaluation.
Average Oracle outperforms DCC+NLS in portfolio optimization.
problem Optimizing portfolio performance in volatile markets.
method Comparing the Average Oracle to various DCC+NLS variants.
result The Average Oracle consistently yields higher Sharpe ratios.
New method assesses financial and cyber risks under uncertainty.
problem Uncertainty in risk assessment for financial and cyber systems.
method Combines stochastic approximation and distorted mix method to compute worst case average value at risk.
result Efficient algorithm for tail uncertainty in multivariate distributions.
New averaging technique speeds up Newton method convergence.
problem Superlinear convergence of stochastic Newton methods with noisy Hessians.
method Hessian averaging to reduce noise and maintain superlinear convergence.
result Hessian averaging achieves superlinear convergence with a non-asymptotic rate.
Deviation inequalities for stochastic approximation methods.
problem Establishing bounds on the deviation of stochastic approximation methods.
method Martingale approximation method for separately Lipschitz functions.
result Established various deviation inequalities for stochastic approximation by averaging and minimization.
IEA improves CNN models by averaging multiple convolutional layers.
problem Improving CNN model accuracy through ensemble learning.
method Replacing single convolutional layers with Inner Average Ensembles (IEA) of multiple convolutional layers.
result CNN models using IEA outperform those with regular convolutional layers.
A fast method for Lasso and Logistic Lasso problems.
problem Solving Lasso and Logistic Lasso regression problems efficiently.
method Iterative active set approach using solver updates.
result 31.41 times faster on average for compressed sensing.
Proposes α-integration pooling for CNNs to improve performance.
problem Finding optimal pooling method for CNNs is challenging.
method Introduces α-integration pooling with a trainable parameter α. result Demonstrates α-integration pooling outperforms other pooling methods in image recognition. New RL algorithms improve average-reward performance.
problem Improving RL algorithms for average-reward criteria.
method Developed novel algorithms addressing average-reward criterion directly.
result ATRPO significantly outperforms TRPO in challenging MuJuCo environments.
clusterBMA combines clustering results from multiple models using Bayesian model averaging.
problem Uncertainty in model selection for clustering.
method Bayesian model averaging to combine results from multiple clustering algorithms.
result ClusterBMA offers probabilistic cluster allocations and quantifies model-based uncertainty.
Paper uses LSTM neural networks to forecast commodity prices.
problem Forecasting accuracy of traditional methods like ARIMA.
method Long Short-Term Memory (LSTM) neural networks complement traditional methods.
result Forecast averaging of LSTM and ARIMA models improves forecast accuracy.
Partial model averaging improves Federated Learning performance.
problem Periodic model averaging causes significant model discrepancy in Federated Learning.
method Proposes a partial model averaging framework that encourages local models to stay close to each other.
result Partial averaging achieves up to 2.2% higher validation accuracy than full averaging.
Paper proposes dense average network for improved power load forecasting.
problem Improving power load forecasting accuracy to save millions for the power industry.
method Introduces dense average connection and constructs dense average network for power load forecasting.
result Proposed model outperforms existing methods on public datasets.
The paper investigates length averages in foliations, contrasting with time averages in dynamical systems.
problem Investigate the existence and non-existence of length averages in foliations.
method Generalize the existence problem of time averages in dynamical systems to foliations and introduce the concept of length averages.
result Length averages exist everywhere for codimension one orientable singular foliations without degenerate singularities on compact surfaces under a mild condition.
This paper studies robust estimation methods in high dimensions, comparing model-averaged and composite quantile estimators.
problem Understanding robustness in high-dimensional regularized estimation.
method Optimal weights are determined by minimizing the asymptotic mean squared error, incorporating regularization effects without perfect selection.
result Model-averaged and composite quantile estimators often outperform least-squares methods in prediction quality.
Local averaging accurately distills manifold structure from noisy data.
problem Tackles the challenge of uncovering manifold structure from noisy data.
method Two-round mini-batch local averaging method applied to noisy samples.
result Achieves accuracy bound of $d(\hat{\mathbf q}, \mathcal M) \leq σ\sqrt{d\left(1+\frac{κ\mathrm{diam}(\mathcal {M})}{\log(D)}
ight)}$.
In mixture model-based clustering applications, it is common to fit several models from a family and report clustering results from only the `best' one. In such circumstances, selection of this best model is achieved using a model selection criterion, most often the Bayesian information criterion. Rather than throw awa…
Unified analysis of finite weight averaging methods in deep learning.
problem Theoretical explanation of finite weight averaging methods in deep learning.
method Generalized SGD and LAWA, establishing convergence and generalization bounds.
result FWA achieves faster convergence and better generalization compared to SGD.
New TD method stabilizes average-reward learning.
problem Stability issues in average-reward TD learning.
method Implicit fixed point update for average-reward TD(λ). result Improved numerical stability and broader step-size range.
Enhances Fourier estimator performance for asynchronous event-data.
problem Improving correlation and covariance estimation on event-data.
method Implement and test NUFFT methods with different averaging kernels.
result Demonstrates improved performance and relationship between averaging scales.
A new method for averaging data on manifolds is proposed, offering simplicity and efficiency.
problem The difficulty of computing Fréchet means on manifolds, especially Stiefel and Grassmann.
method Proposed RL-barycenters, simpler arithmetic means projected onto the manifold.
result RL-barycenters yield simple yet effective means on Stiefel and Grassmann manifolds.
Develops first-order methods for average-reward MDPs with strong guarantees.
problem Lack of strong theoretical guarantees for first-order methods in AMDPs.
method Average-reward stochastic policy mirror descent (SPMD) and variance-reduced temporal difference (VRTD) methods.
result Establishes sample complexity results for solving AMDPs.
Determinantal averaging corrects inversion bias in distributed Newton's method.
problem Inverting a sum of distributed matrices is biased; local averages are incorrect.
method Reweighting local estimates of the Newton's step proportionally to the determinant of the local Hessian estimate, then averaging them.
result Determinantal averaging provides the first known asymptotically consistent distributed Newton step.
Develops unbiased averaging methods for second order optimization in distributed systems.
problem Computing the Hessian is challenging and communication is a bottleneck in distributed optimization.
method Unbiased parameter averaging methods using sampling and sketching of the Hessian.
result Provably minimizes bias for sketched Newton directions.
This work analyzes mini-batching and averaging in SGD for least squares regression.
problem Improving parallelization and reducing variance in SGD for least squares regression.
method Characterizes mini-batching and tail-averaging for least squares regression, providing non-asymptotic excess risk bounds.
result Proves near-linear parallelization speedups and minimax risk for SGD with nearly the same number of updates as batch gradient descent.
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…
Researchers combined linear classifiers using score functions and found simple and trimmed averages to be the best combination strategies.
problem Combining linear classifiers using their score functions.
method Two score functions tested; four combination strategies investigated; comparison with majority voting and model averaging.
result Simple and trimmed average combination strategies were the best.
The study compares various methods for stochastic optimization, finding that sample average approximation is effective but outperformed by Bayesian approaches.
problem Improving on sample average approximation for stochastic optimization when distributional information is available.
method Compared bagging, kernel smoothing, maximum likelihood estimation, and a Bayesian approach.
result Sample average approximation is effective but outperformed by Bayesian approaches in certain scenarios.
We propose a method for pricing American options whose pay-off depends on the moving average of the underlying asset price. The method uses a finite dimensional approximation of the infinite-dimensional dynamics of the moving average process based on a truncated Laguerre series expansion. The resulting problem is a fin…
Distributed learning for related tasks using graph weights.
problem Learning multiple tasks on different machines.
method Weighted averaging of messages with skewing or stepsize control.
result Different tasks can be learned on different machines.
New methods optimize functions faster with less gradient accuracy needed.
problem Optimizing complex functions with limited gradient accuracy.
method Hessian averaging and adaptive gradient sampling methods.
result Improved convergence rates for various function types.
A new method averages neural network parameters to rank features robustly.
problem Neural networks' sensitivity to random initialization affects feature ranking robustness.
method Parameter averaging of multiple shallow networks trained with different random seeds.
result The averaged model discovers ground-truth feature importance consistently.
Bayesian model averaging has become a widely used approach to accounting for uncertainty about the structural form of the model generating the data. When data arrive sequentially and the generating model can change over time, Dynamic Model Averaging (DMA) extends model averaging to deal with this situation. Often in ma…
We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multi-core machines. In order to be as hardware-agnostic as possible, we needed a way to use multiple machines without …
Accelerates machine learning algorithms for sparse data.
problem Efficiently solving composite convex minimization problems.
method Accelerated dual-averaging primal-dual method for composite convex minimization.
result Demonstrates advantages in handling sparse data both theoretically and empirically.
Proposes a CVaR-based method to learn robust options in uncertain environments.
problem Learning options from inaccurate models with uncertain parameters.
method Extends CVaR-based policy gradient method to robust Markov decision processes.
result Produces options that perform well in both average and worst cases.
We present a geometric proof of the averaging theorem for perturbed dynamical systems on a Riemannian manifold, in the case where the flow of the unperturbed vector field is periodic and the S1-action associated to this vector field is not necessarily trivial. We generalize the averaging procedure \cite{A…
Two-Tailed Averaging improves generalization by optimizing the number of leading iterates to ignore.
problem Improving generalization in stochastic optimization with limited resources and hyperparameters.
method An anytime adaptive algorithm that balances the number of leading iterates to ignore for better generalization.
result Approximates the optimal tail at all optimization steps, improving generalization without hyperparameters.