PARyOpt optimizes functions asynchronously, reducing wall clock time.
problem Efficiently optimizing functions on distributed systems with asynchronous evaluations.
method Parallel asynchronous Bayesian optimization.
result Reduces total optimization time for various test problems.
This study compares parallel SMC and MCMC for Bayesian deep learning, showing SMC parallel is faster.
problem Efficiently performing Bayesian deep learning with parallel computing.
method Compared sequential Monte Carlo (SMC) and Markov chain Monte Carlo (MCMC) in parallel settings.
result Parallel SMC achieves similar convergence as a single SMC but with reduced communication time.
New method trains neural networks to optimize faster than tuned methods.
problem Training learned optimizers is difficult and often leads to poor performance.
method Dynamic weighting of unbiased gradient estimators for a variational loss.
result Trained neural networks optimize faster than tuned first-order methods.
Large batch sizes don't improve training time for most models.
problem The inefficiency of large batch sizes in stochastic gradient descent.
method Empirical analysis of network training across various architectures and domains.
result Increasing batch size beyond a certain point does not reduce training time for either train or test loss.
Warm-starting neural networks can lead to worse performance than fresh starts.
problem Warm-starting neural networks can degrade performance compared to fresh starts.
method Analyzed and provided a simple trick to overcome the degradation of warm-starting.
result A simple trick can overcome the degradation of warm-starting in several important situations.
Improves meta-learning efficiency with mixed-mode differentiation.
problem Efficiently calculating complex derivatives in meta-learning.
method Mixed-Flow Meta-Gradients (MixFlow-MG) for scalable differentiation.
result Significant memory and time improvements in meta-learning tasks.
Moniqua improves SGD convergence with quantized communication.
problem Efficiently communicating in decentralized SGD with limited bandwidth.
method Modulo quantized communication in decentralized SGD.
result Moniqua converges at the same rate as full-precision communication with less bits.
PowerSGD compresses gradients for faster distributed optimization.
problem Communication bottleneck in data-parallel distributed optimization.
method Low-rank gradient compressor based on power iteration.
result Achieves test performance on par with SGD and consistent speedups.
Pipe-SGD improves deep net training time by 5.4x with pipelined AllReduce.
problem Efficiently training deep neural networks on distributed systems.
method Pipelined SGD with two worker nodes, considering network latency and bandwidth.
result Up to 5.4x improvement in wall-clock training time.
NAC-FL optimizes model updates in FL systems by adapting compression to network congestion.
problem Federated Learning systems face congestion and delays in data exchanges.
method NAC-FL dynamically adjusts client compression based on network congestion.
result NAC-FL reduces training time and achieves robust performance improvements.
New algorithms optimize multiple machine learning metrics in real-world tasks.
problem Optimizing multiple conflicting performance criteria in real-world applications.
method Extends ASHA to multi-objective hyperparameter optimization.
result MO ASHA enables scalable multi-objective hyperparameter optimization.
D-Wave hybrid quantum-classical portfolio optimization shows classical decomposition is key, not quantum sampling.
problem Optimizing portfolios with constraints using hybrid quantum-classical methods.
method Operational decomposition audit of D-Wave's hybrid quantum-classical service on mean-variance-turnover instances.
result Classical decomposition and feasibility-aware reassembly are key to hybrid quantum-classical performance.
A new approach to protein language models combines latent space prediction with masked language modeling.
problem Improving protein language models by predicting amino acid identities at masked positions.
method A variant of masked language modeling that predicts latent targets only at masked positions, retaining the MLM cross-entropy.
result The new approach outperforms pure masked language modeling on 11 out of 16 downstream tasks.
A new ES method improves reinforcement learning speed and accuracy.
problem Slow convergence and local maxima in reinforcement learning.
method Directional Gaussian Smoothing Evolution Strategy (DGS-ES)
result DGS-ES accelerates RL training with high accuracy and nonlocal search direction.
K-FAC doesn't improve large batch training efficiency.
problem Inefficiency of K-FAC in large batch size training.
method Empirical analysis of K-FAC and SGD on ResNet and AlexNet.
result K-FAC doesn't exhibit improved scalability to large batch sizes.
This work provides a scaling rule for model EMA optimization across batch sizes.
problem Training dynamics and performance differences across batch sizes when using model EMA.
method Developed a scaling rule for model EMA optimization, demonstrating its validity across various architectures and data modalities.
result Enabled SSL methods like BYOL to train at larger batch sizes without performance degradation.
The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empiri…
The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic settings, together with hyperparameter learning. We conclude that the natural gradient can signific…
Improves BO efficiency by allowing asynchronous parallel computing.
problem Wasteful use of resources in batch Bayesian optimisation.
method Developed PLAyBOOK for asynchronous local penalisation.
result Asynchronous BO often outperforms synchronous BO.
We present a fast variational Bayesian algorithm for performing non-negative matrix factorisation and tri-factorisation. We show that our approach achieves faster convergence per iteration and timestep (wall-clock) than Gibbs sampling and non-probabilistic approaches, and do not require additional samples to estimate t…
GPU-accelerated particle methods outperform neural samplers in LFT benchmarks.
problem High-dimensional multimodal sampling problems in lattice field theory.
method GPU-accelerated particle Monte Carlo methods (Sequential Monte Carlo and nested sampling).
result These methods match or outperform neural samplers in sample quality and wall-clock time.
MSLs use parallelizable root-finding for efficient ODE and PDE solutions.
problem Efficiently solving initial value problems for ODEs and PDEs.
method Leveraging time-parallel methods, MSLs use parallelizable root-finding algorithms.
result MSLs offer significant speedups in NFEs and inference time.
We propose RoBiRank, a ranking algorithm that is motivated by observing a close connection between evaluation metrics for learning to rank and loss functions for robust classification. The algorithm shows a very competitive performance on standard benchmark datasets against other representative algorithms in the litera…
We present an improved Bayesian framework for performing inference of affine transformations of constrained functions. We focus on quadrature with nonnegative functions, a common task in Bayesian inference. We consider constraints on the range of the function of interest, such as nonnegativity or boundedness. Although …
Efficiently marginalizes over Gaussian Process kernels for better model flexibility and uncertainty.
problem Inefficient marginalization over Gaussian Process kernels for large datasets.
method Bayesian Quadrature scheme with maximum mean discrepancies and invariances between Spectral Mixture kernels.
result Achieves more accurate predictions and better calibrated uncertainty than state-of-the-art baselines.
Noise-Contrastive Estimation improves efficiency for estimating log-likelihood of complex point processes.
problem Estimating log-likelihood of complex multivariate point processes is computationally expensive.
method Noise-Contrastive Estimation adapted for multivariate point processes, with provable guarantees.
result Our method achieves similar log-likelihood with fewer evaluations and less time.
IW-ES improves ES efficiency by using Importance Sampling.
problem Data inefficiency in Evolution Strategies.
method Importance Sampling to perform multiple updates per batch.
result IW-ES shows promising results in efficiency.
Training for just one epoch improves model performance without overfitting.
problem Training large unsupervised models is costly and time-consuming.
method Train models for only one epoch, adjust model size and iterations.
result Performance significantly improved with one epoch training, especially for large models.
Deep-learning method solves BSVIEs and coupled systems.
problem High-dimensional, time-inconsistent stochastic control problems.
method Trains a neural network to approximate solution fields directly.
result Non-asymptotic error bound and scalable performance.
Linear classification has been widely used in many high-dimensional applications like text classification. To perform linear classification for large-scale tasks, we often need to design distributed learning methods on a cluster of multiple machines. In this paper, we propose a new distributed learning method, called f…
EKM solves the K-medoids problem in polynomial time.
problem The K-medoids problem in data analysis. method EKM is a novel algorithm using transformational programming and combinatorial generation.
result EKM solves the K-medoids problem in worst-case $O\left(N^{K+1}
ight)$ time complexity. Paper proposes SCott optimizer to reduce forecasting model training variance.
problem Large variance in gradient estimation for forecasting models.
method Stratified sampling and control variate to reduce gradient variance.
result SCott optimizer converges faster on time series forecasting problems.
New method trains neural ODEs faster with fewer layers.
problem Training neural ODEs on large datasets is computationally expensive.
method Combines optimal transport and stability regularizations.
result Significant reductions in training time with no performance loss.
ProMP improves Meta-RL by better handling credit assignment.
problem Poor credit assignment in Meta-RL leads to suboptimal performance.
method Develops a novel meta-learning algorithm controlling pre-adaptation and adapted policies.
result Significantly improves sample-efficiency and asymptotic performance.
We introduce a new, high-throughput, synchronous, distributed, data-parallel, stochastic-gradient-descent learning algorithm. This algorithm uses amortized inference in a compute-cluster-specific, deep, generative, dynamical model to perform joint posterior predictive inference of the mini-batch gradient computation ti…
Improved stochastic clocks for financial models without increasing trades.
problem Dealing with asymmetrical and tail risks in financial returns.
method Proposes a new approach to regulate Lévy subordinators for financial models.
result Achieves arbitrarily large skewness and excess kurtosis of returns.
AdaComm optimizes SGD by dynamically adjusting communication frequency for faster convergence.
problem Achieving optimal error-runtime trade-off in distributed SGD.
method Adaptive communication strategy that starts with infrequent averaging to save delay and improve speed, then increases frequency.
result AdaComm reduces training time by 3x while maintaining the same final loss.
Enhances SGLD for log-concave posteriors with asynchronous computation.
problem Sampling log-concave posterior distributions efficiently.
method Integrates asynchronous computation into SGLD with delayed gradients.
result Convergence in measure is not significantly affected by delayed gradient information.
The Kelly rule fails to maximize growth in a time-changed return setting.
problem Performance of the Kelly rule in a time-changed return process.
method Investigated the Kelly rule in a semi-martingale setting with a time change process.
result The Kelly rule does not maximize average growth rate in a non-normal log-return setting.
A new uncertainty principle helps traders better understand market activity.
problem Understanding high-frequency market activity and correlation.
method Integrates market activity, order-flow overlap, and response time into a clock-dependent uncertainty principle.
result Six rules of thumb for traders operating at market-making frequencies.
New LNS framework improves integer program solving.
problem Solving large-scale integer linear programs efficiently.
method Large neighborhood search with imitation and reinforcement learning.
result Framework significantly outperforms commercial solvers.
We propose a novel sampling framework for inference in probabilistic models: an active learning approach that converges more quickly (in wall-clock time) than Markov chain Monte Carlo (MCMC) benchmarks. The central challenge in probabilistic inference is numerical integration, to average over ensembles of models or unk…
ES improves training efficiency by dynamically selecting data samples.
problem Efficiently selecting informative data samples for faster learning.
method Evolved Sampling (ES) dynamically selects data samples based on loss dynamics and differences.
result ES achieves significant training acceleration without compromising model performance.
New method trains reflected Schrödinger bridges without complex derivatives.
problem Training reflected Schrödinger bridges efficiently in high dimensions.
method Partially simulation-free framework with new sampling method.
result Generative performance maintained or slightly improved with reflected dynamics.
This paper improves parallel belief propagation for scalable machine learning.
problem Efficient parallelization of belief propagation for large-scale machine learning tasks.
method Use of scalable relaxed schedulers to parallelize belief propagation.
result Our approach outperforms previous methods in scalability and convergence time.
A new algorithm reduces medoid computation time by 4-5 orders of magnitude.
problem Computing the medoid of a set of points efficiently.
method Adapting multi-armed bandit sampling strategy to reduce distance computations.
result Significant gains in both number of distance computations and wall clock time.
We describe a computationally efficient, stochastic graph-regularization technique that can be utilized for the semi-supervised training of deep neural networks in a parallel or distributed setting. We utilize a technique, first described in [13] for the construction of mini-batches for stochastic gradient descent (SGD…
Recency Bias selects recent uncertain samples for faster, more accurate deep learning.
problem Improving the accuracy of deep neural networks through better mini-batch selection.
method Uses historical label predictions to evaluate predictive uncertainty and selects samples proportionally.
result Reduces test error by up to 20.97% compared to existing methods in the same training time.