One-pass optimisation for high-dimensional hyperparameters.
problem Efficient optimisation of hyperparameters in machine learning models.
method Approximate hypergradient-based optimisation for any continuous hyperparameter, requiring only one training episode.
result Competitive performance on various datasets without hyperparameter restarts.
One-pass algorithm finds small subset for ℓp subspace approximation with additive error.
problem Finding a small subset of data points for ℓp subspace approximation. method One-pass subset selection with additive approximation guarantee for p∈[1,∞). result First one-pass algorithm with additive error for ℓp subspace approximation. Bayesian online learning algorithm for one-pass data, achieving frequentist validity and uncertainty quantification.
problem Theoretical limitations in Bayesian online learning, especially in the one-pass setting.
method Proposed a new Bayesian online learning algorithm with a warm-start phase for the one-pass regime, establishing convergence rates and valid uncertainty quantification.
result The sequentially updated posterior attains optimal convergence rates and valid uncertainty quantification without diverging mini-batch sample sizes.
Full-batch GD outperforms one-pass SGD in learning a single-index model with quadratic activation.
problem Learning a single-index model with quadratic activation using gradient descent.
method Full-batch gradient descent compared to one-pass stochastic gradient descent (SGD) on a correlation loss.
result Full-batch GD requires only n≃d samples for strong recovery, while one-pass SGD requires n≳dlogd samples. One-pass private sketch supports various machine learning tasks.
problem Efficiently supporting multiple machine learning tasks with differential privacy.
method Randomized contingency tables indexed with locality-sensitive hashing, constructed in one pass.
result Competitive error bounds for DP kernel density estimation, faster than existing methods.
We study distribution testing with communication and memory constraints in the following computational models: (1) The {\em one-pass streaming model} where the goal is to minimize the sample complexity of the protocol subject to a memory constraint, and (2) A {\em distributed model} where the data samples reside at mul…
We present a one-pass sparsified Gaussian mixture model (SGMM). Given N data points in P dimensions, X, the model fits K Gaussian distributions to X and (softly) classifies each point to these clusters. After paying an up-front cost of O(NPlogP) to precondition the data, we subsample Q entries…
New algorithm achieves nearly optimal regret with one-pass updates for GLB problems.
problem Generalized linear bandits with non-linear reward distributions.
method Jointly efficient algorithm using OMD estimator with one-pass updates.
result Nearly optimal regret bound with O(1) time and space complexities per round. One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.
problem Slow escape from poor generalization solutions in overparameterized neural networks.
method Analysis of one-pass SGD dynamics using ordinary differential equations for overlap matrices.
result Overparameterization only modestly accelerates escape from poor solutions.
ORFit trains models on streaming data with one pass, minimizing memory and computational costs.
problem Training large models on a stream of data without retraining on previous data.
method Orthogonal Recursive Fitting (ORFit) using orthogonal gradient descent and recursive least-squares.
result ORFit updates parameters orthogonally to past gradients, leading to efficient memory and computational usage.
One-pass SGD converges in overparametrized neural networks with random data.
problem Understanding convergence of SGD in neural networks with streaming data.
method Overparameterized two-layer neural networks, one-pass SGD, random initialization, NTK eigen-decomposition, VC dimension, McDiarmid's inequality.
result Prediction error converges in expectation under one-pass SGD in overparametrized neural networks.
Consider a sequence of closed, orientable surfaces of fixed genus g in a Riemannian manifold M with uniform upper bounds on mean curvature and area. We show that on passing to a subsequence and choosing appropriate parametrisations, the inclusion maps converge in C0 to a map from a surface of genus g to M. W…
We develop an efficient alternating framework for learning a generalized version of Factorization Machine (gFM) on steaming data with provable guarantees. When the instances are sampled from d dimensional random Gaussian vectors and the target second order coefficient matrix in gFM is of rank k, our algorithm conve…
In this paper, we propose a one-pass algorithm on MapReduce for penalized linear regression \[f_λ(α, β) = \|Y - α\mathbf{1} - Xβ\|_2^2 + p_λ(β)\] where α is the intercept which can be omitted depending on application; β is the coefficients and pλ is the penalized function with penalizing parameter λ. $f_λ(α, β…
In many large-scale machine learning applications, data are accumulated with time, and thus, an appropriate model should be able to update in an online paradigm. Moreover, as the whole data volume is unknown when constructing the model, it is desired to scan each data item only once with a storage independent with the …
New method for RLHF reduces costs by integrating new data in one pass.
problem Continuous integration and re-optimization of models in RLHF leads to high computational and storage costs.
method Proposes a one-pass reward modeling method using online mirror descent with a tailored local norm.
result Achieves constant-time updates per iteration, enhancing both statistical and computational efficiency.
A new data-oblivious sketch for logistic regression reduces data size while maintaining approximation accuracy.
problem Efficiently solving logistic regression in one pass over a data stream.
method Data-oblivious sketching approach that reduces data size to poly(μdlog n) weighted points.
result Sketching reduces data size significantly and provides approximation guarantees.
This paper explores optimising acquisition functions in Bayesian optimisation.
problem Optimising acquisition functions in Bayesian optimisation is challenging due to their non-convex nature.
method The authors derive compositional forms for acquisition functions and use them to recast maximisation as a compositional optimisation problem.
result The compositional approach to maximising acquisition functions shows empirical advantages across various tasks.
New framework converts multi-objective to single-objective optimisation.
problem Solving multi-objective optimisation problems.
method Formalises scalarisation into mathematical framework, uses R2 utilities.
result R2 utilities are monotone and submodular, optimised by greedy algorithms.
Unified Bayesian Optimisation for mixed variables improves performance.
problem Efficient optimisation of problems with both categorical and continuous variables.
method Derive value proposals from the Expected Improvement criterion to optimise both categorical and continuous variables under a single acquisition metric.
result Unified approach significantly outperforms existing methods across mixed-variable tasks.
A new method learns priors for Bayesian optimisation to improve performance.
problem Bayesian optimisation tasks often assume strong similarity, which is violated in many cases.
method Replace strong similarity assumption with shape similarity, learn priors for hyperparameters.
result PLeBO and prior transfer find good inputs in fewer evaluations.
Real data are often with multiple modalities or from multiple heterogeneous sources, thus forming so-called multi-view data, which receives more and more attentions in machine learning. Multi-view clustering (MVC) becomes its important paradigm. In real-world applications, some views often suffer from instances missing…
Study improves Bayesian optimisation with ensemble transfer learning.
problem Improving sample efficiency in Bayesian optimisation of expensive functions.
method Empirical analysis of ensemble-based transfer learning methods and pipeline components.
result Two components (warm start initialisation and positive weight constraint) improve transfer learning Bayesian optimisation performance.
Stochastic Gradient Descent can overfit after just a few passes, contrary to initial expectations.
problem Understanding the out-of-sample performance of multi-pass SGD in stochastic convex optimization.
method Analysis of multi-pass SGD in the stochastic convex optimization model.
result Multi-pass SGD can lead to significant overfitting after just a few passes, contrary to initial expectations.
AdamZ optimiser improves neural network training efficiency.
problem Challenges in optimisation like overshooting and stagnation.
method Dynamic learning rate adjustment based on overshoot and stagnation factors.
result Consistently minimises loss function, improving model performance.
Wind farm layout optimisation tackles space constraints with Bayesian multi-objective approach.
problem Optimizing wind farm layout due to limited space and conflicting objectives.
method Set-based multi-objective Bayesian optimisation using Gaussian process.
result Demonstrates potential of set-based Bayesian multi-objective optimisation for wind farm layout.
Book covers tools for zeroth-order convex optimisation.
problem Zeroth-order convex optimisation.
method Cutting plane methods, interior point methods, continuous exponential weights, gradient descent, online Newton step.
result Improved existing bounds and algorithms.
Bayesian optimisation tackles high-dimensional categorical and mixed search spaces.
problem Bayesian optimisation on high-dimensional categorical and mixed search spaces is challenging.
method Combining local optimisation with a tailored kernel design.
result Empirically outperforms current baselines in performance and computational costs.
O3 optimizes samples from generative models without extra training.
problem Optimizing specific criteria within rich data distributions.
method Surrogate latent spaces for efficient black-box optimisation.
result Surrogate-space optimisation finds higher-scoring samples.
A new optimisation method efficiently scales Hessian-vector products for neural networks.
problem Challenges in applying second-order quasi-Newton methods due to large Hessian and non-convexity.
method Proposes an optimisation algorithm that asymptotically uses the exact inverse Hessian with modified eigenvalues.
result Demonstrates scalability and comparable performance to other optimisation methods in neural networks.
Bayesian optimisation framework for graph functions.
problem Optimizing functions on graph structures efficiently.
method Learning suitable kernels and local modelling.
result Demonstrated effectiveness on synthetic and real-world graphs.
Two new approaches for point prediction in streaming data, showing consistency and performance.
problem Predicting points in streaming data without a true model.
method Count-Min sketch and Gaussian process priors with random bias.
result CMS-based estimates are consistent under i.i.d. samples assumption.
Bayesian optimisation tackles expensive black-box functions with constraints.
problem Optimizing constrained black-box functions in machine learning and simulation.
method Proposes a new Knowledge Gradient acquisition function for constrained Bayesian optimisation.
result Demonstrates superior performance over four state-of-the-art constrained Bayesian optimisation algorithms.
MTL2L learns to adapt optimisation rules for unseen data.
problem Learners need to adapt to unseen data domains.
method Introduces MTL2L, a context-aware neural optimiser.
result MTL2L can adapt optimisation rules for unseen data.
Real world experiments are expensive, and thus it is important to reach a target in minimum number of experiments. Experimental processes often involve control variables that changes over time. Such problems can be formulated as a functional optimisation problem. We develop a novel Bayesian optimisation framework for s…
Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current information-theoretic approaches require many approximations in implementation, introduce often-prohibitive computational overhead and limit the choi…
Bayesian optimisation algorithm for unknown search spaces with sub-linear regret.
problem Efficient optimisation of expensive black-box functions in unknown search spaces.
method Expands search space over iterations based on a hyperharmonic series, scales to high dimensions.
result Sub-linear regret growth for both algorithms.
New methods improve global optimisation for expensive functions using lookahead strategies.
problem Optimising expensive functions without gradient info in high dimensions.
method Nonmyopic acquisition strategies based on approximate dynamic programming.
result Nonmyopic methods outperform myopic approaches in various applications.
Deep Optimisation (DO) combines evolutionary search with Deep Neural Networks (DNNs) in a novel way - not for optimising a learning algorithm, but for finding a solution to an optimisation problem. Deep learning has been successfully applied to classification, regression, decision and generative tasks and in this paper…
Bayesian optimisation is improved by incorporating expert prior through space warping.
problem Cold start phase in expensive function optimisation.
method Prior distribution warps the search space around high probability regions of function optimum.
result Improves optimisation performance through acquisition agnostic approach.
The equivariant Gromov--Hausdorff convergence of metric spaces is studied. Where all isometry groups under consideration are compact Lie, it is shown that an upper bound on the dimension of the group guarantees that the convergence is by Lie homomorphisms. Additional lower bounds on curvature and volume strengthen this…
Given a compact Riemann surface X and a complex reductive Lie group G equipped with real structures, we define antiholomorphic involutions on the moduli space of G-Higgs bundles over X. We investigate how the various components of the fixed point locus match up, as one passes from G to its Langlands dual $^LG…
We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for whic…
A novel optimisation framework through quadratic nonlinear projection is introduced for credit portfolio when the portfolio risk is measured by Conditional Value-at-Risk (CVaR). The whole optimisation procedure to search toward the optimal portfolio state is conducted by a series of single-step optimisations under the …
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
New approach separates VAE and GP for better molecular optimisation.
problem Optimizing complex structured domains like molecular spaces using VAEs.
method Decouples VAE for structure generation and GP for predictive modelling, combining them with a Bayesian update rule.
result Improves identification of high-potential candidates in molecular optimisation.
A new feature selection method using attention for neural networks.
problem Feature selection for neural networks with a budget constraint.
method Sequential Attention: greedy forward selection with attention weights.
result Achieves state-of-the-art results for neural networks.
Optimizes data power control in cell-free networks for better spectral efficiency.
problem Maximizing overall spectral efficiency in cell-free networks with multi-objective optimisation.
method Applied scalable multi-objective Bayesian optimisation to solve convergence-time limitations.
result Improved radio resource management in cell-free networks.