Warm-up improves training by adapting learning rate based on curvature.
problem Improving training efficiency in deep learning models.
method Introducing a curvature condition to explain warm-up, and showing empirically that it leads to faster convergence.
result Adapting learning rate based on curvature condition naturally induces warm-up-like schedule, leading to faster convergence.
One of the most pressing questions in climate science is that of the effect of anthropogenic aerosol on the Earth's energy balance. Aerosols provide the `seeds' on which cloud droplets form, and changes in the amount of aerosol available to a cloud can change its brightness and other physical properties such as optical…
New algorithm speeds up HMC by generating a warm start in O(d^1/4) iterations.
problem Unclear how many iterations of HMC are needed for high-dimensional sampling.
method Developed a non-Metropolized HMC that generates a warm start in O(d^1/4) iterations, followed by Metropolized HMC.
result Final complexity of O(d^1/4) is the fastest algorithm for high-accuracy sampling under strong log-concavity assumptions.
Paper eliminates warm-up phase for PO in linear MDPs, achieving optimal regret.
problem Costly warm-up phase in PO algorithms for linear MDPs.
method Simple contraction mechanism replaces warm-up phase.
result Achieves rate-optimal regret with improved dependence on problem parameters.
Warm starts improve Gaussian process regression by up to 16x.
problem Optimizing hyperparameters for Gaussian processes.
method Iterative Gaussian processes with warm start optimization.
result Warm starts achieve the same results as conventional methods but significantly speed up computations.
Warm-start strategies speed up GP inference by 19x.
problem Efficient sequential inference in Gaussian processes.
method Three warm-start strategies exploiting smaller linear systems.
result Warm-starting achieves up to 19x speed-up in convergence.
Layer normalization placement affects training stability and warm-up stage necessity.
problem Training instability and the necessity of a learning rate warm-up stage in Transformers.
method Theoretical analysis and mean field theory to prove gradient behavior at initialization.
result Removing the warm-up stage for Pre-LN Transformers can achieve comparable results with less time and tuning.
New method achieves optimal sample complexity without warm-start in bilevel optimization.
problem Optimizing smooth objective functions with fixed point constraints in meta-learning and equilibrium models.
method Fixed point iterations at lower-level and projected inexact gradient descent at upper-level.
result Achieves near optimal sample complexity O(ε−2) and ildeO(ε−1) samples. When society maintains a competitive system to promote an abstract goal, competition by necessity relies on imperfect proxy measures. For instance profit is used to measure value to consumers, patient volumes to measure hospital performance, or the Journal Impact Factor to measure scientific value. Here we note that \t…
Reweighted ALPS improves sampling from multimodal distributions using warm start points.
problem Sampling from multimodal distributions is hard due to exponential mixing times.
method Introduces Reweighted ALPS, a modified Annealed Leap-Point Sampler that uses warm start points.
result First polynomial-time bound for Re-ALPS in a general setting, under a natural assumption.
We develop a framework for warm-starting Bayesian optimization, that reduces the solution time required to solve an optimization problem that is one in a sequence of related problems. This is useful when optimizing the output of a stochastic simulator that fails to provide derivative information, for which Bayesian opt…
WSD uses a deterministic model to accelerate diffusion-based sampling.
problem Slow refinement process in diffusion models.
method Warm-start model that predicts an informed prior conditioned on input context.
result Significantly reduces the number of diffusion steps required for realistic samples.
A hybrid strategy forecasts short-term loads using Warm-start Gradient Tree Boosting.
problem Lack of effective short-term load forecasting methods.
method Hybrid strategy integrating four different inference models: tree-based ensemble method Warm-start Gradient Tree Boosting (WGTB).
result Demonstrates effectiveness of hybrid strategy on real datasets.
Faster algorithm for sampling logconcave densities in high dimensions.
problem Cubic barrier in sampling logconcave densities from a cold start.
method Two key ingredients: weaker distance sampling and refined log-Sobolev inequality.
result First sub-cubic sampling algorithms for isotropic position.
Warm starts improve variational quantum algorithms by avoiding barren plateaus.
problem Barren plateaus in variational quantum algorithms limit scaling.
method Exploring warm starts in iterative variational methods for quantum circuits.
result Warm starts can lead to substantial gradients in small regions, suggesting trainability.
Globally normalized neural sequence models are considered superior to their locally normalized equivalents because they may ameliorate the effects of label bias. However, when considering high-capacity neural parametrizations that condition on the whole input sequence, both model classes are theoretically equivalent in…
Paper proposes a holistic optimization for civil structures considering uncertainties.
problem Designing civil structures requires integration of material and structural design.
method Holistic optimization combining concrete mixture design and structural simulations.
result Inverted workflow allows consideration of new mixtures and uncertainties.
DP-SGD can update fewer coordinates while maintaining privacy.
problem How to update fewer coordinates in DP-SGD without losing optimization signal.
method TP-TopK (Two-Phase TopK DP-SGD), a two-phase method for coordinate-sparse private training.
result Private training can update fewer coordinates without losing optimization signal, scaling noise with active dimension \(k\) instead of full dimension \(d\).
This paper speeds up iterative GP inference with warm starting.
problem Improving scalability of Gaussian process inference.
method Warm starting sequential posteriors using known solutions.
result Significant speed-ups and improved Bayesian optimisation performance.
Proposes a Carbon Equivalence Principle for financial products to align incentives and drive sustainability.
problem Align financial market incentives with carbon emissions to limit global warming.
method Introduces a Carbon Equivalence Principle requiring financial products to describe equivalent carbon flows alongside cash flows.
result Transparency of carbon flows in financial products can align incentives and reduce future costs, necessitating project re-structuring and financial net-zero designs.
In many real-world deployments of machine learning systems, data arrive piecemeal. These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily financial data) or active, where samples are selected according to a measure of their quality (e.g., experi…
Enhances genetic programming for stock alpha discovery with warm start and structural constraints.
problem Overwhelming search space and computational burden in traditional genetic programming for alpha factor discovery.
method Proposes a new GP framework with warm start and structural constraints to enhance search performance and interpretability.
result Superior out-of-sample prediction results and higher portfolio returns compared to benchmarks.
Flow-VQE uses generative flows to optimize VQE efficiently.
problem Complex objective functions and expensive optimization in VQE.
method Generative normalizing flows with parameterized quantum circuits.
result Flow-VQE accelerates convergence and reduces circuit evaluations.
Improved log-concave sampling to O(d1/2) with warm starts.
problem Sampling from strongly log-concave distributions efficiently.
method Warm starts and discretized underdamped Langevin diffusion.
result Achieved O(d1/2) complexity for high-accuracy sampling. Paper presents algorithm for optimal job selection with dynamic scoring.
problem Optimal job assignment in a sequential selection process with dynamic scores.
method Developed using dynamic programming, with extensions for partial and no-information cases.
result Algorithm allows for optimal job assignment with limited information.
Develops a contraction framework for MCMC mixing rates.
problem Proving mixing-time bounds for MCMC algorithms.
method Global and local contraction coefficients under Eγ-divergence. result Explicit global contraction coefficients for Gaussian smoothing.
Click-through rate (CTR) prediction has been one of the most central problems in computational advertising. Lately, embedding techniques that produce low-dimensional representations of ad IDs drastically improve CTR prediction accuracies. However, such learning techniques are data demanding and work poorly on new ads w…
The paper speeds up hyperparameter optimisation in Gaussian processes.
problem Scaling hyperparameter optimisation to large datasets.
method Improvements to linear system solvers (pathwise gradient, warm starting, early stopping).
result Speed-ups of up to 72x and residual norm decreases of up to 7x.
New method for sparse kernel selection improves prediction accuracy.
problem Sparse Multiple Kernel Learning for binary classification.
method Alternating best response algorithm with semidefinite relaxations.
result Method outperforms state-of-the-art MKL approaches in prediction accuracy.
Hyperparameter optimization aims to find the optimal hyperparameter configuration of a machine learning model, which provides the best performance on a validation dataset. Manual search usually leads to get stuck in a local hyperparameter configuration, and heavily depends on human intuition and experience. A simple al…
Accelerates optimal transport computation by 10x with spectral insights.
problem Exponential slow-down of convergence in Entropic Optimal Transport as regularization weakens.
method Spectral insights and spectral warm-start strategy to mitigate convergence issues.
result Faster convergence compared to the reference method Sinkhorn algorithm.
Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization. Typically, BO is powered by a Gaussian process (GP), whose algorithmic complexity is cubic in the number of evaluations. Hence, GP-based BO cannot leverage large amounts of past or related function evaluations, for e…
A dry decade in the Navajo Nation has killed vegetation, dessicated soils, and released once-stable sand into the wind. This sand now covers one-third of the Nation's land, threatening roads, gardens and hundreds of homes. Many arid regions have similar problems: global warming has increased dune movement across farmla…
This work explores the ability of collective matrix factorization models in recommender systems to make predictions about users and items for which there is side information available but no feedback or interactions data, and proposes a new formulation with a faster cold-start prediction formula that can be used in rea…
New model uses pretrained biochemical language models to generate drug compounds.
problem Developing novel compounds targeting specific proteins.
method Exploits pretrained language models to initialize and fine-tune targeted molecule generation models.
result Warm-started models outperform baseline models, with one-stage strategy showing better generalization.
This paper analyzes EM algorithm for softmax mixture models in high dimensions.
problem Modeling heterogeneous populations choosing from multiple attributes.
method Comprehensive analysis of the EM algorithm for softmax mixture models (SMMs), proving identifiability and convergence.
result EM algorithm recovers mixture atoms at near-parametric rate under suitable initialization.
Method for initializing Gaussian mixtures for variational inference with multi-modal distributions.
problem Challenges in variational inference with Gaussian mixtures due to multimodality and nonconvex loss functions.
method Optimization to find local maxima, local Gaussian approximations, and constrained least squares regression.
result Robust initialization improves variational inference performance and scalability.
We present an affine-invariant random walk for drawing uniform random samples from a convex body K⊂Rn that uses maximum volume inscribed ellipsoids, known as John's ellipsoids, for the proposal distribution. Our algorithm makes steps using uniform sampling from the John's ellipsoid of the …
A new screening rule improves lasso solving speed.
problem Efficiently solving lasso problems with high correlation.
method Uses second-order information from the Hessian to screen predictors.
result Outperforms alternatives on simulated and real data.
We consider solving the ℓ1-regularized least-squares (ℓ1-LS) problem in the context of sparse recovery, for applications such as compressed sensing. The standard proximal gradient method, also known as iterative soft-thresholding when applied to this problem, has low computational cost per iteration but a r…
Zigzag sampling algorithm efficiently samples from strongly log-concave distributions with low computational cost.
problem Sampling from strongly log-concave distributions efficiently and with low computational complexity.
method Zigzag sampling algorithm with warm start assumption, focusing on gradient evaluations.
result Achieves ε error in chi-square divergence with computational cost of O(κ²d^(1/2)(log(1/ε))^(3/2)) gradient evaluations.
We introduce a generic scheme for accelerating gradient-based optimization methods in the sense of Nesterov. The approach, called Catalyst, builds upon the inexact accelerated proximal point algorithm for minimizing a convex objective function, and consists of approximately solving a sequence of well-chosen auxiliary p…
New screening rules improve lasso model fitting efficiency.
problem Efficiently solving high-dimensional lasso problems.
method Look-ahead screening rules to discard predictors.
result Look-ahead screening rules outperform existing methods.
Paper optimizes MVE network convergence and regularization.
problem Optimizing Mean Variance Estimation networks for better performance.
method Presented two key insights: warm-up period for mean optimization and separate regularization of mean and variance.
result Warm-up period and separate regularization improve MVE network performance.
Active learning reduces smart meter data needs for better electric load predictions.
problem Inaccurate and costly electric load predictions due to insufficient data.
method Active learning to collect more informative data subsets.
result Electric load predictions can be made with about half the data using active learning.
Providing long-range forecasts is a fundamental challenge in time series modeling, which is only compounded by the challenge of having to form such forecasts when a time series has never previously been observed. The latter challenge is the time series version of the cold-start problem seen in recommender systems which…
Model predicts EU carbon prices using market and political factors.
problem Predict future carbon prices for EU market management.
method Support vector regression with grid search and cross validation.
result Model predicts carbon prices accurately for 2030.
Machine learning predicts perovskite formability and classifies crystal structures.
problem Predicting and classifying perovskite formability and crystal structures.
method Machine learning, specifically Random Forest, with 5-fold cross-validation.
result 98.57% accuracy in predicting perovskite formability and 90.53% in classifying crystal structures.