ANT improves TS diffusion models by automatically determining noise schedules.
problem Suboptimal performance of TS diffusion models due to lack of domain-specific noise schedules.
method ANT proposes an adaptive noise schedule that automatically determines proper noise schedules for TS datasets based on their statistics.
result ANT achieves state-of-the-art performance on various TS tasks, including forecasting, refinement, and generation.
Proposes CDTD, a diffusion model for mixed-type tabular data.
problem Adapting diffusion models to mixed-type tabular data.
method Score matching and score interpolation for continuous features, adaptive noise schedules for categorical features.
result Consistently outperforms state-of-the-art models in mixed-type tabular data.
Generative models improve for multiscale scientific data with new noise and interpolation techniques.
problem Numerical challenges in generating high-fidelity samples for multiscale scientific data.
method Design of noise distributions and interpolation schedules in function space to ensure Lipschitz regularity and finite noise roughness.
result Scale-adaptive noise and interpolation schedules improve numerical efficiency and fidelity of generated samples.
This work sets lower bounds on the number of score queries needed for diffusion sampling.
problem Establishing information-theoretic limits on the number of score evaluations required for diffusion sampling.
method Proving lower bounds on the number of adaptive score queries needed for sampling.
result Any sampling algorithm requires at least \(\widetilde{\Omega}(\sqrt{d})\) adaptive score queries for \(d\)-dimensional distributions.
Adaptive batch size schedules improve language model training efficiency and generalization.
problem Dilemma of choosing batch sizes in large-scale model training.
method General-purpose adaptive batch size schedules compatible with data and model parallelism.
result Adaptive batch size schedules outperform constant batch sizes and heuristic warmup schedules.
This paper optimizes diffusion schedules for better sampling from data distributions.
problem Choosing an optimal discretization schedule for denoising diffusion models.
method Adaptive algorithm that selects an optimal schedule based on a work cost measure.
result The learned schedule recovers and outperforms manually tuned schedules.
This study shows how DDPM can be represented by the OU process.
problem Designing optimal noise schedules for DDPM.
method Formal equivalence between DDPM and OU process, heuristic designs based on Fisher Information.
result Fisher-Information-motivated schedule corresponds to cosine noise schedule.
Improves diffusion models by controlling total variance and signal-to-noise-ratio.
problem Long sampling time in diffusion models.
method Total-Variance/Signal-to-Noise-Ratio (TV/SNR) disentangled framework.
result Improves generation performance by controlling TV and SNR independently.
The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.
problem The variability of SGD performance due to step size choice.
method Analysis and empirical evaluation of exponential and cosine step sizes.
result Exponential and cosine step sizes are adaptive to noise and achieve optimal performance without tuning hyperparameters.
We analyze DMs using spectral methods to design effective noise schedules.
problem Lack of theoretical foundation for synthesis process decisions in DMs.
method Introduced a frequency response perspective based on Gaussianity assumption.
result Proposed a spectral transfer function to understand DM inference process.
Researchers establish bounds for SGMs' KL and Wasserstein divergences under various noise schedules.
problem Estimating the error between target and estimated distributions in SGMs.
method Established upper bounds for KL divergence and Wasserstein distance, incorporating target distribution properties and SGM hyperparameters.
result Optimal noise schedules identified for SGMs, improving generative quality.
MLR-SNet learns flexible LR schedules for diverse tasks.
problem Adapting LR schedules for non-convex optimization problems.
method Parameterizes LR schedules with an explicit mapping formulation.
result Meta-learned LR schedules are transferable and adaptable.
We improve private training accuracy with learning rate schedules and matrix factorizations.
problem Private training with learning rate schedules and correlated noise.
method General upper and lower bounds for learning rate schedules, memory-efficient constructions, and schedule-aware factorizations.
result Schedule-aware factorizations improve accuracy in private training.
To train neural machine translation models simultaneously on multiple tasks (languages), it is common to sample each task uniformly or in proportion to dataset sizes. As these methods offer little control over performance trade-offs, we explore different task scheduling approaches. We first consider existing non-adapti…
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
MDMs train to decode tokens in a random order, which affects performance; we show they can be optimized for a favorable order.
problem Performance of MDMs is affected by the random order in which tokens are decoded.
method We show that MDMs can be optimized for a favorable decoding order by equipping their continuous-time variational objective with multivariate noise schedules.
result MDMs can be decomposed into a weighted auto-regressive losses over orders, making them auto-regressive models with learnable orders.
Optimal learning rate schedules derived for various tasks.
problem Inadequate learning rate schedules in practice compared to theory.
method Refined analysis of learning rate schedules for optimization algorithms.
result Derives new problem-adaptive learning rate schedules.
BDDMs eliminate noise conditioning in diffusion models, simplifying training and sampling.
problem Noise conditioning in diffusion models is ad hoc and requires unprincipled noise embeddings.
method Introduce blind denoising diffusion models (BDDMs) that do not require noise conditioning.
result BDDMs simplify training and sampling by eliminating noise conditioning.
BLiE optimizes hyperparameters with theoretical guarantees and superior performance.
problem Hyperparameter optimization in machine learning.
method Lipschitz bandit approach exploiting Lipschitz continuity.
result BLiE finds ε-optimal hyperparameters with theoretical complexity.
AIHT improves online high-dimensional quantile regression by separating support discovery and refinement.
problem Online high-dimensional quantile regression with structural sparsity.
method Adaptive Iterative Hard Thresholding (AIHT) alternates stochastic updates with adaptive hard-thresholding steps.
result AIHT achieves logarithmic regret for the sliding-window objective in high-dimensional settings.
New damping technique improves deep learning models by reducing noise in flat directions.
problem Improving generalization in deep learning models by reducing estimation noise in flat directions.
method Developed a novel random matrix theory based damping learner to reduce the shrinkage coefficient and improve generalization.
result Significant generalization improvements in logistic regression and deep neural networks experiments.
The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.
problem Analyzing Adam and SGD in nonstationary optimization problems.
method Theoretical analysis of Adam and SGD under non-stationary stochastic objectives, separating two regimes.
result Characterizes the tradeoff between noise and drift in Adam and SGD, revealing when adaptive step-sizing is beneficial or harmful.
New method reduces Gibbs partition function estimation complexity.
problem Estimating partition functions of Gibbs distributions.
method Doubly-adaptive MCMC with adaptive cooling schedule and mean estimator.
result Outperforms state-of-the-art algorithms in computational complexity and robustness.
Study shows annealing with adaptive schedule reduces mode collapse in NFs for parameter estimation.
problem Mode collapse in normalizing flows for multimodal distributions.
method Annealing with an adaptive schedule based on effective sample size (ESS).
result Our approach reduces mode collapse and converges marginal likelihood faster than MCMC methods.
Study finds optimal learning rate schedules for sub-100M quantization-aware training across bit-widths.
problem Optimal learning rate schedules for quantization-aware training depend on bit-width.
method Factorial grid testing over bit-width, warmdown fraction, LR magnitude, model size, and seed.
result INT6 QAT requires a different schedule than higher-precision training, falsifying the primary hypothesis.
Auto-Ensemble automates deep learning model ensembling with adaptive learning rate scheduling.
problem Difficulty in collecting diverse and accurate deep learning models through single training.
method Auto-Ensemble collects model checkpoints and uses adaptive learning rate scheduling to ensemble them.
result Ensembled models converge to various local optima, improving performance on few-shot learning.
We propose a variance reduction framework for variational inference using the Multilevel Monte Carlo (MLMC) method. Our framework is built on reparameterized gradient estimators and "recycles" parameters obtained from past update history in optimization. In addition, our framework provides a new optimization algorithm …
The learning rate is one of the most important hyper-parameters for model training and generalization. However, current hand-designed parametric learning rate schedules offer limited flexibility and the predefined schedule may not match the training dynamics of high dimensional and non-convex optimization problems. In …
New adaptive scheduler improves SAM for better model training.
problem Training machine learning models requires selecting a learning rate, which is often difficult and time-consuming.
method Derive Polyak schedulers tailored to SAM-style updates, proving linear convergence for strongly convex objectives and an O(1/T) rate for convex objectives.
result Polyak schedulers achieve comparable or better performance than tuned SAM baselines, reducing the need for learning-rate tuning.
Adaptive model scheduling boosts data labeling efficiency.
problem Efficiently labeling diverse data with limited resources.
method Adaptive Model Scheduling framework using deep reinforcement learning and heuristic algorithms.
result 53% reduction in execution time with no loss of labels.
Improved diffusion models achieve state-of-the-art likelihoods in image density estimation.
problem Improving likelihood-based performance of diffusion models.
method Joint optimization of noise schedule and model parameters, using signal-to-noise ratio simplification.
result State-of-the-art likelihoods on image density estimation benchmarks, faster optimization.
This paper proposes a system-agnostic policy for dynamic scheduling.
problem Dynamic scheduling in changing systems is challenging due to system-specific optimal policies.
method Descriptive policy that learns a system-agnostic scheduling principle.
result System-agnostic meta-learning enables adaptation to unseen system characteristics.
Study separates learning rate effects from adaptive gradient methods.
problem Understanding the impact of learning rates on neural network training.
method Introduced a 'grafting' experiment to isolate learning rate effects.
result Many existing beliefs about adaptive gradient methods may be incorrect.
Unified framework for distributed compressed SGD under (L0,L1)-smoothness.
problem Understanding the joint effect of batch noise, adaptivity, and compression in distributed stochastic optimization.
method Developed a unified theoretical framework using SDEs that incorporate curvature-dependent terms.
result Normalizing updates in DCSGD stabilizes convergence, with normalization degree determined by noise structure and landscape regularity.
Spaced repetition is among the most studied learning strategies in the cognitive science literature. It consists in temporally distributing exposure to an information so as to improve long-term memorization. Providing students with an adaptive and personalized distributed practice schedule would benefit more than just …
Adaptive optimization algorithms such as Adam are widely used in deep learning. The stability of such algorithms is often improved with a warmup schedule for the learning rate. Motivated by the difficulty of choosing and tuning warmup schedules, recent work proposes automatic variance rectification of Adam's adaptive l…
Proposes an automatic cyclical scheduling for gradient-based discrete sampling.
problem Gradient-based sampling in high-dimensional models can get stuck in local modes.
method Cyclical step size and balancing schedules with automatic hyperparameter tuning.
result Proves non-asymptotic convergence and inference guarantees for general discrete distributions.
AdaPID optimizes diffusion-based samplers by dynamically adjusting schedules.
problem Optimizing the intermediate-time dynamics in diffusion-based samplers.
method Develops a time-varying stiffness schedule using Piece-Wise-Constant (PWC) parametrizations and a hierarchical refinement approach.
result QoS-driven PWC schedules consistently improve sampling fidelity and accuracy.
We present a representation learning method that learns features at multiple different levels of scale. Working within the unsupervised framework of denoising autoencoders, we observe that when the input is heavily corrupted during training, the network tends to learn coarse-grained features, whereas when the input is …
Deep learning approach for efficient IoT task scheduling in MEC networks.
problem Minimizing task latency in IoT users with large-scale MEC systems.
method Stacked auto-encoder for data compression, adaptive simulated annealing, experience replay.
result Near-optimal performance with significantly reduced computational time.
ADCMs adaptively discretize CMs for efficient training.
problem Manual discretization schemes cause repeated adjustments for different noise schedules and datasets.
method Unified framework with optimization problem, local and global consistency constraints, and Gauss-Newton method.
result Significantly improve training efficiency and generative performance of CMs.
Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
problem Optimizing training efficiency for large language models with adaptive optimizers.
method Develops a principled framework for batch-size scheduling, introducing Seesaw which multiplies learning rate by 1/√2 and doubles batch size.
result Empirically, Seesaw reduces wall-clock time by approximately 36% compared to cosine decay, matching theoretical limits.
AdaS adapts SGD learning rate based on knowledge gain metrics.
problem Empirical step-size selection in SGD optimization lacks consistency and insight.
method Introduces AdaS algorithm that adapts SGD learning rate based on knowledge gain metrics.
result AdaS outperforms existing adaptive learning methods in convergence and generalization.
AdaAnn optimizes annealing for efficient probability density approximation.
problem Efficiently approximating complex probability distributions with multiple modes.
method AdaAnn is an adaptive annealing scheduler that adjusts temperature increments based on KL divergence.
result AdaAnn improves computational efficiency in variational inference and parameter estimation.
New method adapts DLMs to intrinsic data dependence without prior knowledge.
problem Understanding how unmasking schedules affect DLM generation quality.
method Adapts unmasking schedule to target data distribution's dependence structure.
result Sampling convergence guarantees improve for low-complexity distributions.
Deployment of emerging technologies and rapid change in industries has created a lot of risk for initiating the new projects. Many techniques and suggestions have been introduced but still lack the gap from various prospective. This paper proposes a reliable project scheduling approach. The objectives of project schedu…
This work formalizes guidance in diffusion models and introduces a stochastic control framework.
problem Lack of a solid theoretical foundation for guidance scheduling in diffusion models.
method Introduces a stochastic optimal control framework to cast guidance scheduling as an adaptive optimization problem.
result Establishes a principled foundation for more effective guidance in diffusion models.
Meta-SAGE improves deep RL scalability for CO tasks by adapting pre-trained models to larger-scale problems.
problem Improving scalability of deep reinforcement learning models for combinatorial optimization tasks.
method Meta-SAGE combines a scale meta-learner and scheduled adaptation with guided exploration to adjust model parameters for larger-scale problems.
result Meta-SAGE outperforms previous methods and significantly improves scalability in CO tasks.