Generative models improve for multiscale scientific data with new noise and interpolation techniques.
problem Numerical challenges in generating high-fidelity samples for multiscale scientific data.
method Design of noise distributions and interpolation schedules in function space to ensure Lipschitz regularity and finite noise roughness.
result Scale-adaptive noise and interpolation schedules improve numerical efficiency and fidelity of generated samples.
This work sets lower bounds on the number of score queries needed for diffusion sampling.
problem Establishing information-theoretic limits on the number of score evaluations required for diffusion sampling.
method Proving lower bounds on the number of adaptive score queries needed for sampling.
result Any sampling algorithm requires at least \(\widetilde{\Omega}(\sqrt{d})\) adaptive score queries for \(d\)-dimensional distributions.
Bayesian model learns multiscale interactions in complex systems.
problem Understanding dynamic interplay between processes at different time scales.
method Bayesian learning framework with Particle Gibbs with Ancestor Sampling (PGAS) algorithm.
result Demonstrated the effectiveness of the proposed approach through simulations.
This study shows how DDPM can be represented by the OU process.
problem Designing optimal noise schedules for DDPM.
method Formal equivalence between DDPM and OU process, heuristic designs based on Fisher Information.
result Fisher-Information-motivated schedule corresponds to cosine noise schedule.
ANT improves TS diffusion models by automatically determining noise schedules.
problem Suboptimal performance of TS diffusion models due to lack of domain-specific noise schedules.
method ANT proposes an adaptive noise schedule that automatically determines proper noise schedules for TS datasets based on their statistics.
result ANT achieves state-of-the-art performance on various TS tasks, including forecasting, refinement, and generation.
Improves diffusion models by controlling total variance and signal-to-noise-ratio.
problem Long sampling time in diffusion models.
method Total-Variance/Signal-to-Noise-Ratio (TV/SNR) disentangled framework.
result Improves generation performance by controlling TV and SNR independently.
The variational autoencoder (VAE) framework remains a popular option for training unsupervised generative models, especially for discrete data where generative adversarial networks (GANs) require workaround to create gradient for the generator. In our work modeling US postal addresses, we show that our discrete VAE wit…
We analyze DMs using spectral methods to design effective noise schedules.
problem Lack of theoretical foundation for synthesis process decisions in DMs.
method Introduced a frequency response perspective based on Gaussianity assumption.
result Proposed a spectral transfer function to understand DM inference process.
Researchers establish bounds for SGMs' KL and Wasserstein divergences under various noise schedules.
problem Estimating the error between target and estimated distributions in SGMs.
method Established upper bounds for KL divergence and Wasserstein distance, incorporating target distribution properties and SGM hyperparameters.
result Optimal noise schedules identified for SGMs, improving generative quality.
This paper proposes a novel multiscale estimator for the integrated volatility of an Ito process, in the presence of market microstructure noise (observation error). The multiscale structure of the observed process is represented frequency-by-frequency and the concept of the multiscale ratio is introduced to quantify t…
We improve private training accuracy with learning rate schedules and matrix factorizations.
problem Private training with learning rate schedules and correlated noise.
method General upper and lower bounds for learning rate schedules, memory-efficient constructions, and schedule-aware factorizations.
result Schedule-aware factorizations improve accuracy in private training.
MDMs train to decode tokens in a random order, which affects performance; we show they can be optimized for a favorable order.
problem Performance of MDMs is affected by the random order in which tokens are decoded.
method We show that MDMs can be optimized for a favorable decoding order by equipping their continuous-time variational objective with multivariate noise schedules.
result MDMs can be decomposed into a weighted auto-regressive losses over orders, making them auto-regressive models with learnable orders.
Graph Prolongation Convolutional Networks improve model performance in microtubule bending simulations.
problem Improving prediction accuracy in coarse-grained mechanochemical simulations of microtubule bending.
method Defines a novel ensemble Graph Convolutional Network model using optimized linear projection operators to map between graph scales.
result Graph Prolongation-Convolutional Network outperforms other GCN ensemble models in predicting microtubule bending potential energy.
Deterministic GD can behave stochastically in large learning rates for multiscale functions.
problem Understanding deterministic GD's stochastic behavior in large learning rates for multiscale objectives.
method Established a sufficient condition for deterministic GD to converge to a rescaled Gibbs distribution in large learning rates for multiscale functions.
result Deterministic GD can converge to a statistical distribution in large learning rates for multiscale functions.
Multigrid modeling algorithms are a technique used to accelerate relaxation models running on a hierarchy of similar graphlike structures. We introduce and demonstrate a new method for training neural networks which uses multilevel methods. Using an objective function derived from a graph-distance metric, we perform or…
A new sampling method estimates scores without training or nested MCMC.
problem Efficient sampling from complex, unnormalised distributions.
method Multiscale averaging in SDEs for score estimation.
result Empirical results show competitive accuracy and efficiency.
Study finds optimal learning rate schedules for sub-100M quantization-aware training across bit-widths.
problem Optimal learning rate schedules for quantization-aware training depend on bit-width.
method Factorial grid testing over bit-width, warmdown fraction, LR magnitude, model size, and seed.
result INT6 QAT requires a different schedule than higher-precision training, falsifying the primary hypothesis.
Proposes CDTD, a diffusion model for mixed-type tabular data.
problem Adapting diffusion models to mixed-type tabular data.
method Score matching and score interpolation for continuous features, adaptive noise schedules for categorical features.
result Consistently outperforms state-of-the-art models in mixed-type tabular data.
Improved diffusion models achieve state-of-the-art likelihoods in image density estimation.
problem Improving likelihood-based performance of diffusion models.
method Joint optimization of noise schedule and model parameters, using signal-to-noise ratio simplification.
result State-of-the-art likelihoods on image density estimation benchmarks, faster optimization.
This paper optimizes diffusion schedules for better sampling from data distributions.
problem Choosing an optimal discretization schedule for denoising diffusion models.
method Adaptive algorithm that selects an optimal schedule based on a work cost measure.
result The learned schedule recovers and outperforms manually tuned schedules.
MLR-SNet learns flexible LR schedules for diverse tasks.
problem Adapting LR schedules for non-convex optimization problems.
method Parameterizes LR schedules with an explicit mapping formulation.
result Meta-learned LR schedules are transferable and adaptable.
We present a representation learning method that learns features at multiple different levels of scale. Working within the unsupervised framework of denoising autoencoders, we observe that when the input is heavily corrupted during training, the network tends to learn coarse-grained features, whereas when the input is …
Adjoint Matching improves flow and diffusion models with reward fine-tuning.
problem Improving generative models with reward fine-tuning.
method Casting reward fine-tuning as stochastic optimal control (SOC) and enforcing a specific noise schedule.
result Adjoint Matching outperforms existing SOC algorithms.
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
Unified diffusion framework enhances generative models flexibility.
problem Improving generative models' design freedom and efficiency.
method Unified framework incorporating choice of representation, prior distribution, and noise scheduling.
result Enhanced flexibility leading to more efficient training and data generation.
The paper proves Gorenstein contractions for multiscale differentials on nodal curves.
problem Proving Gorenstein contractions for multiscale differentials on nodal curves.
method Addressing the conjecture by Ranganathan and Wise, showing contractions level by level.
result Multiscale differentials can be contracted to Gorenstein singularities, level by level, from the top down.
A method predicts GNS of transformer layers using normalization layer norms.
problem Estimating gradient noise scale with minimal variance.
method Simultaneously compute per-example gradient norms and parameter gradients.
result Total GNS is predicted well by normalization layer GNS.
Proposes a new algorithm for non-stationary bandits.
problem Non-stationary reward distributions in contextual bandits.
method Multiscale changepoint detection for adaptive learning.
result Regret bound analysis and superior performance in experiments.
Adaptive batch size schedules improve language model training efficiency and generalization.
problem Dilemma of choosing batch sizes in large-scale model training.
method General-purpose adaptive batch size schedules compatible with data and model parallelism.
result Adaptive batch size schedules outperform constant batch sizes and heuristic warmup schedules.
DMGNN predicts 3D human motions using adaptive multiscale graphs.
problem Predicting 3D skeleton-based human motions accurately.
method Dynamic multiscale graph neural networks (DMGNN) with adaptive multiscale graphs and MGCU.
result DMGNN outperforms state-of-the-art methods in short and long-term predictions.
Neural network approach simplifies multiscale problem homogenization.
problem Homogenizing multiscale problems with varying microscale structures.
method Derivative-free neural network with Brownian walkers.
result Neural network method is computationally efficient and robust.
New framework optimizes deep learning training by deferring large batch sizes to late stages.
problem Optimizing batch size scheduling for deep learning training efficiency.
method Introduced the functional scaling law (FSL) framework to analyze and optimize batch size scheduling.
result Large batch sizes can be deferred to late training stages without sacrificing performance.
Diffusion models accurately recover mixture weights from generated samples despite score function insensitivity.
problem Score-based generative models often fail to learn correct relative mode amplitudes (mixture weights) from generated samples.
method Relate diffusion score matching (DSM) loss to mixture weight estimation error, define diffusion score sensitivity index (DSSI), and prove its governing role in mixture weight recovery.
result Generated samples can accurately recover mixture weights from the DSM loss, even when the target score is insensitive to mixture weights.
BLiE optimizes hyperparameters with theoretical guarantees and superior performance.
problem Hyperparameter optimization in machine learning.
method Lipschitz bandit approach exploiting Lipschitz continuity.
result BLiE finds ε-optimal hyperparameters with theoretical complexity.
We discuss multiscale representations of discrete manifold-valued data. As it turns out that we cannot expect general manifold-analogues of biorthogonal wavelets to possess perfect reconstruction, we focus our attention on those constructions which are based on upscaling operators which are either interpolating or midp…
Masking diffusion outperforms other discrete diffusion models by incorporating jump times into the model.
problem Improving the performance of discrete diffusion models.
method Conditioning on the jump schedule of discrete Markov processes.
result Schedule-conditioned discrete diffusion (SCUD) models outperform classical and masking diffusion models.
We present a graph-theoretical approach to data clustering, which combines the creation of a graph from the data with Markov Stability, a multiscale community detection framework. We show how the multiscale capabilities of the method allow the estimation of the number of clusters, as well as alleviating the sensitivity…
MELD model clusters data at multiple scales, improving understanding of latent structure.
problem Understanding latent multiscale structure in datasets.
method Multiscale Learning by Unsupervised Nonlinear Diffusion (M-LUND) algorithm.
result M-LUND detects latent structure in synthetic and real datasets.
New algorithm learns switching dynamics from multiple neural signals.
problem Learning accurate switching dynamical system models from multimodal neural data.
method Unsupervised learning algorithm for multiscale switching dynamical system models.
result Switching multiscale dynamical system models outperform single-scale models in behavior decoding.
A new method optimizes diffusion models for fine-tuning tasks efficiently.
problem Optimizing diffusion models for downstream tasks using nested bilevel structures.
method Formalizes the challenge as a generative bilevel optimization problem and introduces a first-order bilevel framework.
result Our method outperforms existing fine-tuning and hyperparameter search baselines.
Optimal multiscale learning of linear operators
problem Statistical and computational limits of learning bounded linear operators between Sobolev spaces
method Reformulate as an infinite-dimensional matrix regression problem with heterogeneous multiscale structure
result Establish minimax rates and construct a finite-resolution blockwise least-squares estimator attaining these rates
MsIGN tackles high-dimensional Bayesian inference using multiscale structure.
problem High-dimensional Bayesian inference challenges due to the curse of dimensionality.
method MsIGN generates samples from coarse to fine scale, minimizing Jeffreys divergence.
result MsIGN outperforms previous approaches in posterior approximation and mode capture.
Kernel analog forecasting studied for multiscale systems.
problem Interpreting data-driven predictions in multiscale dynamical systems.
method Kernel analog forecasting methods applied to multiscale systems with varying Markovian closures.
result Guidance provided for interpreting data-driven predictions in practice.
The paper extends entropy maximization to multiscale settings and applies it to neural networks.
problem Achieving optimal risk bounds in neural networks using multiscale entropy.
method Generalizing maximum entropy to multiscale settings and applying it to neural networks.
result The multiscale Gibbs posterior can achieve a smaller excess risk than the single-scale Gibbs posterior in a teacher-student scenario.
New method analyzes knots and links using multiscale Gauss link integral.
problem Lack of localization and quantization in knot theory applications.
method Integrates curve segmentation and multiscale analysis into the Gauss link integral.
result Significantly outperforms other methods in protein flexibility analysis.
The paper provides an efficient method to price path-dependent derivatives using multiscale stochastic volatility models.
problem Pricing path-dependent derivatives under multiscale stochastic volatility models.
method Derives a Malliavin representation for the first-order approximation of the price of path-dependent derivatives.
result An efficient Monte Carlo approximation for pricing path-dependent derivatives is derived.
Novel graph network learns hierarchical network structure.
problem Lack of information in hierarchical network topology.
method Hierarchical clustering for multiscale decomposition, graph convolutional layers.
result Competitive performance on citation network benchmark.
Extracts causal brain dynamics across multiple scales.
problem Statistical associations do not reflect causal mechanisms in brain dynamics.
method Multiscale causal backbone (MCB) extraction using advanced causal structure learning.
result Sparse MCBs reveal distinct causal roles at different brain frequency bands.