Cosine schedule is optimal for discrete diffusion models.
problem Choosing the best discretization schedule for diffusion models.
method Optimized using Fisher-Rao geometry.
result Cosine schedule is Fisher-Rao optimal.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Cosine Schedule”
The study introduces anytime learning schedules for large language models without fixed horizons.
problem Training large language models without knowing the total training horizon.
method Theoretical analysis and weight averaging to create anytime learning schedules.
result Theoretical and empirical evidence shows that weight averaging with simple step sizes can achieve comparable final loss to well-tuned cosine schedules.
This study shows how DDPM can be represented by the OU process.
problem Designing optimal noise schedules for DDPM.
method Formal equivalence between DDPM and OU process, heuristic designs based on Fisher Information.
result Fisher-Information-motivated schedule corresponds to cosine noise schedule.
The paper presents a multi-power law for predicting loss curves across different learning rate schedules.
problem Understanding and optimizing the relationship between model performance and hyperparameters, especially learning rates.
method Proposes a multi-power law that combines power laws based on the sum of learning rates and additional laws for loss reduction due to decay.
result The multi-power law accurately predicts loss curves for unseen learning rate schedules and finds a schedule that outperforms cosine learning rate.
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
Robust Transformer-Based One-Step Stock Index Forecasting via Shifted Data Augmentation
problem Robust stock index forecasting
method Modified Transformer architecture with Shifted Data Augmentation
result Best performance on benchmark datasets
The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.
problem The variability of SGD performance due to step size choice.
method Analysis and empirical evaluation of exponential and cosine step sizes.
result Exponential and cosine step sizes are adaptive to noise and achieve optimal performance without tuning hyperparameters.
Optimal learning rate schedules derived for various tasks.
problem Inadequate learning rate schedules in practice compared to theory.
method Refined analysis of learning rate schedules for optimization algorithms.
result Derives new problem-adaptive learning rate schedules.
Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (\textsc{Demon}) rule. We conduct the first large-scale empirical analysis of momentum decay methods for modern neural network optimization, in addition to the most popular learning rate dec…
WSqD extends learning rate schedules for large model training without fixed horizons.
problem Fixed learning rate schedules limit training horizon extension.
method WSqD replaces constant stable phase with a shifted inverse-square-root base, retaining linear cooldown.
result WSqD achieves minimax-optimal convergence rate and horizon-independence.
Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
problem Optimizing training efficiency for large language models with adaptive optimizers.
method Develops a principled framework for batch-size scheduling, introducing Seesaw which multiplies learning rate by 1/√2 and doubles batch size.
result Empirically, Seesaw reduces wall-clock time by approximately 36% compared to cosine decay, matching theoretical limits.
Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcs.LG
New method SF-AdamW trains large models without decay phases or memory overhead.
problem Inadequate fixed compute budgets for large-scale training.
method Schedule-Free (SF) method revisited and refined.
result SF-AdamW effectively navigates loss landscape without decay phases or memory overhead.
WSD schedule improves model training efficiency by adapting learning rates dynamically.
problem Fixed compute budgets limit training efficiency of language models.
method Introduces a WSD schedule that uses a constant learning rate followed by a rapid decay phase.
result WSD schedule generates a non-traditional loss curve with stable and decay phases.
Study finds optimal learning rate schedules for sub-100M quantization-aware training across bit-widths.
problem Optimal learning rate schedules for quantization-aware training depend on bit-width.
method Factorial grid testing over bit-width, warmdown fraction, LR magnitude, model size, and seed.
result INT6 QAT requires a different schedule than higher-precision training, falsifying the primary hypothesis.
Learning rate annealing improves robustness in stochastic optimization.
problem Tuning learning rates in large-scale models is costly and prone to errors.
method We analyze and demonstrate the benefits of learning rate annealing schemes.
result Stochastic gradient descent with annealed schedules converges more robustly to the optimal solution.
Sharp 2-Wasserstein bounds for DDPMs derived from Föllmer process.
problem Sampling error bounds for DDPMs in 2-Wasserstein distance.
method Lipschitz-type conditions on score function, Föllmer process, and log-concave target distributions.
result Sharp upper bounds for DDPMs in 2-Wasserstein distance, optimal in dimension and steps.
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strate…
Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaystat.ML
Optimal learning rates decay to zero in easy tasks and maintain a warmup phase in hard tasks.
problem Optimizing learning rates under functional scaling laws for model training.
method Deriving optimal learning-rate schedules based on exponents and .
result Sharp phase transition between easy and hard tasks, with different decay behaviors.
GradPower speeds up language model training with minimal code changes.
problem Slower training of large language models.
method Elementwise sign-power transformation applied to gradients.
result Consistently lower terminal loss across various models and datasets.
Modified cosine distance improves similarity performance in data with variance and correlation.
problem Limitations of traditional cosine similarity in random variable spaces with variance and correlation.
method Proposed a variance-adjusted cosine distance metric to overcome limitations of traditional cosine similarity.
result Modified cosine distance shows 100% test accuracy in KNN model on the Wisconsin Breast Cancer Dataset.
Improved MoE performance through perturbing cosine router.
problem Representation collapse and parameter redundancy in MoE models.
method Least square estimation of cosine router in MoE, followed by noise addition to improve convergence rates.
result Perturbed cosine router leads to polynomial convergence rates for MoE models.
Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…
Researchers establish bounds and continuity of decomposed Möbius energies using cosine formula.
problem Estimating the bounds and continuity of decomposed Möbius energies.
method Using the cosine formula to evaluate upper and lower bounds and modulus of continuity of decomposed energies.
result Affirmative answer to the question of estimating decomposed energies using the cosine formula.
Derives hyperbolic laws of cosines and sines with fermionic corrections.
problem Deriving hyperbolic laws of cosines and sines with new mathematical corrections.
method Using Minkowski supergeometry, the laws of cosines and sines are derived in the super hyperbolic plane.
result Identical formulae to classical cases with fermionic corrections for cosines and sines.
Seismic Data Interpolation via Denoising Diffusion Implicit Models with Coherence-corrected Resamplingphysics.geo-ph
A new method uses denoising diffusion models to improve seismic data interpolation.
problem Improving the accuracy of seismic data interpolation to enhance imaging and interpretation.
method The approach combines denoising diffusion probabilistic models with coherence-corrected resampling strategies.
result The proposed method achieves superior performance and generalization to various missing patterns and noise levels.
Cosine similarity can force points to grow in magnitude, causing convergence issues.
problem Cosine similarity loss can lead to convergence issues in deep learning.
method Analyzing under-explored settings and proposing cut-initialization.
result Cosine similarity optimization forces points to grow in magnitude, leading to convergence issues.
We study the rigidity of polyhedral surfaces using variational principle. The action functionals are derived from the cosine laws. The main focus of this paper is on the cosine law for a non-triangular region bounded by three possibly disjoint geodesics. Several of these cosine laws were first discovered and used by Fe…
New method for European option pricing faster and more robust.
problem Pricing European options efficiently and accurately.
method Fourier cosine series expansions for models with known characteristic functions.
result More robust and faster than the original COS method.
PEARL uses reinforcement learning to improve matrix preconditioners.
problem Learning effective preconditioners for iterative solvers is challenging.
method PEARL employs an actor-critic reinforcement learning framework to learn preconditioners dynamically.
result PEARL outperforms traditional and neural preconditioners in flexibility and solving speed.
Feedback alignment methods need to be evaluated for accuracy and gradient cosine similarity.
problem Evaluating feedback alignment methods
method Proposed diagnostic evaluation protocol
result Identified silent failures in standard reporting pair
This paper tackles noise in raw datasets to improve representation learning efficiency.
problem Noise in real-world datasets degrades representation learning quality.
method Proposes denoising Cosine-Similarity (dCS) loss to learn robust representations.
result Empirical results show the dCS loss outperforms baseline objective functions.
T-PSDA improves speaker recognition accuracy on toroidal submanifolds.
problem Improving speaker recognition accuracy on hypersphere embeddings.
method Extends PSDA to model within and between-speaker variabilities in toroidal submanifolds of the hypersphere.
result T-PSDA achieves accuracy on par with cosine scoring on VoxCeleb and large accuracy gains on NIST SRE'21.
We do further investigation in a certain cosine function defined for smooth Minkowski spaces. We prove that such function is symmetric if and only if the referred space is Euclidean, and also that it can be given in terms of the Gateaux derivative of the norm. As an application we use it to study the ratio between the …
The spherical Radon transform on the unit sphere can be regarded as a member of the analytic family of suitably normalized generalized cosine transforms. We derive new formulas for these transforms and apply them to study classes of intersections bodies in convex geometry.
New method schedules learning rate without stopping time, outperforming existing methods.
problem Learning rate schedules that require stopping time are outperformed.
method Schedule-Free approach that avoids stopping time and introduces no additional hyper-parameters.
result Exhibits state-of-the-art performance across various problems.
We extend the Fourier cosine method to discrete probability distributions, achieving faster convergence rates.
problem Extending Fourier cosine method to discrete probability distributions.
method Spectral filters and convergence rates analysis.
result Spectral filters achieve one order faster convergence rates than previously recognized.
This paper proposes a method to select project schedules with the lowest risk.
problem Selecting schedules that meet project deadlines while minimizing risk.
method Integrating aleatory uncertainty into project scheduling to quantify and compare risks.
result Proposes a method to select schedules with the lowest risk.
Person recognition aims at recognizing the same identity across time and space with complicated scenes and similar appearance. In this paper, we propose a novel method to address this task by training a network to obtain robust and representative features. The intuition is that we directly compare and optimize the cosi…
In this short article, we extend the cosine formula for the Möbius energy to generalized O'Hara energies. The newly derived formula gives us a condition for which the right circle minimizes the energy under the length-constraint. Furthermore, it shows us how far the energy is from the Möbius invariant property.
Optimizes financial auditor schedules to reduce time and costs.
problem Efficiently scheduling financial auditors with multiple constraints.
method Used Integer Linear Programming and compared two exact formulations.
result Multi-commodity network flow formulation is 24 times faster.
ScheduleFree+ improves large language model training without schedules or learning rates.
problem Scaling up Schedule-Free Learning to large language models.
method Learning-rate-free and schedule-free method for training large language models.
result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.
The Analysis from Nonlinear Distance Metric to Kernel-based Drug Prescription Prediction Systemcs.LG
The study compares Euclidean and cosine distances in medical drug prescription prediction.
problem Comparing Euclidean and cosine distances in medical drug prescription prediction.
method Established geometric properties and compared distances in real-world medical data.
result Different distances lead to different optimizing nonlinear kernel embedding frameworks.
A large body of research into semantic textual similarity has focused on constructing state-of-the-art embeddings using sophisticated modelling, careful choice of learning signals and many clever tricks. By contrast, little attention has been devoted to similarity measures between these embeddings, with cosine similari…
The paper optimizes interpolation schedules in generative models to improve sampling accuracy.
problem Improving sampling accuracy in generative models with fewer resources.
method Minimizing the averaged squared Lipschitzness of the drift field, using transfer formulas.
result Designed schedules yield more accurate fine-scale statistics at fixed integrator budget.
New method converts and optimizes sampling schedules for generative models.
problem Optimizing sampling schedules for generative models like flows and diffusions.
method Unified framework for stochastic interpolants, including point mass schedules.
result Demonstrated efficient generation of images with fewer steps.
Sensors which use electromagnetic induction (EMI) to excite a response in conducting bodies have long been investigated for subsurface explosive hazard detection. In particular, EMI sensors have been used to discriminate between different types of objects, and to detect objects with low metal content. One successful, p…
This paper proposes a system-agnostic policy for dynamic scheduling.
problem Dynamic scheduling in changing systems is challenging due to system-specific optimal policies.
method Descriptive policy that learns a system-agnostic scheduling principle.
result System-agnostic meta-learning enables adaptation to unseen system characteristics.
Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss fun…