TomOpt optimizes muon detector designs using differentiable programming.
problem Designing efficient particle detectors for muon tomography.
method Differentiable programming for muon interaction modeling, inference, and optimisation.
result Demonstrated end-to-end differentiable and inference-aware optimisation of particle physics instruments.
CMS uses neural networks to monitor muon detector anomalies.
problem Monitoring anomalies in muon detector data for physics analysis.
method Supervised and semi-supervised artificial neural networks, convolutional autoencoders.
result Unprecedented efficiency in detecting known anomalous behaviors.
Muon optimizer outperforms GD in neural networks.
problem Optimizing matrix-structured parameters in neural networks.
method Muon optimizer specifically designed for matrix parameters, analyzing convergence rate and low-rank Hessian structure.
result Muon can outperform Gradient Descent due to its ability to leverage the low-rank structure of Hessian matrices.
Muon fails to converge on convex Lipschitz functions.
problem Understanding the convergence of Muon on convex and Lipschitz functions.
method Analyzing Muon's performance on convex and Lipschitz functions without error feedback.
result Muon does not converge on convex and Lipschitz functions, regardless of learning rate schedule.
Muon optimizer simplifies matrix optimization with spectral orthogonalization.
problem Matrix optimization challenges, especially with large condition numbers.
method Simplified Muon optimizer using spectral orthogonalization of gradients.
result Simplified Muon converges linearly with independent scalar sequences, outperforming gradient descent and Adam.
Muon outperforms GD in associative memory learning by balancing frequency components.
problem Training dynamics and scaling behavior of Muon in associative memory learning.
method Study of Muon in a linear associative memory model with softmax retrieval and hierarchical frequency spectrum over query-answer pairs.
result Muon achieves exponential speedup over GD in noiseless case and superior scaling efficiency in noisy case.
Muon optimizer improves deep learning with spectral norm constraints.
problem Improving optimization algorithms in deep learning.
method Theoretical analysis of Muon optimizer within the Lion-K family. result Muon implicitly solves an optimization problem enforcing spectral norm constraints.
Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
problem Improving the convergence rate of Muon optimizer.
method Using Newton-Schulz steps for momentum orthogonalization, proving convergence rate and constant factor.
result Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
MiMuon optimizer improves generalization for large models by reducing generalization error.
problem Improving generalization of Muon optimizer for large models.
method Enhanced Muon optimizer using orthogonalization of gradient, proving lower generalization error.
result MiMuon optimizer has a lower generalization error of O(N1) compared to Muon's O(NκT1). EF21-Muon optimizes deep learning with error feedback, improving efficiency and accuracy.
problem Lack of principled distributed frameworks for non-Euclidean LMO-based optimizers.
method Introduces EF21-Muon, a communication-efficient, non-Euclidean LMO-based optimizer with convergence guarantees.
result First efficient distributed implementation of non-Euclidean LMO-based optimizers, achieving up to 7x communication savings.
Muon replaces matrix gradient with polar factor, optimizing flat spectrum updates
problem Optimization bias in matrix updates
method Using polar factor of gradient
result Muon update maximizes entropy among bounded updates
Develops a gradient flow for Muon optimizer, a method for optimization.
problem Optimization of complex systems with matrix-valued parameters.
method Gradient flow on probability measures induced by regularized Muon optimizer.
result Derives continuous-time limits and proves Hamiltonian dissipation.
Muon optimizes training efficiency by improving data retention at large batch sizes.
problem Improving training efficiency and data retention at large batch sizes.
method Introducing Muon, a second-order optimizer, and combining it with muP for efficient hyperparameter transfer.
result Muon outperforms AdamW in retaining data efficiency at large batch sizes, enabling more economical training.
New Muon and Momo variants improve neural network optimization robustness.
problem Improving neural network optimization methods.
method Systematic exploration of non-Euclidean gradient descent variants.
result Momo variants of Muon are more robust to hyperparameter tuning.
Muon optimizer improves federated learning performance.
problem Improving federated learning performance with Muon optimizer.
method Proposed FedMuon algorithm with orthonormalized update direction.
result FedMuon's learning rate is independent of problem-specific parameters.
Shampoo achieves higher token efficiency than Muon in language models.
problem Understanding the relationship and relative data efficiency of Shampoo and Muon compared to Adam and Signum.
method Extensive experiments on language models, demonstrating Shampoo's higher efficiency and decomposing its updates.
result Shampoo's benefits are attributed to its application to weight matrices, challenging interpretations based on variance adaptation and whitening.
New analysis of Muon and SignSGD on matrix-valued least squares problems.
problem Understanding the behavior of Muon and SignSGD on matrix-valued least squares problems.
method Derive explicit deterministic dynamics to study learning behavior of Muon and SignSGD.
result Muon and SignSGD exhibit different optimal learning rates and convergence characteristics based on batch size and data covariance.
New SAM method improves model robustness with spectral inner perturbation and Muon optimizer.
problem Improving model robustness to small parameter perturbations.
method Introducing a spectral inner perturbation step in SAM combined with Muon optimizer.
result Spectral inner perturbation combined with Muon optimizer achieves best validation accuracy on ImageNet-1K.
Muon spectral optimizer outperforms SGD in associative memory tasks.
problem Understanding the advantage of spectral optimizers in learning associative memory.
method Linear associative memory problem, Gaussian inputs and outputs, power law frequency distribution, thresholded gradient approximation.
result Muon significantly outperforms SGD in storage capacity and recovery rates.
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
Unified view of Lion and Muon as Stochastic Frank-Wolfe methods.
problem Optimization of constrained problems in deep learning.
method Interpreting Lion and Muon as Stochastic Frank-Wolfe methods and extending the approach to heavy-tailed noise.
result Convergence guarantees and KKT point convergence for Lion and Muon.
Muon optimizes Transformer training with heavy-tailed data, achieving optimal sample complexity.
problem Theoretical understanding of non-Euclidean optimisation methods for heavy-tailed data in training Transformers.
method Addressing the gap in theoretical understanding, we show Muon achieves optimal sample complexity under heavy-tailed noise.
result Muon finds an ε-stationary point in nuclear norm with optimal sample complexity, absorbing heavy-tailed noise without dimension dependence.
ML PCA detects phase transitions in muon spectroscopy data.
problem Detecting phase transitions in materials using muon spectroscopy data.
method Unsupervised Machine Learning (PCA) applied to asymmetry functions.
result PCA method effectively detects phase transitions in muon spectroscopy experiments.
Muon dynamics study uses spectral Wasserstein flow for optimization stability.
problem Optimizing deep learning models with gradient normalization.
method Introduces Spectral Wasserstein distances for matrix flows, proving equivalence with Benamou--Brenier formulation.
result Gradient-flow interpretation of mean-field normalized training dynamics.
MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.
problem Training matrix-valued parameters with orthogonalized-update optimizers like Muon.
method MuonEq introduces three lightweight pre-orthogonalization equilibration schemes: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).
result Row/column normalization acts as a zeroth-order surrogate for whitening and improves the geometry seen by orthogonalization.
Muons and random optimizers perform similarly, challenging geometric optimization theory.
problem Empirical success of Muon optimizer challenges geometric optimization theory.
method Introducing Freon and Kaon optimizers, demonstrating performance without precise geometric structure.
result Performance of optimizers is controlled by alignment and descent potential, not geometric structure.
Drop-Muon updates only some layers, speeding up training.
problem Conventional deep learning optimizers update all layers at once, which can be inefficient.
method Drop-Muon updates only a subset of layers per step, with randomized schedules.
result Drop-Muon achieves up to 1.4x faster training time with similar accuracy.
New optimizer MARS-M combines variance reduction with Muon for faster LLM training.
problem Training large-scale neural networks efficiently.
method Integrates MARS variance reduction with Muon optimizer.
result MARS-M converges to a first-order stationary point at a rate of ildeO(T−1/3). AuON is a linear-time optimizer that improves upon Muon's performance without approximate orthogonal matrices.
problem High memory and computational costs of orthogonal momentum updates.
method AuON uses normalized nonlinear scaling and a 'emergency brake' to handle exploding attention logits.
result AuON achieves strong performance without approximate orthogonal matrices, preserving structural alignment and reconditioning.
PolarGrad optimizes deep learning models by considering matrix structure, outperforming Adam and Muon.
problem Efficient optimization of large-scale neural networks and language models.
method A unifying framework for analyzing matrix-aware preconditioned methods, including PolarGrad.
result PolarGrad outperforms Adam and Muon in various tasks.
Muon optimizes deep learning models on imbalanced data by learning all components equally.
problem Generalization issues in deep learning models on imbalanced data.
method Introduced Spectral Gradient Descent (SpecGD) as a canonical form of Muon and Shampoo, and studied its performance on imbalanced data.
result SpecGD learns all principal components of imbalanced data at equal rates, unlike vanilla GD which prioritizes dominant components.
Improved tensor GLM estimation for complex data.
problem Complex tensor data in GLMs leads to high-dimensional, ill-posed estimation.
method Proposed LSRTR-M algorithm using Muon updates for faster convergence and lower errors.
result LSRTR-M converges faster and achieves lower errors than LSRTR.
A new Riemannian framework optimizes LoRA for faster convergence and better performance.
problem Optimizing low-rank adapters in neural networks to improve convergence and performance.
method Integrates Riemannion optimizer, LoRA initialization, and efficient implementation for geometrically treating low-rank adapters.
result Consistent and noticeable improvements in convergence speed and final task performance over standard LoRA and its modifications.
PCA improves detection of phase transitions in muon spectroscopy data from various materials.
problem Subtle changes in asymmetry function indicate phase transitions, but existing methods require material-specific knowledge.
method Applied unsupervised PCA to muon spectroscopy asymmetry data from multiple materials.
result PCA can recover phase transition indicators and improve detection of material-specific variations.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
This letter presents the sparse vector signal detection from one bit compressed sensing measurements, in contrast to the previous works which deal with scalar signal detection. In this letter, available results are extended to the vector case and the GLRT detector and the optimal quantizer design are obtained. Also, a …
Gluon optimizes LMO-based methods for large-scale tasks, improving performance and theory-practice gap.
problem LMO-based methods lack theoretical support for practical implementation and smoothness assumptions.
method Introduces Gluon, a new LMO-based method with refined smoothness model.
result Gluon's theoretical stepsizes match fine-tuned values, closing the theory-practice gap.
Spectral gradient methods outperform Euclidean in certain deep learning scenarios.
problem When do spectral gradient updates outperform Euclidean in deep learning?
method Layerwise condition comparing squared nuclear-to-Frobenius ratio to stable rank of activations.
result Spectral updates can be more effective than Euclidean in deep networks and transformers.
Adversarial perturbations fool deepfake detectors with high accuracy.
problem Improving deepfake detection accuracy against adversarial attacks.
method Used adversarial perturbations and two defenses: Lipschitz regularization and Deep Image Prior (DIP).
result Deepfake detectors achieved 27% accuracy on perturbed images, compared to 95% on unperturbed.
This paper studies concept drift detectors for financial time series.
problem Improving accuracy on financial time series with concept drifts.
method Three simple concept drift detectors tailored to financial time series.
result Two of the detectors are as effective as state-of-the-art detectors.
Graph networks improve particle reconstruction in irregular detectors.
problem Handling irregular particle-detector geometries in particle reconstruction.
method Introduce distance-weighted graph network architectures (GarNet, GravNet layers) for irregular geometry detectors.
result The proposed graph networks provide equally performing or less resource-demanding solutions compared to existing methods.
Bayesian detector improves defect detection on rough surfaces.
problem Defect detection on motion blurred images with rough texture.
method Developed Bayesian detector using reflected non-local prior distributions.
result Significantly enhanced accuracy in identifying defects.
ATSDLN adapts to time series data for anomaly detection.
problem Challenges in selecting and optimizing anomaly detectors for time series data.
method Adaptive Time Series Detector Learning Network (ATSDLN) that selects and optimizes detectors and parameters.
result ATSDLN outperforms other methods in anomaly detection across various datasets.
New method uses eigenspectrum to optimize object detector architectures.
problem Understanding the effects of ImageNet pre-training on object detectors.
method Analysis of eigenspectrum dynamics of feature maps in object detectors.
result Object detectors trained from scratch and ImageNet pre-trained models behave differently.
New method uses neural networks to estimate parameters without needing detector simulations.
problem Estimating parameters in high-energy physics with detector effects.
method Two-level fitting approach: SRGN (Simulation-level fit based on Reweighting Generator-level events with Neural networks).
result Demonstrated using simulated datasets, SRGN can estimate parameters without detector effects.
Deep neural networks improve MIMO detection performance.
problem Improving detection accuracy in massive MIMO systems.
method Introduced a neural network architecture based on BP algorithms, optimized with deep learning techniques.
result DNN MIMO detectors achieve lower bit error rates compared to other detectors.
DCSO dynamically selects top-performing base detectors for outlier ensembles.
problem Challenges in selecting and combining outlier scores from different detectors.
method DCSO dynamically selects top-performing base detectors based on local k-nearest neighbors.
result DCSO provides consistent performance improvement over static combination approaches.
Improves relevancy of black-box anomaly detectors with user feedback.
problem Users often ignore many detected anomalies, requiring a method to identify and prioritize relevant ones.
method Uses user feedback to adjust anomaly selection process based on identified anomaly types.
result Significant improvements in precision and recall over various anomaly detectors.