TomOpt optimizes muon detector designs using differentiable programming.
problem Designing efficient particle detectors for muon tomography.
method Differentiable programming for muon interaction modeling, inference, and optimisation.
result Demonstrated end-to-end differentiable and inference-aware optimisation of particle physics instruments.
Muon optimizer outperforms GD in neural networks.
problem Optimizing matrix-structured parameters in neural networks.
method Muon optimizer specifically designed for matrix parameters, analyzing convergence rate and low-rank Hessian structure.
result Muon can outperform Gradient Descent due to its ability to leverage the low-rank structure of Hessian matrices.
Muon fails to converge on convex Lipschitz functions.
problem Understanding the convergence of Muon on convex and Lipschitz functions.
method Analyzing Muon's performance on convex and Lipschitz functions without error feedback.
result Muon does not converge on convex and Lipschitz functions, regardless of learning rate schedule.
Muon optimizer simplifies matrix optimization with spectral orthogonalization.
problem Matrix optimization challenges, especially with large condition numbers.
method Simplified Muon optimizer using spectral orthogonalization of gradients.
result Simplified Muon converges linearly with independent scalar sequences, outperforming gradient descent and Adam.
Muon outperforms GD in associative memory learning by balancing frequency components.
problem Training dynamics and scaling behavior of Muon in associative memory learning.
method Study of Muon in a linear associative memory model with softmax retrieval and hierarchical frequency spectrum over query-answer pairs.
result Muon achieves exponential speedup over GD in noiseless case and superior scaling efficiency in noisy case.
Muon optimizer improves deep learning with spectral norm constraints.
problem Improving optimization algorithms in deep learning.
method Theoretical analysis of Muon optimizer within the Lion- K \mathcal{K} K family. result Muon implicitly solves an optimization problem enforcing spectral norm constraints.
Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
problem Improving the convergence rate of Muon optimizer.
method Using Newton-Schulz steps for momentum orthogonalization, proving convergence rate and constant factor.
result Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
MiMuon optimizer improves generalization for large models by reducing generalization error.
problem Improving generalization of Muon optimizer for large models.
method Enhanced Muon optimizer using orthogonalization of gradient, proving lower generalization error.
result MiMuon optimizer has a lower generalization error of O ( 1 N ) O\big(\frac{1}{N}\big) O ( N 1 ) compared to Muon's O ( 1 N κ T ) O\big(\frac{1}{Nκ^{T}}\big) O ( N κ T 1 ) . EF21-Muon optimizes deep learning with error feedback, improving efficiency and accuracy.
problem Lack of principled distributed frameworks for non-Euclidean LMO-based optimizers.
method Introduces EF21-Muon, a communication-efficient, non-Euclidean LMO-based optimizer with convergence guarantees.
result First efficient distributed implementation of non-Euclidean LMO-based optimizers, achieving up to 7x communication savings.
Muon replaces matrix gradient with polar factor, optimizing flat spectrum updates
problem Optimization bias in matrix updates
method Using polar factor of gradient
result Muon update maximizes entropy among bounded updates
Develops a gradient flow for Muon optimizer, a method for optimization.
problem Optimization of complex systems with matrix-valued parameters.
method Gradient flow on probability measures induced by regularized Muon optimizer.
result Derives continuous-time limits and proves Hamiltonian dissipation.
Muon optimizes training efficiency by improving data retention at large batch sizes.
problem Improving training efficiency and data retention at large batch sizes.
method Introducing Muon, a second-order optimizer, and combining it with muP for efficient hyperparameter transfer.
result Muon outperforms AdamW in retaining data efficiency at large batch sizes, enabling more economical training.
New Muon and Momo variants improve neural network optimization robustness.
problem Improving neural network optimization methods.
method Systematic exploration of non-Euclidean gradient descent variants.
result Momo variants of Muon are more robust to hyperparameter tuning.
Muon optimizer improves federated learning performance.
problem Improving federated learning performance with Muon optimizer.
method Proposed FedMuon algorithm with orthonormalized update direction.
result FedMuon's learning rate is independent of problem-specific parameters.
Shampoo achieves higher token efficiency than Muon in language models.
problem Understanding the relationship and relative data efficiency of Shampoo and Muon compared to Adam and Signum.
method Extensive experiments on language models, demonstrating Shampoo's higher efficiency and decomposing its updates.
result Shampoo's benefits are attributed to its application to weight matrices, challenging interpretations based on variance adaptation and whitening.
New analysis of Muon and SignSGD on matrix-valued least squares problems.
problem Understanding the behavior of Muon and SignSGD on matrix-valued least squares problems.
method Derive explicit deterministic dynamics to study learning behavior of Muon and SignSGD.
result Muon and SignSGD exhibit different optimal learning rates and convergence characteristics based on batch size and data covariance.
Muon spectral optimizer outperforms SGD in associative memory tasks.
problem Understanding the advantage of spectral optimizers in learning associative memory.
method Linear associative memory problem, Gaussian inputs and outputs, power law frequency distribution, thresholded gradient approximation.
result Muon significantly outperforms SGD in storage capacity and recovery rates.
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
Unified view of Lion and Muon as Stochastic Frank-Wolfe methods.
problem Optimization of constrained problems in deep learning.
method Interpreting Lion and Muon as Stochastic Frank-Wolfe methods and extending the approach to heavy-tailed noise.
result Convergence guarantees and KKT point convergence for Lion and Muon.
Muon optimizes Transformer training with heavy-tailed data, achieving optimal sample complexity.
problem Theoretical understanding of non-Euclidean optimisation methods for heavy-tailed data in training Transformers.
method Addressing the gap in theoretical understanding, we show Muon achieves optimal sample complexity under heavy-tailed noise.
result Muon finds an ε-stationary point in nuclear norm with optimal sample complexity, absorbing heavy-tailed noise without dimension dependence.
ML PCA detects phase transitions in muon spectroscopy data.
problem Detecting phase transitions in materials using muon spectroscopy data.
method Unsupervised Machine Learning (PCA) applied to asymmetry functions.
result PCA method effectively detects phase transitions in muon spectroscopy experiments.
Muon dynamics study uses spectral Wasserstein flow for optimization stability.
problem Optimizing deep learning models with gradient normalization.
method Introduces Spectral Wasserstein distances for matrix flows, proving equivalence with Benamou--Brenier formulation.
result Gradient-flow interpretation of mean-field normalized training dynamics.
MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.
problem Training matrix-valued parameters with orthogonalized-update optimizers like Muon.
method MuonEq introduces three lightweight pre-orthogonalization equilibration schemes: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).
result Row/column normalization acts as a zeroth-order surrogate for whitening and improves the geometry seen by orthogonalization.
Muons and random optimizers perform similarly, challenging geometric optimization theory.
problem Empirical success of Muon optimizer challenges geometric optimization theory.
method Introducing Freon and Kaon optimizers, demonstrating performance without precise geometric structure.
result Performance of optimizers is controlled by alignment and descent potential, not geometric structure.
Drop-Muon updates only some layers, speeding up training.
problem Conventional deep learning optimizers update all layers at once, which can be inefficient.
method Drop-Muon updates only a subset of layers per step, with randomized schedules.
result Drop-Muon achieves up to 1.4x faster training time with similar accuracy.
New optimizer MARS-M combines variance reduction with Muon for faster LLM training.
problem Training large-scale neural networks efficiently.
method Integrates MARS variance reduction with Muon optimizer.
result MARS-M converges to a first-order stationary point at a rate of i l d e O ( T − 1 / 3 ) ilde{\mathcal{O}}(T^{-1/3}) i l d e O ( T − 1/3 ) . AuON is a linear-time optimizer that improves upon Muon's performance without approximate orthogonal matrices.
problem High memory and computational costs of orthogonal momentum updates.
method AuON uses normalized nonlinear scaling and a 'emergency brake' to handle exploding attention logits.
result AuON achieves strong performance without approximate orthogonal matrices, preserving structural alignment and reconditioning.
PolarGrad optimizes deep learning models by considering matrix structure, outperforming Adam and Muon.
problem Efficient optimization of large-scale neural networks and language models.
method A unifying framework for analyzing matrix-aware preconditioned methods, including PolarGrad.
result PolarGrad outperforms Adam and Muon in various tasks.
Muon optimizes deep learning models on imbalanced data by learning all components equally.
problem Generalization issues in deep learning models on imbalanced data.
method Introduced Spectral Gradient Descent (SpecGD) as a canonical form of Muon and Shampoo, and studied its performance on imbalanced data.
result SpecGD learns all principal components of imbalanced data at equal rates, unlike vanilla GD which prioritizes dominant components.
Improved tensor GLM estimation for complex data.
problem Complex tensor data in GLMs leads to high-dimensional, ill-posed estimation.
method Proposed LSRTR-M algorithm using Muon updates for faster convergence and lower errors.
result LSRTR-M converges faster and achieves lower errors than LSRTR.
A new Riemannian framework optimizes LoRA for faster convergence and better performance.
problem Optimizing low-rank adapters in neural networks to improve convergence and performance.
method Integrates Riemannion optimizer, LoRA initialization, and efficient implementation for geometrically treating low-rank adapters.
result Consistent and noticeable improvements in convergence speed and final task performance over standard LoRA and its modifications.
PCA improves detection of phase transitions in muon spectroscopy data from various materials.
problem Subtle changes in asymmetry function indicate phase transitions, but existing methods require material-specific knowledge.
method Applied unsupervised PCA to muon spectroscopy asymmetry data from multiple materials.
result PCA can recover phase transition indicators and improve detection of material-specific variations.
Study travel time tomography for transversely isotropic media using modified pseudodifferential calculus.
problem Travel time tomography problem for transversely isotropic media.
method Modified scattering pseudodifferential calculus to solve the tomography problem.
result Construction and use of modified pseudodifferential calculus to solve the tomography problem.
Summary of tensor tomography proofs on manifolds with boundaries.
problem Proving injectivity of tensor tomography on compact Riemannian manifolds with boundaries.
method Summarized proofs from previous studies.
result Summary of proofs for s-injectivity.
Paper addresses travel time tomography stability and statistical inversion.
problem Determining conformal factors of metrics from geodesic lengths.
method Established forward and inverse stability estimates; applied to Bayesian statistical inversion.
result Consistency of statistical inversion technique for travel time tomography.
Study uses machine learning to solve photoacoustic tomography's inverse problem.
problem Solving the full inverse problem in photoacoustic tomography.
method Developed an approach using variational autoencoders for Bayesian estimation of the posterior distribution.
result Evaluated the approach with numerical simulations and compared it to a Bayesian solution.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
These are lecture notes for the course "Analysis and X-ray tomography". The course is a broad overview of various tools in analysis that can be used to study X-ray tomography. The focus is on tools and ideas, not so much on technical details and minimal assumptions. Only very basic functional analysis is assumed as bac…
Sketch Tomography improves quantum state estimation accuracy.
problem Efficiently estimating quantum states, especially MPS states.
method Hybridizes classical shadow protocol with tensor train ansatz.
result Proven convergence with quadratic sample complexity.
Deep network improves electrical tomography across multiple frequencies.
problem Nonlinear multi-frequency electrical impedance tomography (mfEIT) for tissue conductivity estimation.
method Integrates graph neural networks (GNNs) into the iterative Proximal Regularized Gauss Newton (PRGN) framework to reconstruct tissue concentrations accurately.
result Accurate reconstruction of overlapping tissue fraction concentrations across multiple frequencies.
Gluon optimizes LMO-based methods for large-scale tasks, improving performance and theory-practice gap.
problem LMO-based methods lack theoretical support for practical implementation and smoothness assumptions.
method Introduces Gluon, a new LMO-based method with refined smoothness model.
result Gluon's theoretical stepsizes match fine-tuned values, closing the theory-practice gap.
Spectral gradient methods outperform Euclidean in certain deep learning scenarios.
problem When do spectral gradient updates outperform Euclidean in deep learning?
method Layerwise condition comparing squared nuclear-to-Frobenius ratio to stable rank of activations.
result Spectral updates can be more effective than Euclidean in deep networks and transformers.
A new method speeds up quantum state estimation.
problem Exponential growth in sample size and dimension for quantum state tomography.
method Stochastic mirror descent with Burg entropy.
result Optimization error vanishes at a O ( ( 1 / t ) d log t ) O (\sqrt{ ( 1 / t ) d \log t }) O ( ( 1/ t ) d log t ) rate. Improved computed tomography reconstruction with deep learning and deep image prior.
problem Low data efficiency in computed tomography reconstruction.
method Combining learned primal-dual methods with deep image prior for improved quality and generalization.
result Proposed methods outperform state-of-the-art in low data regime.
We develop a 2D travel time tomography method which regularizes the inversion by modeling groups of slowness pixels from discrete slowness maps, called patches, as sparse linear combinations of atoms from a dictionary. We propose to use dictionary learning during the inversion to adapt dictionaries to specific slowness…
New method uses MRI data to improve PET tomography uncertainty quantification.
problem Improving uncertainty quantification in emission tomography with multimodal data.
method Nonparametric posterior learning technique adapted for Poisson-type data.
result Sampling algorithms are scalable, parallelizable, and easy to implement.
Sharp stability estimate for tensor tomography in non-positive curvature.
problem Stability estimate for tensor tomography on manifolds with non-positive curvature.
method Pestov identity with localized frequency boundary term.
result Stability estimate of the form L 2 ↦ H T 1 / 2 L^2\mapsto H^{1/2}_{T} L 2 ↦ H T 1/2 . Study inverse problems for twisted geodesic flows on manifolds.
problem Understanding inverse problems for twisted geodesic flows.
method Generalized ray transforms and tensor tomography.
result New insights into rigidity problems for twisted geodesic flows.