RAME adapts learning rates using recent first moment of gradients.
problem Training deep neural networks efficiently and adaptively.
method RAME computes individual learning rates using the most recent first moment of gradients.
result RAME outperforms SHB, Adam, and RMSprop in convergence speed and generalization performance.
This thesis studies domain adaptation under minimal distribution similarity assumptions using moments.
problem Learning from samples with distributions different from training samples.
method Uses minimal similarity assumptions modeled by moments.
result Establishes learning bounds and algorithms for domain adaptation.
The paper improves generalization bounds for domain adaptation.
problem Improving generalization bounds for domain adaptation under practical conditions.
method Derives generalization bounds for domain adaptation based on finitely many moments and smoothness conditions.
result Obtains generalization bounds for domain adaptation.
Adaptive moment methods have been remarkably successful in deep learning optimization, particularly in the presence of noisy and/or sparse gradients. We further the advantages of adaptive moment techniques by proposing a family of double adaptive stochastic gradient methods~\textsc{DASGrad}. They leverage the complemen…
New method improves Bayesian cross-validation.
problem Finding good proposal distributions for importance sampling.
method Implicitly adaptive importance sampling that iteratively matches moments.
result Better than many existing parametric adaptive importance sampling methods.
NovoGrad improves deep learning training with adaptive moments and layer-wise normalization.
problem Training deep neural networks efficiently and effectively.
method Layer-wise adaptive moments with gradient normalization and decoupled weight decay.
result NovoGrad outperforms well-tuned SGD with momentum and Adam/AdamW in various tasks.
Enhanced Adam uses higher-order moments for better performance.
problem Improving the performance of Adam optimization algorithm.
method Proposes HAdam, an extension of Adam using higher-order moments of the stochastic gradient.
result Higher-order moments of the stochastic gradient can lead to better performance than vanilla Adam.
Adaptive t-distribution estimates nonstationary time series using moving moments.
problem Nonstationary time series with varying dependence structure.
method Moving estimator optimizing a weighted log-likelihood, using exponential moving averages for moments.
result Evolution of ν parameter in Student's t-distribution, capturing tail behavior and extreme events.
New method for adaptive estimation and inference in econometric models without knowing smoothness.
problem Adaptive estimation and inference in ill-posed linear inverse problems with unknown smoothness.
method Discrepancy principle-based framework for adaptive hyperparameter selection.
result Achieves optimal rates in weak and strong metrics for linear functionals.
Constructs a moment map flow for isotropic maps on surfaces.
problem Understanding isotropic maps on surfaces and their properties.
method Develops a Kähler moment map geometry and a modified moment map flow.
result Polyhedral modified moment map flow induces a strong deformation retraction.
We present a proof due to Duistermaat that the gradient flow of the norm squared of the moment map defines a deformation retract of the appropriate piece of the manifold onto the zero level set of the moment map. Duistermaat's proof is an adaptation of Lojasiewicz's argument for analytic functions to functions which ar…
New method AdaMod stabilizes deep neural network training by limiting adaptive learning rates.
problem Adaptive learning rates can produce extremely large values at the start of training, hindering learning.
method AdaMod uses adaptive and momental upper bounds to restrict learning rates dynamically.
result AdaMod eliminates large learning rates and improves training on complex networks.
Estimates RL data for dynamic treatment effects using GMM.
problem Estimating dynamic treatment effects from RL data with nonstationary behavior policies.
method Weighted GMM approach to stabilize variance in adaptive RL settings.
result Valid hypothesis testing and confidence regions for dynamic treatment effects.
Unified framework for distribution shift estimation, explanation, and improvement.
problem Estimating, explaining, and improving model performance on target domains with distribution shift.
method Entropic Projection Alignment (EPA) aligns source and target distributions by matching moments and minimizing KL divergence.
result EPA consistently outperforms state-of-the-art baselines while offering computational efficiency.
Python package ajdmom simplifies moment formula derivation for jump diffusions.
problem Deriving moment formulae for complex jump diffusion processes.
method Automatically generates closed-form expressions and derivatives for any order of moments.
result Enhances usability and usability of affine jump diffusion models.
New adaptive stepsize method for stochastic approximation converges to target point.
problem Finding optimal step sizes for stochastic approximation algorithms.
method Adaptive block-coordinate stepsizes using online estimates of second moment.
result New method converges almost surely to a small neighborhood of the target point.
We introduce a notion of moment map adapted to actions of Lie groups that preserve a closed three-form. We show existence of our multi-moment maps in many circumstances, including mild topological assumptions on the underlying manifold. Such maps are also shown to exist for all groups whose second and third Lie algebra…
ELSA efficiently adapts to label shift without post-prediction calibrations.
problem Domain adaptation with label shift across training and testing datasets.
method Moment-matching framework based on influence function geometry; solves linear systems for adaptation weights.
result ELSA estimator is n \sqrt{n} n -consistent and asymptotically normal, achieving state-of-the-art estimation performance. A new adaptive splitting method improves accuracy for Cox-Ingersoll-Ross model.
problem Improving numerical solution accuracy for Cox-Ingersoll-Ross model.
method Adaptive splitting method over deterministic and random meshes, with uniform moment bound and strong error results.
result Uniform moment bound and strong error results of order 1/4 in L1 and L2 for κθ>σ^2, and order 1 for large noise.
New flow connects symplectic maps to hyperKähler geometry.
problem Understanding symplectic maps and their geometry.
method Established a correspondence between symplectic diffeomorphisms and hyperKähler moment maps.
result Introduced a new flow, the modified moment map flow.
New optimizer Eve uses examplewise gradients for better second-moment estimates.
problem Improving optimization methods for machine learning.
method Adaptive optimization with examplewise gradients.
result Eve optimizer slightly outperforms Adam on small scale benchmarks.
The learning of domain-invariant representations in the context of domain adaptation with neural networks is considered. We propose a new regularization method that minimizes the discrepancy between domain-specific latent feature representations directly in the hidden activation space. Although some standard distributi…
This paper analyzes MaskGIT sampler and introduces a moment sampler for faster masked diffusion sampling.
problem Efficiently sampling from masked diffusion models.
method Theoretical analysis of MaskGIT sampler, introduction of moment sampler, and two innovations for improving choose-then-sample efficiency.
result The moment sampler is an asymptotically equivalent, more interpretable alternative to MaskGIT.
Proposes DWMD for better matching of hidden representations across domains.
problem Measuring data distribution discrepancy between semantically related domains for feature representation matching.
method DWMD, a moment-based probability distribution metric that explicitly orders and weights higher-order moments.
result DWMD is error-free and can strictly reflect distribution differences without feature distribution assumptions.
Synthesizes robust estimators for domain adaptation.
problem Improving prediction accuracy in target domain with limited data.
method Synthesizes a family of robust least squares estimators using convex optimization.
result Robust strategies can outperform non-robust interpolations.
Learning rate needs to decrease with higher data moments for effective ICA in high dimensions.
problem Slower convergence of ICA in high-dimensional data with high-order moments.
method High-dimensional ODE analysis of ICA algorithm under controlled moment structure.
result Critical learning rate threshold for effective ICA when moments are high.
Deform moment map on symplectic connections using star product algebras.
problem Understanding symplectic connections and their deformations.
method Study vector bundle of Fedosov star product algebras, formal connection, curvature, and star product trace.
result Showed star product trace as a formal symplectic form and moment map.
The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.
problem Analyzing Adam and SGD in nonstationary optimization problems.
method Theoretical analysis of Adam and SGD under non-stationary stochastic objectives, separating two regimes.
result Characterizes the tradeoff between noise and drift in Adam and SGD, revealing when adaptive step-sizing is beneficial or harmful.
In this paper we investigate the adaptive market efficiency of the agricultural commodity futures market, using a sample of eight futures contracts. Using a battery of nonlinear tests, we uncover the nonlinear serial dependence in the returns series. We run the Hinich portmanteau bicorrelation test to uncover the momen…
Adaptive gradient-based optimization methods such as \textsc{Adagrad}, \textsc{Rmsprop}, and \textsc{Adam} are widely used in solving large-scale machine learning problems including deep learning. A number of schemes have been proposed in the literature aiming at parallelizing them, based on communications of periphera…
Huber regression assessed for robustness in statistical learning.
problem Understanding Huber regression in nonparametric statistical learning.
method Assessment from statistical learning perspective, focusing on risk consistency, adaptive tuning, and convergence rates.
result Huber regression can be asymptotically mean regression calibrated under ( 1 + ε ) (1+ε) ( 1 + ε ) -moment conditions, justifying its robustness. Let Ξ \,Ξ\, Ξ be the crown domain associated with a non-compact irreducible hermitian symmetric space G / K \,G/K G / K . We give an explicit description of the unique G \,G G -invariant adapted hyper-Kähler structure on Ξ \,Ξ Ξ , \ i. \, e. \ compatible with the adapted complex structure J a d \,J_{ad}\, J a d and with the G \,G G -invariant Kähle…
Optimal ability estimation in adaptive testing with binary responses.
problem Estimating a continuous ability parameter from sequential binary responses.
method Adaptive selection of questions to maximize Fisher information, updating estimate using method-of-moments, and deciding accuracy with a test statistic.
result Fisher-tracking strategy achieves optimal performance in fixed-confidence and fixed-budget regimes.
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For …
Researchers compute the index of a specific operator on contact manifolds.
problem Computing the index of a twisted Dolbeault operator on toric contact manifolds.
method Using equivariant techniques, they localized the symbol to Reeb orbits and applied polytope decomposition.
result They derived an Atiyah-Bott-Lefschetz type formula for the index.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
Adaptive estimation of alpha-Stable distribution and Hurst exponent for nonstationary time series.
problem Nonstationary time series require adaptive models to avoid bias.
method Moving estimator with exponentially weakening weights of old values, optimized using EMA of absolute central moments.
result Continuous adaptive estimation of alpha-Stable distribution and Hurst exponent for market stability evaluation.
New protocols show 1-bit mean estimation can be order-optimal without interaction.
problem Can 1-bit mean estimation be optimal without interaction?
method Adaptive and non-adaptive threshold and interval queries, with one adaptive transition.
result Arbitrary non-adaptive quantizers can match the adaptive rate, suggesting interaction is not necessary.
AGMMNs improve learning of copula models by adaptively selecting kernels.
problem Learning dependence structures in copula models.
method Adaptive bandwidth selection for MMD in GMMNs, increasing kernels based on validation loss.
result AGMMNs significantly improve training performance over GMMNs and parametric models.
A novel approach for unsupervised domain adaptation for neural networks is proposed. It relies on metric-based regularization of the learning process. The metric-based regularization aims at domain-invariant latent feature representations by means of maximizing the similarity between domain-specific activation distribu…
Paper proposes ClipSMT algorithm for better ATE estimation.
problem Adaptive estimation of Average Treatment Effect (ATE).
method ClipSMT algorithm for improved Neyman regret.
result Achieves exponential improvements in Neyman regret.
The paper develops a mathematical model for strategic shifts.
problem Finding optimal moments for strategy changes in market dynamics.
method Explicit strategy formulation using fluctuation theory.
result Analytical results predict optimal strategy shifts.
New algorithms avoid a dominant lower-order term in heavy-tailed loss settings.
problem Prediction with heavy-tailed losses without prior knowledge.
method Adaptive algorithms that avoid the maximum of losses as a lower-order term in regret.
result Improved regret bounds of O ( θ T log ( K ) ) \mathcal{O}(\sqrt{θT\log(K)}) O ( θ T log ( K ) ) and O ( θ log ( K T ) / Δ min ) \mathcal{O}(θ\log(KT)/Δ_{\min}) O ( θ log ( K T ) / Δ m i n ) . Adaptive algorithm AMSGrad converges for weakly convex constrained optimization problems.
problem Solving constrained stochastic optimization problems with weakly convex objectives.
method Analysis of AMSGrad algorithm for a specific class of problems.
result AMSGrad achieves a convergence rate of i l d e O ( t − 1 / 4 ) \mathcal{ ilde O}(t^{-1/4}) i l d e O ( t − 1/4 ) for the norm of the gradient of the Moreau envelope. Many scientific and engineering challenges -- ranging from pharmacokinetic drug dosage allocation and personalized medicine to marketing mix (4Ps) recommendations -- require an understanding of the unobserved heterogeneity in order to develop the best decision making-processes. In this paper, we develop a hypothesis te…
Second-order optimization speeds up deep hedging for complex options.
problem Hedging exotic options with market frictions in realistic markets.
method Second-order optimization scheme leveraging pathwise differentiability and Kronecker-factoring.
result Our method optimizes the policy in 1/4 the steps of standard optimization.
A new method for density estimation using mixture discrepancy and moments.
problem Generalizing histogram statistics to higher dimensions.
method Density estimation via mixture discrepancy and moments (DSP-mix and MSP).
result DSP-mix and MSP are computationally tractable and maintain accuracy with increased speed.
In this work, we study the problem of aggregating a finite number of predictors for nonstationary sub-linear processes. We provide oracle inequalities relying essentially on three ingredients: (1) a uniform bound of the ℓ 1 \ell^1 ℓ 1 norm of the time varying sub-linear coefficients, (2) a Lipschitz assumption on the predict…