Paper proposes ClipSMT algorithm for better ATE estimation.
problem Adaptive estimation of Average Treatment Effect (ATE).
method ClipSMT algorithm for improved Neyman regret.
result Achieves exponential improvements in Neyman regret.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “clipped second moment tracking”
In this paper, we consider the problem of prediction with expert advice in dynamic environments. We choose tracking regret as the performance metric and develop two adaptive and efficient algorithms with data-dependent tracking regret bounds. The first algorithm achieves a second-order tracking regret bound, which impr…
The study of random walks on hyperbolic spaces and Teichmüller spaces, proving central limit theorems and geodesic tracking.
problem Analyzing random walks on hyperbolic and Teichmüller spaces.
method Proving central limit theorems and geodesic tracking using finite moments and logarithmic moments.
result Translation lengths of random isometries satisfy a central limit theorem if and only if the random walk has finite second moment.
VR-GHAL method solves stochastic fixed-point equations with high probability.
problem Solving stochastic fixed-point equations in normed spaces with nonexpansive or contractive operators.
method VR-GHAL, a variance-reduced gradual Halpern method for quadratically smoothable Banach spaces, using clipped stochastic differences.
result The method achieves a high-probability residual bound, reducing the residual nearly geometrically across epochs.
Improved Clipped-SGD achieves near-optimal heavy-tailed statistical estimation in streaming settings.
problem High-dimensional heavy-tailed statistical estimation in streaming with memory constraints.
method Stochastic convex optimization with Clipped-SGD, proving near-optimal sub-Gaussian statistical rates.
result Clipped-SGD achieves an error of with probability .
Improved analysis for clipped gradient methods in nonsmooth convex optimization under heavy-tailed noise.
problem Optimization under heavy-tailed noise in nonsmooth convex problems.
method Refined analysis of Clipped Stochastic Gradient Descent (Clipped SGD) with new rates and improved utilization of Freedman's inequality.
result New rates and for nonsmooth convex and strongly convex problems, respectively.
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For …
New method makes reinforcement learning robust to heavy-tailed rewards.
problem Heavy-tailed rewards cause statistical outliers in reinforcement learning.
method Dynamic gradient clipping in TD learning and NAC.
result Provably robust TD and NAC achieve optimal sample complexities.
New algorithm detects changes in heavy-tailed data streams.
problem Detecting changes in heavy-tailed data streams.
method Clipped Stochastic Gradient Descent (SGD) combined with union bound.
result First algorithm with finite-sample false-positive rate guarantees for heavy-tailed data.
New method for estimating covariance with robustness to outliers.
problem Estimating covariance from noisy data with outliers.
method Cross-fitted clipped covariance estimator with computable Bernstein certificates.
result The method balances certified stochastic error and robust hold-out proxy for clipping bias.
Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FL) settings involve bounding the contribution of each user's model update by clipping it to some constant value. However there is no good a priori setting of the clipping norm acr…
Adaptive gradient methods such as Adam have been shown to be very effective for training deep neural networks (DNNs) by tracking the second moment of gradients to compute the individual learning rates. Differently from existing methods, we make use of the most recent first moment of gradients to compute the individual …
FSD50K provides an open dataset of over 51k audio clips for sound event recognition.
problem Small and domain-specific sound event recognition datasets.
method Creation of an open dataset with over 51k audio clips manually labeled using 200 classes.
result FSD50K is a new open benchmark for sound event recognition research.
Paper introduces TAP-Vid, a benchmark for tracking any point in videos.
problem Tackles the problem of tracking arbitrary physical points on surfaces over longer video clips.
method Formalizes the problem as TAP, introduces TAP-Vid benchmark, uses crowdsourced pipeline with optical flow estimates, proposes TAP-Net model.
result TAP-Net outperforms all prior methods on TAP-Vid benchmark when trained on synthetic data.
Fisher GANcs.LG
Generative Adversarial Networks (GANs) are powerful models for learning complex distributions. Stable training of GANs has been addressed in many recent works which explore different metrics between distributions. In this paper we introduce Fisher GAN which fits within the Integral Probability Metrics (IPM) framework f…
Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
problem Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
method Analyzing the behavior of memory-efficient optimizers like GaLore, which project gradients onto a rank-r subspace recomputed every T steps.
result Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.
problem Analyzing Adam and SGD in nonstationary optimization problems.
method Theoretical analysis of Adam and SGD under non-stationary stochastic objectives, separating two regimes.
result Characterizes the tradeoff between noise and drift in Adam and SGD, revealing when adaptive step-sizing is beneficial or harmful.
Paper explores whether gradient normalization can replace clipping for SGD in heavy-tailed noise.
problem Ensuring convergence of SGD in heavy-tailed noise.
method Revisits gradient clipping and normalization, proving their sufficiency and effectiveness.
result Gradient normalization alone is sufficient for nonconvex SGD convergence under smoothness assumptions.
New method for differentially private optimization with general Lipschitz conditions.
problem Differentially private optimization under general Lipschitz conditions.
method Generalized Lipschitz condition for per-sample gradients, tuning clip norm based on minimum per-sample Lipschitz constant.
result Efficacy of the recommended clip norm tuning method verified on 8 datasets.
New theory explains why normalization is preferred in SGD under heavy-tailed noise.
problem Understanding why normalization is preferred in stochastic gradient descent (SGD) under heavy-tailed noise.
method Developed a worst-case complexity theory for stochastically preconditioned SGD and its variants.
result Normalization guarantees convergence at optimal rates, while clipping may fail in the worst case.
Nonlinear SGD achieves high-probability rates in non-convex optimization with heavy-tailed noise.
problem Optimization in non-convex problems with heavy-tailed noise.
method General nonlinear framework for SGD, including symmetrization techniques.
result Achieves rate for heavy-tailed noise.
New method improves convergence of SGD for heavy-tailed noise.
problem Heavy-tailed noise in machine learning applications.
method Normalized SGD (NSGD) to handle heavy-tailed noise.
result Improved sample complexity and high-probability convergence.
New convergence bounds for online learning with heavy-tailed noise.
problem Learning on streaming data with heavy-tailed noise.
method Nonlinear stochastic gradient descent (SGD) for non-convex and strongly convex costs.
result Strong convergence rates for various nonlinearities and noise distributions.
Recently, the study of heavy-tailed noises in first-order nonconvex stochastic optimization has gotten a lot of attention since it was recognized as a more realistic condition as suggested by many empirical observations. Specifically, the stochastic noise (the difference between the stochastic and true gradient) is con…
U-Clip clips gradients to reduce bias without discarding them.
problem Reduces bias in stochastic gradient descent.
method Maintains a buffer of clipped gradients to add to next iteration.
result U-Clip updates are unbiased on average.
Rod flow models Adam's behavior at the edge of stability.
problem Modeling adaptive gradient methods like Adam at the edge of stability.
method Extended rod flow to Adam, considering parameters, first moment, and second moment as variables.
result Rod flow accurately tracks Adam's behavior through the edge-of-stability regime.
Paper introduces a method to create robust representations against covariate shifts.
problem Distribution shift between training and testing data in machine learning.
method Introduces a variational objective with two components: discriminative representation and invariant support.
result Optimal representations ensure robustness to covariate shifts, improving performance on DomainBed.
This paper presents GRASTA (Grassmannian Robust Adaptive Subspace Tracking Algorithm), an efficient and robust online algorithm for tracking subspaces from highly incomplete information. The algorithm uses a robust -norm cost function in order to estimate and track non-stationary subspaces when the streaming data …
Driving datasets accelerate the development of intelligent driving and related computer vision technologies, while substantial and detailed annotations serve as fuels and powers to boost the efficacy of such datasets to improve learning-based models. We propose D-City, a large-scale comprehensive collection of dash…
Improved differentially private deep learning with group-wise clipping techniques.
problem Efficiency and privacy trade-offs in deep learning models.
method Group-wise clipping techniques (per-layer and per-device) to reduce compute time and memory overhead.
result Private learning with group-wise clipping achieves similar or better performance than non-private learning with less wall time.
Study on gradient clipping in SGD for high-dimensional problems.
problem Understanding and optimizing gradient clipping in high-dimensional machine learning models.
method Theoretical analysis of streaming SGD with gradient clipping in a least squares problem, focusing on large intrinsic dimensionality.
result Developed a deterministic equation to describe the loss evolution in clipped SGD, showing benefits under certain conditions.
JME continually estimates data moments privately and accurately.
problem Private and accurate continual estimation of data moments.
method Uses matrix mechanism and joint sensitivity analysis.
result Improves accuracy in estimating mean and covariance with reduced noise.
PMT uses public data moments to make DP feasible for unbounded data.
problem Applying differential privacy to unbounded data distributions.
method Public-moment-guided Truncation (PMT) using second-moments from public data.
result PMT improves the accuracy and stability of DP models.
Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clippingcs.LG
New method improves fairness in DP learning by preventing excessive gradient suppression.
problem Disparate impact on model predictions for minority groups in DP learning.
method Bounded adaptive clipping to prevent excessive gradient suppression.
result Improves worst-class accuracy by over 10 percentage points compared to existing methods.
This paper analyzes convergence of DP-SGD with adaptive quantile clipping.
problem Empirical success of adaptive clipping methods lacks theoretical understanding.
method Comprehensive convergence analysis of SGD with quantile clipping (QC-SGD).
result Establishes theoretical guarantees for DP-QC-SGD, revealing relationships between quantile selection, step size, and convergence.
A new method improves training speed and robustness for deep learning models.
problem Training deep learning models with heavy-tailed gradient noise.
method Per-sample gradient clipping combined with SGD.
result PS-Clip-SGD achieves optimal convergence rates and high-probability guarantees.
Enhances DP linear regression using public data moments.
problem Limited utility of traditional DP methods in linear regression.
method Transform private data using public second-moment matrix for a better OLSE.
result Improved accuracy and robustness of OLSE in DP linear regression.
New algorithm finds critical points in non-convex optimization with heavy-tailed gradients.
problem Non-convex stochastic optimization with heavy-tailed gradient estimates.
method Gradient clipping, momentum, and normalized gradient descent.
result High-probability convergence to critical points with best-known rates.
Paper improves Fisher information estimation methods.
problem Estimating Fisher information for location parameters.
method Revisits and improves Bhattacharya estimator, introduces clipped estimator.
result Clipped estimator shows superior convergence rates in Gaussian noise.
CLIP models robustness to spurious features is re-evaluated using a new dataset.
problem Existing robustness tests of CLIP models may not fully reflect their performance on spurious features.
method Crafted a new dataset (CounterAnimal) to reveal CLIP models' reliance on realistic spurious features.
result CLIP models are robust to spurious features learned from their training data, not ImageNet.
CLIP learns joint image-text representations for zero-shot learning.
problem Understanding and improving zero-shot transfer performance in CLIP.
method Formal study of transferrable representation learning and analysis of zero-shot transfer performance.
result Proposes a new CLIP-type approach that outperforms existing methods.
We consider two stage estimation with a non-parametric first stage and a generalized method of moments second stage, in a simpler setting than (Chernozhukov et al. 2016). We give an alternative proof of the theorem given in (Chernozhukov et al. 2016) that orthogonal second stage moments, sample splitting and -…
Paper introduces Latent-CLIP for efficient text-image comparison in latent space.
problem Efficiently compare text and images in latent space without costly decoding.
method Trains CLIP model in latent space, uses Latent-CLIP rewards for noise optimization, and guides generation away from harmful content.
result Latent-CLIP matches CLIP performance on text-image classification and harmful content detection.
Gradient clipping helps private SGD converge despite potential bias.
problem Gradient clipping in private SGD can bias convergence.
method Theoretical analysis and empirical evaluation of gradient clipping effects.
result Gradient clipping can prevent convergence to stationary points and introduces bias.
A new memory-efficient Adam variant reduces second moments when feasible.
problem Memory constraints in training machine learning models.
method Signal-to-Noise Ratio (SNR) analysis to identify dimensions where second moments can be replaced by means.
result Memory-efficient Adam variant (SlimAdam) matches performance and stability of Adam while saving up to 98% of second moments.
This paper improves analysis of clipping algorithms for non-convex optimization.
problem Improving convergence analysis of clipping algorithms for non-convex optimization.
method Develops a general framework to study clipping algorithms, including momentum methods, and provides convergence analysis in deterministic and stochastic settings.
result Demonstrates that clipping methods can maintain efficiency even in highly non-smooth regions.
Enhances CLIP's similarity computation using PMI's linear structure.
problem CLIP's similarity computation misses the optimal linear structure of PMI.
method KME-CLIP, utilizing inner product in a reproducing kernel Hilbert space.
result KME-CLIP approximates PMI with arbitrary accuracy and outperforms CLIP.
Sketchy reduces memory and compute requirements for adaptive regularization in deep learning.
problem Prohibitive memory and running time for adaptive regularization methods in deep learning.
method Low-rank sketching approach using Frequent Directions (FD) to reduce memory and compute requirements.
result Efficient interpolation between resource requirements and degradation in regret guarantees with rank .