In statistical learning theory, convex surrogates of the 0-1 loss are highly preferred because of the computational and theoretical virtues that convexity brings in. This is of more importance if we consider smooth surrogates as witnessed by the fact that the smoothness is further beneficial both computationally- by at…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Excess-Risk”
The paper explores the information-theoretic nature of excess risk in machine learning.
problem Understanding the excess risk in machine learning models.
method Formulates the minimax excess risk as a zero-sum game and modifies it to allow swapping of the order of play.
result Proves that under certain conditions, the duality gap is zero, allowing for the application of Bayesian results to provide bounds on minimax excess risk.
A Geometric Analysis of PCAmath.ST
The paper analyzes the excess risk of PCA and provides a precise characterization.
problem Understanding the excess risk of principal component analysis (PCA).
method Established a central limit theorem for PCA error and derived the excess risk distribution.
result Obtained a non-asymptotic upper bound on the excess risk of PCA.
New tool detects 'fleeting modes' causing excess risk in financial markets.
problem Detecting portfolios with statistically significant excess risk in financial markets.
method Random Matrix Theory to identify 'fleeting modes' independent of underlying correlation structure.
result Fleeting modes exist in both futures and equity markets, and momentum is a source of excess risk.
Study excess risk in statistical inference with transformations.
problem Excess risk in estimating random variables from feature vectors and transformations.
method Characterize lossless transformations, develop test statistics, and information-theoretic bounds.
result Strongly consistent partitioning test statistic for lossless transformations.
New method estimates model risk without knowing function class.
problem Evaluating model risk for complex, opaque models.
method Wild refitting with Bregman losses and randomized symmetrization.
result Valid upper bound on excess risk for opaque models.
The paper bounds the excess risk of deep neural networks for weakly dependent processes.
problem Learning with weakly dependent data using deep neural networks.
method Approximation of smooth functions by deep neural networks and a bound on excess risk.
result The excess risk bound for deep learning under weak dependence is close to for sufficiently smooth functions.
Study non-asymptotic bounds for robust estimators under misspecified models.
problem Evaluate performance of robust estimators under adversarial conditions.
method Propose a general approach to adversarial risk analysis, including investigations on generalization and approximation errors.
result Establish non-asymptotic upper bounds for adversarial excess risk under Lipschitz loss functions.
Non-Euclidean Differentially Private Stochastic Convex Optimization: Optimal Rates in Linear Timecs.LG
New algorithm achieves optimal privacy and efficiency in non-Euclidean convex optimization.
problem Optimizing convex functions while maintaining privacy in non-Euclidean settings.
method Developed a linear-time algorithm for -setups, leveraging geometric properties.
result Optimal excess risk achieved in linear time for .
This paper analyzes multi-pass SGD for least squares, improving generalization bounds.
problem Improving generalization bounds for multi-pass SGD in the least squares problem.
method Develops an instance-dependent excess risk bound for least squares in the interpolation regime.
result SGD performs worse than GD instance-wise but saves computational time.
New method for unbiased regression reduces excess risk.
problem Least squares regression with optimal solution and Hessian matrix.
method Averaged stochastic gradient descent with time-average estimator.
result Unbiased estimator with O(1/k) expected excess risk.
Paper bounds excess risk in robust empirical risk minimization for heavy-tailed distributions.
problem Risk bounds for robust empirical risk minimization in heavy-tailed distributions.
method Proposes robust proxies for expectation to bound excess risk.
result Excess risk of robust estimators can converge to 0 at fast rates.
The paper analyzes the generalization performance of spectral clustering algorithms and proposes new methods to improve their effectiveness.
problem Theoretical analysis of spectral clustering's generalization performance.
method Theoretical analysis and development of new spectral clustering algorithms.
result The excess risk bounds of spectral clustering algorithms have a convergence rate.
Lower bounds show OLS outperforms basis pursuit in overparameterized linear regression.
problem Excess risk of sparse interpolating procedures in overparameterized linear regression.
method Proved lower bounds on excess risk for OLS and basis pursuit.
result Excess risk of basis pursuit can converge at an exponentially slower rate than OLS.
Full-batch GD achieves generalization close to any stationary point with fewer assumptions.
problem Generalization and excess risk bounds for smooth losses, including non-Lipschitz and nonconvex cases.
method Path-dependent analysis of GD's generalization error, focusing on optimization error and stability.
result Generalization error is tightly bound in terms of optimization error and iteration count, bypassing common assumptions.
New method refines model-free evaluation of complex machine learning models.
problem Evaluating the excess risk of opaque machine learning predictors.
method Perturbing derivatives to create pseudo-outcomes and refitting the model twice.
result Upper bound on excess risk derived efficiently without prior function class knowledge.
Paper improves privacy in SGD with low noise, achieving optimal risk rates.
problem Privacy-preserving machine learning with good performance.
method Differentially private SGD with low-noise analysis.
result Achieves optimal excess risk rates for non-smooth losses.
Gibbs-ERM learning is a natural idealized model of learning with stochastic optimization algorithms (such as Stochastic Gradient Langevin Dynamics and ---to some extent--- Stochastic Gradient Descent), while it also arises in other contexts, including PAC-Bayesian theory, and sampling mechanisms. In this work we study …
SMP estimator improves density and logistic regression under misspecification.
problem Improper estimator for optimal excess risk in misspecified models.
method SMP minimizes a new excess risk bound for statistical learning.
result SMP achieves optimal excess risk of for logistic regression.
We give improved constants for data dependent and variance sensitive confidence bounds, called empirical Bernstein bounds, and extend these inequalities to hold uniformly over classes of functionswhose growth function is polynomial in the sample size n. The bounds lead us to consider sample variance penalization, a nov…
Optimizes differentially private kernel learning with random projection.
problem Privacy-preserving learning algorithms with optimal performance.
method Differentially private kernel ERM algorithm based on random projection in reproducing kernel Hilbert space.
result Achieves minimax-optimal excess risk rates for various loss functions.
This research improves PAC-Bayesian bounds for classification tasks using convexified loss.
problem Deriving generalization bounds for classification tasks with non-convex loss functions.
method Shift focus to misclassification excess risk bounds for PAC-Bayesian classification using convex surrogate loss and leveraging PAC-Bayesian relative bounds in expectation.
result Improved PAC-Bayesian bounds for classification tasks with convex surrogate loss.
Concentration and excess risk bounds for imbalanced classification with synthetic oversamplingstat.ML
The paper analyzes SMOTE for imbalanced classification, providing theoretical bounds and guidelines.
problem The challenge of imbalanced classification problems, especially with minority classes.
method Theoretical analysis of SMOTE and related oversampling techniques for minority classes.
result Derives concentration and excess risk bounds for SMOTE and kernel-based classifiers.
Paper proposes ZO-SMD for MERO, achieving optimal convergence rates.
problem Minimizing excess risk across all test distributions.
method Zeroth-order stochastic mirror descent algorithm for both smooth and non-smooth MERO.
result Converges at optimal rates of for estimates and optimization errors.
Data-driven method for error estimation without needing class complexity.
problem Constructing confidence intervals for a class of estimates.
method Data-driven approach to derive high-probability upper bounds on maximum error.
result Method naturally adapts to unknown correlation structures and works for finite and infinite classes.
Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.
problem Optimal risk in bounded distributions for constrained least squares.
method Comparison of least squares and non-linear predictors.
result Non-linear predictors can achieve optimal risk in bounded distributions.
Trace norm regularization is a popular method of multitask learning. We give excess risk bounds with explicit dependence on the number of tasks, the number of examples per task and properties of the data distribution. The bounds are independent of the dimension of the input space, which may be infinite as in the case o…
The paper analyzes the performance of empirical risk minimization for -norm linear regression.
problem Empirical risk minimization on -norm linear regression.
method Analyzes performance under various conditions and moment assumptions.
result High probability excess risk bounds for empirical risk minimizer, matching asymptotic rates.
New DP algorithm improves privacy and efficiency for convex optimization.
problem Efficient, DP algorithms for convex optimization with strong excess risk bounds.
method Output perturbation for a broad class of tilted loss functions.
result Near optimal DP excess risk and runtime bounds for convex optimization.
Fine-tuning with pre-training data improves performance.
problem Limited training data for tasks.
method Theoretical analysis of excess risk bound and selection of pre-training data subset.
result Improvement in generalization performance with pre-training data.
Improved algorithm reduces excess risk in selective learning.
problem Selective learning with windowed model selection.
method Hybrid Exponential Weights Algorithm and bounded-recall ERM.
result Achieves expected excess risk of O((log log |L| + log log n) / log n).
The paper explores how linear neural networks can overfit without bias when data is well-behaved.
problem Understanding why linear neural networks can generalize well despite fitting noisy data.
method Analyzing two-layer linear neural networks trained with gradient flow, deriving bounds on excess risk.
result The excess risk depends on initialization quality and data covariance matrix properties.
The overarching goal of this paper is to derive excess risk bounds for learning from exp-concave loss functions in passive and sequential learning settings. Exp-concave loss functions encompass several fundamental problems in machine learning such as squared loss in linear regression, logistic loss in classification, a…
Deep linear networks can closely approximate interpolants without improving risk.
problem Understanding the risk bounds of deep linear networks compared to minimum -norm solutions.
method Bounding excess risk of interpolating deep linear networks trained using gradient flow.
result Deep linear networks can closely approximate or match minimum -norm solutions in terms of risk.
Repeated self-distillation improves model performance significantly.
problem How much gain is possible by applying multiple steps of self-distillation?
method Investigated linear regression tasks, applied multiple steps of self-distillation, analyzed excess risk reduction.
result Multi-step self-distillation reduces excess risk by a factor as large as , where is the input dimension.
Privacy-preserving binary classification using locally differential private data.
problem Classifying data while protecting individual privacy.
method Locally differential private mechanism followed by a universally consistent classifier.
result Minimax rates of convergence are slower when using private data.
Paper provides optimal statistical guarantees for adversarial robustness in Gaussian classification.
problem Understanding statistical risks for adversarial robustness in Gaussian classification models.
method Established minimax lower bounds and designed an efficient estimator for excess risk.
result Optimal minimax guarantees for excess risk under Gaussian mixture model with AdvSNR.
This paper analyzes neural network classifiers' performance in binary classification.
problem Performance of neural network classifiers in binary classification problems.
method Plug-in classifiers based on neural networks, considering a more general function class and surrogate loss.
result Dimension-free, uniform rate of convergence for the excess risk of neural networks, showing minimax optimality.
We consider a standard binary classification problem. The performance of any binary classifier based on the training data is characterized by the excess risk. We study Bahadur's type exponential bounds on the minimax accuracy confidence function based on the excess risk. We study how this quantity depends on the comple…
Study uses spectral risk for learning with heavy-tailed data.
problem Learning with heavy-tailed loss distributions.
method Spectral risk with Lipschitz-continuous density, derivative-free learning.
result Excess risk guarantees and improved performance over traditional methods.
This paper examines error bounds for deep learning classifiers with noisy labels.
problem Understanding the performance of classifiers trained on noisy data.
method Derives error bounds for excess risk, decomposing it into statistical and approximation errors. Uses independent block construction for statistical dependencies and vector-valued setting for approximation error.
result Established theoretical results for error bounds in deep learning with noisy labels, mitigating the impact of high-dimensional input spaces.
The paper analyzes the generalization of deep neural networks for metric and similarity learning.
problem Lack of rigorous understanding of generalization performance in metric and similarity learning.
method Derive explicit form of true metric, construct structured deep ReLU neural network, establish excess risk bounds.
result Explicit excess risk bounds for metric and similarity learning are derived.
Paper proposes a new framework to improve stability-based bounds in deep learning.
problem Explaining generalization in overparameterized neural networks.
method Decomposes excess risk dynamics into signal and noise components, applying stability-based bounds only to the noise.
result The decomposition framework improves stability-based bounds and explains generalization in neural networks.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
Study shows how SGD's implicit regularization relates to ridge regression.
problem Least squares regression optimization with mini-batch SGD.
method Analyzes stochastic gradient flow as a continuous-time model of SGD.
result Bound on excess risk of SGD flow over ridge regression, revealing how parameters drive risk.
The paper provides theoretical guarantees for neural network-based anomaly detection.
problem Theoretical guarantees for unsupervised neural network-based anomaly detection.
method Casting anomaly detection as a binary classification problem, establishing non-asymptotic upper bounds and convergence rates.
result The convergence rate on the excess risk matches the minimax optimal rate.
Improved DP SO with large Lipschitz parameters, handling outliers and heavy-tailed data.
problem Differential privacy in stochastic optimization with large Lipschitz parameters.
method Assumes bounded k-th order moments, provides linear-time algorithms for smooth convex and non-smooth convex losses.
result Improved risk bounds scaling with k-th moment, not uniform Lipschitz parameter.
Improves statistical learning bounds with self-concordant losses.
problem Statistical prediction with nuisance components.
method Orthogonal statistical learning with self-concordant loss.
result Non-asymptotic bounds on excess risk improved by a dimension factor.