In statistical learning theory, convex surrogates of the 0-1 loss are highly preferred because of the computational and theoretical virtues that convexity brings in. This is of more importance if we consider smooth surrogates as witnessed by the fact that the smoothness is further beneficial both computationally- by at…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Excess-risk bounds”
Study excess risk in statistical inference with transformations.
problem Excess risk in estimating random variables from feature vectors and transformations.
method Characterize lossless transformations, develop test statistics, and information-theoretic bounds.
result Strongly consistent partitioning test statistic for lossless transformations.
The paper bounds the excess risk of deep neural networks for weakly dependent processes.
problem Learning with weakly dependent data using deep neural networks.
method Approximation of smooth functions by deep neural networks and a bound on excess risk.
result The excess risk bound for deep learning under weak dependence is close to for sufficiently smooth functions.
Study non-asymptotic bounds for robust estimators under misspecified models.
problem Evaluate performance of robust estimators under adversarial conditions.
method Propose a general approach to adversarial risk analysis, including investigations on generalization and approximation errors.
result Establish non-asymptotic upper bounds for adversarial excess risk under Lipschitz loss functions.
This research improves PAC-Bayesian bounds for classification tasks using convexified loss.
problem Deriving generalization bounds for classification tasks with non-convex loss functions.
method Shift focus to misclassification excess risk bounds for PAC-Bayesian classification using convex surrogate loss and leveraging PAC-Bayesian relative bounds in expectation.
result Improved PAC-Bayesian bounds for classification tasks with convex surrogate loss.
We give improved constants for data dependent and variance sensitive confidence bounds, called empirical Bernstein bounds, and extend these inequalities to hold uniformly over classes of functionswhose growth function is polynomial in the sample size n. The bounds lead us to consider sample variance penalization, a nov…
Trace norm regularization is a popular method of multitask learning. We give excess risk bounds with explicit dependence on the number of tasks, the number of examples per task and properties of the data distribution. The bounds are independent of the dimension of the input space, which may be infinite as in the case o…
A Geometric Analysis of PCAmath.ST
The paper analyzes the excess risk of PCA and provides a precise characterization.
problem Understanding the excess risk of principal component analysis (PCA).
method Established a central limit theorem for PCA error and derived the excess risk distribution.
result Obtained a non-asymptotic upper bound on the excess risk of PCA.
Lower bounds show OLS outperforms basis pursuit in overparameterized linear regression.
problem Excess risk of sparse interpolating procedures in overparameterized linear regression.
method Proved lower bounds on excess risk for OLS and basis pursuit.
result Excess risk of basis pursuit can converge at an exponentially slower rate than OLS.
Paper improves privacy in SGD with low noise, achieving optimal risk rates.
problem Privacy-preserving machine learning with good performance.
method Differentially private SGD with low-noise analysis.
result Achieves optimal excess risk rates for non-smooth losses.
This paper analyzes multi-pass SGD for least squares, improving generalization bounds.
problem Improving generalization bounds for multi-pass SGD in the least squares problem.
method Develops an instance-dependent excess risk bound for least squares in the interpolation regime.
result SGD performs worse than GD instance-wise but saves computational time.
Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.
problem Optimal risk in bounded distributions for constrained least squares.
method Comparison of least squares and non-linear predictors.
result Non-linear predictors can achieve optimal risk in bounded distributions.
The overarching goal of this paper is to derive excess risk bounds for learning from exp-concave loss functions in passive and sequential learning settings. Exp-concave loss functions encompass several fundamental problems in machine learning such as squared loss in linear regression, logistic loss in classification, a…
Deep linear networks can closely approximate interpolants without improving risk.
problem Understanding the risk bounds of deep linear networks compared to minimum -norm solutions.
method Bounding excess risk of interpolating deep linear networks trained using gradient flow.
result Deep linear networks can closely approximate or match minimum -norm solutions in terms of risk.
The paper analyzes the generalization performance of spectral clustering algorithms and proposes new methods to improve their effectiveness.
problem Theoretical analysis of spectral clustering's generalization performance.
method Theoretical analysis and development of new spectral clustering algorithms.
result The excess risk bounds of spectral clustering algorithms have a convergence rate.
Concentration and excess risk bounds for imbalanced classification with synthetic oversamplingstat.ML
The paper analyzes SMOTE for imbalanced classification, providing theoretical bounds and guidelines.
problem The challenge of imbalanced classification problems, especially with minority classes.
method Theoretical analysis of SMOTE and related oversampling techniques for minority classes.
result Derives concentration and excess risk bounds for SMOTE and kernel-based classifiers.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
This paper examines error bounds for deep learning classifiers with noisy labels.
problem Understanding the performance of classifiers trained on noisy data.
method Derives error bounds for excess risk, decomposing it into statistical and approximation errors. Uses independent block construction for statistical dependencies and vector-valued setting for approximation error.
result Established theoretical results for error bounds in deep learning with noisy labels, mitigating the impact of high-dimensional input spaces.
Full-batch GD achieves generalization close to any stationary point with fewer assumptions.
problem Generalization and excess risk bounds for smooth losses, including non-Lipschitz and nonconvex cases.
method Path-dependent analysis of GD's generalization error, focusing on optimization error and stability.
result Generalization error is tightly bound in terms of optimization error and iteration count, bypassing common assumptions.
The paper explores the information-theoretic nature of excess risk in machine learning.
problem Understanding the excess risk in machine learning models.
method Formulates the minimax excess risk as a zero-sum game and modifies it to allow swapping of the order of play.
result Proves that under certain conditions, the duality gap is zero, allowing for the application of Bayesian results to provide bounds on minimax excess risk.
New method estimates model risk without knowing function class.
problem Evaluating model risk for complex, opaque models.
method Wild refitting with Bregman losses and randomized symmetrization.
result Valid upper bound on excess risk for opaque models.
Improves statistical learning bounds with self-concordant losses.
problem Statistical prediction with nuisance components.
method Orthogonal statistical learning with self-concordant loss.
result Non-asymptotic bounds on excess risk improved by a dimension factor.
Improved DP SO with large Lipschitz parameters, handling outliers and heavy-tailed data.
problem Differential privacy in stochastic optimization with large Lipschitz parameters.
method Assumes bounded k-th order moments, provides linear-time algorithms for smooth convex and non-smooth convex losses.
result Improved risk bounds scaling with k-th moment, not uniform Lipschitz parameter.
Paper proposes a new framework to improve stability-based bounds in deep learning.
problem Explaining generalization in overparameterized neural networks.
method Decomposes excess risk dynamics into signal and noise components, applying stability-based bounds only to the noise.
result The decomposition framework improves stability-based bounds and explains generalization in neural networks.
Data-driven method for error estimation without needing class complexity.
problem Constructing confidence intervals for a class of estimates.
method Data-driven approach to derive high-probability upper bounds on maximum error.
result Method naturally adapts to unknown correlation structures and works for finite and infinite classes.
SGD in linear regression overfits but performs well due to bias-variance trade-off.
problem Understanding overfitting in SGD for linear regression.
method Constant-stepsize SGD with iterate averaging or tail averaging, analyzing full eigenspectrum of data covariance matrix.
result Sharp excess risk bounds revealing bias-variance decomposition for SGD in linear regression.
Upper bounds and lower bounds show ERM outperforms DG methods in various settings.
problem Limitations of domain generalisation methods in various settings.
method Upper bounds and lower bounds on excess risk of ERM, and analysis of DG settings.
result It is not possible to significantly outperform ERM in DG settings.
New DP algorithm improves privacy and efficiency for convex optimization.
problem Efficient, DP algorithms for convex optimization with strong excess risk bounds.
method Output perturbation for a broad class of tilted loss functions.
result Near optimal DP excess risk and runtime bounds for convex optimization.
Fine-tuning with pre-training data improves performance.
problem Limited training data for tasks.
method Theoretical analysis of excess risk bound and selection of pre-training data subset.
result Improvement in generalization performance with pre-training data.
Improved algorithm reduces excess risk in selective learning.
problem Selective learning with windowed model selection.
method Hybrid Exponential Weights Algorithm and bounded-recall ERM.
result Achieves expected excess risk of O((log log |L| + log log n) / log n).
We introduce a procedure for conditional density estimation under logarithmic loss, which we call SMP (Sample Minmax Predictor). This estimator minimizes a new general excess risk bound for statistical learning. On standard examples, this bound scales as with the model dimension and the sample size, and c…
Optimizes differentially private kernel learning with random projection.
problem Privacy-preserving learning algorithms with optimal performance.
method Differentially private kernel ERM algorithm based on random projection in reproducing kernel Hilbert space.
result Achieves minimax-optimal excess risk rates for various loss functions.
Improved PAC-Bayesian bounds by considering example difficulty.
problem Improving generalization bounds in machine learning.
method Introducing a modified excess risk that leverages example difficulty to reduce variance and tighten PAC-Bayesian bounds.
result Tighter PAC-Bayesian generalization bounds for machine learning models.
Gibbs-ERM learning is a natural idealized model of learning with stochastic optimization algorithms (such as Stochastic Gradient Langevin Dynamics and ---to some extent--- Stochastic Gradient Descent), while it also arises in other contexts, including PAC-Bayesian theory, and sampling mechanisms. In this work we study …
New method refines model-free evaluation of complex machine learning models.
problem Evaluating the excess risk of opaque machine learning predictors.
method Perturbing derivatives to create pseudo-outcomes and refitting the model twice.
result Upper bound on excess risk derived efficiently without prior function class knowledge.
Last SGD iterate bounds for overparameterized linear regression.
problem Analyzing the last iterate risk bounds of SGD with decaying stepsize for overparameterized linear regression.
method Problem-dependent analysis of last iterate risk bounds of SGD with geometrically decaying stepsize.
result Proved nearly matching upper and lower bounds on the excess risk for last iterate SGD with geometrically decaying stepsize.
We consider a standard binary classification problem. The performance of any binary classifier based on the training data is characterized by the excess risk. We study Bahadur's type exponential bounds on the minimax accuracy confidence function based on the excess risk. We study how this quantity depends on the comple…
Non-Euclidean Differentially Private Stochastic Convex Optimization: Optimal Rates in Linear Timecs.LG
New algorithm achieves optimal privacy and efficiency in non-Euclidean convex optimization.
problem Optimizing convex functions while maintaining privacy in non-Euclidean settings.
method Developed a linear-time algorithm for -setups, leveraging geometric properties.
result Optimal excess risk achieved in linear time for .
Study shows how SGD's implicit regularization relates to ridge regression.
problem Least squares regression optimization with mini-batch SGD.
method Analyzes stochastic gradient flow as a continuous-time model of SGD.
result Bound on excess risk of SGD flow over ridge regression, revealing how parameters drive risk.
The paper analyzes variational autoencoders for state space models with risk bounds.
problem Analyzing the risk associated with variational autoencoders for state space models.
method Backward factorization of variational distributions to analyze excess risk, providing oracle inequalities and upper bounds.
result Explicit upper bounds on variational estimation error for state space models under strong mixing assumptions.
The paper analyzes the performance of empirical risk minimization for -norm linear regression.
problem Empirical risk minimization on -norm linear regression.
method Analyzes performance under various conditions and moment assumptions.
result High probability excess risk bounds for empirical risk minimizer, matching asymptotic rates.
New research shows that binary classification can be done with noisy data, but only if there are clean samples available.
problem Learning binary classification with instance and label dependent label noise.
method Theoretical analysis and empirical risk minimization.
result Empirical risk minimization achieves the optimal excess risk bound without additional assumptions.
The paper explores how linear neural networks can overfit without bias when data is well-behaved.
problem Understanding why linear neural networks can generalize well despite fitting noisy data.
method Analyzing two-layer linear neural networks trained with gradient flow, deriving bounds on excess risk.
result The excess risk depends on initialization quality and data covariance matrix properties.
This study tightens bounds on how GD and SGD generalize in smooth convex optimization problems.
problem Understanding how GD and SGD generalize in smooth stochastic convex optimization problems.
method Provided tight excess risk lower bounds for GD and SGD under different conditions.
result Lower bounds suggest overfitting occurs and gaps remain in some cases.
New bounds show polyhedral surrogates are optimal for generalization.
problem Proving generalization rates for polyhedral loss functions.
method Developed two general results for polyhedral surrogates.
result Polyhedral surrogates provide linear surrogate regret bounds, translating directly to target rates.
The paper analyzes the generalization of deep neural networks for metric and similarity learning.
problem Lack of rigorous understanding of generalization performance in metric and similarity learning.
method Derive explicit form of true metric, construct structured deep ReLU neural network, establish excess risk bounds.
result Explicit excess risk bounds for metric and similarity learning are derived.
New bounds for non-convex estimators without Bernstein condition.
problem Sharp excess risk bounds for non-convex and improper estimators.
method Exponential-tail local Rademacher complexity risk bounds with offset condition.
result Sharp bounds for non-convex and improper estimators without Bernstein condition.
Gradient methods improve deep network training with tighter bounds and faster convergence.
problem Improving convergence and generalization of gradient methods for neural networks.
method Algorithmic stability analysis and novel bounds on excess risk.
result Gradient descent achieves optimal excess risk for deep nets with polynomial width conditions.