This paper examines error bounds for deep learning classifiers with noisy labels.
problem Understanding the performance of classifiers trained on noisy data.
method Derives error bounds for excess risk, decomposing it into statistical and approximation errors. Uses independent block construction for statistical dependencies and vector-valued setting for approximation error.
result Established theoretical results for error bounds in deep learning with noisy labels, mitigating the impact of high-dimensional input spaces.
Paper establishes a universal growth rate for smooth surrogate losses in classification.
problem Analyzing growth rates of consistency bounds for various surrogate losses.
method Proves square-root growth rate for smooth margin-based losses; extends to multi-class classification.
result Demonstrates a universal square-root growth rate for smooth comp-sum and constrained losses.
Study non-asymptotic bounds for robust estimators under misspecified models.
problem Evaluate performance of robust estimators under adversarial conditions.
method Propose a general approach to adversarial risk analysis, including investigations on generalization and approximation errors.
result Establish non-asymptotic upper bounds for adversarial excess risk under Lipschitz loss functions.
Full-batch GD achieves generalization close to any stationary point with fewer assumptions.
problem Generalization and excess risk bounds for smooth losses, including non-Lipschitz and nonconvex cases.
method Path-dependent analysis of GD's generalization error, focusing on optimization error and stability.
result Generalization error is tightly bound in terms of optimization error and iteration count, bypassing common assumptions.
Data-driven method for error estimation without needing class complexity.
problem Constructing confidence intervals for a class of estimates.
method Data-driven approach to derive high-probability upper bounds on maximum error.
result Method naturally adapts to unknown correlation structures and works for finite and infinite classes.
Paper improves risk bounds for nonconvex-strongly-concave minimax problems.
problem Achieving sharper risk bounds for nonconvex-strongly-concave minimax problems.
method Using uniform localized convergence to derive high probability generalization error bounds.
result Derives n times faster excess primal risk bounds for popular algorithms.
In statistical learning theory, convex surrogates of the 0-1 loss are highly preferred because of the computational and theoretical virtues that convexity brings in. This is of more importance if we consider smooth surrogates as witnessed by the fact that the smoothness is further beneficial both computationally- by at…
This article studies the achievable guarantees on the error rates of certain learning algorithms, with particular focus on refining logarithmic factors. Many of the results are based on a general technique for obtaining bounds on the error rates of sample-consistent classifiers with monotonic error regions, in the real…
The paper explores the information-theoretic nature of excess risk in machine learning.
problem Understanding the excess risk in machine learning models.
method Formulates the minimax excess risk as a zero-sum game and modifies it to allow swapping of the order of play.
result Proves that under certain conditions, the duality gap is zero, allowing for the application of Bayesian results to provide bounds on minimax excess risk.
This research evaluates generalization measures in deep learning.
problem Understanding why deep learning models generalize well despite small training error.
method Empirical evaluation of generalization bounds and measures.
result Generalization measures should be evaluated using distributional robustness.
Study excess capacity in neural networks using Rademacher complexity.
problem Understanding how much capacity deep networks have beyond what's needed for classification.
method Unified Rademacher complexity bounds for function composition and convolutional layers, considering Lipschitz constants and initialization norms.
result There is substantial excess capacity per task, and capacity can be kept similar across different tasks.
Study of estimation errors in surrogate loss minimizers, providing stronger guarantees than existing methods.
problem Estimation errors in surrogate loss minimizers for various hypothesis sets.
method Detailed study of H-consistency estimation error bounds, proving general theorems for distribution-dependent and independent settings. result Explicit bounds for zero-one and adversarial losses, showing enhancements under distributional assumptions.
The paper analyzes the excess risk of PCA and provides a precise characterization.
problem Understanding the excess risk of principal component analysis (PCA).
method Established a central limit theorem for PCA error and derived the excess risk distribution.
result Obtained a non-asymptotic upper bound on the excess risk of PCA.
ASGD outperforms SGD in overparameterized linear regression, especially in subspaces of small eigenvalues.
problem Generalization of ASGD for overparameterized linear regression.
method Established instance-dependent excess risk bound for ASGD in each eigen-subspace of the data covariance matrix.
result ASGD outperforms SGD in subspaces of small eigenvalues, exhibiting faster decay of bias error.
The study analyzes transfer learning using information theory.
problem Transfer learning in different distributions.
method Information-theoretic analysis, focusing on KL divergence.
result Upper bounds for general transfer learning algorithms and specific ERM.
Deep neural networks classify unbounded Gaussian mixture data without dimensionality issues.
problem Binary classification of unbounded Gaussian mixture data.
method Deep ReLU neural networks with non-asymptotic upper bounds and convergence rates.
result Deep ReLU networks can classify unbounded Gaussian mixture data without dimensionality constraints.
Paper establishes bounds for RNN-TPPs, showing four-layer networks can achieve vanishing errors.
problem Understanding theoretical limits of RNN-TPPs.
method Characterized RNN complexity, constructed neural approximations, applied truncation technique.
result Four-layer RNN-TPPs can achieve vanishing generalization errors.
This paper tightens information-theoretic bounds on generalization errors.
problem Understanding the discrepancy between training and testing data losses.
method Investigates the tightness of information-theoretic bounds on generalization error.
result The individual sample mutual information bound can be asymptotically tight under specific assumptions.
The paper analyzes the generalization of deep neural networks for metric and similarity learning.
problem Lack of rigorous understanding of generalization performance in metric and similarity learning.
method Derive explicit form of true metric, construct structured deep ReLU neural network, establish excess risk bounds.
result Explicit excess risk bounds for metric and similarity learning are derived.
This paper analyzes multi-pass SGD for least squares, improving generalization bounds.
problem Improving generalization bounds for multi-pass SGD in the least squares problem.
method Develops an instance-dependent excess risk bound for least squares in the interpolation regime.
result SGD performs worse than GD instance-wise but saves computational time.
We consider the problem of learning a binary classifier from only positive and unlabeled observations (called PU learning). Recent studies in PU learning have shown superior performance theoretically and empirically. However, most existing algorithms may not be suitable for large-scale datasets because they face repeat…
New framework analyzes deep learning optimization with finite width networks, revealing generalization gaps and excess risks.
problem Analyzing generalization error of deep learning with finite width networks.
method Formulating neural network training as transportation map estimation and analyzing via infinite dimensional Langevin dynamics.
result Achieves fast learning rate and minimax optimal rates for classification and regression problems.
The study bounds the excess of disjoint nonorientable surfaces in a 4-manifold.
problem Bounding the excess of disjoint nonorientable surfaces in a 4-manifold.
method Combining tubing construction with signature and Euler-characteristic formulas for 2-fold branched covers.
result The normal-Euler excess is bounded by a constant depending only on the ambient 4-manifold.
Gradient methods improve deep network training with tighter bounds and faster convergence.
problem Improving convergence and generalization of gradient methods for neural networks.
method Algorithmic stability analysis and novel bounds on excess risk.
result Gradient descent achieves optimal excess risk for deep nets with polynomial width conditions.
The paper analyzes risk bounds and Rademacher complexity in batch RL.
problem Estimating/minimizing Bellman error with general value function approximation.
method Characterizes generalization performance using Rademacher complexities of function classes.
result Risk bounds and Rademacher complexities provide insights into batch RL.
Defines MER for Bayesian learning, a gap between achievable and optimal performance.
problem Analyzing the best performance of Bayesian learning under generative models.
method Two methods for deriving upper bounds for MER: conditional mutual information and minimum estimation error.
result Quantifies the rate at which MER decays to zero with more data and relates it to model richness.
The paper analyzes variational autoencoders for state space models with risk bounds.
problem Analyzing the risk associated with variational autoencoders for state space models.
method Backward factorization of variational distributions to analyze excess risk, providing oracle inequalities and upper bounds.
result Explicit upper bounds on variational estimation error for state space models under strong mixing assumptions.
Paper proposes deep neural networks for nonparametric regression from dependent data.
problem Nonparametric regression from strongly mixing observations.
method Minimum error entropy principle applied to deep neural networks.
result Deep neural networks achieve minimax optimal convergence rates for Gaussian errors.
Paper introduces a novel measure to analyze excess error in classification under covariate shift.
problem Analyzing excess error in classification under covariate shift.
method Utilizes vicinity information to characterize excess error.
result Faster or competitive convergence rates compared to previous techniques.
New method estimates tensors from noisy data with missing entries.
problem Tensor estimation from noisy observations with missing entries.
method Sign series representation for tensor completion, addressing low- and high-rank signals.
result Excess risk bounds, estimation error rates, and sample complexities established.
We introduce a procedure for conditional density estimation under logarithmic loss, which we call SMP (Sample Minmax Predictor). This estimator minimizes a new general excess risk bound for statistical learning. On standard examples, this bound scales as d/n with d the model dimension and n the sample size, and c…
New method bounds high-dimensional regression without estimating design covariance.
problem High-dimensional linear regression with random design.
method Error-in-operator approach that incorporates design covariance into empirical risk minimization.
result Dimension-free bounds on excess prediction risk derived.
Study optimizes prediction error for growing-dimensional PFLM models.
problem Optimizing prediction error for growing-dimensional PFLM models.
method Penalized least-squares approach in RKHS with effective dimension consideration.
result Shows exact upper bound for excess prediction risk in non-asymptotic form.
New method for multiclass classification reduces error bounds.
problem High-dimensional multiclass classification with sparse coefficients.
method Sparse multinomial logistic regression with convex penalties.
result Plug-in classifiers achieve minimax generalization error bounds.
This paper improves risk bounds and calibration for smart predict-then-optimize method.
problem Improving risk bounds and calibration for smart predict-then-optimize method.
method Develops risk bounds and uniform calibration results for the SPO+ loss relative to the SPO loss.
result Empirical minimizer of the SPO+ loss achieves low excess true risk with high probability.
This study tightens bounds on how GD and SGD generalize in smooth convex optimization problems.
problem Understanding how GD and SGD generalize in smooth stochastic convex optimization problems.
method Provided tight excess risk lower bounds for GD and SGD under different conditions.
result Lower bounds suggest overfitting occurs and gaps remain in some cases.
We revisit the problem of differentially private release of classification queries. In this problem, the goal is to design an algorithm that can accurately answer a sequence of classification queries based on a private training set while ensuring differential privacy. We formally study this problem in the agnostic PAC …
New method improves understanding of machine learning model performance.
problem Understanding how well machine learning models generalize from training data to unseen data.
method Auxiliary Distribution Method to derive new generalization error bounds.
result Upper bounds on generalization errors are tighter and more applicable.
The study examines Kernel Ridge Regression error rates across noiseless and noisy conditions.
problem Characterizing Kernel Ridge Regression error rates in different noise levels.
method Unified analysis of Kernel Ridge Regression under various noise and regularization conditions.
result A crossover from noiseless to noisy error rates is observed as sample complexity increases.
In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking,…
The paper analyzes kernel classifiers' performance in Sobolev spaces and proves their optimality.
problem Theoretical analysis of kernel classifiers' performance in Sobolev spaces.
method Deriving upper and lower bounds on classification excess risk using kernel regression theory and estimating interpolation smoothness.
result The proposed kernel classifier is optimal in Sobolev spaces, with theoretical bounds confirmed by real data.
Paper tackles robust deep learning from weakly dependent data with unbounded loss and input.
problem Tackles robust deep learning from weakly dependent data with unbounded loss and input.
method Establishes non-asymptotic bounds for expected excess risk under strong mixing and ψ-weak dependence assumptions. result Derives a relationship between bounds and r, and shows convergence rate close to i.i.d. results for r=∞. Investigates fast prediction rates with limited expert advice.
problem Minimizing excess generalization error with limited expert access.
method Assumes Lipschitz and strongly convex loss, designs novel algorithms.
result Achieves fast rates of O(1/T) with optimal number of expert advices.
We derive PAC-Bayesian learning guarantees for heavy-tailed losses, and obtain a novel optimal Gibbs posterior which enjoys finite-sample excess risk bounds at logarithmic confidence. Our core technique itself makes use of PAC-Bayesian inequalities in order to derive a robust risk estimator, which by design is easy to …
Novel bounds for SGLD show generalization error decreases with more samples.
problem Understanding the generalization error of SGLD in non-convex optimization.
method Information-theoretic approach focusing on Kullback-Leibler divergence and sub-exponential loss function.
result Time-independent generalization bounds for SGLD, independent of step size and number of iterations.
New L1 regularization controls neural network generalization error and sparsifies input dimensions.
problem Selecting the optimal number of hidden neurons in neural networks.
method Theoretical analysis of L1 regularization in two-layer neural networks. result Appropriate L1 regularization leads to near minimax optimal generalization risk bounds. Study improves robust nonparametric regression in heavy-tailed noise.
problem Robust nonparametric regression with heavy-tailed noise and unbounded functions.
method Huber regression in reproducing kernel Hilbert spaces (RKHS), probabilistic effective hypothesis space, new comparison theorems.
result Explicit finite-sample error bounds and convergence rates for Huber regression in RKHS under heavy-tailed noise.
Framework for domain adaptation using pseudo-labels from unlabeled data.
problem Improving prediction accuracy in target domain with covariate shift.
method Kernel GLMs with labeled and pseudo-labeled data, using imputation model for target data.
result Non-asymptotic excess-risk bounds for effective labeled sample size.