Statistical learning theory connects to spin glass models via Rademacher complexity and replica theory.
problem Bounding generalization gap in statistical learning theory.
method Linking Rademacher complexity in statistical learning to synthetic models in statistical physics.
result Rademacher complexity is closely related to ground state energy in spin glass models.
The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend thi…
New bounds for non-convex estimators without Bernstein condition.
problem Sharp excess risk bounds for non-convex and improper estimators.
method Exponential-tail local Rademacher complexity risk bounds with offset condition.
result Sharp bounds for non-convex and improper estimators without Bernstein condition.
For a finite function class we describe the large sample limit of the sequential Rademacher complexity in terms of the viscosity solution of a G-heat equation. In the language of Peng's sublinear expectation theory, the same quantity equals to the expected value of the largest order statistics of a multidimensional $…
New method calculates winding of geodesics on surfaces.
problem Understanding the distribution of geodesics on surfaces.
method Introducing a new construction of winding numbers for geodesics on cusped hyperbolic orbifolds.
result Winding numbers can be expressed by Rademacher symbols for various arithmetic families of surfaces.
The paper analyzes risk bounds and Rademacher complexity in batch RL.
problem Estimating/minimizing Bellman error with general value function approximation.
method Characterizes generalization performance using Rademacher complexities of function classes.
result Risk bounds and Rademacher complexities provide insights into batch RL.
Paper improves performance guarantees for Rademacher projections.
problem Improving statistical guarantees for Rademacher random projections.
method Algebraic framework for proving Schur-concavity properties.
result Novel Schur-concavity property of Rademacher projections with improved performance.
ACL improves robustness with unlabeled data, and we analyze its generalization using Rademacher complexity.
problem Improving robustness of deep networks against adversarial attacks using unlabeled data.
method We analyze the generalization performance of Adversarial Contrastive Learning (ACL) using Rademacher complexity.
result The average adversarial risk of the downstream tasks can be upper bounded by the adversarial unsupervised risk of the upstream task.
The paper analyzes adversarial robustness for linear models and neural networks using Rademacher complexity.
problem Understanding adversarial robustness of linear models and neural networks.
method The paper uses Rademacher complexity to provide upper and lower bounds for adversarial robustness of linear hypotheses and neural networks.
result The paper provides bounds on adversarial Rademacher complexity for linear hypotheses and neural networks, offering a finer analysis of input dimensionality.
The paper provides bounds for regression schemes using nonstationary training samples.
problem Developing confidence intervals for nonparametric regression with nonstationary data.
method The approach involves Rademacher and Vapnik-Chervonenkis theories to analyze the cost and optimality of regression schemes.
result The paper establishes nonasymptotic bounds for regression schemes and optimality in L2-distance. Transductive learning considers situations when a learner observes m labelled training points and u unlabelled test points with the final goal of giving correct answers for the test points. This paper introduces a new complexity measure for transductive learning called Permutational Rademacher Complexity (PRC) and …
New bound explains why high-rank neural nets generalize well.
problem Understanding why high-rank neural networks generalize well.
method Using Koopman operators, group representations, and RKHSs, a new Rademacher complexity bound is derived.
result Derives a bound for a wider range of realistic models.
É.Ghys proved that the linking numbers of modular knots and the "missing" trefoil K2,3 in S3 coincide with the values of a highly ubiquitous function called the Rademacher symbol for SL2Z. In this paper, we replace SL2Z=Γ2,3 by the triangle group Γp,q for any coprime …
Majorizing measures control sequential complexities for online learning.
problem Extending classical empirical processes theory to sequential cases.
method Generic chaining, majorizing measures, fractional covering numbers.
result Sharp control of worst-case sequential Rademacher complexity.
New bound on Rademacher complexity for vector functions.
problem Bounding Rademacher complexity for vector-valued functions.
method Bounding Rademacher complexity by coordinate-wise complexity with a factor of sqrt(K).
result Rademacher complexity is bounded by the maximum coordinate-wise complexity times sqrt(K).
Extends inequality for Rademacher complexities using p-stable variables.
problem Improving Rademacher complexity bounds using p-stable variables. method Extends contraction inequality to p-stable variables for 1<p<2. result New bounds for Rademacher complexities with p-stable variables. We present a novel notion of complexity that interpolates between and generalizes some classic existing complexity notions in learning theory: for estimators like empirical risk minimization (ERM) with arbitrary bounded losses, it is upper bounded in terms of data-independent Rademacher complexity; for generalized Baye…
The paper proves Rademacher's theorem for Heisenberg groups.
problem Proving Rademacher's theorem for Heisenberg groups.
method New definition of intrinsic Lipschitz graphs, extension and approximation theorems, use of Heisenberg currents and Rumin's complex.
result Rademacher's theorem for intrinsic Lipschitz graphs in Heisenberg groups.
Study improves generalization bounds for equivariant networks on Markov data.
problem Challenges in integrating equivariance with Markov dependencies in neural networks.
method Applied McDiarmid's inequality and computed covering number using group theory.
result Derived upper bound on Rademacher complexity for equivariant neural networks on Markov datasets.
Paper introduces new neural network models and theories.
problem Understanding neural networks beyond over-parameterized regime.
method Develops two exact models and a novel representor theory.
result Provides insights into neural network training and kernel evolution.
The resilience of low-degree Rademacher chaos is studied, providing probabilistic lower bounds.
problem Understanding how much a Rademacher chaos can withstand adversarial sign-flips without significant probability changes.
method Probabilistic lower-bound guarantees for the resilience of Rademacher chaos of arbitrary degree.
result Probabilistic lower-bound guarantees for the resilience of Rademacher chaos of arbitrary degree, especially meaningful for constant degree.
We develop a novel family of algorithms for the online learning setting with regret against any data sequence bounded by the empirical Rademacher complexity of that sequence. To develop a general theory of when this type of adaptive regret bound is achievable we establish a connection to the theory of decoupling inequa…
A formula for Rademacher symbols in triangle groups is provided.
problem No specific problem stated; focuses on a mathematical formula.
method Presentation of an explicit formula for Rademacher symbols.
result Generalizes Ghys' proof of modular knot linking numbers.
Paper extends learning theory to dependent data with uniform risk bounds.
problem Learning with dependent data sequences.
method Derives uniform risk bounds for dependent data using VC-dimension and Rademacher complexity.
result Standard classification risk bounds hold for dependent data, same as for independent data.
We develop a technique for deriving data-dependent error bounds for transductive learning algorithms based on transductive Rademacher complexity. Our technique is based on a novel general error bound for transduction in terms of transductive Rademacher complexity, together with a novel bounding technique for Rademacher…
Improved generalization bounds for CNNs using Rademacher complexity.
problem Establishing non-vacuous generalization bounds for deep learning models.
method Rademacher complexity framework with novel contraction lemmas for high-dimensional mappings.
result Enhanced generalization bounds for a broader class of activation functions.
We propose Rademacher complexity bounds for multiclass classifiers trained with a two-step semi-supervised model. In the first step, the algorithm partitions the partially labeled data and then identifies dense clusters containing κ predominant classes using the labeled training examples such that the proportion of t…
Survey on learning with graph-dependent data, deriving new generalization bounds.
problem Traditional i.i.d. data assumption fails in many real-life applications.
method Collect and analyze graph-dependent concentration bounds, derive generalization bounds.
result New generalization bounds for graph-dependent data.
The paper bounds the complexity of GCNs using Rademacher complexity.
problem Understanding the sample complexity of GCNs.
method Derived tight upper and lower bounds of Rademacher complexity for GCN models.
result The derived bounds depend on the largest eigenvalue of the graph filter and the degree distribution.
Graph neural networks generalize well under certain conditions, explained by learning theory.
problem Understanding why graph neural networks generalize well in transductive inference.
method Analysis of transductive Rademacher complexity to explain generalization properties of graph convolutional networks.
result Transductive Rademacher complexity can explain the generalization of graph convolutional networks for node classification in stochastic block models.
Analyzes the complexity of linear hypothesis sets using Rademacher complexity.
problem Understanding the complexity of linear hypothesis sets for various norms.
method Tight analysis of empirical Rademacher complexity for linear hypothesis classes with bounded weights.
result Improved bounds on Rademacher complexity for linear hypothesis sets, matching or improving existing results.
The contraction inequality for Rademacher averages is extended to Lipschitz functions with vector-valued domains, and it is also shown that in the bounding expression the Rademacher variables can be replaced by arbitrary iid symmetric and sub-gaussian variables. Example applications are given for multi-category learnin…
Tensorized Rademacher projections outperform Gaussian projections in reducing tensor dimensions.
problem Reducing the dimension of high-dimensional tensors for machine learning.
method Tensorized Rademacher random projections using Tensor Train decomposition.
result Tensorized Rademacher projections can replace Gaussian projections in tensor compression.
We show how to control the generalization error of time series models wherein past values of the outcome are used to predict future values. The results are based on a generalization of standard i.i.d. concentration inequalities to dependent data without the mixing assumptions common in the time series setting. Our proo…
Study reveals adversarially robust domain adaptation is harder to generalize across domains.
problem Hardness of transferring adversarial robustness across different domains.
method Analysis of adversarial Rademacher complexity over symmetric difference hypothesis space.
result Adversarial Rademacher complexity is always greater than non-adversarial, indicating intrinsic hardness.
Improved bounds for Monte Carlo Rademacher Averages using self-bounding functions.
problem Proving sharper concentration bounds for MCERA.
method Deriving new bounds through self-bounding functions and concentration of measure.
result Novel bounds depend on data-dependent quantities, improving over standard methods.
New approach for testable learning using moment matching and Rademacher complexity.
problem Replacing hard-to-verify distributional assumptions with testable ones.
method Moment matching and metric distances in probability.
result Improved sample complexity bounds for various concept classes and distributions.
New findings show Rademacher complexities are not crucial for learning complexities.
problem Understanding the sample complexity of learning with squared loss in convex classes.
method Novel learning procedure combining mean estimation and Talagrand's generic chaining method.
result Sample complexity is determined by the limiting Gaussian process, not Rademacher complexities.
Great successes of deep neural networks have been witnessed in various real applications. Many algorithmic and implementation techniques have been developed, however, theoretical understanding of many aspects of deep neural networks is far from clear. A particular interesting issue is the usefulness of dropout, which w…
Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii [arXiv:0708.0083]
Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii [arXiv:0708.0083]
Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii [arXiv:0708.0083]
Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii [arXiv:0708.0083]
Discussion of "2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization" by V. Koltchinskii [arXiv:0708.0083]
This paper provides a general result on controlling local Rademacher complexities, which captures in an elegant form to relate the complexities with constraint on the expected norm to the corresponding ones with constraint on the empirical norm. This result is convenient to apply in real applications and could yield re…
Logistic regression gets a new, simpler uniform bound.
problem Finding a uniform bound for logistic regression's empirical risk.
method PAC-Bayes approach with second-order expansion and Rademacher-complexity bounds.
result Provides a dimension-free uniform concentration bound.
We present a study of generalization for data-dependent hypothesis sets. We give a general learning guarantee for data-dependent hypothesis sets based on a notion of transductive Rademacher complexity. Our main result is a generalization bound for data-dependent hypothesis sets expressed in terms of a notion of hypothe…
Many machine learning models are vulnerable to adversarial attacks; for example, adding adversarial perturbations that are imperceptible to humans can often make machine learning models produce wrong predictions with high confidence. Moreover, although we may obtain robust models on the training dataset via adversarial…