VC dimensions of group CNNs are infinite for certain kernels and groups.
problem Estimating the generalization capacity of group convolutional neural networks.
method Identifying precise VC dimension estimates for simple sets of group CNNs.
result Two-parameter families of convolutional neural networks have an infinite VC dimension for infinite groups and certain kernels.
We will establish that the VC dimension of the class of d-dimensional ellipsoids is (d^2+3d)/2, and that maximum likelihood estimate with N-component d-dimensional Gaussian mixture models induces a geometric class having VC dimension at least N(d^2+3d)/2. Keywords: VC dimension; finite dimensional ellipsoid; Gaussian m…
Study on VC dimension of GCNNs with input resolution effects.
problem Understanding the generalization capabilities of GCNNs.
method Derived upper and lower bounds for VC dimension, analyzed factors affecting it.
result Extended previous results on VC dimension of GCNNs, providing insights into input resolution dependence.
One of the earliest conjectures in computational learning theory-the Sample Compression conjecture-asserts that concept classes (equivalently set systems) admit compression schemes of size linear in their VC dimension. To-date this statement is known to be true for maximum classes---those that possess maximum cardinali…
Contradiction graphs reveal VC dimension threshold.
problem Determining VC dimension of concept classes.
method Study contradiction graphs of binary concept classes.
result Single contradiction graph Gm(H) determines VC dimension. New neural network class reduces VC dimension, leading to better generalization.
problem VC theory struggles with explaining small generalization errors in overparametrized neural networks.
method Developed hyperplane arrangement neural networks (HANNs) and used sample compression analysis.
result HANNs can have significantly smaller VC dimension than the number of weights, yet remain highly expressive.
Vapnik-Chervonenkis (VC) dimension is a fundamental measure of the generalization capacity of learning algorithms. However, apart from a few special cases, it is hard or impossible to calculate analytically. Vapnik et al. [10] proposed a technique for estimating the VC dimension empirically. While their approach behave…
Improved multi-group learning with group-realizable concepts.
problem Enhancing multi-group learning efficiency.
method Empirical risk minimization over group-realizable concepts.
result Improved sample complexity in group-realizable settings.
The paper connects GNNs to VC dimension theory to study their generalization performance.
problem Understanding GNNs' ability to make meaningful predictions beyond the training set.
method Using Vapnik-Chervonenkis (VC) dimension theory in two settings: no upper bound on graph order and known upper bound.
result Tight connections between GNNs' bitlength, number of colors, and VC dimension in different settings.
Develops higher arity VC theory and characterizes PAC learning in product spaces.
problem Characterizing PAC learning in multi-dimensional product spaces.
method Introduces higher arity VC dimension, generalizes Haussler packing lemma, and develops hypergraph regularity lemma.
result Characterizes higher arity PAC learning in n-fold product spaces.
Investigates the impact of finite VC dimension on neural network approximation and learning.
problem The influence of VC dimension on neural network approximation and learning from samples.
method Analysis of high-dimensional geometry and statistical learning theory, focusing on VC dimension.
result Finite VC dimension is beneficial for uniform convergence of empirical errors but not for approximation of functions from a probability distribution.
The existence of evasion attacks during the test phase of machine learning algorithms represents a significant challenge to both their deployment and understanding. These attacks can be carried out by adding imperceptible perturbations to inputs to generate adversarial examples and finding effective defenses and detect…
For any family of measurable sets in a probability space, we show that either (i) the family has infinite Vapnik-Chervonenkis (VC) dimension or (ii) for every epsilon > 0 there is a finite partition pi such the pi-boundary of each set has measure at most epsilon. Immediate corollaries include the fact that a family wit…
We study the question of learning an adversarially robust predictor. We show that any hypothesis class H with finite VC dimension is robustly PAC learnable with an improper learning rule. The requirement of being improper is necessary as we exhibit examples of hypothesis classes H with finite VC…
Adversarial robust learning improved for transductive setting.
problem Adversarial robust learning in transductive setting.
method Simple transductive learner for bounded VC dimension classes.
result Robust error rate linear in VC dimension, adaptive to perturbation complexity.
In Statistical Learning, the Vapnik-Chervonenkis (VC) dimension is an important combinatorial property of classifiers. To our knowledge, no theoretical results yet exist for the VC dimension of edited nearest-neighbour (1NN) classifiers with reference set of fixed size. Related theoretical results are scattered in the …
Learnable multiclass hypothesis classes don't always have a sample compression scheme of fixed size.
problem The limitation of sample compression schemes for multiclass hypothesis classes.
method Analysis of DS dimension and sample compression schemes.
result Learnable multiclass hypothesis classes do not always have a sample compression scheme of fixed size.
Paper generalizes strategic classification framework and introduces SVC for PAC-learning.
problem Strategic manipulation of testing data to fool classifiers.
method Unified framework for strategic classification, strategic VC-dimension (SVC).
result Characterizes the learnability and computational tractability of linear classifiers.
In response to a 1997 problem of M. Vidyasagar, we state a criterion for PAC learnability of a concept class C under the family of all non-atomic (diffuse) measures on the domain Ω. The uniform Glivenko--Cantelli property with respect to non-atomic measures is no longer a necessary condition, and consisten…
We show that the sets in a family with finite VC dimension can be uniformly approximated within a given error by a finite partition. Immediate corollaries include the fact that VC classes have finite bracketing numbers, satisfy uniform laws of averages under strong dependence, and exhibit uniform mixing. Our results ar…
The VC-dimension of a set system is a way to capture its complexity and has been a key parameter studied extensively in machine learning and geometry communities. In this paper, we resolve two longstanding open problems on bounding the VC-dimension of two fundamental set systems: k-fold unions/intersections of half-s…
Study on proper learning under relaxed worst-case robust loss for VC classes.
problem Proper adversarially robust PAC learning under relaxed worst-case robust loss.
method Introduced a family of robust loss relaxations and showed their effectiveness for proper learnability.
result VC classes are properly PAC learnable with sample complexity close to standard PAC learning setup.
The recently proposed Minimal Complexity Machine (MCM) finds a hyperplane classifier by minimizing an exact bound on the Vapnik-Chervonenkis (VC) dimension. The VC dimension measures the capacity of a learning machine, and a smaller VC dimension leads to improved generalization. On many benchmark datasets, the MCM gene…
The Vapnik-Chervonenkis (VC) dimension of a collection of subsets of a set is an important combinatorial concept in settings such as discrete geometry and machine learning. In this paper we prove that the VC dimension of the family of d-dimensional cubes in Rd is ⌊(3d+1)/2⌋.
Study extends GNN VC dimension bounds to Pfaffian activation functions.
problem Bounding GNN VC dimension for new activation functions.
method Pfaffian function theory applied to GNNs with sigmoid and hyperbolic tangent activations.
result Bounds on GNN VC dimension for various architectures and graph properties.
New robust estimators achieve subgaussian bounds using VC-dimension.
problem Robust estimation of sparse and corrupted data.
method Use of VC-dimension to measure statistical complexity.
result First robust estimators for sparse estimation with subgaussian rate.
Reduces multiclass and regression compression schemes to binary ones.
problem Developing efficient learning algorithms for multiclass and regression problems.
method Reduces sample compression schemes for binary classes to multiclass and regression settings.
result Establishes new compression schemes for multiclass and regression problems.
The study analyzes decision trees on real and categorical features, deriving bounds on their VC dimension and proposing improved pruning algorithms.
problem Understanding the generalization properties of decision trees on different types of features.
method Introducing partitioning functions, relating them to growth functions and VC dimension, and deriving bounds for decision stumps and trees of various structures.
result Exact VC dimension of decision stumps and improved pruning algorithms for binary trees.
New complexity measure ADL connects to classical complexity measures.
problem Deriving generalization bounds for neural networks.
method Exploring ADL's relationship to Covering Numbers and VC Dimension.
result ADL is equivalent to Covering Numbers and VC Dimension for real-valued functions.
Lower bounds set for infinite-precision transformers.
problem Understanding limitations of infinite-precision transformers.
method Used VC dimension technique to prove lower bounds.
result First lower bounds for two tasks: function composition and SUM2. Algorithm learns from both labeled and arbitrary test examples, giving guarantees for bounded VC dimension classes.
problem Learning from arbitrary test examples, not just perturbations.
method Selective transductive learning algorithm that outputs abstaining predictions.
result Nontrivial guarantees for bounded VC dimension classes with arbitrary train and test distributions.
Adversarial attacks during the testing phase of neural networks pose a challenge for the deployment of neural networks in security critical settings. These attacks can be performed by adding noise that is imperceptible to humans on top of the original data. By doing so, an attacker can create an adversarial sample, whi…
Study private query release with public data, reducing sample sizes.
problem Answering a wide range of statistical queries while maintaining privacy.
method Combines public and private samples to answer queries with differential privacy.
result Private and public sample complexities for different query classes.
The paper explores how to reduce classification tasks to optimization problems in Euclidean space.
problem Understanding the minimum dimension needed for reducing classification tasks to optimization problems.
method Developed a generalization of the Borsuk-Ulam Theorem to analyze the expressivity of reductions.
result The minimum Euclidean dimension required can be exponentially larger than the VC dimension, even for slightly non-trivial reductions.
A new model for sequential prediction handles adversarial examples by allowing abstention.
problem Sequential prediction algorithms fail with adversarial examples, leading to incorrect predictions.
method Proposes a new model that allows abstention from predictions on adversarial examples, scaling error with VC dimension.
result A learner's error scales with the VC dimension of the hypothesis class, matching the stochastic setting.
Novel framework for teaching complexity in machine teaching models.
problem Understanding and comparing teaching models in batch and sequential settings.
method Developed a novel framework using preference functions to capture teaching complexity.
result Identified preference functions leading to linear teaching complexity in sequential models.
New bounds on learning from multiple distributions for VC classes.
problem Understanding the sample complexity of learning from multiple data distributions.
method Analyzing the gap between known upper and lower bounds for PAC-learnable classes.
result Recent progress on sample complexity for VC dimension d classes on k distributions.
Deep Heaviside networks are limited but can be improved with connections or linear neurons.
problem Limited expressivity of deep Heaviside networks.
method Including skip connections or linear activation neurons improves expressivity.
result Lower and upper bounds for VC dimensions and approximation rates are derived.
Comparative learning combines realizable and agnostic settings for two hypothesis classes, reducing sample complexity.
problem Learning with two hypothesis classes in a more general setting than single hypothesis classes.
method Introduces comparative learning, defines mutual VC dimension and Littlestone dimension, and applies insights to multiaccuracy and multicalibration.
result Sample complexity of comparative learning is characterized by mutual VC dimension and Littlestone dimension.
We explore in some detail the notion of algorithmic stability as a viable framework for analyzing the generalization error of learning algorithms. We introduce the new notion of training stability of a learning algorithm and show that, in a general setting, it is sufficient for good bounds on generalization error. In t…
New method learns robustly with less data, bridging theory and practice.
problem Adversarial robust learning with metric perturbation.
method Tolerant adversarial PAC-learning with perturb-and-smooth approach and compression-based algorithm.
result First PAC-type guarantees for popular adversarial learning techniques.
Study robust learning without knowing perturbation sets, using interactions with attackers.
problem Learning robust predictors against unknown adversarial perturbations.
method Examined different interaction models with adversarial attackers, derived bounds on sample complexity and interactions.
result Upper bounds on sample complexity and lower bounds on interactions in various models.
Study shows how many domains are needed for generalization, using a new measure called domain shattering dimension.
problem How many domains are needed for domain generalization?
method Introduced a new combinatorial measure called the domain shattering dimension to model domain sample complexity.
result Established a tight quantitative relationship between domain shattering dimension and classic VC dimension.
New algorithms achieve better regret bounds for online classification with relaxed benchmarks.
problem Competing with worst-case optimal binary loss in online classification.
method Comparing against predictors robust to small input perturbations, performing well under Gaussian smoothing, or maintaining a prescribed output margin.
result Regret guarantees depend only on VC dimension and instance space complexity, with an O(log(1/γ)) dependence on the generalized margin. New insights into learning from only positive examples.
problem Characterizing proper learning from positive-only samples.
method Introducing a new combinatorial condition for proper positive-only learning.
result Proper positive-only learning is characterized by finite VC dimension and uniform exterior separability.
Characterizes distribution-free rates in unbalanced classification problems.
problem Minimizing error under two different distributions in unbalanced settings.
method Characterizes minimax rates over all pairs of distributions using a geometric condition.
result Identifies a dichotomy between hard and easy classes based on a three-points-separation condition.
New Sauer inequality improves multiclass hypothesis class bounds.
problem Bounding the size of multiclass hypothesis classes.
method Polynomial method and combinatorial parameters (DS, list-DS dimensions).
result Sharp Sauer inequality with optimal polynomial dependence on list size and alphabet size.
New bounds show agnostic multiclass learning depends on two dimensions: Natarajan and Daniely-Shalev-Shwartz.
problem Understanding sample complexity in multiclass classification with agnostic learning.
method Developed a novel online procedure based on a self-adaptive multiplicative-weights algorithm.
result Agnostic sample complexity bounds are in the form of DS^(1.5)/ε + Nat/ε^2, nearly tight up to a √DS factor.