Study shows transductive learning is equivalent to PAC learning for most natural loss functions.
problem Understanding the relationship between transductive and PAC learning models.
method Extending existing results and developing new techniques to analyze the equivalence of the two models.
result Transductive learning is essentially equivalent to PAC learning for realizable learning with most natural loss functions.
Paper analyzes error exponent in agnostic PAC learning.
problem Analyzing performance of agnostic PAC learning.
method Using error exponent from Information Theory to analyze PAC learning.
result Improved distribution-dependent error exponent for agnostic learning.
Paper proves L2 regression can learn k-juntas without distributional assumptions.
problem Learning k-juntas using L2 regression without distributional restrictions.
method L2 polynomial regression and minimum mean square estimation (MMSE).
result Agnostic PAC learning of k-juntas using L2 polynomial regression.
New algorithm for reliable learning of Gaussian halfspaces with improved sample and computational complexity.
problem Learning halfspaces under Gaussian marginals with reliable agnostic model.
method Developed a new algorithm for reliable learning of Gaussian halfspaces with specific sample and computational complexity.
result Achieved a new algorithm with improved sample and computational complexity for reliable learning of Gaussian halfspaces.
Paper develops PAC verification for hypothesis classes and statistical algorithms.
problem Verifying machine learning models interactively.
method Develops interactive proof for PAC verification, proves lower bounds, and introduces a generalization.
result Improved protocol for verifying unions of intervals and statistical query algorithms.
Study on learning halfspaces under adversarial perturbations, finding computational hardness.
problem Learning halfspaces in the presence of adversarial noise.
method Introduced an efficient learning algorithm and proved a nearly matching computational hardness result.
result The L∞ perturbations case is provably computationally harder than 2≤p<∞. We clarify measurability assumptions in the agnostic PAC learning theorem.
problem Measurability assumptions in the Fundamental Theorem of Statistical Learning.
method Measure-theoretic scrutiny of existing proofs to extract minimal assumptions.
result Sound statement and detailed proof of the Fundamental Theorem in the agnostic setting.
Improved private sample complexity for answering classification queries.
problem Designing an algorithm to accurately answer classification queries while maintaining differential privacy.
method Formally studied in agnostic PAC model, derived new upper bound on private sample complexity.
result Improved private sample complexity bound for answering classification queries.
New complexity measure helps in agnostic reinforcement learning with or without access to MDP dynamics.
problem Understanding the number of rounds needed to learn an ε-suboptimal policy in unknown MDPs.
method Introducing spanning capacity as a new complexity measure and developing POPLER algorithm.
result There is a separation between generative and online access models for agnostic learnability.
Study shows realizable learnability doesn't imply agnostic learnability for distributions.
problem Learnability and robustness of distribution classes.
method Analyzes the relationship between learnability and robustness for distribution learning.
result Realizable learnability does not imply agnostic learnability for distributions.
Compact learning results across various loss functions.
problem Understanding sample complexity in transductive learning.
method Analyzing finite projections and sample complexities for different loss functions.
result Exact compactness of sample complexity holds broadly across realizable and agnostic learning.
New algorithm learns disjunctions faster than previous methods.
problem Learning Boolean disjunctions in the agnostic PAC model.
method Developed an agnostic learner with complexity 2ildeO(n1/3). result First separation between SQ and CSQ models in distribution-free agnostic learning.
New learner achieves optimal agnostic error in small error regime.
problem Optimizing agnostic learning in the small error regime.
method Careful aggregations of ERM classifiers.
result Achieves error $c \cdot τ+ O \left(\sqrt{\frac{τ(d + \log(1 / δ))}{m}} + \frac{d + \log(1 / δ)}{m}
ight)$, matching lower bound when τ≈d/m. Study robust regression learning under adversarial attacks.
problem Understanding which function classes are learnable in the presence of adversarial attacks.
method Introduced a novel agnostic sample compression scheme and used fat-shattering dimension to construct adversarially robust sample compression schemes.
result Finite fat-shattering dimension classes are learnable in both realizable and agnostic settings.
Query access significantly speeds up learning Multi-Index Models under Gaussian distribution.
problem Agnostically learning Multi-Index Models (MIMs) under Gaussian distribution.
method Query access for MIMs with complexity O(k)poly(1/ε)poly(d) under standard regularity assumptions. result Query access gives significant runtime improvements over random examples for agnostically learning MIMs.
Study learning and refutation in non-interactive LDP, showing sample complexity equivalence.
problem Characterize sample complexity for learning and refutation in non-interactive LDP.
method Characterize sample complexity for agnostic PAC learning in non-interactive LDP protocols.
result Optimal sample complexity for any concept class is captured by the approximate γ2~norm of a natural matrix associated with the class. Improved PAC learning algorithm for multiclass classification with bandit feedback.
problem Efficiently learning multiclass classification with limited feedback.
method Novel algorithm using stochastic optimization and Frank-Wolfe updates.
result Improved sample complexity bounds for multiclass PAC learning.
Non-convex SGD learns halfspaces with adversarial label noise efficiently.
problem Agnostically learning halfspaces in adversarial label noise settings.
method Non-convex SGD optimization for halfspace learning.
result Non-convex SGD achieves misclassification error close to optimal with adversarial noise.
PAC learning, dating back to Valiant'84 and Vapnik and Chervonenkis'64,'74, is a classic model for studying supervised learning. In the agnostic setting, we have access to a hypothesis set H and a training set of labeled samples (x1,y1),…,(xn,yn)∈X×{−1,1} drawn i.i.d. from a…
Two different views on machine learning problem: Applied learning (machine learning with business applications) and Agnostic PAC learning are formalized and compared here. I show that, under some conditions, the theory of PAC Learnable provides a way to solve the Applied learning problem. However, the theory requires t…
Study selective classification with halfspaces, achieving error bounds under Gaussian distributions.
problem Modeling relationships in subsets of data defined by selection rules.
method Sparse linear classifiers for subsets defined by halfspaces, focusing on Gaussian feature distributions.
result First PAC-learning algorithm for homogeneous halfspace selectors with error guarantee $\bigO*{\sqrt{\mathrm{opt}}}$.
This research improves binary classification by balancing overfitting and generalization with a novel Bayesian approach.
problem Improving binary classification models to avoid overfitting and generalize well.
method Introduces a PAC-Bayes type learning rule with a balancing parameter λ to balance training error and KL divergence to a prior.
result A choice of λ ensures uniformly vanishing excess loss, even in the agnostic case, by under-regularizing or over-regularizing appropriately.
New algorithm reduces contextual bandit identification to argmax calls.
problem Best-arm identification in stochastic contextual bandits.
method Instance-optimal PAC algorithm using argmax oracle calls.
result First instance-dependent PAC sample complexity for contextual bandits.
Boosting algorithm reduces error in noisy data.
problem Improving weak learners in the presence of Massart noise.
method First computationally efficient boosting algorithm for Massart noise.
result Achieves misclassification error arbitrarily close to Massart noise threshold.
New learning rules achieve optimal sample complexity for weakly supervised classification.
problem Learning from label proportions in weakly supervised settings.
method Debiased proportional square loss and EasyLLP learning rule.
result Achieves optimal sample complexity in both realizable and agnostic settings.
Study robust online learning with adversarial perturbations.
problem Learning robust classifiers in the presence of adversarial perturbations.
method Formulated as an online learning problem, considered both realizable and agnostic learnability, defined new dimension controlling mistake/regret bounds.
result Showed new dimension controls mistake/regret bounds, generalized to multiclass hypothesis classes.
New findings show modern neural networks have finite sample complexity in o-minimal structures.
problem Understanding the learnability of modern neural networks in a broad context.
method Analyzing feedforward neural networks definable in o-minimal structures.
result Modern neural networks, including MLPs, CNNs, GNNs, and transformers, have finite sample complexity in the agnostic PAC setting.
Unified PAC-Bayesian framework for deep learning generalization.
problem Limitations of existing PAC-Bayesian norm-based bounds for deep neural networks.
method Unified framework using anisotropic Gaussian posteriors and sensitivity matrix.
result Comparable or tighter generalization bounds compared to state-of-the-art approaches.
New bounds show agnostic multiclass learning depends on two dimensions: Natarajan and Daniely-Shalev-Shwartz.
problem Understanding sample complexity in multiclass classification with agnostic learning.
method Developed a novel online procedure based on a self-adaptive multiplicative-weights algorithm.
result Agnostic sample complexity bounds are in the form of DS^(1.5)/ε + Nat/ε^2, nearly tight up to a √DS factor.
New method extends low-rank MDPs to continuous action spaces.
problem Limited applicability of current low-rank MDP methods to continuous action spaces.
method Extending FLAMBE algorithm to continuous action spaces with Hölder smoothness conditions.
result Similar PAC bound achieved for continuous actions with polynomial dependence on smoothness order.
Characterizes the sample complexity of list regression tasks.
problem Understanding the sample complexity of list learning tasks in regression.
method Introducing two combinatorial dimensions: k-OIG dimension and k-fat-shattering dimension.
result These dimensions characterize realizable and agnostic k-list regression.
Unified framework for realizable and agnostic learning.
problem Lack of a unified theory for realizable and agnostic learnability.
method Three-line blackbox reduction.
result Unified understanding across various learning settings.
We provide new results concerning label efficient, polynomial time, passive and active learning of linear separators. We prove that active learning provides an exponential improvement over PAC (passive) learning of homogeneous linear separators under nearly log-concave distributions. Building on this, we provide a comp…
Proves SQ lower bounds for learning two-hidden-layer neural networks.
problem Learning two-hidden-layer ReLU networks with Gaussian inputs.
method Refined lifting procedure to reduce Boolean PAC learning to Gaussian.
result Superpolynomial SQ lower bounds for Gaussian inputs.
Algorithm learns halfspaces in noisy data efficiently.
problem Learning halfspaces with Tsybakov noise.
method Novel semi-definite programming and online convex optimization.
result First non-trivial PAC learning algorithm for Tsybakov noise.
This paper studies universal rates of ERM for binary classification under agnostic learning.
problem The challenge of achieving universal rates of ERM for binary classification under agnostic learning.
method The paper explores the agnostic universal rates of ERM for binary classification, revealing three possible rates: e−n, o(n−1/2), or arbitrarily slow. result The paper provides a complete characterization of which concept classes fall into each of the three categories of agnostic universal rates.
New method learns robustly with less data, bridging theory and practice.
problem Adversarial robust learning with metric perturbation.
method Tolerant adversarial PAC-learning with perturb-and-smooth approach and compression-based algorithm.
result First PAC-type guarantees for popular adversarial learning techniques.
New algorithm learns optimal policies without explicit rewards.
problem Learning optimal policies without explicit rewards.
method Developed a computationally tractable algorithm for reward-free navigation using linear value iteration.
result PAC guarantees on learning near optimal value functions and policies.
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
We analyze MDL for binary classification, quantifying overfitting and underfitting.
problem Understanding the trade-off between underfitting and overfitting in MDL for binary classification.
method Complete characterization of the regularization curve for MDL, extending previous work to all λ. result Precise quantitative description of the worst case limiting error as a function of λ and noise level. New algorithm reduces prediction error in online learning without knowing base measure.
problem Smoothed online learning without knowledge of base measure.
method R-Cover algorithm based on recursive coverings.
result First algorithm to guarantee sublinear regret for agnostic smoothed online learning without prior knowledge of base measure.
New algorithm learns halfspaces with large margins efficiently.
problem Learning halfspaces with large margins in noisy data.
method Proper learning algorithm with tight complexity bounds.
result Nearly tight complexity characterization for α=1.01-approximate learning. An active learner is given a class of models, a large set of unlabeled examples, and the ability to interactively query labels of a subset of these examples; the goal of the learner is to learn a model in the class that fits the data well. Previous theoretical work has rigorously characterized label complexity of activ…
Paper proposes a new approach to Model Selection using a U-curve algorithm.
problem Model Selection in Machine Learning.
method Data-driven, non-exhaustive search of a Learning Space L(H). result Conditions for non-exhaustive search to return optimal solutions.
New algorithm learns efficiently with a simple 'yes/no' oracle.
problem Can efficient learning be achieved with a simpler oracle than ERM?
method Developed an oracle that returns 'yes' or 'no' for realizable datasets.
result Learnability is possible with a polynomial price in VC dimension.
Gradient descent learns a single neuron without knowing the relationship between inputs and labels.
problem Learning a single neuron without knowing the relationship between inputs and labels.
method Using gradient descent to minimize empirical risk over i.i.d. samples, with a nonconvex and nonsmooth optimization problem.
result Gradient descent achieves near-optimal population risk in polynomial time and sample complexity.
Comparative learning combines realizable and agnostic settings for two hypothesis classes, reducing sample complexity.
problem Learning with two hypothesis classes in a more general setting than single hypothesis classes.
method Introduces comparative learning, defines mutual VC dimension and Littlestone dimension, and applies insights to multiaccuracy and multicalibration.
result Sample complexity of comparative learning is characterized by mutual VC dimension and Littlestone dimension.
PAC learning simplified as bipartite matching.
problem Efficiently solving PAC learning problems.
method Transductive learning and one-inclusion graphs.
result PAC learning can be reduced to bipartite matching.