New learnability criteria for non-iid processes equivalent to online learning.
problem Statistical learning under non-iid stochastic processes is underdeveloped.
method Defined two learnability notions and showed their equivalence to online learning.
result Learnability criteria for non-iid processes are equivalent to online learning.
New approach to nonuniform learnability using measure theory.
problem Nonuniform learnability of hypotheses with varying sample sizes.
method Measure theoretic approach to redefine nonuniform learnability, introducing a new algorithm (Generalize Measure Learnability).
result Achieved statistical consistency in learning countable hypothesis classes.
New learnability concept shown independent of ZFC axioms.
problem Learnability of finite subsets on [0,1] interval is undecidable.
method Set-theoretic techniques to prove independence of ZFC.
result EMX learnability of finite subsets on [0,1] is independent of ZFC.
Survey of statistical queries and their applications.
problem Understanding statistical queries and their applications.
method Exploration of statistical queries model, definitions, and connections to learnability.
result Connections to learnability and applications in optimization, evolvability, and differential privacy.
New fair PCA method using streaming algorithms with statistical guarantees.
problem Perform PCA while ensuring fair projected distributions.
method Formulated new notion PAFO learnability, proposed fair noisy power method (FNPM) for memory efficiency.
result First statistical guarantee for fair PCA in streaming setting.
Study shows limitations and possibilities of learning quantum circuit output distributions.
problem Learnability of output distributions of local quantum circuits.
method Investigated within two oracle models: statistical query model and direct sample access model.
result Output distributions of super-logarithmic depth Clifford circuits are not efficiently learnable in the statistical query model.
The paper demonstrates that falsifiability is fundamental to learning. We prove the following theorem for statistical learning and sequential prediction: If a theory is falsifiable then it is learnable -- i.e. admits a strategy that predicts optimally. An analogous result is shown for universal induction.
Unified model for interactive estimation with improved learnability measure.
problem Improving learnability in interactive estimation models.
method Introducing a combinatorial measure (dissimilarity dimension) and a general algorithm with polynomial bounds.
result Unified model subsumes statistical-query learning and structured bandits.
Study reveals how spectral bias affects learnability on real-world data.
problem Understanding how well complex datasets can be learned using kernel methods.
method Use eigenvalues and eigenfunctions from idealized data to reveal spectral bias on real-world data.
result Bound learnability on real-world data using symmetries of realistic kernels.
The paper analyzes the statistical learnability of GAMs using TV regularization.
problem Statistical learnability of generalized additive models with TV regularization.
method Total variation (TV) as a complexity measure for functions in Lmc1(R)-space, and Rademacher complexity analysis. result Generalization error bounds for finite samples are derived, showing tight complexity in terms of m and p. Deep networks can learn functions approximated by shallow networks, but not all functions.
problem The learnability of functions by deep neural networks and the approximation capacity of simpler classes.
method Study the connection between learnability and approximation capacity of functions by deep neural networks and simpler classes.
result A necessary condition for a function to be learnable by deep neural networks is to be approximable by shallow networks.
It is usual to consider data protection and learnability as conflicting objectives. This is not always the case: we show how to jointly control inference --- seen as the attack --- and learnability by a noise-free process that mixes training examples, the Crossover Process (cp). One key point is that the cp~is typicall…
Characterizes statistical complexity of realizable regression in PAC and online learning.
problem Understanding the statistical complexity of realizable regression in both PAC and online learning settings.
method Introduces minimax instance optimal learners, novel and combinatorial dimensions to characterize learnability.
result Characterizes which classes of real-valued predictors are learnable and provides necessary conditions for learnability.
New insights into learning from distributional adversaries and private data.
problem Understanding minimal assumptions for learning and generalization under distributional constraints.
method Generalized smoothness as a characterization of learnability and privacy under distributional adversaries.
result Near complete characterization of families that admit learnability and privacy under distributional adversaries.
New findings show learnable distributions remain learnable even with noisy or adversarial perturbations.
problem Learning from perturbed samples in high-dimensional spaces.
method Developed a perturbation-quantization framework to analyze additive noise and adversarial corruption models.
result Sample compressible families remain learnable even under noisy or adversarial perturbations.
New method generates private synthetic data without domain size dependence.
problem Private synthetic data generation for unbounded query classes.
method Constructs a private synthetic data generator for privately PAC learnable query classes.
result Sample complexity independent of domain size for privately PAC learnable query classes.
Study shows realizable learnability doesn't imply agnostic learnability for distributions.
problem Learnability and robustness of distribution classes.
method Analyzes the relationship between learnability and robustness for distribution learning.
result Realizable learnability does not imply agnostic learnability for distributions.
We consider the problem of sequential prediction and provide tools to study the minimax value of the associated game. Classical statistical learning theory provides several useful complexity measures to study learning with i.i.d. data. Our proposed sequential complexities can be seen as extensions of these measures to …
Study extends learnability equivalence to multi-class and regression, overcoming binary classification limits.
problem Equivalence of online and private learnability in multi-class and regression settings.
method Introduced a novel Littlestone dimension variant and threshold functions for multi-class classification.
result Online learnability implies private learnability in multi-class classification but not in regression.
New complexity measure helps in agnostic reinforcement learning with or without access to MDP dynamics.
problem Understanding the number of rounds needed to learn an ε-suboptimal policy in unknown MDPs.
method Introducing spanning capacity as a new complexity measure and developing POPLER algorithm.
result There is a separation between generative and online access models for agnostic learnability.
Adversarial examples arise from computational constraints in high-dimensional spaces.
problem Why classifiers in high dimensions are vulnerable to adversarial perturbations.
method Proved computational intractability of robust learning in high-dimensional space.
result Adversarial examples are due to computational limitations, not information theory.
Unified framework for realizable and agnostic learning.
problem Lack of a unified theory for realizable and agnostic learnability.
method Three-line blackbox reduction.
result Unified understanding across various learning settings.
Example shows learnable distributions not privately learnable.
problem Learnable distributions under non-private conditions not transferable to differential privacy.
method Example of a distribution class learnable up to constant error in total variation distance but not under differential privacy.
result Contradicts conjecture of Ashtiani on learnability under differential privacy.
Efficiently learns private models using public data.
problem Improving private learning performance with public data.
method Proves computationally efficient algorithms for private learning with public data.
result First computationally efficient algorithms for private learning with public data.
We clarify measurability assumptions in the agnostic PAC learning theorem.
problem Measurability assumptions in the Fundamental Theorem of Statistical Learning.
method Measure-theoretic scrutiny of existing proofs to extract minimal assumptions.
result Sound statement and detailed proof of the Fundamental Theorem in the agnostic setting.
The study defines backdoor detection in ML and proves its infeasibility.
problem Backdoor detection in machine learning systems.
method Formal statistical definition and analysis of feasibility.
result Backdoor detection is impossible except for very small alphabet sizes.
Characterizes learnability of multioutput functions in various settings.
problem Learning multioutput function classes in batch and online settings.
method Characterizes learnability based on single-output restrictions.
result Complete characterization of learnability in multioutput classification and regression.
Research characterizes learnability of multilabel ranking problems.
problem Learnability of multilabel ranking problems with relevance-score feedback.
method Characterizes learnability in batch and online settings for a large family of ranking losses.
result Characterizes two equivalence classes of ranking losses based on learnability.
Study on learnability of deep random networks, showing practical limitations with depth.
problem Learnability of deep random networks with sign activation.
method Theoretical and practical analysis of random deep networks with sign activation.
result Learnability of random deep networks drops exponentially with depth.
Model explains Netflix Prize success with structured correlation.
problem Understanding structured correlation in high-dimensional data.
method Developed a new statistical learning model.
result Characterized learnability in terms of VCNk,k-dimension. Smooth activations enable optimal error rates in neural networks for Sobolev function classes.
problem Achieving optimal approximation and estimation error rates for neural networks in Sobolev function classes.
method Study of neural networks with smooth activations, proving optimal rates via approximation and statistical properties.
result Constant-depth networks with smooth activations achieve optimal rates of approximation and estimation, demonstrating smoothness adaptivity.
Measures sample learnability across DNNs, showing consistency.
problem Estimating the learnability of each sample in a training set.
method Train DNN on training set, aggregate hits and misses over epochs.
result Sample-wise learnability measure is highly correlated across DNN models.
Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
problem Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
method Formal study of PAC learnability of MLP layers with activation sparsity.
result Classes of functions with activation sparsity lead to provable computational and statistical advantages over their non-sparse counterparts.
No single parameter characterizes the learnability of probability distributions.
problem Finding a parameter to characterize the learnability of probability distributions.
method Analyzing various notions of learnability and showing impossibility results.
result No such parameter exists for characterizing learnability of probability distributions.
We consider the fundamental question of learnability of a hypotheses class in the supervised learning setting and in the general learning setting introduced by Vladimir Vapnik. We survey classic results characterizing learnability in term of suitable notions of complexity, as well as more recent results that establish …
New insights into Valiant's learnability model reveal classes learnable with membership queries.
problem Which classes are learnable in Valiant's original model?
method Characterization using poly-size adaptive query-compression schemes and techniques for arbitrary domains.
result Learnability in Valiant's model is sandwiched between PAC and query-less variants, with halfspaces learnable with queries.
Learnable token perturbations boost extrapolation in LLMs.
problem Limited flexibility of current discrete perturbations in large language models.
method Learnable continuous latent vector transformations in embedding space, unbiased estimating equations, stochastic gradient descent optimization.
result Significant gains in out-of-domain settings over state-of-the-art methods.
New approach to linear regression using universal learning.
problem Applying universal learning to linear regression.
method Using a Gaussian error hypothesis class and Predictive Normalized Maximum Likelihood (pNML) solution.
result Linear regression can generalize even with over-parametrized models under certain conditions.
Deep models learn to parse complex language structures from local data patterns.
problem Understanding how deep models parse and represent language structures.
method Introduced tunable probabilistic context-free grammars and a learning algorithm inspired by deep networks.
result Data correlations across scales enable hierarchical language representations.
New algorithm learns regression models privately under growth condition.
problem Private learning of nonparametric regression models.
method Novel filtering procedure to output stable hypotheses for nonparametric function classes.
result Established first nonparametric private learnability guarantee for diverging fat shattering dimensions.
New method optimizes complex models with minimal data, proving global optimality.
problem Optimizing complex models with unknown cost functions and prior distributions.
method Introduces 'coarse learnability' and an iterative MBO algorithm with sample correction.
result Achieves global optimality with polynomial sample complexity.
Bayesian neural networks learn efficiently at infinite width, matching polynomial-width performance.
problem Understanding the inductive bias of infinite-width neural networks.
method Analyzing the reduced entropy and using subsampling techniques.
result The Bayesian mean-field learner generalizes exactly on polynomially-bounded targets.
This work characterizes when a hypothesis class can be k-list learned.
problem Characterizing when a hypothesis class can be k-list learned.
method Introducing the k-DS dimension and proving the equivalence of k-list learnability and the finiteness of the k-DS dimension.
result A hypothesis class is k-list learnable if and only if the k-DS dimension is finite.
Characterizes learnability of forgiving 0-1 loss functions in multiclass settings.
problem Understanding when multiclass learning with forgiving 0-1 loss functions is possible.
method Introduces a new combinatorial dimension based on Natarajan Dimension to determine learnability.
result A hypothesis class is learnable if and only if the Generalized Natarajan Dimension is finite.
Study on approximability and generalization in machine learning.
problem Understanding how approximation affects learning and generalization in machine learning.
method Introducing a notion of sensitivity to analyze the impact of approximation operators on predictors and proving upper bounds on generalization.
result Proven that approximable target concepts are learnable with fewer labelled samples and sufficient unlabelled data.
New findings on depth vs. width in neural networks, showing depth can improve learnability.
problem Understanding the role of depth in neural networks, especially when width is unbounded.
method Analyzing sample complexity for learnability in norm-controlled depth-2 and depth-3 ReLU networks.
result Depth can improve learnability of functions that are otherwise unlearnable with depth-2 networks.
The paper solves open questions in computable PAC learning, providing a complete landscape.
problem Understanding the boundaries and capabilities of computable PAC learning.
method Analyzing and constructing decidable hypothesis classes with different sample complexities and Littlestone dimensions.
result A complete understanding of CPAC learnability, answering open questions and confirming conjectures.
This work proves DP learnability implies online learnability for general classification tasks.
problem Link between differential privacy and online learning for general classification tasks.
method Establishes Ramsey-type theorems for trees to prove DP learnability implies online learnability.
result DP learnability implies online learnability for general classification tasks.