We present a study of generalization for data-dependent hypothesis sets. We give a general learning guarantee for data-dependent hypothesis sets based on a notion of transductive Rademacher complexity. Our main result is a generalization bound for data-dependent hypothesis sets expressed in terms of a notion of hypothe…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “data-dependent hypothesis sets”
PAC-Bayesian theory applied to data-dependent hypothesis sets yields uniform generalization bounds.
problem Proving uniform generalization bounds for data-dependent hypothesis sets.
method Applying PAC-Bayesian framework on 'random sets' and considering data-dependent hypothesis sets.
result Data-dependent uniform generalization bounds are proven, providing tighter and unified results.
Survey on new data-dependent bounds for neural networks.
problem Generalization of overparameterized neural networks.
method Extending PAC-Bayesian theory, refining complexity terms, and replacing information-theoretic terms with stability assumptions.
result Unified template inequality and comparison of bounds.
Analyzes the complexity of linear hypothesis sets using Rademacher complexity.
problem Understanding the complexity of linear hypothesis sets for various norms.
method Tight analysis of empirical Rademacher complexity for linear hypothesis classes with bounded weights.
result Improved bounds on Rademacher complexity for linear hypothesis sets, matching or improving existing results.
A new learning method uses data to learn from large model sets.
problem Learning with large sets of candidate models where uniform convergence is hard.
method Data-dependent learning that incorporates empirical data less reliant on prior assumptions.
result Demonstrates improved generalization in various learning assumptions.
Improved pruning method finds winning neural network subnetworks.
problem Finding a small subnetwork that performs as well as a full neural network.
method Data-dependent pruning criterion using gradient of training loss.
result Data-dependent pruning improves existing pruning algorithms.
New bounds explain modern machine learning algorithms' generalization.
problem Explaining generalization behavior of modern machine learning algorithms.
method Proposes a new complexity measure based on empirical Rademacher complexity of an algorithm- and data-dependent hypothesis class.
result Obtains novel bounds with finite fractal dimension, simplifies proofs, and recovers known results.
The paper shows robustness and generalization are closely connected via data-dependent bounds.
problem Connecting robustness and generalization in machine learning.
method Data-dependent generalization bounds that reduce dependence on covering number and hypothesis space.
result Proves robustness implies generalization, with near-exponential improvements in various situations.
Develops bounds for deep learning risk via Hilbert coresets.
problem Risk estimation for complex deep learning models.
method Hilbert coreset approach for transductive risk bounds.
result Effective and meaningful bounds for deep neural networks.
This work proves generalization bounds for neural networks without Lipschitz assumptions.
problem Proving generalization guarantees for neural networks without Lipschitz continuity.
method Introduces a data-dependent fractal dimension and uses it to prove generalization bounds.
result Generalization bounds are proven without requiring Lipschitz continuity.
Paper introduces new bounds linking data compressibility to generalization error.
problem Establishing data-dependent generalization bounds.
method Variable-size compressibility framework linking generalization error to compression rate of input data.
result New bounds depend on empirical data measure, subsuming existing PAC-Bayes and intrinsic dimension bounds.
Study expands multiclass classification models with new rates and partial concept classes.
problem Multiclass classification with a bounded number of labels under various conditions.
method Extends traditional PAC model to distribution-dependent and data-dependent learning rates, characterizes optimal rates for universal and partial concept classes.
result Characterizes three types of learning rates (exponential, linear, arbitrarily slow) for fixed distributions and complexity measures for partial concept classes.
New method certifies deep graph classifiers with tighter risk bounds.
problem Certifying the reliability of deep graph classifiers.
method Linearized deep assignment flows with random initial conditions, using PAC-Bayes risk certification.
result Computes tighter out-of-sample risk certificates efficiently.
We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound's p…
LOO prediction method improves generalization guarantees for arbitrary datasets.
problem Understanding LOO error guarantees in fully transductive settings for arbitrary datasets.
method Median of Level-Set Aggregation (MLSA) for empirical-risk level sets.
result Multiplicative oracle inequality for LOO error with complexity scaling.
Recently, the \textit{Tensor Nuclear Norm~(TNN)} regularization based on t-SVD has been widely used in various low tubal-rank tensor recovery tasks. However, these models usually require smooth change of data along the third dimension to ensure their low rank structures. In this paper, we propose a new definition of da…
aLTT selects hyperparameters efficiently with statistical guarantees.
problem Statistical validity and efficiency in hyperparameter selection.
method Sequential data-dependent multiple hypothesis testing with early termination.
result Reduces testing rounds while maintaining statistical validity.
The paper analyzes prediction error in nonstationary settings using weighted risk minimization.
problem Prediction under distribution drift and nonstationary conditions.
method General decomposition of excess risk into learning and drift terms, proving oracle inequalities under mixing conditions.
result Oracle inequalities for the learning error, providing bounds that hold uniformly over arbitrary weight classes.
The paper improves PAC-Bayes bounds for data-dependent predictors.
problem Guaranteeing the quality of predictions on unseen examples.
method Basic PAC-Bayes inequality for stochastic kernels, leading to various bounds.
result Validates PAC-Bayes bounds without fixed 'data-free' priors and bounded losses.
Feed-forward neural networks can be understood as a combination of an intermediate representation and a linear hypothesis. While most previous works aim to diversify the representations, we explore the complementary direction by performing an adaptive and data-dependent regularization motivated by the empirical Bayes m…
Optimal kernel in KR can be data-dependent, improving model performance.
problem Fixed kernel in KR limits model performance.
method Considered data-dependent kernels for KR, using posterior covariance.
result Data-dependent kernel choice leads to optimal performance.
We develop a novel family of algorithms for the online learning setting with regret against any data sequence bounded by the empirical Rademacher complexity of that sequence. To develop a general theory of when this type of adaptive regret bound is achievable we establish a connection to the theory of decoupling inequa…
Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data. Proving tight generalization error bounds is a central question in statistical learning theory. In this paper, we obtain generalization error bounds for learnin…
Reweighting improves risk bounds in certain data regions.
problem Improving risk bounds in classification and heteroscedastic regression.
method Weighted empirical risk minimization with a data-dependent weight function.
result A weighted ERM estimator can achieve superior performance in specific sub-regions.
Unified framework controls false discovery rate in bandit multiple testing.
problem Designing adaptive algorithms to identify true discoveries in multiple hypothesis testing.
method Unified modular framework using e-processes for FDR control in arbitrary settings.
result Unified framework ensures FDR control for dependent and simultaneous arm queries.
In this paper we analyze a budgeted learning setting, in which the learner can only choose and observe a small subset of the attributes of each training example. We develop efficient algorithms for ridge and lasso linear regression, which utilize the geometry of the data by a novel data-dependent sampling scheme. When …
In recent studies, the generalization properties for distributed learning and random features assumed the existence of the target concept over the hypothesis space. However, this strict condition is not applicable to the more common non-attainable case. In this paper, using refined proof techniques, we first extend the…
We reformulate data-dependent constraints to ensure they are always met with high probability.
problem Ensuring fairness and stability in machine learning models with data-dependent constraints.
method Calibrated reformulation of constraints to guarantee satisfaction with a specified probability.
result Our method guarantees that fairness constraints are met at test time with high probability.
Kernel methods offer the flexibility to learn complex relationships in modern, large data sets while enjoying strong theoretical guarantees on quality. Unfortunately, these methods typically require cubic running time in the data set size, a prohibitive cost in the large-data setting. Random feature maps (RFMs) and the…
Fast robust subspace tracking in sparse data-dependent noise with near-optimal delay.
problem Robustly tracking time-varying subspaces in the presence of sparse outliers.
method Introduces a fast mini-batch robust ST solution under mild assumptions.
result Provably correct subspace tracking with near-optimal delay and same time complexity as simple PCA.
New bounds use IPMs to improve generalization in machine learning.
problem Improving generalization bounds in machine learning.
method PAC-Bayes bounds with Integral Probability Metrics (IPM).
result Natural interpolation between worst-case and favorable cases.
A key learning scenario in large-scale applications is that of federated learning, where a centralized model is trained based on data originating from a large number of clients. We argue that, with the existing training and inference, federated models can be biased towards different clients. Instead, we propose a new f…
The Gibbs algorithm's generalization error is bounded, improving with prior volume in low temperatures.
problem Bounding the generalization error of the Gibbs algorithm in low temperature regimes.
method Analyzes the Gibbs algorithm's performance, extending known high-temperature bounds to low-temperature scenarios.
result With high probability, the generalization error decreases with the total prior volume of similar hypotheses.
New decentralized KRR algorithm adapts to node-specific data.
problem Consistent node-specific data in decentralized KRR.
method Data-dependent random features for adaptive RF generation.
result Average regression accuracy improved by 25.5% across six datasets.
Study on using random subspaces for ERM with various loss functions.
problem Improving learning accuracy with computational savings from random subspaces.
method Random subspaces of a hypothesis space, considering data-dependent subspaces.
result Unified analysis showing computational efficiency can be improved without performance loss.
Paper introduces data-dependent SSP for private linear and logistic regression.
problem Private linear and logistic regression with better performance.
method Data-dependent sufficient statistic perturbation (SSP) for linear and logistic regression.
result Data-dependent SSP outperforms state-of-the-art methods for linear and logistic regression.
In this dissertation we propose alternative analysis of distributed stochastic gradient descent (SGD) algorithms that rely on spectral properties of the data covariance. As a consequence we can relate questions pertaining to speedups and convergence rates for distributed SGD to the data distribution instead of the regu…
We improve adversarial robustness calibration analysis for broader hypothesis sets.
problem Improving calibration for adversarial robustness in machine learning.
method A finer definition of calibration for adversarial robustness.
result Our results cover most common hypothesis sets in machine learning.
In this paper, we consider the problem of recovering a graph that represents the statistical data dependency among nodes for a set of data samples generated by nodes, which provides the basic structure to perform an inference task, such as MAP (maximum a posteriori). This problem is referred to as structure learning. W…
Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priorsstat.ML
The study improves representation learning bounds using data-dependent Gaussian mixtures.
problem Improving generalization in representation learning.
method Established bounds using relative entropy and MDL of latent variables.
result The approach significantly improves generalization over existing methods.
The randomized-feature approach has been successfully employed in large-scale kernel approximation and supervised learning. The distribution from which the random features are drawn impacts the number of features required to efficiently perform a learning task. Recently, it has been shown that employing data-dependent …
Meta-learning bounds derived using PAC-Bayes theory for improved generalization.
problem Uncertainty in generalization performance for meta-learning with new tasks.
method PAC-Bayes relative entropy bounds and empirical risk minimization (ERM) method.
result Competitive generalization performance and rapid convergence with data-dependent prior.
The Probably Approximately Correct (PAC) Bayes framework (McAllester, 1999) can incorporate knowledge about the learning algorithm and (data) distribution through the use of distribution-dependent priors, yielding tighter generalization bounds on data-dependent posteriors. Using this flexibility, however, is difficult,…
Research on predicting with lists of labels, characterizing learnability and providing algorithms.
problem Multiclass online prediction with multiple labels.
method Characterization using -ary Littlestone dimension, adaptation of classical algorithms, combinatorial results.
result Achievement of negative regret in some scenarios, complete characterization of learnability.
Comparative learning combines realizable and agnostic settings for two hypothesis classes, reducing sample complexity.
problem Learning with two hypothesis classes in a more general setting than single hypothesis classes.
method Introduces comparative learning, defines mutual VC dimension and Littlestone dimension, and applies insights to multiaccuracy and multicalibration.
result Sample complexity of comparative learning is characterized by mutual VC dimension and Littlestone dimension.
We study Principal Component Analysis (PCA) in a setting where a part of the corrupting noise is data-dependent and, as a result, the noise and the true data are correlated. Under a bounded-ness assumption on the true data and the noise, and a simple assumption on data-noise correlation, we obtain a nearly optimal samp…
Algorithm provides online learning guarantees against general comparators in full and bandit feedback.
problem Adversarial online learning with data-dependent regret guarantees.
method Completely online algorithm with data-dependent regret guarantees for full and bandit feedback.
result Algorithm achieves expected performance against arbitrary comparator sequences in full and bandit feedback settings.
New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.
problem Achieving best-of-both-worlds guarantees with data-dependent regret bounds in MDPs with unknown transitions.
method Optimistic follow-the-regularized-leader algorithm with new optimistic Q-function estimators and transition bonus.
result First-order, second-order, and path-length bounds with polylog(T) regret in the stochastic regime.