Formalizes weak and strong verification for LLMs, controlling errors without assumptions.
problem Balancing cost and reliability in reasoning with LLMs.
method Formalizes weak-strong verification policies, introduces metrics, develops online algorithm.
result Optimal policies admit a two-threshold structure, and calibration and sharpness govern value of weak verifiers.
The study explores the strengths and weaknesses of models that generalize from weak to strong supervision.
problem Understanding the limitations and capabilities of models that generalize from weak to strong supervision.
method Theoretical analysis and experimental validation in both classification and regression settings.
result Theoretical bounds reveal the importance of strong generalization and calibration of the weak model and a careful balance in the training process.
Improved machine learning models outperform their simpler counterparts by using imperfect labels.
problem Improving model performance using imperfect labels.
method Random feature ridge regression (RFRR) with a deterministic equivalent for excess test error.
result The student model can outperform the teacher model regardless of the teacher's scaling law, achieving the minimax optimal rate.
New theory explains how strong models can learn from weak ones.
problem Learning from weak, incomplete, or incorrect labels.
method New bounds based on data distribution and student hypothesis class.
result Existing weak supervision theory fails to account for pseudolabel correction and coverage expansion.
CB-SLICE identifies concept-based error slices in deep learning models.
problem Systematic errors in deep learning models on specific groups.
method Concept Bottleneck Models (CBMs) and concept representations.
result CB-SLICE outperforms state-of-the-art methods in error slice identification.
A new adaptive splitting method improves accuracy for Cox-Ingersoll-Ross model.
problem Improving numerical solution accuracy for Cox-Ingersoll-Ross model.
method Adaptive splitting method over deterministic and random meshes, with uniform moment bound and strong error results.
result Uniform moment bound and strong error results of order 1/4 in L1 and L2 for κθ>σ^2, and order 1 for large noise.
We establish the first nonasymptotic error bounds for Kaplan-Meier-based nearest neighbor and kernel survival probability estimators where feature vectors reside in metric spaces. Our bounds imply rates of strong consistency for these nonparametric estimators and, up to a log factor, match an existing lower bound for c…
We consider the approximation of stochastic differential equations (SDEs) with non-Lipschitz drift or diffusion coefficients. We present a modified explicit Euler-Maruyama discretisation scheme that allows us to prove strong convergence, with a rate. Under some regularity and integrability conditions, we obtain the opt…
Although kernel methods are widely used in many learning problems, they have poor scalability to large datasets. To address this problem, sketching and stochastic gradient methods are the most commonly used techniques to derive efficient large-scale learning algorithms. In this study, we consider solving a binary class…
Analyzes deep neural networks training errors with SGD and random init.
problem Lack of rigorous understanding of deep learning algorithms.
method Mathematical analysis of deep learning with SGD and random init.
result First full error analysis for deep learning with SGD and random init.
New tests for identifying the number of latent factors in short panels with small time dimensions.
problem Determining the number of latent factors in short panels with small time dimensions.
method Eigenvalue tests based on variance-covariance matrices of asset returns, with assumptions on spherical errors or instrumental variables for factor betas.
result Established asymptotic distributional results and proposed a novel statistical test for weak factors.
Boosting improves accuracy by combining weak learners into a voting classifier.
problem Boosting's theoretical performance is sub-optimal, especially for voting classifiers.
method Proposes a randomized boosting algorithm that outputs voting classifiers with a single logarithmic dependency on sample size.
result Randomized boosting achieves a generalization error with a single logarithmic dependency on the sample size.
Finding biologically plausible alternatives to back-propagation of errors is a fundamentally important challenge in artificial neural network research. In this paper, we propose a learning algorithm called error-driven Local Representation Alignment (LRA-E), which has strong connections to predictive coding, a theory t…
New method improves optimization and DP in FL.
problem Combining strong DP and optimization in FL.
method Combining clipping, momentum, and error feedback.
result Optimal convergence rate and near optimal DP guarantees.
Self-training improves weak classifiers in mixture models.
problem Improving weak classifiers in mixture models.
method Iterative self-training algorithm using pseudolabels and unlabeled data.
result Self-training converts weak learners to strong learners in mixture models.
While active learning offers potential cost savings, the actual data efficiency---the reduction in amount of labeled data needed to obtain the same error rate---observed in practice is mixed. This paper poses a basic question: when is active learning actually helpful? We provide an answer for logistic regression with t…
New algorithm optimally evaluates policies with linear approximations.
problem Policy evaluation with linear function approximation.
method Accelerated, variance-reduced fast temporal difference algorithm (VRFTD).
result VRFTD matches both deterministic and stochastic lower bounds.
A contraction analysis improves model-based RL's error recovery.
problem Theoretical understanding of model-based reinforcement learning.
method Contraction analysis applied to both stochastic and deterministic state transitions.
result Error reduction in cumulative reward using branched rollouts.
Improves test set performance and reduces out-of-sample disappointment for unstable models.
problem Ensuring strong test set performance via cross-validation for unstable models.
method Nested k-fold cross-validation with hyperparameter selection based on a weighted sum of cross-validation metric and model stability measure.
result Improves out-of-sample MSE for sparse ridge regression and CART by 4% and 2% respectively, compared to k-fold cross-validation.
Paper proposes deep neural networks for nonparametric regression from dependent data.
problem Nonparametric regression from strongly mixing observations.
method Minimum error entropy principle applied to deep neural networks.
result Deep neural networks achieve minimax optimal convergence rates for Gaussian errors.
Bayesian sequence prediction is a simple technique for predicting future symbols sampled from an unknown measure on infinite sequences over a countable alphabet. While strong bounds on the expected cumulative error are known, there are only limited results on the distribution of this error. We prove tight high-probabil…
W2S FT often outperforms weak teachers due to low intrinsic dimensionality.
problem Understanding why weak-to-strong finetuning outperforms weak models.
method Analyzing W2S in ridgeless regression setting, focusing on variance reduction.
result Weak teacher's variance is inherited by strong student in shared feature subspace, reduced in discrepancy subspace.
New ensemble SVM model reduces prediction error without choosing best kernel.
problem Reducing prediction error in regression problems.
method Bagged-weighted support vector regression model with random machines.
result Regression Random Machines achieve lower generalization error.
SGD-trained deep nets often generalize well due to a strong inductive bias towards low-error, low-complexity functions.
problem Understanding why overparameterized deep nets generalize well despite fitting training data perfectly.
method Empirical investigation of PSGD(f∣S) and PB(f∣S) for various architectures and datasets. result The probability of SGD-converging on a function consistent with training data correlates well with the Bayesian posterior probability of expressing that function.
Bayesian Additive Regression Trees (BART) is a fully Bayesian approach to modeling with ensembles of trees. BART can uncover complex regression functions with high dimensional regressors in a fairly automatic way and provide Bayesian quantification of the uncertainty through the posterior. However, BART assumes IID nor…
New algorithms improve community detection in network data with strong consistency.
problem Challenges in effectively adapting spectral clustering techniques and achieving strong consistency in label recovery.
method Proposed Thresholded Cosine Spectral Clustering (TCSC) and one-step Refined TCSC algorithms, with strong consistency proofs.
result One-step Refined TCSC achieves strong consistency in community detection under PABM, correctly recovering all labels with high probability.
This article analyzes the weak error of SGD optimization schemes.
problem Analyzing the error in SGD optimization schemes with respect to a test function.
method Weak error analysis for SGD type optimization schemes.
result The weak error decays at the same speed as in the strong sense.
New findings show privacy affects generalization error in a non-monotonic way.
problem Privacy and robustness in distributed learning.
method Theoretical analysis and matching lower/upper bounds on algorithmic stability.
result Generalization error is non-monotonically affected by privacy, depending on noise level.
We derive error estimates for multinomial approximations of American options in a multidimensional jump--diffusion Merton's model. We assume that the payoffs are Markovian and satisfy Lipschitz type conditions. Error estimates for such type of approximations were not obtained before. Our main tool is the strong approxi…
Bayesian framework tackles measurement error in covariates.
problem Misleading inference due to corrupted covariates.
method Bayesian Nonparametric Learning framework robust to misspecification.
result General framework for Classical and Berkson error models.
We justify and give error estimates for binomial approximations of game (Israeli) options in the Black--Scholes market with Lipschitz continuous path dependent payoffs which are new also for usual American style options. We show also that rational (optimal) exercise times and hedging self-financing portfolios of binomi…
End-to-end ASR error detection using audio-transcript entailment.
problem Detecting transcription errors in ASR systems to prevent error propagation.
method Proposes a novel end-to-end approach using audio-transcript entailment, with acoustic and linguistic encoders.
result Achieves CER of 26.2% on all transcription errors and 23% on medical errors specifically, improving by 12% and 15.4% respectively over a strong baseline.
This paper investigates tradeoffs among optimization errors, statistical rates of convergence and the effect of heavy-tailed errors for high-dimensional robust regression with nonconvex regularization. When the additive errors in linear models have only bounded second moment, we show that iteratively reweighted $\ell_1…
K-fold cross-validation (CV) with squared error loss is widely used for evaluating predictive models, especially when strong distributional assumptions cannot be taken. However, CV with squared error loss is not free from distributional assumptions, in particular in cases involving non-i.i.d. data. This paper analyzes …
Spectral feature learning improves IV regression for causal effect estimation.
problem Estimating causal effects in the presence of hidden confounders.
method Two-stage least squares estimator based on spectral features.
result Performance of the method depends on strong spectral alignment and slow eigenvalue decay.
In this paper, we are interested in the strong convergence properties of the Ninomiya-Victoir scheme which is known to exhibit weak convergence with order 2. We prove strong convergence with order 1/2. This study is aimed at analysing the use of this scheme either at each level or only at the finest level of a multil…
The paper reveals three mechanisms for weak-to-strong generalization.
problem Understanding the mechanisms behind weak-to-strong generalization in imperfect labeling scenarios.
method Theoretical analysis of simple models including ridge regression and weighted ridge regression, and a nonlinear multi-index setting.
result A student model can compensate for a teacher's under-regularization and achieve lower test error.
An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses…
Strong inductive biases prevent harmless interpolation in overparameterized models.
problem Understanding the conditions under which overparameterized models can interpolate noise without overfitting.
method Theoretical analysis of high-dimensional kernel regression and deep neural networks, focusing on the role of inductive biases.
result The strength of an estimator's inductive bias determines whether interpolation is harmless or requires fitting noise for good generalization.
We study high-dimensional asymptotic performance limits of binary supervised classification problems where the class conditional densities are Gaussian with unknown means and covariances and the number of signal dimensions scales faster than the number of labeled training samples. We show that the Bayes error, namely t…
Study provides error estimates for approximating game options with diffusion asset prices.
problem Approximating fair prices of game options with diffusion asset prices.
method Error estimates for discrete approximations of diffusion processes, applied to game options.
result Effective tool for computing fair prices of game options in multi-asset markets.
New algorithm solves saddle point problems in Banach spaces.
problem Solving saddle point problems in real reflexive Banach spaces.
method Stochastic Bregman Primal-Dual Splitting Algorithm with relative smoothness and strong convexity assumptions.
result Almost sure convergence to saddle points under various conditions.
In high dimensions, most machine learning methods are brittle to even a small fraction of structured outliers. To address this, we introduce a new meta-algorithm that can take in a base learner such as least squares or stochastic gradient descent, and harden the learner to be resistant to outliers. Our method, Sever, p…
Develops algorithms for multi-class Neyman-Pearson classification with cost sensitivity.
problem Asymmetric misclassification costs in multi-class classification problems.
method Establishes connection with cost-sensitive learning, proposes two algorithms, extends NP oracle properties.
result Proposes algorithms with theoretical guarantees for multi-class Neyman-Pearson classification.
ECN framework improves training on noisy structured labels.
problem Structured errors in fine-grained annotations lead to biased models.
method Error-Correcting Networks (ECN) framework.
result ECN improves fine-grained annotation prediction.
Study linear regression with missing or corrupted data, showing error bounds.
problem Linear regression under missing or corrupted data.
method Information-theoretic lower bounds and efficient algorithms.
result Error bounds match in missing and corruption settings.
Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itse…
We show that the sets in a family with finite VC dimension can be uniformly approximated within a given error by a finite partition. Immediate corollaries include the fact that VC classes have finite bracketing numbers, satisfy uniform laws of averages under strong dependence, and exhibit uniform mixing. Our results ar…