RAVEN improves weak-to-strong generalization under distribution shifts.
problem Weak models fail to supervise strong models effectively under distribution shifts.
method RAVEN dynamically learns optimal combinations of weak models and strong model parameters.
result RAVEN outperforms existing methods by over 30% on out-of-distribution tasks.
Weak labels can significantly speed up learning for strong tasks.
problem Learning with limited strong labels.
method Using weak labels to accelerate learning of strong tasks.
result Weak labels can accelerate learning to O(icefrac1n) rate. New model shows weak teachers can help strong students learn even with imperfect labels.
problem Improving strong student's performance with weak teacher's imperfect pseudolabels.
method Stylized overparameterized spiked covariance model with Gaussian covariates, proving two phases of generalization.
result Provable successful and random guessing phases of strong student's generalization.
Optimal algorithm converts weak to strong learner with less data.
problem Constructing a strong learner from a weak learner with minimal data.
method New algorithm that uses less training data than AdaBoost.
result Optimal sample complexity for converting weak to strong learner.
New mechanism detects overlap density for weak-to-strong generalization.
problem Understanding what aspects of data enable weak-to-strong generalization.
method Data-centric mechanism and overlap detection algorithm.
result Overlap density is a key factor in weak-to-strong generalization.
New framework transfers latent knowledge from weak to strong models.
problem Aligning superhuman LLMs with human feedback.
method Transfer learning framework using refinement approach.
result Proves weak-to-strong generalization is possible.
New theory explains how strong models can learn from weak ones.
problem Learning from weak, incomplete, or incorrect labels.
method New bounds based on data distribution and student hypothesis class.
result Existing weak supervision theory fails to account for pseudolabel correction and coverage expansion.
Random feature models can outperform a weak teacher with early stopping.
problem Generalization from a weak to a strong model in random feature networks.
method Random feature models, early stopping, proving weak-to-strong generalization.
result Random feature models can outperform a weak teacher with early stopping.
Boosting weak learners to strong ones from aggregate labels is possible for LLP but not for MIL.
problem Boosting weak learners to strong ones from aggregate labels in learning from label proportions (LLP).
method Using a weak learner on large enough bags to obtain a strong learner for small bags in polynomial time.
result Boosting is possible for LLP but not for MIL.
Paper analyzes weak-to-strong generalization in CNNs, identifying data-scarce and data-abundant regimes.
problem Weak-to-strong generalization in CNNs trained on weak models.
method Formal analysis of gradient descent dynamics in data-scarce and data-abundant regimes.
result Identifies two regimes and distinct mechanisms of generalization in each.
An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses…
Study shows how a strong model can learn a task's feature while retaining other capabilities.
problem How to align superhuman AI systems using weak-to-strong generalization.
method Two-layer neural networks, reward-model learning, multi-step SGD, feature learning.
result The strong model efficiently learns task features while retaining general capabilities.
Formalizes weak and strong verification for LLMs, controlling errors without assumptions.
problem Balancing cost and reliability in reasoning with LLMs.
method Formalizes weak-strong verification policies, introduces metrics, develops online algorithm.
result Optimal policies admit a two-threshold structure, and calibration and sharpness govern value of weak verifiers.
The study explores the strengths and weaknesses of models that generalize from weak to strong supervision.
problem Understanding the limitations and capabilities of models that generalize from weak to strong supervision.
method Theoretical analysis and experimental validation in both classification and regression settings.
result Theoretical bounds reveal the importance of strong generalization and calibration of the weak model and a careful balance in the training process.
Boosts weak online learners to strong ones with sublinear regret.
problem Online learning agnostic setting without strong guarantees.
method Reduction to online convex optimization, boosting via marginally-better-than-trivial regret guarantees.
result First agnostic online boosting algorithm with sublinear regret.
Improved machine learning models outperform their simpler counterparts by using imperfect labels.
problem Improving model performance using imperfect labels.
method Random feature ridge regression (RFRR) with a deterministic equivalent for excess test error.
result The student model can outperform the teacher model regardless of the teacher's scaling law, achieving the minimax optimal rate.
The study examines different types of equilibria for stopping problems in one-dimensional diffusion processes.
problem Characterizing and comparing different types of equilibria for time-inconsistent stopping problems.
method Analyzes log sub-additive discount functions and one-dimensional diffusion processes to derive necessary and sufficient conditions for weak equilibria and other types of equilibria.
result Conditions for weak equilibria and their implications for other types of equilibria are provided.
Boosting improves accuracy with fewer calls to weak learners for certain concept classes.
problem Improving accuracy of learning algorithms with limited weak learner calls.
method Combines boosting and list-decodable codes to achieve better performance for specific concept classes.
result A new boosting algorithm that achieves strong learning with fewer calls to weak learners and additional samples.
W2S FT often outperforms weak teachers due to low intrinsic dimensionality.
problem Understanding why weak-to-strong finetuning outperforms weak models.
method Analyzing W2S in ridgeless regression setting, focusing on variance reduction.
result Weak teacher's variance is inherited by strong student in shared feature subspace, reduced in discrepancy subspace.
The paper defines a new equivalence relation for knot projections and finds an infinite number of distinct classes.
problem Classifying knot projections based on weak homotopy equivalence.
method Defining weak (1, 2, 3) homotopy and using it to find an invariant.
result There are an infinite number of weak (1, 2, 3) homotopy equivalence classes of knot projections.
The paper reveals three mechanisms for weak-to-strong generalization.
problem Understanding the mechanisms behind weak-to-strong generalization in imperfect labeling scenarios.
method Theoretical analysis of simple models including ridge regression and weighted ridge regression, and a nonlinear multi-index setting.
result A student model can compensate for a teacher's under-regularization and achieve lower test error.
Online boosting method improves weak to strong learner.
problem Online learning of weak to strong learner.
method Extends batch GentleAdaBoost to online approach with line search.
result Online boosting performs better than other methods.
A new definition of continuous-time equilibrium controls is introduced. As opposed to the standard definition, which involves a derivative-type operation, the new definition parallels how a discrete-time equilibrium is defined, and allows for unambiguous economic interpretation. The terms "strong equilibria" and "weak …
Boosting improves accuracy by combining weak learners into a voting classifier.
problem Boosting's theoretical performance is sub-optimal, especially for voting classifiers.
method Proposes a randomized boosting algorithm that outputs voting classifiers with a single logarithmic dependency on sample size.
result Randomized boosting achieves a generalization error with a single logarithmic dependency on the sample size.
Weak supervision challenges black-box models, suggesting fusion of modeling cultures.
problem Challenges of strong supervision in achieving accurate predictions.
method Integrating data modeling into algorithmic modeling for weak supervision.
result Integration of data modeling culture improves model stability and accuracy.
New homotopy types and invariants defined for knots.
problem Defining and characterizing different homotopy types of knot projections.
method Introducing strong and weak (1, 2) homotopies and defining new invariants.
result New necessary and sufficient conditions for homotopy equivalence of knot projections.
Large learning rates cause oscillations in NN weights that improve generalization.
problem Improving generalization of neural networks trained with large learning rates.
method Theoretical analysis and feature-noise data generation model.
result Oscillating SGD with large learning rates benefits NN generalization by effectively learning weak features.
We propose a projected semi-stochastic gradient descent method with mini-batch for improving both the theoretical complexity and practical performance of the general stochastic gradient descent method (SGD). We are able to prove linear convergence under weak strong convexity assumption. This requires no strong convexit…
Proves inextendibility of weak null singularities from curvature blow-up.
problem Inextendibility of weak null singularities in the context of curvature blow-up.
method Introduces a new strategy to infer Cloc0,1-inextendibility from curvature blow-up. result Expected to contribute to the resolution of strong cosmic censorship conjecture.
The paper defines new homotopy relations on knot projections and classifies certain knot types.
problem Defining and classifying knot homotopy relations.
method Introducing cross chord numbers and using them to define strong and weak (1, 3) homotopies.
result Complete classification of knot projections with trivializing number two.
New research shows label refinement and weak training have limitations for aligning LLMs.
problem Limitations of refinement methods for aligning large language models.
method Analyzed probabilistic assumptions and alternative approaches to label refinement and weak training.
result Label refinement and weak training suffer from irreducible error, leaving a performance gap.
Studied SGD convergence under weak conditions.
problem Convergence of SGD in nonconvex optimization.
method Analyzed biased nonconvex SGD under mild conditions.
result Provided convergence rates and complexities.
New varifold solutions for mean curvature flow converge and are unique.
problem Mean curvature flow and Allen-Cahn equation convergence and uniqueness.
method Evolving varifolds coupled to phase volumes, weak-strong uniqueness principle.
result Limits of Allen-Cahn solutions are varifold solutions, and classical flows are unique.
We propose a weak formulation for the binormal curvature flow of curves in R3. This formulation is sufficiently broad to consider integral currents as initial data, and sufficiently strong for the weak-strong uniqueness property to hold, as long as self-intersections do not occur. We also prove a global existence t…
Boosting is a popular way to derive powerful learners from simpler hypothesis classes. Following previous work (Mason et al., 1999; Friedman, 2000) on general boosting frameworks, we analyze gradient-based descent algorithms for boosting with respect to any convex objective and introduce a new measure of weak learner p…
Self-training improves weak classifiers in mixture models.
problem Improving weak classifiers in mixture models.
method Iterative self-training algorithm using pseudolabels and unlabeled data.
result Self-training converts weak learners to strong learners in mixture models.
The strong maximum principle is proved to hold for weak (in the sense of support functions) sub- and super-solutions to a class of quasi-linear elliptic equations that includes the mean curvature equation for C0 spacelike hypersurfaces in a Lorentzian manifold. As one application a Lorentzian warped product splittin…
Sharp analysis of knowledge distillation for high-dimensional regression.
problem Characterizing the risk of target models in high-dimensional settings.
method Sharp non-asymptotic bounds for ridgeless regression under model and distribution shifts.
result Identifies optimal surrogate models and reveals benefits and limitations of discarding weak features.
Unified approach for sample aggregation in transfer learning across various divergence measures.
problem Optimizing sample aggregation from source to target distributions for improved target performance.
method Unified algorithmic approach that adapts to multiple divergence measures via a weak modulus of transfer.
result Unified approach achieves near optimal rates in terms of the unknown strong modulus, applicable in more general settings.
Study shows offline RL with partial coverage and weak function classes is possible.
problem Learning optimal policies from logged data with function approximation.
method Marginalized Importance Sampling (MIS) with additional covering distribution.
result Finite-sample guarantees for sample-efficient offline RL for general MDPs.
The paper explores criteria for positivity of forms and proves their strong positivity in specific cases.
problem Understanding positivity of exterior forms on complex vector spaces.
method Dimensionality reduction and criteria based on Hermitian matrices.
result Strong positivity of certain forms proven by duality.
An algorithm learns from multiple models to match an oracle's risk.
problem Learning from multiple noisy models to estimate a target parameter.
method Elimination rounds algorithm for adaptive learning.
result Risk of weak-oracle learner matches that of an oracle in multiple source case.
Paper proves weak unique continuation for harmonic functions on RCD spaces but finds counterexample for strong uniqueness.
problem Unique continuation of harmonic functions on RCD spaces, especially strong uniqueness.
method Establishes weak unique continuation theorem and provides counterexample for strong uniqueness.
result Found counterexample for strong unique continuation in RCD(K,N) spaces for N≥4 and K∈R.
A powerful network teaches a weak one, improving its performance.
problem Improving the performance of a weak neural network using a more powerful one.
method During training, a weak network learns features from a strong network to minimize feature distance.
result A weak neural network can increase its performance by learning from a more powerful network.
Novel weak solutions for volume-preserving mean curvature flow established.
problem Existence and uniqueness of solutions to volume-preserving mean curvature flow.
method Introducing varifold solutions coupled with phase volumes and new calibrations.
result Uniqueness of classical solutions among varifold solutions.
Compactifies Calabi-Yau to weak Fano manifolds.
problem Compactifying Calabi-Yau manifolds to weak Fano manifolds.
method Generalized Tian-Yau construction and asymptotically Calabi metrics.
result Calabi-Yau structure arises from compactification.
Extends boosting to multiclass online agnostic classification.
problem Online multiclass classification with weak learners.
method Reduces multiclass online agnostic boosting to online convex optimization.
result First boosting algorithm for online agnostic multiclass classification.
Study shows tractable generalization in RL is impossible but possible with Strong Proximity.
problem RL agents struggle to generalize to new environments.
method Introduced Weak and Strong Proximity conditions to capture similarity between environments.
result Proved Strong Proximity is sufficient for efficient generalization.