Ridgeless ReLU networks interpolate datasets and extrapolate based on curvature signs.
problem Interpolating and extrapolating 1D datasets with ReLU networks.
method Minimizes ℓ2-norm of weights, extrapolates based on curvature signs. result Ridgeless ReLU interpolants extrapolate as nearest neighbor curvature extrapolation.
Lower bound proves ridgeless regression performs poorly near interpolation threshold.
problem Proving performance of ridgeless regression near interpolation threshold.
method Distribution-independent lower bound for mean squared error in noisy ridgeless linear regression.
result Lower bound implies ridgeless regression performs poorly near interpolation threshold.
Interpolating models can have heavy-tailed risk, leading to rare but severe errors.
problem Interpolating models' tail risk is poorly understood, affecting rare but impactful errors.
method Large-deviation methods to study the fragility of high-dimensional linear interpolators.
result Ridgeless regression exhibits heavy-tailed risk, while ridge-regularized estimators have better tail behavior.
Kernel ridgeless regression with random features shows good generalization without explicit regularization.
problem Generalization of kernel ridgeless regression without explicit regularization.
method Investigation of ridgeless regression with random features and stochastic gradient descent, exploring the effect of random features error and spectral density optimization.
result Random features error exhibits the double-descent curve, leading to improved generalization.
Study on ridgeless interpolation in high-dimensional regression models.
problem Understanding interpolation in high-dimensional least squares regression.
method Analyzes ridgeless interpolation in two models: linear and neural network.
result Reveals double descent behavior and benefits of overparametrization.
In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for…
Study shows that ridgeless Gaussian kernel regression overfits even with varying bandwidth or dimensionality.
problem Analyzing overfitting in Gaussian kernel ridgeless regression with varying bandwidth or dimensionality.
method Examined the behavior of minimum norm interpolating solutions for fixed and increasing dimensions under varying bandwidth and sample size.
result Ridgeless solutions are never consistent and can be worse than null predictor with large enough noise, even with varying bandwidth or dimensionality.
The paper examines prediction and estimation risks of ridgeless least squares under general error assumptions.
problem Prediction and estimation risks of ridgeless least squares under realistic error structures.
method Analysis of prediction and estimation risks under general regression error assumptions, including clustered or serial dependence.
result The benefits of overparameterization extend to time series, panel, and grouped data.
Enhanced kernel ridgeless regression improves performance with LAB RBF kernels.
problem Lack of flexibility in kernel ridgeless regression.
method Locally-Adaptive-Bandwidths (LAB) RBF kernels and kernel learning techniques.
result Functions learned from LAB RBF kernels belong to an integral space of RKHSs, demonstrating robust generalization.
Paper explores how DPP sampling can implicitly regularize kernel regression.
problem Improving kernel regression by reducing redundancy in data.
method Using Determinantal Point Processes (DPPs) to sample subsets implicitly regularizes ridgeless Kernel Regression.
result Ensemble of ridgeless regressors can be effective for datasets with redundant information.
Downsampling can improve generalization in ridgeless linear regression, especially with optimal sketching size.
problem Improving generalization in ridgeless linear regression with limited data.
method Investigating the effects of downsampling on the sketched ridgeless least square estimator in the proportional regime.
result Optimal sketching size minimizes out-of-sample prediction risks and stabilizes risk curves.
Data splitting enhances model performance in overparametrized ridgeless regression.
problem Computational inefficiency in training models with large datasets.
method Data splitting as a regularization technique in overparametrized ridgeless regression.
result Data splitting improves statistical performance and computational complexity.
New bounds for KRR condition number reveal overfitting phenomena.
problem Characterizing overfitting in KRR with varying kernel spectral decay.
method Derived new bounds for kernel matrices, enhanced test error bounds, and identified feature independence role.
result Identified tempered and catastrophic overfitting phenomena.
This work analyzes how preconditioning affects generalization in machine learning models.
problem The impact of preconditioning on the generalization of machine learning models.
method An asymptotic bias-variance decomposition of the generalization error for ridgeless regression under various preconditioners.
result The optimal preconditioner depends on label noise, model specification, and signal alignment, with NGD potentially better under certain conditions.
Study the cost of overfitting in noisy KRR models.
problem Cost of overfitting in noisy kernel ridge regression.
method An agnostic view of overfitting cost as a function of sample size for any target function, using Gaussian universality ansatz and task eigenstructure.
result Characterization of benign, tempered, and catastrophic overfitting.
Linear models can be poisoned by shifting a fraction of one class's data, revealing scaling laws and weight alignment.
problem Understanding and quantifying data poisoning in linear models.
method Analysis of ridge least squares with an unpenalized intercept, using resolvent techniques and random matrix theory.
result Closed-form limits for the poisoned score, revealing scaling laws and weight alignment with the poisoning direction.
The paper analyzes bagging in overparameterized learning, deriving risk properties and optimal subsample sizes.
problem Characterizing the risk of bagged predictors in overparameterized settings.
method General strategy using classical results on simple random sampling, specialized for ridge and ridgeless predictors.
result Derives exact asymptotic risk of bagged ridge and ridgeless predictors under various conditions.
Sharp analysis of knowledge distillation for high-dimensional regression.
problem Characterizing the risk of target models in high-dimensional settings.
method Sharp non-asymptotic bounds for ridgeless regression under model and distribution shifts.
result Identifies optimal surrogate models and reveals benefits and limitations of discarding weak features.
New method stabilizes machine learning for physics-informed inverse problems.
problem Reconstructing physical quantities from PDE-compliant measurements.
method Physics-informed learning with smooth inductive bias.
result PDE operators stabilize variance and prevent overfitting in fixed dimensions.
Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.
problem Characterizing and optimizing ridge ensembles in proportional feature-to-sample size regimes.
method Proportional asymptotics analysis, GCV for tuning, proving risk equivalence.
result Risk of optimal full ridgeless ensemble matches optimal ridge predictor's risk.
Knoop enhances variable selection with over-parameterization and knockoffs.
problem Challenges of variable selection in high-dimensional datasets.
method Generates knockoff variables, integrates them into an over-parameterized model, and uses anomaly-based significance tests.
result Superior performance in variable selection compared to existing methods.
New method estimates precision matrices without models, achieving dense, consistent, and model-free properties.
problem Lack of methods that are dense, consistent, and model-free for precision matrix estimation.
method General class of estimators that unify dense, consistent, and model-free properties within a nonasymptotic framework.
result Ridgeless regression exhibits the double descent phenomenon, establishing a precision matrix analogue to linear regression's double descent.
Locality helps in learning from high-dimensional data.
problem Understanding how convolutional neural networks learn from high-dimensional data.
method Teacher-student framework for kernel regression with convolutional kernels.
result Locality is key to determining the learning curve exponent in high-dimensional data.
Optimal recovery framework for non-IID data in Hilbert spaces.
problem Generalization in non-IID data scenarios.
method Optimal recovery perspective, semidefinite programming, kernel ridgeless regression.
result Optimal recovery formula coincides with kernel ridgeless regression in some cases.
Overparameterized ensembles don't offer generalization benefits over single large models.
problem Theoretical limitations of ensembles in overparameterized settings.
method Using ensembles of random feature (RF) regressors, the paper clarifies how modern ensembles differ from underparameterized counterparts.
result Infinite ensembles of overparameterized RF regressors become pointwise equivalent to single infinite-width RF regressors, and finite width ensembles converge to single models with the same parameter budget.
Lecture notes on advanced linear regression methods.
problem Understanding the properties of linear regression estimators in high dimensions.
method Proposition-proof exploration of least squares, ridgeless, ridge, and lasso estimators.
result Detailed analysis of the existence, uniqueness, relations, computation, and non-asymptotic properties of these estimators.
W2S FT often outperforms weak teachers due to low intrinsic dimensionality.
problem Understanding why weak-to-strong finetuning outperforms weak models.
method Analyzing W2S in ridgeless regression setting, focusing on variance reduction.
result Weak teacher's variance is inherited by strong student in shared feature subspace, reduced in discrepancy subspace.
A new method using negative-shifted gradient descent improves overparameterized linear regression by avoiding structural limitations of negative ridge endpoints.
problem Structural limitations of negative ridge endpoints in overparameterized linear regression.
method Negative-shifted gradient descent, which avoids the pole constraint of negative ridge endpoints.
result The method improves over all admissible endpoints by a polynomial factor in risk under explicit conditions.
The paper analyzes how generated data improves adversarial training in high-dimensional regression.
problem Improving adversarial training in high-dimensional regression.
method Theoretical analysis of a two-stage training approach with generated data and pseudo-labels.
result Two-stage adversarial training achieves better performance than ridgeless training in high-dimensional linear regression.
This paper shows how forward rate interpolations are equivalent to discount factor interpolations in yield curve construction.
problem The challenge of choosing between different interpolation methods for yield curve construction.
method Demonstrates the equivalence between forward rate interpolations and discount factor interpolations.
result Some popular interpolation methods on forward rates are equivalent to classical interpolation methods on discount factors.
The paper proposes a method for generating uniform interpolations on data manifolds.
problem Generating high-quality interpolations between data samples on complex manifolds.
method Autoencoder network with interpolation network, regularized by a Riemannian metric.
result The method generates interpolations that remain within the manifold's distribution.
Study finds conditions for Legendre curves to be interpolating sesqui-harmonic in Sasakian space forms.
problem Characterizing Legendre curves in Sasakian space forms.
method Analyzes necessary and sufficient conditions for Legendre curves to be interpolating sesqui-harmonic.
result Obtains an example of an interpolating sesqui-harmonic Legendre curve in a Sasakian space form.
New FX option interpolations impact implied volatilities.
problem Different interpolations of FX option quotes lead to varying implied volatilities.
method Analysis of various exact interpolations of broker quotes.
result Different interpolations result in different implied volatilities.
Kernel interpolation is inconsistent for norms with smoothness above a constant.
problem Inconsistency of kernel interpolation in reproducing kernel Hilbert spaces.
method Lower bounds for generalization error in Sobolev norms.
result Kernel interpolation is always inconsistent for norms with smoothness above a constant.
Bayesian interpolants explain neural network inferences concisely.
problem Understanding neural network inferences.
method Adapting Craig interpolants for neural networks.
result Produces precise, understandable explanations.
Near-interpolating models grow norms quickly, affecting generalization.
problem Understanding the trade-off between interpolation and generalization in near-interpolating models.
method Random matrix theory and eigendecay analysis of data covariance matrix.
result Near-interpolating models exhibit rapid norm growth and worse generalization trade-offs.
The paper improves interpolation in generative models by using specific base distributions.
problem Unexpected side effects in linear interpolations of normalizing flows.
method Enforces a specific manifold using Dirichlet and von Mises-Fisher base distributions.
result Superior performance in terms of bits per dimension, FID, and KID scores for interpolation.
Paper presents a unified approach to interpolation and geodesics in latent spaces of generative models.
problem Finding geodesics and interpolating in latent spaces of non-Gaussian densities.
method General approach to interpolation and geodesics in latent space for arbitrary density.
result Maximizing quality measure of an interpolating curve is equivalent to finding geodesic.
Improved sparse-view CT images with deep learning sinogram interpolation.
problem Sparse-view CT images quality improvement with limited projection data.
method Combination of U-Net and residual learning for sinogram interpolation.
result Significantly improved CT image quality (RMSE and SSIM metrics) over standard methods.
Deep neural networks can interpolate any dataset in the overparametrized regime.
problem Interpolating any dataset with deep neural networks in the overparametrized regime.
method Proving universal approximations and interpolating any dataset with deep neural networks, considering specific conditions on activation functions.
result Interpolation of any dataset is possible in the overparametrized regime with deep neural networks.
Interpolation hurts robust generalization even without noise.
problem The challenge of robust generalization in the absence of noise.
method Avoiding interpolation through ridge regularization.
result Ridge regularization improves robust generalization.
Uniform convergence of interpolators proven for Gaussian data.
problem Interpolation learning in high-dimensional linear regression with Gaussian data.
method Generic uniform convergence guarantee in terms of Gaussian width.
result Consistency of interpolators for minimum-norm and near-minimal-norm cases.
Interpolation improves nearest neighbor algorithms' performance.
problem Improving nearest neighbor algorithms' performance.
method Considered a class of interpolated weighting schemes and characterized their asymptotic performances.
result Mild degree of data interpolation strictly improves prediction accuracy and statistical stability.
The paper characterizes vector fields as interpolating sesqui-harmonic maps on Riemannian manifolds.
problem Characterizing vector fields as interpolating sesqui-harmonic maps on Riemannian manifolds.
method Characterization theorem and critical point condition for interpolating sesqui-harmonic vector fields.
result Conditions for vector fields to be interpolating sesqui-harmonic maps on compact manifolds.
SoftKI combines SKI and variational methods for scalable GP regression.
problem Scalable Gaussian Process regression on high-dimensional datasets.
method SoftKI approximates kernel via softmax interpolation from a smaller number of learned points.
result SoftKI is competitive with other approximated GP methods for modest data dimensions.
Proves a new law of robustness for interpolating arbitrary data distributions.
problem Understanding robust interpolation for arbitrary data distributions.
method Proves a Lipschitzness lower bound for robust interpolation.
result Demonstrates a two-fold law of robustness for interpolating functions.
Interpolating estimators in nonparametric regression become suboptimal under adversarial attacks.
problem Adversarial robustness of interpolating estimators in nonparametric regression.
method Investigation of adversarial robustness of interpolating estimators in a nonparametric regression framework.
result Interpolating estimators must be suboptimal even under a subtle future X-attack. Paper investigates optimal interpolation methods in linear regression.
problem Understanding when interpolating methods generalize well in linear regression.
method Investigates optimal response-linear interpolators using functions linear in the response variable.
result Provides a closed-form expression for the optimal interpolator and shows it can be derived as the limit of gradient descent.