Uniform convergence of interpolators proven for Gaussian data.
problem Interpolation learning in high-dimensional linear regression with Gaussian data.
method Generic uniform convergence guarantee in terms of Gaussian width.
result Consistency of interpolators for minimum-norm and near-minimal-norm cases.
Kernel interpolation is inconsistent for norms with smoothness above a constant.
problem Inconsistency of kernel interpolation in reproducing kernel Hilbert spaces.
method Lower bounds for generalization error in Sobolev norms.
result Kernel interpolation is always inconsistent for norms with smoothness above a constant.
Near-interpolating models grow norms quickly, affecting generalization.
problem Understanding the trade-off between interpolation and generalization in near-interpolating models.
method Random matrix theory and eigendecay analysis of data covariance matrix.
result Near-interpolating models exhibit rapid norm growth and worse generalization trade-offs.
Study tightens bounds for interpolating noisy data using minimum l1-norm.
problem Predicting noisy data with minimum l1-norm interpolation.
method Provided matching upper and lower bounds for prediction error.
result Tight consistency up to negligible terms for d ≫ n d \gg n d ≫ n . Study shows how networks converge to minimum norm solutions with regularization.
problem Interpolating between known regions in shallow ReLU networks.
method Investigates empirical risk minimizers and weight decay regularizers.
result Empirical risk minimizers converge to minimum norm interpolants under specific conditions.
The paper explores why a specific type of predictor works well in noisy data.
problem Understanding why a specific type of predictor (minimum-norm interpolator) works well in noisy data.
method The paper uses uniform convergence and zero-error predictors in a norm ball to explain the success of the minimum-norm interpolator.
result The minimum-norm interpolator is consistent, and this can be explained by uniform convergence of zero-error predictors in a norm ball.
The paper studies the minimum ℓ₁-norm interpolator's risk behavior in over-parameterized settings.
problem Understanding the risk behavior of minimum ℓ₁-norm interpolators in high-dimensional settings.
method Exact characterization of the risk behavior through a system of two non-linear equations.
result Observation of a multi-descent phenomenon in the generalization risk of the minimum ℓ₁-norm interpolator.
Loose bounds found for least-norm interpolant in over-parameterized settings.
problem Failures of model-dependent generalization bounds for least-norm interpolation.
method Analysis of generalization performance of least-norm linear regressor in over-parameterized regime.
result Generalization bounds for least-norm interpolant can be very loose, even when true excess risk goes to zero.
The study analyzes robustness of estimators in linear models with adversarial errors.
problem Analyzing robustness of estimators in linear models with adversarial errors.
method Develops a general theory for minimum norm interpolating estimators and RERM in linear models without conditions on errors.
result Quantitative bound for the prediction error relating it to Rademacher complexity, norm of minimum norm interpolator of errors, and subdifferential size.
Interpolation hurts robust generalization even without noise.
problem The challenge of robust generalization in the absence of noise.
method Avoiding interpolation through ridge regularization.
result Ridge regularization improves robust generalization.
The paper analyzes the generalization error of min-norm interpolators in transfer learning with limited test samples.
problem Characterizing the generalization error of min-norm interpolators in transfer learning with limited test samples.
method Characterizes the bias and variance of pooled min- ℓ 2 \ell_2 ℓ 2 -norm interpolation under covariate shift and model shift. result Shows that adding data can hurt when SNR is low and is beneficial at higher SNR levels under certain conditions.
Paper analyzes mistake and generalization of MNIC classifiers.
problem Understanding the performance of interpolating classifiers.
method Elementary analyses of MNIC's regret and generalization.
result MNIC generalizes with a rate proportional to the norm of the interpolating solution and inversely proportional to the number of data points.
We introduce a new family of matrix norms, the "local max" norms, generalizing existing methods such as the max norm, the trace norm (nuclear norm), and the weighted or smoothed weighted trace norms, which have been extensively used in the literature as regularizers for matrix reconstruction problems. We show that this…
Deep linear networks can closely approximate interpolants without improving risk.
problem Understanding the risk bounds of deep linear networks compared to minimum ℓ 2 \ell_2 ℓ 2 -norm solutions. method Bounding excess risk of interpolating deep linear networks trained using gradient flow.
result Deep linear networks can closely approximate or match minimum ℓ 2 \ell_2 ℓ 2 -norm solutions in terms of risk. Study shows interpolating predictor's risk is optimal in low-dimensional factor regression models.
problem Understanding the risk of interpolating predictors in high-dimensional factor regression models.
method Detailed finite-sample analysis of minimum-norm interpolating predictor's risk in factor regression models.
result The risk of the minimum-norm interpolating predictor approaches optimal benchmarks in low-dimensional factor regression models.
We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape for the various scalings of d = n α d = n^α d = n α , α ∈ ( 0 , 1 ) α\in(0,1) α ∈ ( 0 , 1 ) , for the input dimension d d d and sample size n n n . Empirical evidence supports our finding that minimum-norm interpo…
We show that minimum-norm interpolation in the Reproducing Kernel Hilbert Space corresponding to the Laplace kernel is not consistent if input dimension is constant. The lower bound holds for any choice of kernel bandwidth, even if selected based on data. The result supports the empirical observation that minimum-norm …
This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.
problem Exact data interpolation using sparse, infinitely wide neural networks.
method Atomic norm framework to derive convex hulls and equivalent convex formulations.
result Simple characterizations of convex hulls for different constraints on network weights and biases.
Study shows gap between uniform convergence and test error in random feature models.
problem Understanding the gap between uniform convergence and test error in random feature models.
method Analytical expressions for uniform convergence over norm balls, interpolators, and minimum norm interpolator risk derived and proved.
result Uniform convergence over interpolators still gives a non-trivial bound of test error even when classical uniform convergence is vacuous.
New model leads to optimal test loss in sparse linear regression.
problem Sparse linear regression with low test loss despite interpolating training data.
method Developed a new parametrization of the model that combines benefits of ℓ1 and ℓ2 norms.
result Training via gradient descent leads to an interpolator with near-optimal test loss.
A new interpolation method speeds up neural ODE training.
problem Efficiently approximating gradients in neural ODEs.
method Interpolation-based technique to approximate gradients.
result Our method trains neural ODEs faster than the reverse dynamic method.
Lower bounds show OLS outperforms basis pursuit in overparameterized linear regression.
problem Excess risk of sparse interpolating procedures in overparameterized linear regression.
method Proved lower bounds on excess risk for OLS and basis pursuit.
result Excess risk of basis pursuit can converge at an exponentially slower rate than OLS.
Inflating the minimum norm interpolator improves linear regression generalization error.
problem Highly anisotropic covariances and diverging d / n d/n d / n in linear regression. method Inflating the minimum ℓ 2 \ell_2 ℓ 2 norm interpolator by a constant greater than one. result Inflating the minimum norm interpolator improves generalization error.
Study investigates overparametrization in survival models, revealing complex loss behavior.
problem Understanding overparametrization in survival models through interpolation.
method Defined interpolation and finite-norm interpolation, rigorously analyzed four survival models.
result Overparametrization can lead to improved performance in survival models, contrary to classical learning theory.
New optimization method helps models generalize better after achieving near-perfect training performance.
problem Models can achieve near-perfect training performance but fail to generalize well to unseen examples.
method GROKtimizer combines rapid convergence to interpolation with post-interpolation norm minimization using Critically Damped Momentum.
result GROKtimizer provides a quadratic speedup over classical gradient descent, offering a natural solution for selecting low-norm interpolating solutions.
New insights into optimization and generalization for linear models.
problem Understanding the implicit regularization of optimization methods for linear models.
method Investigating the norms minimized by interpolating solutions and using projections to move between solutions.
result Proving that for over-parameterized linear classification, projections onto the data-span enable the use of under-parameterized techniques.
Following Kobayashi, we consider Griffiths negative complex Finsler bundles, naturally leading us to introduce Griffiths extremal Finsler metrics. As we point out, this notion is closely related to the theory of interpolation of norms, and is characterized by an equation of complex Monge--Ampère type, whose correspondi…
This work ensures stability in POD basis interpolation for pMOR in hyperelasticity.
problem Stability of POD basis interpolation on Grassmann manifolds for pMOR in hyperelasticity.
method Stability conditions derived from Grassmannian Exponential map and principal angles.
result Explicit stability conditions for practical pMOR applications and non-monotonic error behavior.
Study shows minimizing the norm of the ERM solution stabilizes kernel ridge-less regression.
problem Stability of kernel ridge-less regression.
method Minimizing the norm of the ERM solution to minimize CV stability.
result Interpolating solution with minimum norm minimizes CV stability.
We refine and generalize several interpolation inequalities bounding the L p L^p L p norm of a probability density with respect to the reference measure μ μ μ by its Sobolev norm and the Kantorovich distance to μ μ μ on a smooth weighted Riemannian manifold satisfying C D ( 0 , ∞ ) CD(0, \infty) C D ( 0 , ∞ ) condition.
Unified algorithm for any p p p -norm experimental design problems.
problem Experimental design problems for various p p p -norm objectives. method Randomized local search approach for all p p p . result First approximation algorithm for general p p p -norm objective. Bagging stabilizes linear interpolators, improving their generalization performance.
problem Unstable linear interpolators fail on noisy data.
method Introduced multiplier-bootstrap-based bagged least square estimator.
result Bagging effectively mitigates variance, leading to bounded prediction risk.
Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.
problem Understanding the learning dynamics and implicit bias in ReLU networks with orthogonal data.
method Gradient flow analysis of two-layer ReLU networks from small initialization with orthogonal training data.
result The learned interpolator has a squared ℓ 2 \ell_2 ℓ 2 -norm scaling as n \sqrt{n} n , close to the minimal interpolator's complexity. The paper characterizes functions of shallow ReLU NN denoisers under minimal norm constraints.
problem Understanding the theoretical success of neural network denoisers.
method Characterization of functions realized by shallow ReLU NN denoisers under minimal norm constraints.
result The functions realized by shallow ReLU NN denoisers are contractive toward clean data points and generalize better than the empirical MMSE estimator at low noise levels.
Task shift from classification to regression is possible in overparameterized linear models with limited additional data.
problem Transferability of latent knowledge from classification to regression in overparameterized linear models.
method Investigation of task shift in overparameterized linear regression, zero-shot and few-shot cases, with a focus on minimum-norm interpolation.
result Minimum-norm interpolators can transfer latent knowledge from classification to regression with limited additional data.
Study resolves conjecture on overparameterized linear models' generalization.
problem Asymptotic generalization of multiclass classification with overparameterized models.
method Gaussian covariates bi-level model, Hanson-Wright inequality variant.
result Min-norm interpolating classifier can be suboptimal compared to noninterpolating classifiers.
The paper improves interpolation in generative models by using specific base distributions.
problem Unexpected side effects in linear interpolations of normalizing flows.
method Enforces a specific manifold using Dirichlet and von Mises-Fisher base distributions.
result Superior performance in terms of bits per dimension, FID, and KID scores for interpolation.
The study uses persistent homology to determine when Voronoi interpolation should stop.
problem Interpolating complex topological data sets accurately.
method Persistent homology is applied to the Voronoi tessellation to detect changes in the data's topology.
result The method effectively identifies when the interpolation has captured the data's topology changes.
Our paper examines binary linear classification under Gaussian mixtures, revealing conditions for optimal performance.
problem Understanding the conditions for optimal performance of binary linear classifiers under Gaussian mixtures.
method We study max-margin SVM and min-norm interpolating classifiers, deriving bounds and conditions for optimal performance.
result Interpolating estimators achieve asymptotically optimal performance under certain conditions, emphasizing the role of SNR and covariance.
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal …
Study optimizes linear regression analysis for high-dimensional settings.
problem Understanding high-dimensional linear regression with interpolation and regularization.
method Localized uniform convergence analysis of optimistic rates for linear regression.
result Recover guarantees for ridge and LASSO regression under random designs.
Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.
problem The impact of noise on interpolating models with strong inductive biases.
method Analyzing linear and classification models with sparse ground truths, proving fast rates for interpolators.
result Strong inductive biases can lead to faster but noisier interpolators, contrary to intuition.
A new tradeoff between regularization and sharpness improves model performance in overparameterized settings.
problem Improving model performance in overparameterized settings with minimum-norm interpolators.
method Proposes a regularization-sharpness tradeoff for overparameterized linear regression with an ℓ^p penalty.
result Empirical validation shows the tradeoff terms can distinguish performant linear interpolators.
In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for…
Level-set optimization formulations with data-driven constraints minimize a regularization functional subject to matching observations to a given error level. These formulations are widely used, particularly for matrix completion and sparsity promotion in data interpolation and denoising. The misfit level is typically …
Paper investigates optimal interpolation methods in linear regression.
problem Understanding when interpolating methods generalize well in linear regression.
method Investigates optimal response-linear interpolators using functions linear in the response variable.
result Provides a closed-form expression for the optimal interpolator and shows it can be derived as the limit of gradient descent.
This paper introduces a new bound to explain generalization in over-parameterized models.
problem Understanding why some over-parameterized models generalize well while others do not.
method PAC-Chernoff bounds and smoothness measures based on large deviation theory.
result Interpolators with smoother structures generalize better, according to the new theoretical framework.
Regression models can interpolate noisy data and still perform well, contrary to the bias-variance tradeoff.
problem Understanding why overparametrized models can generalize well despite the bias-variance tradeoff.
method Analysis of minimum norm solutions and ridge regression, focusing on the smallest singular value of the regression matrix.
result Testing error exhibits double descent behavior as model order increases, contrary to the classical bias-variance tradeoff.