Interpolation hurts robust generalization even without noise.
problem The challenge of robust generalization in the absence of noise.
method Avoiding interpolation through ridge regularization.
result Ridge regularization improves robust generalization.
Study resolves conjecture on overparameterized linear models' generalization.
problem Asymptotic generalization of multiclass classification with overparameterized models.
method Gaussian covariates bi-level model, Hanson-Wright inequality variant.
result Min-norm interpolating classifier can be suboptimal compared to noninterpolating classifiers.
Our paper examines binary linear classification under Gaussian mixtures, revealing conditions for optimal performance.
problem Understanding the conditions for optimal performance of binary linear classifiers under Gaussian mixtures.
method We study max-margin SVM and min-norm interpolating classifiers, deriving bounds and conditions for optimal performance.
result Interpolating estimators achieve asymptotically optimal performance under certain conditions, emphasizing the role of SNR and covariance.
The paper analyzes the generalization error of min-norm interpolators in transfer learning with limited test samples.
problem Characterizing the generalization error of min-norm interpolators in transfer learning with limited test samples.
method Characterizes the bias and variance of pooled min-ℓ2-norm interpolation under covariate shift and model shift. result Shows that adding data can hurt when SNR is low and is beneficial at higher SNR levels under certain conditions.
New research shows SVM and related methods can overfit without harm in multiclass classification.
problem Understanding benign overfitting in multiclass classification.
method Analyzing three training algorithms: ERM with cross-entropy, least-squares, and one-vs-all SVM.
result All three algorithms can lead to classifiers that interpolate training data and have equal accuracy under high overparameterization.
This study reveals fundamental trade-offs between memorization and robustness in neural networks.
problem Understanding the balance between memorization and robustness in neural networks.
method Analyzes two-layer neural networks in various high-dimensional linearized regimes, focusing on Sobolev-seminorm.
result Establishes fundamental trade-offs between memorization and robustness, with lower bounds on Sobolev-seminorm.
Study interpolating estimators for causal learning from observational data.
problem Learning causal models from observational data in complex model classes.
method Investigate min-norm interpolators and ridge-regularized regressors in a linearly confounded model.
result Interpolators cannot be optimal for causal learning under the principle of independent causal mechanisms, requiring stronger regularization.
Study shows interpolating predictor's risk is optimal in low-dimensional factor regression models.
problem Understanding the risk of interpolating predictors in high-dimensional factor regression models.
method Detailed finite-sample analysis of minimum-norm interpolating predictor's risk in factor regression models.
result The risk of the minimum-norm interpolating predictor approaches optimal benchmarks in low-dimensional factor regression models.
GRAIN: Group Aggregation via Min-Norm Objective
problem Learning instability in large models
method Replacing mean aggregation with min-norm convex combination
result Improves performance and reduces variance
Paper analyzes mistake and generalization of MNIC classifiers.
problem Understanding the performance of interpolating classifiers.
method Elementary analyses of MNIC's regret and generalization.
result MNIC generalizes with a rate proportional to the norm of the interpolating solution and inversely proportional to the number of data points.
Interpolation can prevent classifiers from having desired invariance properties.
problem Invariance properties in over-parameterized models are often ineffective.
method Theoretical analysis and algorithm design for non-interpolating classifiers.
result Interpolating classifiers cannot satisfy desired invariance properties.
Fair Mixup improves fairness in classifiers by interpolating between groups.
problem Ensuring fairness in classifiers during training and evaluation.
method Fair Mixup uses interpolation of samples between groups to enforce fairness constraints.
result Fair Mixup ensures better generalization of fairness in various benchmarks.
Study on clustering in high dimensions with anisotropic Gaussian mixtures, showing interpolation can be optimal and robust.
problem Clustering in high-dimensional anisotropic Gaussian mixtures.
method Derive minimax bounds, analyze ℓ2-regularized classifiers, and investigate interpolation's robustness. result Interpolating solutions can be optimal and robust under certain conditions.
Many modern machine learning models are trained to achieve zero or near-zero training error in order to obtain near-optimal (but non-zero) test error. This phenomenon of strong generalization performance for "overfitted" / interpolated classifiers appears to be ubiquitous in high-dimensional data, having been observed …
Geometric theory connects machine learning classifiers to differential geometry.
problem Classifying data points in machine learning.
method Mapping binary classification to vector bundles and differential geometry.
result Harmonic interpolation solves RKHS interpolation problems.
The paper analyzes kernel classifiers' performance in Sobolev spaces and proves their optimality.
problem Theoretical analysis of kernel classifiers' performance in Sobolev spaces.
method Deriving upper and lower bounds on classification excess risk using kernel regression theory and estimating interpolation smoothness.
result The proposed kernel classifier is optimal in Sobolev spaces, with theoretical bounds confirmed by real data.
New method reveals good classifiers are common in over-parameterized models.
problem Understanding how over-parameterized models generalize well.
method Developed a methodology to compute the full distribution of test errors.
result Test errors concentrate around a small typical value ε* rather than the worst-case model.
Study of graphs interpolating curve and pants graphs, providing formulae and geometry classifications.
problem Understanding the large-scale geometry of graphs connecting curve and pants graphs.
method Developed explicit formulae for quasi-flat ranks and classified geometries using twist-free graphs of multicurves.
result Explicit formulae for quasi-flat ranks and classification of geometries into hyperbolic, relatively hyperbolic, and thick cases.
Data augmentation by mixing samples, such as Mixup, has widely been used typically for classification tasks. However, this strategy is not always effective due to the gap between augmented samples for training and original samples for testing. This gap may prevent a classifier from learning the optimal decision boundar…
K-Nearest neighbor classifier (k-NNC) is simple to use and has little design time like finding k values in k-nearest neighbor classifier, hence these are suitable to work with dynamically varying data-sets. There exists some fundamental improvements over the basic k-NNC, like weighted k-nearest neighbors classifier (wh…
New method interpolates training data and is consistent for various data distributions.
problem Establishing generalization guarantees for ensemble methods in the interpolating regime.
method Developed manifold-Hilbert kernel for Riemannian manifolds and used it in ensemble classification.
result Consistent ensemble classification method for broad data distributions.
We propose to study the generalization error of a learned predictor h^ in terms of that of a surrogate (potentially randomized) predictor that is coupled to h^ and designed to trade empirical risk for control of generalization error. In the case where h^ interpolates the data, it is interesting to con…
There is a large literature explaining why AdaBoost is a successful classifier. The literature on AdaBoost focuses on classifier margins and boosting's interpretation as the optimization of an exponential likelihood function. These existing explanations, however, have been pointed out to be incomplete. A random forest …
New method uses graph-based interpolation for few-shot classification.
problem Learning models with limited labeled examples.
method Graph-based feature vector interpolation for transductive learning.
result Significant gains in few-shot classification compared to other methods.
New method certifies generative models' robustness.
problem Certifying generative models' robustness is challenging due to non-convex sets.
method ApproxLine, a scalable certification method capturing infinite sets or distributions over them.
result ApproxLine provides sound deterministic and probabilistic guarantees.
Paper defines a new distance metric for comparing learning tasks.
problem Comparing difficulty of learning tasks between source and target.
method Information geometry, optimal transport, coupled transfer distance.
result Coupled transfer distance correlates with fine-tuning difficulty.
Ensembling improves performance when classifiers disagree more than average.
problem When do ensembles provide significant performance improvements in classification tasks?
method Theoretical and empirical analysis of ensemble improvement rate and disagreement-error ratio.
result Ensembling improves performance significantly when the disagreement rate is large relative to the average error rate.
Paper shows SVMs can interpolate data in various settings.
problem Understanding SVM performance and generalization.
method Flexible analysis framework for proving SVM interpolation in diverse settings.
result Support vector machines can interpolate data in many cases not previously covered.
New generalization concept considers distribution of errors, not just average error.
problem Classical generalization fails to capture distributional differences in classifier outputs.
method Formal conjectures about distributional generalization based on model architecture, training procedure, and data distribution.
result Distributional generalization can be expected in specific conditions, as evidenced by empirical results.
Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
problem Why randomly trained neural networks generalize well despite interpolating training data.
method Examined a random neural network that interpolates training data and showed it generalizes well if there's a simpler underlying teacher model.
result Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
New insights into optimization and generalization for linear models.
problem Understanding the implicit regularization of optimization methods for linear models.
method Investigating the norms minimized by interpolating solutions and using projections to move between solutions.
result Proving that for over-parameterized linear classification, projections onto the data-span enable the use of under-parameterized techniques.
Study shows neural collapse is invariant to class imbalances under certain conditions.
problem Neural collapse properties are only valid for balanced data.
method Adopted UFM and introduced SELI for invariant characterization.
result Embeddings and classifiers always interpolate a simplex-encoded label matrix regardless of class imbalances.
The study classifies surfaces with specific curvature properties.
problem Classifying surfaces with a particular curvature equation.
method Analyzing surfaces in 3D Euclidean space with a specific curvature equation.
result A one-parameter family of surfaces meeting the unit ball orthogonally.
Method generates intermediate domains to align source and target domains.
problem Challenges of domain adaptation with significant domain divergence.
method Progressive domain augmentation via domain interpolation and multiple subspace alignment.
result Achieves state-of-the-art performance on multiple domain adaptation tasks.
We classify generalised supersymmetric fluxbranes in type II string theory obtained as Kaluza-Klein reductions of the Minkowski space vacuum of eleven-dimensional supergravity. We obtain two families of smooth solutions which contains all the known solutions, new solutions called nullbranes, and solutions interpolating…
Neural networks can interpolate noisy data and still generalize well.
problem Generalization of neural networks trained on noisy data.
method Two-layer neural networks trained to interpolation by gradient descent on corrupted labels.
result Neural networks can achieve zero training error and optimal test error.
We prove overfitting in minimal and random NNs, tempering the effect.
problem Overfitting in minimal and random neural networks.
method Analyzing binary weight fitting to noisy data, proving overfitting is tempered.
result The overfitting of minimal and random neural networks is tempered.
Study calibrates high-dimensional binary classifiers using angle between estimator and true weights.
problem Calibrating high-dimensional binary classifiers with provable properties.
method Interpolates with a chance classifier to construct well-calibrated predictor based on angle between estimator and true weights.
result Angular calibration approach is provably well-calibrated in high dimensions, minimizing Bregman divergence.
Gradient descent on shallow neural networks achieves near-optimal generalization error.
problem Optimizing shallow neural networks with minimal width for generalization and stability.
method Gradient descent in the interpolating regime with minimal width.
result Gradient descent achieves near-optimal generalization error with minimal width.
Classifies Real line bundles with Real connections on manifolds with involution.
problem Classifying Real line bundles with Real connections on manifolds with involution.
method Defines Real smooth Deligne cohomology to interpolate between equivariant sheaf cohomology and smooth imaginary-valued forms.
result Classifies Real line bundles with Real connections on manifolds with involution.
New insights into how linear classifiers and leaky ReLU networks can overfit without harming generalization.
problem Understanding conditions for benign overfitting in linear classifiers and leaky ReLU networks.
method Utilizing Karush--Kuhn--Tucker (KKT) conditions for margin maximization.
result Satisfaction of KKT conditions leads to benign overfitting in linear classifiers and leaky ReLU networks.
The paper analyzes boosting and minimum-ℓ1-norm classifiers in high dimensions.
problem Understanding the generalization error and optimal Bayes error in boosting.
method High-dimensional asymptotic theory, Gaussian comparison techniques, uniform deviation argument.
result Precise characterizations of boosting test error and optimal Bayes error.
We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by au…
Research into automated systems for detecting and classifying marine mammals in acoustic recordings is expanding internationally due to the necessity to analyze large collections of data for conservation purposes. In this work, we present a Convolutional Neural Network that is capable of classifying the vocalizations o…
Several recent works have shown that state-of-the-art classifiers are vulnerable to worst-case (i.e., adversarial) perturbations of the datapoints. On the other hand, it has been empirically observed that these same classifiers are relatively robust to random noise. In this paper, we propose to study a \textit{semi-ran…
DCK improves air quality index prediction with probabilistic spatial models.
problem Non-Gaussian, complex spatial structure of air quality index.
method Deep classifier kriging (DCK) for non-Gaussian, nonlinear spatial prediction.
result DCK outperforms conventional methods in predictive accuracy and uncertainty quantification.
New research shows the maximum ℓ1-margin classifier doesn't adapt to sparse ground truths.
problem Understanding the limitations of the maximum ℓ1-margin classifier in high-dimensional settings.
method Analyzing convergence and prediction error rates of the maximum ℓ1-margin classifier.
result Proves tight upper and lower bounds for prediction error, showing benign overfitting.
This paper shows how forward rate interpolations are equivalent to discount factor interpolations in yield curve construction.
problem The challenge of choosing between different interpolation methods for yield curve construction.
method Demonstrates the equivalence between forward rate interpolations and discount factor interpolations.
result Some popular interpolation methods on forward rates are equivalent to classical interpolation methods on discount factors.