arXiv research

Study tightens bounds for interpolating noisy data using minimum l1-norm.

The paper studies the minimum ℓ₁-norm interpolator's risk behavior in over-parameterized settings.

Deep linear networks can closely approximate interpolants without improving risk.

The study analyzes robustness of estimators in linear models with adversarial errors.

Lower bounds show OLS outperforms basis pursuit in overparameterized linear regression.

New model leads to optimal test loss in sparse linear regression.

The paper analyzes boosting and minimum-$\ell_1$-norm classifiers in high dimensions.

Inflating the minimum norm interpolator improves linear regression generalization error.

Study shows how networks converge to minimum norm solutions with regularization.

Uniform convergence of interpolators proven for Gaussian data.

The paper analyzes the robustness of a minimum $\ell_2$ interpolator in high-dimensional linear regression.

Unified analysis of parameter norms in overparameterized linear models, revealing scaling laws and thresholds.

A new tradeoff between regularization and sharpness improves model performance in overparameterized settings.

Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.

Transfer learning improves MNI's performance in high-dimensional linear regression.

The paper characterizes functions of shallow ReLU NN denoisers under minimal norm constraints.

Our paper characterizes how ReLU affects GD's implicit bias in high-dimensional neural networks.

We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape for the various scalings of $d = n^α$, $α\in(0,1)$, for the input dimension $d$ and sample size $n$. Empirical evidence supports our finding that minimum-norm interpo…

Bagging stabilizes linear interpolators, improving their generalization performance.

The paper explores why a specific type of predictor works well in noisy data.

Kernel interpolation is inconsistent for norms with smoothness above a constant.

The paper shows how multi-task learning in neural networks is similar to kernel regression and Hilbert spaces.

We show that minimum-norm interpolation in the Reproducing Kernel Hilbert Space corresponding to the Laplace kernel is not consistent if input dimension is constant. The lower bound holds for any choice of kernel bandwidth, even if selected based on data. The result supports the empirical observation that minimum-norm …

Study shows interpolating predictor's risk is optimal in low-dimensional factor regression models.

A data filtering method for cluster analysis is proposed, based on minimizing a least squares function with a weighted $\ell_0$-norm penalty. To overcome the discontinuity of the objective function, smooth non-convex functions are employed to approximate the $\ell_0$-norm. The convergence of the global minimum points o…

Near-interpolating models grow norms quickly, affecting generalization.

New tensor formulation reveals gradient flow's bias in linear neural networks.

The paper analyzes the generalization error of min-norm interpolators in transfer learning with limited test samples.

Adversarial training improves linear regression solutions, offering robustness against small perturbations.

New method approximates complex kernel norms with random features, making learning tractable.

Adversarial training improves linear regression solutions, revealing sparsity and abrupt interpolation.

Task shift from classification to regression is possible in overparameterized linear models with limited additional data.

New findings on how overfitting can be beneficial in ridge regression.

Paper analyzes mistake and generalization of MNIC classifiers.

Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum $\ell_2$ norm ("ridgeless") interpolation in high-dimensional least squares regression. …

Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.

The study prevents model collapse in overparameterized linear regression by mixing real and synthetic labels.

Develops exact convex optimization formulations for neural networks.

The paper tackles multi-armed bandits with vector losses, focusing on minimizing the $\ell^\infty$-norm of relative losses.

We study least squares linear regression over $N$ uncorrelated Gaussian features that are selected in order of decreasing variance. When the number of selected features $p$ is at most the sample size $n$, the estimator under consideration coincides with the principal component regression estimator; when $p>n$, the esti…

Batching stabilizes risk in high-dimensional linear regression models.

Study shows gap between uniform convergence and test error in random feature models.

Study shows minimizing the norm of the ERM solution stabilizes kernel ridge-less regression.

We give improved algorithms for the $\ell_{p}$-regression problem, $\min_{x} \|x\|_{p}$ such that $A x=b,$ for all $p \in (1,2) \cup (2,\infty).$ Our algorithms obtain a high accuracy solution in $\tilde{O}_{p}(m^{\frac{|p-2|}{2p + |p-2|}}) \le \tilde{O}_{p}(m^{\frac{1}{3}})$ iterations, where each iteration requires s…

This paper considers the fundamental problem of learning a complete (orthogonal) dictionary from samples of sparsely generated signals. Most existing methods solve the dictionary (and sparse representations) based on heuristic algorithms, usually without theoretical guarantees for either optimality or complexity. The r…

Sparse optimization refers to an optimization problem involving the zero-norm in objective or constraints. In this paper, nonconvex approximation approaches for sparse optimization have been studied with a unifying point of view in DC (Difference of Convex functions) programming framework. Considering a common DC appro…

We study the problem of globally recovering a dictionary from a set of signals via $\ell_1$-minimization. We assume that the signals are generated as i.i.d. random linear combinations of the $K$ atoms from a complete reference dictionary $D^*\in \mathbb R^{K\times K}$, where the linear combination coefficients are from…

New method accelerates steepest descent for convex optimization.