The paper explores why a specific type of predictor works well in noisy data.
problem Understanding why a specific type of predictor (minimum-norm interpolator) works well in noisy data.
method The paper uses uniform convergence and zero-error predictors in a norm ball to explain the success of the minimum-norm interpolator.
result The minimum-norm interpolator is consistent, and this can be explained by uniform convergence of zero-error predictors in a norm ball.
Study shows interpolating predictor's risk is optimal in low-dimensional factor regression models.
problem Understanding the risk of interpolating predictors in high-dimensional factor regression models.
method Detailed finite-sample analysis of minimum-norm interpolating predictor's risk in factor regression models.
result The risk of the minimum-norm interpolating predictor approaches optimal benchmarks in low-dimensional factor regression models.
Improves logistic regression performance by reducing dependence on predictor norm.
problem Improper learning in logistic regression with exponential dependence on predictor norm.
method Designing an efficient improper learning algorithm for online logistic regression with doubly-exponential improvement in predictor norm dependence.
result Improves regret bound for online logistic regression with doubly-exponential improvement in dependence on predictor norm.
New sample complexity bounds for linear predictors and neural networks, focusing on initialization.
problem Understanding sample complexity for vector-valued linear predictors and neural networks, especially under initialization-dependent conditions.
method Size-independent bounds on Frobenius norm distance from a fixed reference matrix, applying to vector-valued predictors and neural networks.
result Established new sample complexity bounds for feed-forward neural networks, resolving open questions and introducing a new learnable problem.
The paper predicts survival functions using random survival trees and concordance maximization.
problem Predicting conditional survival functions in right-censored data.
method The approach combines regression strategies with random survival trees and maximizes concordance.
result The proposed weighted predictor outperforms the usual survival cobra in terms of concordance.
Characterizes sample complexity for outcome indistinguishability in machine learning.
problem Outcome indistinguishability in machine learning, focusing on distinguishers and predictors.
method Sample complexity characterized by metric entropy of predictor and distinguisher classes, using dual Minkowski norms.
result Equivalence and tightness of sample complexity characterizations in distribution-specific and distribution-free settings.
New bounds show linear predictors rarely overfit with certain optimization methods.
problem Bounding test error for linear predictors with stochastic optimization methods.
method Coupling argument for fixed point methods like stochastic and batch mirror descent.
result Locally-adapted rates that depend on predictor properties, not global problem structure.
Decision trees are consistent for regression and classification tasks even with many predictors.
problem Consistency of decision trees with many predictors.
method CART and C4.5 methodology, oracle inequality, sparsity constraints.
result Decision trees and random forests are consistent for various types of data.
Paper proposes a blind predictor for unknown PSD Gaussian process.
problem Predicting a circular symmetric zero-mean stationary Gaussian process with unknown PSD.
method Random spectral representation and atomic-norm minimization for blind estimation.
result The proposed blind predictor performs comparably to an MMSE predictor with known PSD.
This paper proves, in very general settings, that convex risk minimization is a procedure to select a unique conditional probability model determined by the classification problem. Unlike most previous work, we give results that are general enough to include cases in which no minimum exists, as occurs typically, for in…
In this work, we study the problem of aggregating a finite number of predictors for nonstationary sub-linear processes. We provide oracle inequalities relying essentially on three ingredients: (1) a uniform bound of the ℓ1 norm of the time varying sub-linear coefficients, (2) a Lipschitz assumption on the predict…
The paper examines how deep linear neural networks behave as they become infinitely wide.
problem Understanding the behavior of deep linear neural networks as they approach infinite width.
method Analyzes the infinite-width limit of deep linear neural networks, proving convergence to deterministic models and providing precise laws for random weights.
result The training dynamics of deep linear neural networks converge to those of a deterministic model, and the weights' behavior is precisely described.
Alpha-norm regularization simplifies marketing demand forecasting.
problem Ultra high-dimensional problems in demand estimation and forecasting.
method Nonconvex alpha-norm objective with coordinate descent and proximal operators.
result Alpha-norm regularization provides accurate out-of-sample estimates for promotion effects.
New probabilistic complexity measures for linear and kernel methods.
problem Limitations of linear and kernel methods in machine learning.
method Introducing approximate notions of dimensional and margin complexity.
result Approximate complexity measures are both sufficient and necessary for learning.
For supervised and unsupervised learning, positive definite kernels allow to use large and potentially infinite dimensional feature spaces with a computational cost that only depends on the number of observations. This is usually done through the penalization of predictor functions by Euclidean or Hilbertian norms. In …
We consider a class of sparsity-inducing regularization terms based on submodular functions. While previous work has focused on non-decreasing functions, we explore symmetric submodular functions and their \lova extensions. We show that the Lovasz extension may be seen as the convex envelope of a function that depends …
We analyze the local Rademacher complexity of empirical risk minimization (ERM)-based multi-label learning algorithms, and in doing so propose a new algorithm for multi-label learning. Rather than using the trace norm to regularize the multi-label predictor, we instead minimize the tail sum of the singular values of th…
Study shows gap between uniform convergence and test error in random feature models.
problem Understanding the gap between uniform convergence and test error in random feature models.
method Analytical expressions for uniform convergence over norm balls, interpolators, and minimum norm interpolator risk derived and proved.
result Uniform convergence over interpolators still gives a non-trivial bound of test error even when classical uniform convergence is vacuous.
Sparse methods for supervised learning aim at finding good linear predictors from as few variables as possible, i.e., with small cardinality of their supports. This combinatorial selection problem is often turned into a convex optimization problem by replacing the cardinality function by its convex envelope (tightest c…
FOLKLORE algorithm speeds up online multiclass logistic regression.
problem Efficiently solving online multiclass logistic regression without high computational cost.
method Developed FOLKLORE algorithm with improved runtime and regret bound.
result First practical algorithm for online multiclass logistic regression.
Study on Transfer Elastic Net error bounds and grouping effect.
problem Estimation error and grouping effect in Transfer Elastic Net.
method Derives non-asymptotic error bound and examines grouping effect scenarios.
result Effective error bounds and grouping effect observed in Transfer Elastic Net.
We propose an approach to multivariate nonparametric regression that generalizes reduced rank regression for linear models. An additive model is estimated for each dimension of a q-dimensional response, with a shared p-dimensional predictor variable. To control the complexity of the model, we employ a functional fo…
Improves shared encoder representations for better multi-task learning performance.
problem Improving quality of shared encoder representations in multi-task learning.
method Dummy Gradient norm Regularization (DGR) to decrease gradient norm of dummy task-specific predictors.
result DGR improves multi-task prediction performances and superior performance compared to existing methods.
iRRR integrates multi-view data for faster prediction.
problem Predicting from multi-view data with high dimensions and sparse relevant views.
method Integrative reduced-rank regression with convex composite nuclear norm penalization.
result iRRR achieves faster convergence and recovers oracle bounds.
We study a norm for structured sparsity which leads to sparse linear predictors whose supports are unions of prede ned overlapping groups of variables. We call the obtained formulation latent group Lasso, since it is based on applying the usual group Lasso penalty on a set of latent variables. A detailed analysis of th…
Study on how initialization scale affects neural network training regimes.
problem Understanding the transition between kernel and rich regimes in overparametrized models.
method Analysis of simple depth-D models and empirical testing on complex models.
result Scale of initialization controls transition between kernel and rich regimes.
We provide a pointwise confidence bound for non-linear least-squares with fixed design.
problem Confidence estimation in non-linear ℓ2-regularized least squares. method Pointwise confidence bound for local minimizers, using weighted norm involving inverse-Hessian.
result The proposed confidence bound scales with the test input's similarity to the training data.
Two models show risk peaks and decreases with feature count.
problem Understanding risk curves in machine learning models with weak features.
method Analyzing two simple data models with least squares/least norm predictors.
result Risk peaks when features are close to sample size, decreases beyond it.
Analyzes the complexity of linear hypothesis sets using Rademacher complexity.
problem Understanding the complexity of linear hypothesis sets for various norms.
method Tight analysis of empirical Rademacher complexity for linear hypothesis classes with bounded weights.
result Improved bounds on Rademacher complexity for linear hypothesis sets, matching or improving existing results.
New method uses nuclear and ℓ1 penalties for matrix regression, improving brain disorder detection.
problem Modeling high-dimensional matrix predictors with binary responses.
method Convex optimization with ADMM for low-rank and sparse structures.
result Effective in identifying brain disorder-related connectivity patterns.
The problem of joint feature selection across a group of related tasks has applications in many areas including biomedical informatics and computer vision. We consider the l2,1-norm regularized regression model for joint feature selection from multiple tasks, which can be derived in the probabilistic framework by assum…
Characterizes inductive bias in multi-channel linear CNNs with bounded weight norm.
problem Understanding the inductive bias in multi-channel linear convolutional networks.
method Function space characterization and empirical testing of gradient descent.
result The inductive bias depends on the number of output channels for multi-channel inputs but not for single-channel inputs.
This study analyzes prediction risk for PCR method in latent factor regression models.
problem Prediction risk analysis in latent factor regression models.
method Adaptive PCR method with risk bounds established under factor regression model.
result Unified framework for analyzing various linear prediction methods under factor regression.
New insights into when benign overfitting occurs in linear and classification tasks.
problem Understanding when benign overfitting happens in linear and classification models.
method Analysis of a generic data model and comparison of predictors (minimum-norm interpolating and max-margin).
result The minimum-norm interpolating predictor is biased towards an inconsistent solution, preventing benign overfitting in linear regression.
Paper develops algorithms for sparse linear regression with generalized elastic net penalty.
problem Sparse linear regression with robust penalty for high-dimensional data.
method Iterative Reweighted Framework based on ADMM and PMM with SNN.
result Efficient algorithms provide superior performance in both simulated and real data.
Improved algorithm for conditional linear regression with heterogeneous covariances.
problem Identifying a linear predictor for a fraction of data with varying covariances.
method Polynomial time algorithm using Disjunctive Normal Form (DNF) to identify a condition and linear predictor.
result Removed requirement for similar covariances in each condition term, improving algorithm applicability.
Diagonal linear networks converge to lasso regularization path during training.
problem Understanding the regularization behavior of diagonal linear networks.
method Analyzing the training trajectory of diagonal linear networks and comparing it to the lasso regularization path.
result The training trajectory of diagonal linear networks is closely related to the lasso regularization path.
A significant hurdle for analyzing large sample data is the lack of effective statistical computing and inference methods. An emerging powerful approach for analyzing large sample data is subsampling, by which one takes a random subsample from the original full sample and uses it as a surrogate for subsequent computati…
Task shift from classification to regression is possible in overparameterized linear models with limited additional data.
problem Transferability of latent knowledge from classification to regression in overparameterized linear models.
method Investigation of task shift in overparameterized linear regression, zero-shot and few-shot cases, with a focus on minimum-norm interpolation.
result Minimum-norm interpolators can transfer latent knowledge from classification to regression with limited additional data.
Forecast dam inflow using sea surface feature weights.
problem Accurate dam inflow forecasting for flood mitigation.
method Extracted sea surface features, applied L2-norm ensemble weighting, used PCA and t-SNE for dimensionality reduction, and calibrated regression models.
result The proposed method improves predictor stability and accuracy in dam inflow forecasting.
Algorithm ensures demographic parity in regression without sensitive attribute data.
problem Performing regression with demographic parity constraints.
method Post-processing algorithm using accurate estimates and sensitive attribute predictor.
result Generates predictions meeting demographic parity constraint.
Deep networks retain initial bias after training, affecting generalization.
problem Understanding how much initial bias in neural networks survives training.
method Introduced initialization memory to measure initial bias's survival.
result SGD can preserve initial bias, while Adam-family methods erase it.
This paper improves SGMs by using a predictor-corrector scheme to converge faster.
problem Theoretical and practical limitations of existing SGMs when T1o∞. method Integrates a predictor-corrector scheme after the forward process to converge in finite time.
result Convergence guarantees for SGMs require only a fixed finite time T1. Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size N after O(Nε1) iterations achieves ε−suboptimality in expected quadratic distance. Optimal ridge penalty can be negative or zero in high-dimensional data.
problem Overfitting in high-dimensional underdetermined linear regression.
method Simulations and real-life data analysis with minimum-norm estimator.
result Optimal ridge penalty can be negative, contradicting conventional wisdom.
Study shows how initialization scale controls transition between kernel and rich regimes in overparametrized models.
problem Understanding the transition between kernel and rich regimes in overparametrized models.
method Analysis of simple two-layer model and complex matrix factorization models, demonstrating the transition.
result Scale of initialization controls the transition between kernel and rich regimes, affecting generalization properties.
Quantum ML predicts data with improved speed and accuracy.
problem Predicting data using maximum likelihood in a quantum setting.
method Quantum states embedding and minimization of quantum relative entropy.
result Unified framework for classical and quantum LLMs with performance guarantees.
The paper introduces a diagnostic method to detect grokking transitions in models before test accuracy improves.
problem Detecting the transition from training to generalization in machine learning models.
method Summarize task-dependent observables as empirical distributions, map them to Wasserstein/quantile coordinates, and analyze using Hankel dynamic mode decomposition.
result The diagnostic method achieves AUROC \(\approx\) 0.93 for grokking-vs-non-grokking discrimination at the run level.