Analyzes SGD dynamics in two-layer networks, bridging different regimes.
problem Understanding SGD dynamics in high-dimensional and mean-field settings.
method Rigorous analysis via deterministic low-dimensional description of sufficient statistics.
result Infinite-width dynamics remains close to a low-dimensional subspace.
Kernel models learn low-dimensional predictive subspaces from input data.
problem Learning effective feature transformations in kernel models.
method Study of a compositional kernel ridge regression model.
result Global minimizers of the objective function identify the subspace with high probability.
Improved generative models using overparametrized shallow neural networks.
problem Improving generative models for data with hidden low-dimensional structure.
method Using energy-based models with overparametrized shallow neural networks as approximators.
result Models trained in the 'active' regime outperform those in the 'lazy' or kernel regime, leading to better adaptivity to hidden structure.
Proposes an EM algorithm for high-dimensional Markov-switching VAR models.
problem Estimating regime shifts in high-dimensional time series data.
method Approximate EM algorithm for Markov-switching VAR models.
result Established consistency of the proposed EM algorithm in high dimensions.
This paper studies the classification of high-dimensional Gaussian signals from low-dimensional noisy, linear measurements. In particular, it provides upper bounds (sufficient conditions) on the number of measurements required to drive the probability of misclassification to zero in the low-noise regime, both for rando…
Single Index Models (SIMs) are simple yet flexible semi-parametric models for classification and regression. Response variables are modeled as a nonlinear, monotonic function of a linear combination of features. Estimation in this context requires learning both the feature weights, and the nonlinear function. While met…
Projective DP-SGD reduces privacy error by identifying low-dimensional gradient subspaces.
problem Differentially private SGD's error rate scales with model's dimensionality, problematic for over-parameterized models.
method Projective DP-SGD, projecting noisy gradients to a low-dimensional subspace identified from a public dataset.
result The method reduces the dependence on model dimensionality, improving accuracy in high privacy regimes.
DOODL learns shared spectral dynamics across related dynamical systems.
problem Learning independent dynamical operators for each system limits discovery of shared structure.
method DOODL learns a dictionary of characteristic spectral dynamics on a manifold of related systems.
result DOODL achieves errors one to two orders of magnitude lower than independent operator estimation methods.
Reduces IB problem to a simpler, lower-dimensional problem.
problem Information bottleneck problem in high-dimensional spaces.
method Identifies sufficient statistic that factors conditional distribution, reducing IB to a lower-dimensional problem.
result Preserves full IB curve and optimal representations, making IB tractable.
New property helps SGD learn sparse functions efficiently in neural networks.
problem Characterizing functions learnable by SGD in non-linear neural networks.
method Mean-field analysis, hierarchical merged-staircase property, dimension-free dynamics approximation.
result Merged-staircase property is necessary and nearly sufficient for SGD learnability.
The paper analyzes the dynamics of tokens in transformer models at moderate interaction levels.
problem Understanding the evolution of tokens in transformer models at moderate interaction levels.
method Modeling transformer models as a system of particles interacting in a mean-field way and studying the corresponding dynamics.
result Characterization and convergence of the limiting dynamics in different phases of the system.
Gaussian equivalence fails for simple polynomial embeddings in quadratic scaling RF models.
problem Failure of Gaussian equivalence in polynomial feature embeddings under quadratic scaling.
method Introduced Conditional Gaussian Equivalent (CGE) model to capture non-Gaussian behavior.
result Correct asymptotics derived for training and test errors in CGE model.
New bounds on learning shared representations improve model performance and efficiency.
problem Improving model performance and efficiency through shared representations across clients.
method Established new upper and lower bounds on statistical error, designed a spectral estimator for non-convex least-squares solutions.
result Optimal statistical rate achieved when shared representation is well covered across clients.
We study the problem of finding the best linear model that can minimize least-squares loss given a data-set. While this problem is trivial in the low dimensional regime, it becomes more interesting in high dimensions where the population minimizer is assumed to lie on a manifold such as sparse vectors. We propose proje…
A large number of online services provide automated recommendations to help users to navigate through a large collection of items. New items (products, videos, songs, advertisements) are suggested on the basis of the user's past history and --when available-- her demographic profile. Recommendations have to satisfy the…
This paper improves adversarial robustness of deep learning models.
problem Vulnerability of machine learning models to adversarial perturbations.
method Analyzes adversarial training for linear regression and neural networks, incorporating L1 penalty.
result Incorporating L1 penalty leads to consistent adversarially robust estimation in high-dimensional settings.
Proposes a method to estimate personalized treatments from high-dimensional data.
problem Estimating individualized treatment regimes (ITRs) from high-dimensional covariates.
method Directly targets the contrast between potential outcomes, using dimension-reduced outcome-weighted learning.
result Achieves universal consistency, converging to the Bayes risk under mild conditions.
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
Sparse connectivity improves generalization in neural networks below the Edge of Stability.
problem Generalization guarantees for fully-connected networks fail at the Edge of Stability.
method Analyzed sparse connectivity's impact on generalization in two-layer ReLU networks.
result Sparse connectivity changes the effective constraint, leading to non-vacuous generalization bounds.
Deep neural networks have had an enormous impact on image analysis. State-of-the-art training methods, based on weight decay and DropOut, result in impressive performance when a very large training set is available. However, they tend to have large problems overfitting to small data sets. Indeed, the available regulari…
The paper finds non-Gaussian directions in high-dimensional data using Wasserstein distance.
problem Locating interesting non-Gaussian features in high-dimensional data.
method Projection pursuit using 2-Wasserstein distance to maximize the difference from Gaussian.
result Statistical guarantees for accurately approximating an unknown low-dimensional non-Gaussian subspace.
Two approaches improve conformal Bayes for label shift, one post-hoc and one in-training.
problem Improving prediction sets for target domain under label shift.
method Two complementary approaches: post-hoc calibration and in-training adaptation.
result In-training adaptation achieves up to 43% width reduction at unchanged coverage.
A new deep hedging framework improves efficiency and robustness.
problem Pricing and hedging of option portfolios with complex models.
method Neural model for training model embeddings using paths of advanced equity option models.
result The proposed method rapidly adapts to new market regimes through recalibration of a low-dimensional embedding vector.
CPCR mitigates bias in PCR for overparameterized models.
problem Bias in Principal Component Regression (PCR) for overparameterized models.
method Calibrated Principal Component Regression (CPCR) learns a low-variance prior in the PC subspace and calibrates the model in the original feature space.
result CPCR outperforms standard PCR in overparameterized settings, improving prediction across multiple problems.
Deep multi-task learning benefits from low intrinsic dimensionality, leading to better generalization.
problem Improving generalization in deep multi-task learning with high-dimensional models.
method Parametrizing multi-task networks in a low-dimensional space using random expansions and weight compression.
result First non-vacuous generalization bounds for deep multi-task networks are derived.
Adding linear layers to ReLU networks favors functions with low mixed variation.
problem Understanding function space bias in overparameterized neural networks.
method Examined a family of networks with varying depths and same capacity but different representation costs, focusing on the effect of adding linear layers to the input side.
result Adding linear layers to shallow ReLU networks results in a bias towards functions with low mixed variation, which can be well approximated by single- or multi-index models.
Paper proposes a new deflation varimax method for vintage factor analysis.
problem Finding a scientifically meaningful low-dimensional representation of data.
method Deflation varimax procedure for orthogonal matrix rotation.
result The proposed method achieves minimax optimal factor loading estimation.
Local averaging accurately distills manifold structure from noisy data.
problem Tackles the challenge of uncovering manifold structure from noisy data.
method Two-round mini-batch local averaging method applied to noisy samples.
result Achieves accuracy bound of $d(\hat{\mathbf q}, \mathcal M) \leq σ\sqrt{d\left(1+\frac{κ\mathrm{diam}(\mathcal {M})}{\log(D)}
ight)}$ .
A new method reduces sample complexity for meta-learning.
problem Efficiently learn new tasks with minimal data.
method Alternating minimization method (MLLAM) for linear regression tasks.
result MLLAM achieves nearly-optimal estimation error with Ω ( log d ) Ω(\log d) Ω ( log d ) samples per task. The paper introduces a method for interpretable principal component analysis of high-dimensional time series.
problem Inconsistent and difficult-to-interpret principal component estimates in high-dimensional regimes.
method Localized sparse principal component analysis of spectral density matrices in frequency domain.
result Efficient algorithm for sparse-localized estimates of principal subspaces.
CoreFlow models matrix-valued distributions efficiently, preserving shared low-rank structure.
problem Learning matrix-valued distributions from high-dimensional and incomplete data.
method Low-rank flow model that learns shared row/column subspaces and trains a normalizing flow on the core.
result CoreFlow improves generation quality in few-sample regimes and remains competitive in data-rich settings.
Algorithm learns means of spherical Gaussian mixtures efficiently in low dimensions.
problem Learning means of spherical Gaussian mixtures with small separation.
method Fourier transform estimation to learn mixture means efficiently.
result Algorithm learns means with separation d / log k d/\sqrt{\log k} d / log k efficiently. Study how data structure impacts classification performance in overparametrized models.
problem Understanding the role of data structure in overparametrized models.
method Analytical and statistical physics methods applied to structured data.
result Data structure significantly affects the performance of overparametrized models, particularly for logistic loss.
The Gauss-Newton method is analyzed for neural networks using Riemannian optimization techniques.
problem Training neural networks with smooth activations and convergence rates.
method Riemannian optimization perspective, analyzing the Gauss-Newton method in both underparameterized and overparameterized regimes.
result Geometric convergence rates independent of conditioning and eigenvalues, demonstrating accelerated convergence.
This paper extends RMT for deep learning models beyond eigenvalues.
problem Challenges in high-dimensional, overparameterized ML models.
method Introduces High-dimensional Equivalent to analyze nonlinear models.
result Unified understanding of training and generalization in deep learning.
This work investigates square loss in overparametrized neural networks, revealing its advantages in robustness and calibration.
problem Theoretical understanding of square loss in overparametrized neural networks.
method Systematic investigation of square loss in the NTK regime for both separable and non-separable classes.
result Square loss shows fast convergence rates and robustness guarantees for overparametrized neural networks.
PCA++ improves robustness to background noise in contrastive learning.
problem Recovering shared signal subspaces from positive pairs in high-dimensional data with structured background noise.
method PCA++ uses hard uniformity-constrained contrastive learning to enforce identity covariance on projected features.
result PCA++ outperforms standard PCA and alignment-only PCA+ in simulations and real-world datasets.
Proposes a new robust expectile regression method for high-dimensional data.
problem Heterogeneity in high-dimensional data with heteroscedastic variance or inhomogeneous covariate effects.
method Iteratively reweighted ℓ1-penalization for robust expectile regression (retire).
result Oracle convergence rate after log(log d) iterations in high-dimensional settings.
The minority game (MG) model introduced recently provides promising insights into the understanding of the evolution of prices, indices and rates in the financial markets. In this paper we perform a time series analysis of the model employing tools from statistics, dynamical systems theory and stochastic processes. Usi…
Algorithm removes specific training data from models efficiently in high-dimensional settings.
problem Efficiently removing specific training data from high-dimensional models without full retraining.
method Starts from original model parameters, performs Newton steps, adds isotropic Laplacian noise.
result Two Newton steps are sufficient for effective unlearning in high-dimensional problems.
New model allows sparse graphs with many triangles to be represented.
problem Sparse graphs with many triangles cannot be accurately represented in finite dimensions.
method Infinite-dimensional inner product model with manifold representations.
result Local neighborhoods can be represented in lower dimensions.
SGD trains NNs to learn low-dimensional representations efficiently.
problem Training two-layer NNs with SGD for multiple-index models.
method Online SGD with weight decay for training.
result First-layer weights converge to the principal subspace of the true model.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
problem Understanding the generalization of linear denoising autoencoders in overparameterized regimes.
method Analyzes two-layer linear denoising autoencoders with a bottleneck layer and skip connection, deriving test risk formulas.
result Bottleneck layers introduce an additional complexity measure, while skip connections can mitigate variance.
Uniform approximations for RHTs improve kernel approximation and distance estimation.
problem Theoretical guarantees for RHTs in low-dimensional applications.
method Proved uniform convergence of average of function over RHTs entries.
result Improved guarantees for kernel approximation and distance estimation.
Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these …
High-dimensional models trained on smooth manifolds achieve optimal rates in Wasserstein metrics.
problem Training score-based generative models on complex, low-dimensional manifolds.
method Proves optimal rates for SGMs on smooth manifolds, separating into noise regimes and using ReLU nearest-projection coordinates.
result Optimal intrinsic Wasserstein rates are achieved, with polynomial ambient dependence for families with controlled geometry and density.
Paper investigates hardness of learning neural networks under manifold hypothesis.
problem Hardness of learning neural networks under the manifold hypothesis.
method Extending proofs of hardness in the SQ and cryptographic settings to the geometric setting.
result Learning is hard under input manifolds of bounded curvature but learnable with additional assumptions on manifold volume.
BGM-IV uses AI to estimate causal effects in complex data.
problem Estimating causal effects in high-dimensional, nonlinear settings with endogeneity.
method Structured latent generative modeling for posterior inference in a causally structured latent space.
result BGM-IV outperforms existing methods in high-dimensional covariate regimes.