New bounds show limitations of sample-wise information-theoretic generalization.
problem Limitations of sample-wise information-theoretic generalization bounds.
method Analysis of existing bounds and derivation of new bounds.
result No sample-wise information-theoretic bounds exist for expected squared generalization gap.
New method enhances neural network robustness against adversarial attacks.
problem Enhancing neural network robustness against adversarial attacks.
method Variational framework with per-sample noise level selector.
result Enhanced empirical robustness and certified robustness.
Estimating the relative importance of each sample in a training set has important practical and theoretical value, such as in importance sampling or curriculum learning. This kind of focus on individual samples invokes the concept of sample-wise learnability: How easy is it to correctly learn each sample (cf. PAC learn…
Estimates interactions between modalities for multimodal data.
problem Accurately quantifying interactions between different data types.
method Developed Lightweight Sample-wise Multimodal Interaction (LSMI) estimator using pointwise information theory.
result LSMI reveals fine-grained dynamics in multimodal data.
ALPCAH improves PCA for noisy data by estimating sample-wise noise variances.
problem Noisy data with varying noise levels in different samples.
method Sample-wise heteroscedastic PCA with tail singular value regularization.
result Improves subspace basis estimation for low-rank data.
AdaTrans adapts to feature and sample transfer in high-dimensional regression.
problem High-dimensional linear regression with more features than samples.
method F-AdaTrans and S-AdaTrans methods using fused-penalties and adaptive weights.
result AdaTrans achieves convergence rates close to oracle estimators and near-minimax optimal rates.
In this paper, we propose a novel domain adaptation method that can be applied without target data. We consider the situation where domain shift is caused by a prior change of a specific factor and assume that we know how the prior changes between source and target domains. We call this factor an attribute, and reformu…
We introduce the localized Lasso, which is suited for learning models that are both interpretable and have a high predictive power in problems with high dimensionality d and small sample size n. More specifically, we consider a function defined by local sparse models, one at each data point. We introduce sample-wis…
ALPCAH improves PCA for noisy data samples.
problem Heteroscedastic data with varying noise levels.
method Subspace learning method estimating sample-wise noise variances.
result Improves subspace basis for low-rank data.
Method removes misleading data to improve ML model accuracy.
problem Unhealthy fear of missing out on data leads to model instability and poor performance.
method Bayesian sequential selection method that identifies and selects critical information.
result Improves sample-wise error convergence and eliminates model instabilities.
The paper studies how more data affects prediction risk in high-dimensional models.
problem The impact of increasing data on prediction risk in high-dimensional models.
method Derives central limit theorem and provides finite-sample distribution and confidence interval for prediction risk.
result Demonstrates 'more data hurt' phenomenon in high-dimensional least squares estimation.
In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimato…
FCDD improves image anomaly detection without post-hoc explainers.
problem Image anomaly detection, especially pixel-wise.
method Fully Convolutional Data Description (FCDD) directly addresses anomaly detection without post-hoc methods.
result FCDD achieves state-of-the-art results on pixel-wise AD tasks.
CDSSL improves representation quality by integrating linear and nonlinear dependencies.
problem Scarcity of labeled data and neglect of nonlinear dependencies in SSL.
method CDSSL combines linear correlations and nonlinear dependencies using HSIC in RKHS.
result CDSSL enhances representation quality on diverse benchmarks.
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
Riemannian metric matching learns the geometry of high-dimensional datasets using neural networks.
problem Estimating the geometry of high-dimensional datasets from samples
method Riemannian metric matching using neural networks
result Riemannian metric matching rivals or improves k-NN-based diffusion geometry estimators Study shows double and triple descent in unsupervised autoencoders, improving performance in various tasks.
problem Exploring the phenomenon of double descent in unsupervised learning.
method Analytical demonstration and extensive experiments on synthetic and real datasets.
result Over-parameterized unsupervised autoencoders exhibit double and triple descent, enhancing performance in downstream tasks.
The paper analyzes how the one-dimensional Wasserstein distance captures pointwise density differences in finite samples.
problem Uncertainty in identifying density differences when supports overlap and densities have substantial pointwise differences.
method Analysis using the Poisson process and neural spike train decoding.
result The one-dimensional Wasserstein distance highlights meaningful density differences related to both rate and support.
ALPCAHUS clusters data from multiple subspaces with varying noise.
problem Heteroscedastic data with varying noise levels.
method Develops a heteroscedastic PCA method for subspace clustering.
result Improves subspace clustering by accounting for sample-wise noise variances.
Noise makes data unlearnable by tricking models.
problem Unauthorized exploitation of personal data by deep learning models.
method Error-minimizing noise to reduce training examples' learnability.
result Error-minimizing noise can make training examples unlearnable by deep learning models.
Study compares random and learned features in deep Bayesian linear models.
problem Understanding how feature learning affects generalization in deep learning.
method Comparing deep random feature models to deep networks with trained layers.
result Random feature models can display double-descent behavior, while deep networks do not.
RID-Noise improves robust design under noisy conditions using neural networks.
problem Design robustness under noisy environments.
method Robust Inverse Design under Noise (RID-Noise) using conditional invertible neural networks (cINNs).
result RID-Noise achieves more effective robust design compared to state-of-the-art methods.
The paper analyzes learning curves for kernel ridge regression with dot-product kernels.
problem Understanding the learning curves for different scaling regimes of data and model.
method Precise formulas for mean test error, bias, and variance in the mo∞ with m/dr constant regime. result A peak in the learning curve at m≈dr/r! for any integer r. The existence of evasion attacks during the test phase of machine learning algorithms represents a significant challenge to both their deployment and understanding. These attacks can be carried out by adding imperceptible perturbations to inputs to generate adversarial examples and finding effective defenses and detect…
The paper analyzes the risk of bagging regularized M-estimators under proportional asymptotics.
problem Characterizing the risk of ensemble estimators trained with subsamples and regularizers.
method Developed a consistent estimator for the risk of ensemble estimators under proportional asymptotics.
result Optimal subsample size k⋆ tends to be in the overparameterized regime for the full-ensemble estimator. The paper explains two distinct peaks in generalization error for neural networks and simpler models, each governed by different factors.
problem Understanding the peaks in generalization error for neural networks and simpler models.
method Analysis of random feature models and comparison with numerical experiments involving deep neural networks.
result The peaks at N=P and N=D are distinct and governed by different factors (noise sensitivity vs. initialization noise).