Improved defense against data poisoning attacks by aggregating smaller subsets.
problem Mitigating the impact of poisoned data on model robustness.
method Finite Aggregation method that combines duplicates of smaller disjoint subsets for training.
result Consistent improvement in certified robustness bounds, up to 4.77% on GTSRB.
Trains neural networks to efficiently solve Navier-Stokes equations across parameter space.
problem Efficiently solving Navier-Stokes equations in parameter space.
method Physics-informed neural networks, active learning algorithm.
result Neural networks can accurately interpolate and aggregate solutions to physical problems.
The paper analyzes how the one-dimensional Wasserstein distance captures pointwise density differences in finite samples.
problem Uncertainty in identifying density differences when supports overlap and densities have substantial pointwise differences.
method Analysis using the Poisson process and neural spike train decoding.
result The one-dimensional Wasserstein distance highlights meaningful density differences related to both rate and support.
Framework solves physics-constrained inverse problems with limited data.
problem Physics-constrained inverse problems with scarce training data.
method Conditional flow matching for Bayesian inverse problems.
result Conditional flow matching mitigates degeneracy in finite training data.
Fewer data weight updates lead to faster convergence in machine learning models.
problem Improving robustness of machine learning models through data mixing.
method Analyzing convergence behavior of data mixing with a finite number of inner steps.
result The optimal number of inner steps scales with the budget and type of gradients used.
Bayesian HMM for protein alignment state estimation.
problem Estimating the MAP state sequence for a Bayesian HMM.
method Training data with Dirichlet priors, iterative algorithms for MAP path estimation.
result Bayesian setup outperforms frequentist estimation in protein alignment.
New theory uses probability sets for data variability, improving machine learning.
problem Variability in data distribution causes learning issues.
method Uses convex sets of probabilities (credal sets) to model data variability.
result Derives bounds for risk of models learned from multiple training sets.
New method uses random features and Tikhonov regularization for operator learning from noisy data.
problem Accurate approximation of mappings between infinite-dimensional function spaces with reduced training time.
method Regularized random Fourier features (RRFF) coupled with finite element reconstruction (RRFF-FEM).
result The method achieves improved performance with reduced training time and noise robustness.
Analyzes DNNs trained with noisy gradients, finding FWCs negligible for large n.
problem Analyzing DNNs trained with noisy gradients.
method Introduced analytical framework to analyze non-Gaussian stochastic process.
result FWCs negligible for large n, improving CNN performance.
New bounds show transformers need longer training for length generalization.
problem Understanding when transformers can generalize to longer inputs.
method Analyzing different settings of transformers, providing quantitative bounds.
result Transformers need training data longer than previously thought for length generalization.
While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary). Previous work has studied this tradeoff between standard and robust accuracy, but only in the setting where no predictor performs well on both objectives in the infinite data…
RMDA trains structured neural networks with regularization and variance reduction.
problem Training structured neural networks with desired properties.
method RMDA algorithm for structured NNs with regularization and variance reduction.
result RMDA achieves desired structures identical to regularizer's at stationary points.
FDNet learns PDEs from data with fast predictions.
problem Discovering complex systems behavior from data.
method Finite difference neural networks (FDNet) to learn PDEs from trajectory data.
result FDNet predicts future behavior with few trainable parameters.
Estimates KRR risk from training data for various kernels and hyperparameters.
problem Predicting the generalization error of Kernel Ridge Regression.
method Introduces SCT and KARE to approximate KRR risk from training data.
result KARE provides an excellent approximation of KRR risk and helps select good kernels.
Non-intrusive algorithm corrects corrupted data in classification models.
problem Classification problems with corrupted training data.
method Post-processes a trained classification model with a correction procedure.
result Corrected models deliver correct classification results even with corrupted data.
CoNNTrA trains DNNs with low-power, low-memory constraints.
problem Training deep neural networks on edge computing systems with low power and memory usage.
method Coordinate gradient descent-based approach for training DNNs with constrained learning parameters.
result CoNNTrA models use 32x less memory and have comparable errors to Backpropagation models.
The paper analyzes how adding gradient data affects overparameterized models' performance.
problem The impact of Sobolev training on high-dimensional predictive models.
method Combining replica method from statistical physics and operator-valued free probability theory.
result Sobolev training does not universally improve performance for target functions described by single-index models.
We present a generative model that is defined on finite sets of exchangeable, potentially high dimensional, data. As the architecture is an extension of RealNVPs, it inherits all its favorable properties, such as being invertible and allowing for exact log-likelihood evaluation. We show that this architecture is able t…
We consider a deep ReLU / Leaky ReLU student network trained from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). The student network is \emph{over-realized}: at each layer l, the number nl of student nodes is more than that (ml) of teacher. Under mild conditions…
NDDV estimates data point value from a single stochastic trajectory.
problem Estimating marginal contributions of data points over stochastic training paths.
method Introduces Neural Dynamic Data Valuation (NDDV) using stochastic state and adjoint equations.
result NDDV provides a one-run, trajectory-conditioned estimator of data point value.
Paper proposes a diagnostic tool for evaluating model performance out-of-sample.
problem Evaluating model performance on unseen data.
method Uses a finite calibration dataset to assess future losses.
result Provides guarantees under weak assumptions and quantifies distribution shifts.
Learn dynamics of a system using auxiliary data from similar systems.
problem Learning dynamics of a linear system with limited data.
method Weighted least squares approach, incorporating auxiliary data.
result Auxiliary data can help reduce intrinsic error due to noise.
The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accura…
Data splitting enhances model performance in overparametrized ridgeless regression.
problem Computational inefficiency in training models with large datasets.
method Data splitting as a regularization technique in overparametrized ridgeless regression.
result Data splitting improves statistical performance and computational complexity.
The paper analyzes the maximum margin algorithm's performance on noisy data.
problem Analyzing the performance of maximum margin algorithm on noisy data.
method Finite-sample analysis of maximum margin algorithm applied to noisy data.
result The maximum margin algorithm can achieve nearly optimal population risk with sufficient over-parameterization.
We propose a learning-based approach for estimating the spectrum of a multisinusoidal signal from a finite number of samples. A neural-network is trained to approximate the spectra of such signals on simulated data. The proposed methodology is very flexible: adapting to different signal and noise models only requires m…
Privacy concerns have led to the development of privacy-preserving approaches for learning models from sensitive data. Yet, in practice, even models learned with privacy guarantees can inadvertently memorize unique training examples or leak sensitive features. To identify such privacy violations, existing model auditin…
Deep networks can be biased to learn top eigenfunctions of the kernel outside the training set.
problem Spectral bias of deep networks in the kernel regime.
method Quantitative bounds on L2 difference between finite-width and infinite-width network trajectories. result Deep networks learn top eigenfunctions of the Neural Tangent Kernel over the entire input space, not just the training set.
Deep neural networks are often trained in the over-parametrized regime (i.e. with far more parameters than training examples), and understanding why the training converges to solutions that generalize remains an open problem. Several studies have highlighted the fact that the training procedure, i.e. mini-batch Stochas…
We implement an all-optical setup demonstrating kernel-based quantum machine learning for two-dimensional classification problems. In this hybrid approach, kernel evaluations are outsourced to projective measurements on suitably designed quantum states encoding the training data, while the model training is processed o…
Infinite-dimensional SBDMs improve image generation across multiple resolutions.
problem Efficient image generation at high resolutions and across different levels.
method Developed SBDMs in infinite-dimensional setting, using trace class operators and operator networks.
result Improved efficiency and generalization across resolution levels.
The paper tackles generalization in machine learning by finding invariant representations of data.
problem Obtaining robust models that generalize well across different training environments.
method The paper introduces the concept of ε-approximate invariance to study the robustness of models to unseen SEMs. result The paper provides finite-sample out-of-distribution generalization guarantees for approximate invariance in linear SEMs.
We propose \emph{Taylorized training} as an initiative towards better understanding neural network training at finite width. Taylorized training involves training the k-th order Taylor expansion of the neural network at initialization, and is a principled extension of linearized training---a recently proposed theory …
Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.
problem Training two-layer networks for binary classification.
method Gradient descent with logistic loss applied to two-layer networks.
result Gradient descent can drive training loss to zero under certain conditions.
A neural network model predicts the critical point of the Ising phase transition.
problem Predicting the critical point of the Ising phase transition using supervised learning.
method Proposed a minimal one-free-parameter neural network model to describe the supervised learning problem for the Ising model.
result Just one free parameter is enough to describe the universal finite-size-scaling function in the network output.
Optimizes weights for better model performance in shifting data.
problem Improper importance weighting leads to poor model performance in data shifts.
method Interprets weights as a bias-variance trade-off and optimizes them simultaneously with model parameters.
result Optimizing weights significantly improves model generalization performance.
Kernel-based GPs model continuous processes with uncountable information.
problem Uncountable information in Gaussian process modeling.
method Kernel-based approach using reproducing kernel Hilbert space.
result Proposed Gaussian process unifies finite and uncountable information.
Machine learning promises methods that generalize well from finite labeled data. However, the brittleness of existing neural net approaches is revealed by notable failures, such as the existence of adversarial examples that are misclassified despite being nearly identical to a training example, or the inability of recu…
We describe an adversarial learning approach to constrain convolutional neural network training for image registration, replacing heuristic smoothness measures of displacement fields often used in these tasks. Using minimally-invasive prostate cancer intervention as an example application, we demonstrate the feasibilit…
Smooth KANs improve model reliability in computational biomedicine.
problem Limited convergence of KANs in representing generic smooth functions.
method Introducing smooth, structurally informed KANs that can approximate MLPs in specific function classes.
result Smooth KANs can achieve equivalence to MLPs in specific function classes, enhancing model reliability and performance.
Study shows neural networks trained with GD converge to Gaussian processes with polynomial decay.
problem Understanding convergence of neural networks to Gaussian processes during training.
method Explicit upper bounds on quadratic Wasserstein distance between trained networks and Gaussian approximations.
result Polynomial decay of approximation error with network width and training time.
Develops a simple model to understand learning curves for arbitrary power laws.
problem Lack of theoretical understanding of scaling laws in machine learning.
method Analyzes a toy model to determine if learning curves are universal or depend on data distribution.
result Determines that learning curves can exhibit n−β for arbitrary power β>0. Self-training improves model accuracy by refining pseudo-labels.
problem Improving semi-supervised learning with self-training.
method Theoretical insights into self-training algorithm with a focus on linear classifiers.
result Self-training iterations can improve model accuracy even if stuck in sub-optimal fixed points.
High-dimensional models trained on smooth manifolds achieve optimal rates in Wasserstein metrics.
problem Training score-based generative models on complex, low-dimensional manifolds.
method Proves optimal rates for SGMs on smooth manifolds, separating into noise regimes and using ReLU nearest-projection coordinates.
result Optimal intrinsic Wasserstein rates are achieved, with polynomial ambient dependence for families with controlled geometry and density.
Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.
problem Optimal policies in finite-time MDPs are not stationary and require epoch-specific training.
method Introduces dynamic policy gradient combining dynamic programming and policy gradient, analyzes convergence for softmax parametrisation.
result Dynamic policy gradient training exploits finite-time structure, leading to better convergence bounds.
Gen-CUDE is a neural network for denoising noisy channels.
problem Denoising in finite-input, general-output noisy channels.
method Unsupervised neural network trained on noisy data.
result Gen-CUDE achieves better denoising results than other methods.
Bayesian ReLU nets fix asymptotic overconfidence with infinite features.
problem Bayesian ReLU nets can be asymptotically overconfident far from training data.
method Extend finite ReLU BNNs with infinite ReLU features via a Gaussian process.
result The resulting model is asymptotically maximally uncertain far from the data.
Convex optimization refines neural network training, improving model performance and reducing hyperparameter sensitivity.
problem Training deep neural networks using non-convex optimization methods often leads to suboptimal solutions and requires extensive tuning.
method Formulate neural network training as convex programs with regularization terms, leveraging sparse recovery models and semi-infinite programming theory.
result Convex models can achieve global optima and outperform traditional non-convex methods, with improved robustness to hyperparameters.