Paper extends nonparametric regression bounds for dependent β-mixing samples.
problem Analyzing error in nonparametric regression with dependent data.
method Extends uniform deviation inequalities from independent to dependent β-mixing samples. result Derives generalization bounds for nonparametric regression with dependent data.
Paper discovers simplicial complexes connecting trained models for improved ensembling.
problem Improving robustness and accuracy of deep learning ensembles.
method Identifies mode-connecting simplicial complexes on loss surfaces.
result Efficiently builds simplicial complexes for ensembling, outperforming independent ensembles.
Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectures, on different variations of spiral datasets, and using different optimization parameters. We used this dataset to train task-independent…
A common assumption in causal modeling posits that the data is generated by a set of independent mechanisms, and algorithms should aim to recover this structure. Standard unsupervised learning, however, is often concerned with training a single model to capture the overall distribution or aspects thereof. Inspired by c…
The paper tackles extrapolation in generative models by enforcing independence of mechanisms.
problem How to make generative models extrapolate to new, unseen environments?
method Developed a theoretical framework for independence of mechanisms, demonstrated on toy examples and real-world data.
result Extrapolation capabilities of generative models can be improved by enforcing independence of mechanisms explicitly during training.
SpeakerStew verifies 46 languages with reduced training and inference costs.
problem Speaker verification for 46 languages with smart speaker interactions.
method Pooling multilingual data, triage between text-dependent and text-independent models.
result Training on multiple languages generalizes well and reduces computational requirements.
DiffusionBlocks trains neural networks by breaking them into independent blocks, reducing memory usage.
problem Memory bottlenecks in end-to-end neural network training.
method Transforming transformer-based networks into independent trainable blocks via a denoising process.
result Independent block-wise training matches end-to-end training performance while reducing memory requirements.
RTFE provides adversarial robustness to multiple models.
problem Adversarial examples can transfer to other models, compromising robustness.
method Proposes RTFE, a deep learning-based pre-processing mechanism.
result RTFE provides adversarial robustness to multiple independently trained classifiers.
Parallel score matching accelerates DPM training and improves density estimation.
problem Extended training periods and limited modeling flexibility in DPMs.
method Partitioning the learning task into independent time sub-intervals and modeling the score at each time point separately.
result Significant acceleration of training process and improved density estimation performance.
We show how to estimate a model's test error from unlabeled data, on distributions very different from the training distribution, while assuming only that certain conditional independencies are preserved between train and test. We do not need to assume that the optimal predictor is the same between train and test, or t…
New method uses MMD estimators to enforce model invariance with missing data.
problem Models trained on missing data can fail on related test distributions.
method Derives MMD estimators for enforcing model invariance under missing nuisances.
result Optimizing through MMD estimates achieves similar test performance to using full data.
Collecting the large datasets needed to train deep neural networks can be very difficult, particularly for the many applications for which sharing and pooling data is complicated by practical, ethical, or legal concerns. However, it may be the case that derivative datasets or predictive models developed within individu…
Distributed machine learning (ML) can bring more computational resources to bear than single-machine learning, thus enabling reductions in training time. Distributed learning partitions models and data over many machines, allowing model and dataset sizes beyond the available compute power and memory of a single machine…
Superposition accelerates training to a universal power-law exponent.
problem Training dynamics in neural networks.
method Teacher-student framework and analytic theory.
result Superposition leads to a universal power-law exponent of ~1, independent of data and channel statistics.
Half-AVAE enhances VAE for underdetermined ICA with adversarial training.
problem Challenges in ICA under underdetermined conditions.
method Encoder-free VAE with adversarial networks and EE terms.
result Half-AVAE outperforms baseline models in underdetermined ICA.
New framework improves model robustness by focusing on stable relations across environments.
problem Standard supervised learning fails under data distribution shift.
method Gradient-based learning framework derived from the principle of independent causal mechanisms (ICM).
result Models generalize well to unseen scenarios, ignoring unstable relations.
Deep ReLU networks generalize well with few parameters.
problem Generalization of overparametrized deep neural networks.
method Explicit bounds on test error independent of overparametrization and VC dimension.
result Generalization error is independent of network architecture and overparametrization.
This research tackles group fairness in predictive process monitoring by ensuring predictions are independent of sensitive group membership.
problem Predictive models using biased historical data can perpetuate unfair behavior in new cases.
method Investigates independence through metrics like ΔDP and a composite loss function balancing predictive performance and fairness.
result Proposes and validates a composite loss function for training models that balance fairness and performance.
Redundancy in AI perception systems doesn't guarantee independent error occurrences.
problem Lack of direct statistical evidence of super-human automated driving performance.
method Investigated the effectiveness of redundancy in neural networks for independent error occurrences.
result Errors in neural networks for computer vision tasks are correlated, not independent.
New method aggregates Gaussian experts by detecting conditional independence violations.
problem Aggregation of dependent Gaussian experts leads to sub-optimal solutions.
method Uses Gaussian graphical model to detect and correct conditional independence violations.
result Improves aggregation of Gaussian experts, outperforming SOTA DGP approaches.
Representing distributions over permutations can be a daunting task due to the fact that the number of permutations of n objects scales factorially in n. One recent way that has been used to reduce storage complexity has been to exploit probabilistic independence, but as we argue, full independence assumptions impo…
The study finds a trade-off between model size, test loss, and training loss for linear predictors.
problem Finding the optimal balance between model size, test loss, and training loss for linear predictors.
method Established an algorithm and distribution-independent trade-off using non-asymptotic analysis.
result Models with low test loss are either classical (close to noise level training loss) or modern (large number of parameters).
In this work, we propose a novel technique to boost training efficiency of a neural network. Our work is based on an excellent idea that whitening the inputs of neural networks can achieve a fast convergence speed. Given the well-known fact that independent components must be whitened, we introduce a novel Independent-…
Estimates individual treatment effects using gradient interpolation and kernel smoothing.
problem Estimating individualized continuous treatment effects in observational data.
method Augment training data with independently sampled treatments and inferred counterfactual outcomes using gradient interpolation and kernel smoothing.
result Our method outperforms state-of-the-art methods on counterfactual estimation error.
Recent works have cast some light on the mystery of why deep nets fit any data and generalize despite being very overparametrized. This paper analyzes training and generalization for a simple 2-layer ReLU net with random initialization, and provides the following improvements over recent works: (i) Using a tighter char…
Gaussian processes adapted for Riemannian manifolds using gauge-independent kernels.
problem Deploying Gaussian processes on non-Euclidean domains like Riemannian manifolds.
method Developed techniques to generalize Gaussian processes to vector fields on Riemannian manifolds using gauge-independent kernels.
result Enabled training of vector-valued Gaussian processes on Riemannian manifolds using standard Gaussian process methods.
We investigate the effects of the unsupervised pre-training method under the perspective of information theory. If the input distribution displays multiple views of the supervision, then unsupervised pre-training allows to learn hierarchical representation which communicates these views across layers, while disentangli…
Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying causes. We propose Recurrent Independent Mechanisms (RIMs), a new recurrent architecture in which multiple groups of recurrent cells operate wit…
We propose a general framework for denoising high-dimensional measurements which requires no prior on the signal, no estimate of the noise, and no clean training data. The only assumption is that the noise exhibits statistical independence across different dimensions of the measurement, while the true signal exhibits s…
We address an anomaly detection setting in which training sequences are unavailable and anomalies are scored independently of temporal ordering. Current algorithms in anomaly detection are based on the classical density estimation approach of learning high-dimensional models and finding low-probability events. These al…
New bounds for KANs trained with DP-SGD, addressing correlated noise.
problem Risk bounds for Kolmogorov-Arnold Networks trained by DP-SGD with correlated noise.
method Established new optimization and population risk analysis for KANs trained with DP-SGD, addressing correlated noise.
result First optimization and population risk analysis of correlated-noise mechanisms for DP training in non-convex settings, including neural networks.
Researchers enhance EfficientNet models for practical efficiency on Graphcore IPU.
problem Improving practical efficiency of EfficientNet models on high-performance accelerators.
method Group convolutions, proxy-normalized activations, and reduced training resolution.
result Improves practical efficiency for both training and inference on Graphcore IPU.
We propose a new approach, called cooperative neural networks (CoNN), which uses a set of cooperatively trained neural networks to capture latent representations that exploit prior given independence structure. The model is more flexible than traditional graphical models based on exponential family distributions, but i…
SIGMA prior enables federated learning for non-factorizable models.
problem Current FL methods assume conditional independence, limiting applicability to non-factorizable models.
method SIGMA prior approximates deep generative model to induce conditional independence structure.
result SIGMA prior expands FL applicability to fields requiring modeling dependencies.
Scalable Gaussian process models trained with unbiased stochastic ELBO.
problem Training large capacity Gaussian process models on huge datasets.
method Unbiased stochastic variational inference for scalable GPs.
result Accurate inference on large datasets with up to 10 million basis functions.
Proposes a new approach to time series representation learning by embedding patches independently.
problem Capturing dependencies between time series patches is not optimal for representation learning.
method 1) Patch reconstruction task, 2) Patch-wise MLP, 3) Complementary contrastive learning.
result Improves time series forecasting and classification performance compared to state-of-the-art models.
This work proposes splitting deep neural networks into smaller sub-networks for faster and more efficient distillation.
problem Challenges in training deep neural networks, including local optima, gradient issues, and computational demands.
method Proposes a non-end-to-end distillation approach by splitting networks into smaller, independent sub-networks (neighbourhoods).
result Independent training of smaller sub-networks can speed up distillation and improve efficiency in various applications.
New method separates objects from images using deep neural networks trained to inpaint.
problem Fully self-supervised instance separation of occluded objects in images.
method Maximizes independence of two image regions given a fully self-supervised inpainting network.
result Method achieves similar segmentation performance to fully supervised methods on microscopy image datasets.
Many machine learning algorithms are based on the assumption that training examples are drawn independently. However, this assumption does not hold anymore when learning from a networked sample because two or more training examples may share some common objects, and hence share the features of these shared objects. We …
The paper uses distance covariance to improve fairness in machine learning models.
problem Improving fairness in machine learning models.
method Using conditional and distance covariance statistics to assess independence and add a penalty for fairness.
result The method effectively reduces the fairness gap in machine learning models.
New methods identify concepts in trained embeddings reliably without human labels.
problem Identifying interpretable concepts in trained embedding spaces without human labels.
method Explicitly connecting concept discovery to PCA and ICA, proposing novel approaches for dependent concepts.
result Proven methods outperform competitors on a variety of experiments, achieving up to 29% better alignment with ground truth.
New metric measures dynamical richness without relying on accuracy.
problem Lack of a reliable metric for measuring dynamical richness.
method Developed a computationally efficient, performance-independent metric based on low-rank bias.
result Metric recovers neural collapse as a special case and captures known transitions without accuracy.
Meta-learning bound uses conditional mutual information.
problem Bounding generalization performance in meta-learning.
method Extends CMI framework to meta-learning with a meta-supersample.
result Explicit bound involving two CMI terms.
Stochastic trace estimation with tensor train random vectors
problem Stochastic trace estimation for large-scale matrices
method Gaussian random tensor train vectors
result Median-of-means variant achieves dimension-independent guarantees
DIET tests conditional independence using marginal dependence measures of residual information.
problem Computational intractability of conditional randomization tests (CRTs).
method DIET avoids fitting large models by leveraging marginal independence statistics of information residuals.
result DIET achieves higher power than other tractable CRTs on synthetic and real benchmarks.
Proposes SVI for covariate-shift generalization with sparse variable independence.
problem Covariate-shift generalization with limited data and unstable variables.
method Introduces sparsity constraint and combines reweighting and selection in an iterative way.
result Improves covariate-shift generalization performance on synthetic and real-world datasets.
Support Vector Data Description (SVDD) is a popular outlier detection technique which constructs a flexible description of the input data. SVDD computation time is high for large training datasets which limits its use in big-data process-monitoring applications. We propose a new iterative sampling-based method for SVDD…
We present a variational renormalization group (RG) approach using a deep generative model based on normalizing flows. The model performs hierarchical change-of-variables transformations from the physical space to a latent space with reduced mutual information. Conversely, the neural net directly maps independent Gauss…