Paper computes MMD for Gaussian kernel and proposes improvements for WAEs.
problem Estimating MMD between distributions for machine learning applications.
method Closed-form expressions for Gaussian kernel MMD, batch normalization at code layer.
result Improved interpretability and performance of WAEs with analytic MMD.
We propose the Wasserstein Auto-Encoder (WAE)---a new algorithm for building a generative model of the data distribution. WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational Aut…
The variational autoencoder (VAE) imposes a probabilistic distribution (typically Gaussian) on the latent space and penalizes the Kullback--Leibler (KL) divergence between the posterior and prior. In NLP, VAEs are extremely difficult to train due to the problem of KL collapsing to zero. One has to implement various heu…
WAEs offer a statistical understanding of density estimation and error bounds.
problem Concurrent density estimation with neural network-induced transformations.
method Statistical analysis of WAEs focusing on upper bounds and error propagation.
result Established deterministic upper bounds on WAE errors and explored their resilience.
This paper provides statistical guarantees for WAE's latent space regeneration.
problem Lack of statistical analysis for Autoencoders, especially WAE.
method Utilizes Vapnik Chervonenkis (VC) theory and Optimal Transport of measures under the Wasserstein metric.
result WAE achieves the target distribution in the latent space and regenerates the input distribution.
We study the role of latent space dimensionality in Wasserstein auto-encoders (WAEs). Through experimentation on synthetic and real datasets, we argue that random encoders should be preferred over deterministic encoders. We highlight the potential of WAEs for representation learning with promising results on a benchmar…
Wasserstein Autoencoders improve model efficiency and interpretability for low-dimensional data.
problem Limited statistical guarantees for WAEs in low-dimensional data.
method Proper network architecture selection and analysis of expected excess risk convergence rates.
result WAEs can learn data distributions efficiently when intrinsic dimension is considered.
Develops a new non-adversarial framework for better generative models.
problem Inaccurate approximation of target distribution in latent space.
method Tessellated Wasserstein Auto-Encoders (TWAE) using centroidal Voronoi tessellation (CVT) to tessellate latent space.
result Significantly enhances generative performance in terms of FID compared to existing models.
We propose a new generative model, Cramer-Wold Autoencoder (CWAE). Following WAE, we directly encourage normality of the latent space. Our paper uses also the recent idea from Sliced WAE (SWAE) model, which uses one-dimensional projections as a method of verifying closeness of two distributions. The crucial new ingredi…
This work reveals a primal-dual relationship between GANs and Autoencoders, improving their theoretical understanding.
problem Improving the theoretical understanding of GANs and Autoencoders.
method Study of f-GAN and WAE models, finding a primal-dual relationship and proving generalization bounds. result The f-GAN and WAE objectives are equivalent under certain assumptions, leading to improved theoretical understanding. A new method for conditional sampling using paired Wasserstein Autoencoders.
problem Conditional sampling from complex data distributions.
method Derive a novel loss function for Wasserstein Autoencoders to enable sampling from OT-type couplings.
result Learned cost-optimal transport maps and conditional sampling from an OT-type coupling.
A new method uses persistent homology to assess auto-encoders' latent manifold quality.
problem Chaos in auto-encoders' latent manifold and failure of current distance measures.
method Persistent Homology for Wasserstein Auto-Encoders (PHom-WAE).
result PHom-WAE improves auto-encoders' performance in credit card transaction data.
Proposes a method to train deep models with one-element batches.
problem Training deep models with small batches (one element) is challenging.
method Splits the batch into historical and current elements for training.
result Allows training on higher resolution images with one-element batches.
In this paper we study generative modeling via autoencoders while using the elegant geometric properties of the optimal transport (OT) problem and the Wasserstein distances. We introduce Sliced-Wasserstein Autoencoders (SWAE), which are generative models that enable one to shape the distribution of the latent space int…
Proposes TCWAE to learn disentangled representations using the Wasserstein Autoencoder.
problem Balancing reconstruction fidelity and disentanglement in learning representations.
method TCWAE (Total Correlation Wasserstein Autoencoder) using different KL estimators.
result Competitive results on data sets with known generative factors, and improved reconstructions on unknown factors.
FlexAE addresses bias-variance trade-off in RAEs by learning latent priors.
problem Improving generation quality of deterministic AE models.
method Introducing flexibly learnable latent priors in WAEs to optimize the latent distribution.
result FlexAE achieves state-of-the-art performance in AE-based generative models.
Method evaluates disentanglement in DLVMs, including those not aligned with latent axes.
problem Evaluate disentanglement in DLVMs, especially those not aligned with latent axes.
method Proposes a statistical method to discover generative factors of a dataset.
result Empirically demonstrates the advantage of the method on two datasets.
C-VAE improves VAE by resolving prior issues and generating better samples.
problem Low-quality samples from VAE due to prior issues.
method Formulates VAE as OT, allows flexible priors, and uses OT formulations.
result C-VAE generates higher quality samples and latent representations.
Proposes a new approach to enforce uniform distribution on torus.
problem Enforcing uniform distribution on torus for generative models.
method Introduces circular spring loss to enforce equally spaced points on torus.
result Enables morphing between points on torus with different paths.
Generative AutoEncoders require a chosen probability distribution in latent space, usually multivariate Gaussian. The original Variational AutoEncoder (VAE) uses randomness in encoder - causing problematic distortion, and overlaps in latent space for distinct inputs. It turned out unnecessary: we can instead use determ…
LANCA uses ANM to learn latent causal factors without supervision.
problem Learning latent causal factors without supervision.
method LANCA employs a deterministic Wasserstein Auto-Encoder coupled with a differentiable ANM Layer.
result LANCA outperforms baselines on physics and photorealistic environments.
Increasingly, discrimination by algorithms is perceived as a societal and legal problem. As a response, a number of criteria for implementing algorithmic fairness in machine learning have been developed in the literature. This paper proposes the Continuous Fairness Algorithm (CFAθ) which enables a continuous interpol…
Taylorized training improves neural network training at finite width.
problem Understanding and improving neural network training at finite width.
method Training the k-th order Taylor expansion of the neural network at initialization.
result Taylorized training agrees with full neural network training better as k increases and can significantly close the performance gap.
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
Un-trained neural networks outperform trained methods in MRI reconstruction.
problem Accelerated MRI reconstruction with minimal training data.
method Variation of Deep Decoder without training data.
result Un-trained approach significantly outperforms other methods in reconstruction accuracy.
Meta-learning performance depends on train-validation split type.
problem Understanding the importance of train-validation split in meta-learning.
method Theoretical and experimental study comparing train-val and train-train methods.
result Train-train method can achieve strictly better excess loss in realizable cases.
New method reveals how training data influence diffusion model outputs.
problem Difficulty in assessing training data impact on diffusion model outputs.
method Use of ensembles trained on carefully engineered splits of training data to identify influential training examples.
result Demonstrated the viability of ensembles as generative models and validity of assessing influence.
Paper shows adversarial training can be fooled by new type of noise.
problem Adversarial training can be fooled by new types of noise.
method Designing ADVIN, a new type of inducing noise.
result ADVIN can degrade adversarial training robustness by 99.9%.
New MIP methods improve training of integer-valued neural networks.
problem Training integer-valued neural networks with limited data and resources.
method Formulated new MIP models to optimize training efficiency and handle more data.
result Significantly outperforms previous state-of-the-art methods in accuracy, training time, and data usage.
Free adversarial training reduces the generalization gap compared to vanilla method.
problem Improving generalization in adversarial training.
method Analysis of algorithmic stability in free adversarial training.
result Free adversarial training shows a lower generalization gap.
ADASS selects adaptive subsets for SGD training acceleration.
problem Fixed sample size in SGD limits training efficiency.
method ADASS selects adaptive subsets based on Lipschitz constants.
result ADASS achieves comparable accuracy with full training set.
Solve-training trains neural nets to map physical solutions efficiently.
problem Representing complex physical solutions with neural networks.
method Variational training using loss functions from physical models.
result Effective neural network representation of solution maps without expensive labels.
Making neural networks robust against adversarial inputs has resulted in an arms race between new defenses and attacks. The most promising defenses, adversarially robust training and verifiably robust training, have limitations that restrict their practical applications. The adversarially robust training only makes the…
learn2mix trains neural nets faster by adjusting class proportions dynamically.
problem Training neural nets efficiently with limited resources and imbalanced classes.
method Adaptive class proportion adjustment during training.
result Neural nets trained with learn2mix converge faster than static methods.
Loss-guided training accelerates node embedding methods on graphs.
problem Training efficiency in graph learning methods with implicit positive examples.
method Dynamic adjustment of training distribution based on loss values.
result Significant acceleration in training and computation over static methods.
Paper explores fast adversarial training to improve robustness with less computation.
problem Efficiently defending against adversarial examples.
method Integrates simple self-attacks for faster training, focusing on overfitting recovery.
result Shows superior robust accuracy with reduced training time compared to strong adversarial training.
Free adversarial training improves robustness without generating adversarial examples.
problem Training robust models against adversarial attacks is costly and impractical for large-scale datasets.
method Recycles gradient information from parameter updates to generate adversarial examples.
result Free adversarial training achieves comparable robustness to PGD training at negligible cost.
Detects backdoors in trained models without poisoned training data.
problem Detecting backdoors in DNNs trained without access to the poisoned training set.
method Proposes a novel detector using the maximum achievable misclassification fraction (MAMF) statistic.
result Detects backdoors and infers source and target classes.
Pre-training improves model robustness and uncertainty.
problem Improving model robustness and uncertainty in machine learning.
method Adversarial pre-training and task-specific methods.
result Approximately a 10% absolute improvement in adversarial robustness.
Crowdsourced training of large neural networks with decentralized Mixture-of-Experts.
problem Expensive training of large neural networks limits research contributions.
method Learning@home: decentralized Mixture-of-Experts for large, poorly connected participants.
result Performance and reliability of Learning@home surpass conventional distributed training.
Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…
New method reconstructs significant parts of training data from neural networks.
problem Understanding and reconstructing training data from neural networks.
method Proposes a novel reconstruction scheme based on recent theoretical results about neural network training.
result Shows that a significant fraction of training data can be reconstructed from neural network parameters.
Reduces deep learning training data for faster testing.
problem Resource-intensive deep learning training with full data sets.
method Evaluated different training set reduction methods.
result Training set reduction is useful in resource-constrained environments.
The paper studies the asymptotic behavior of adversarial training under ℓ∞-perturbation.
problem Theoretical guarantees for sparsity-recovery in adversarial training.
method Investigation of the asymptotic distribution of the adversarial training estimator in generalized linear models.
result The asymptotic distribution of the adversarial training estimator under ℓ∞-perturbation could have a positive probability mass at 0 when the true parameter is 0. This paper introduces 'General Cyclical Training' for neural networks.
problem Improving training efficiency and performance of neural networks.
method Cyclical training phases with varying hyperparameters, batch sizes, loss functions, and data augmentation.
result Cyclical weight decay, softmax temperature, and gradient clipping enhance model accuracy.
Speeds up training and inference by pruning entire channels before training.
problem Training and inference speed in deep neural networks.
method Structured pruning applied before training, focusing on removing entire channels and hidden units.
result 2x speedup in training and 3x speedup in inference.
The paper predicts loss scaling across different datasets and compute scales.
problem Predicting loss scaling across different datasets and compute scales.
method Derive shifted power law relationships between train and test losses.
result Shifted power law relationships hold for various datasets and tasks, improving prediction accuracy.
Theoretical study shows adversarial training improves robustness in deep learning models.
problem Ensuring robustness in pre-trained deep learning models.
method Theoretical analysis of adversarial training and feature purification in two-layer neural networks.
result Adversarial training leads to feature purification, making models more robust to attacks.