This paper establishes a theoretical foundation for consistency training in diffusion models.
problem Lack of a comprehensive theoretical understanding of consistency training in diffusion models.
method Demonstrates the necessity of a number of steps in consistency learning exceeding d5/2/ε for generating samples within ε proximity to the target distribution. result Establishes rigorous insights into the validity and efficacy of consistency models, offering theoretical underpinnings for their utility.
Consistency regularization improves robustness to noisy labels.
problem Improving model robustness to noisy labels in machine learning.
method Empirical study of consistency regularization on noisy datasets.
result Consistency regularization improves model robustness to label noise.
Unified model improves sampling speed and quality.
problem Difficult to balance sampling speed and quality.
method Multistep Consistency Models combining consistency and diffusion models.
result Improved sampling quality with reduced steps.
ADCMs adaptively discretize CMs for efficient training.
problem Manual discretization schemes cause repeated adjustments for different noise schedules and datasets.
method Unified framework with optimization problem, local and global consistency constraints, and Gauss-Newton method.
result Significantly improve training efficiency and generative performance of CMs.
Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedure…
Chronologically consistent models maintain accuracy with time-restricted data.
problem Training data introduces lookahead bias and training leakage in large language models.
method ChronoBERT and ChronoGPT trained with only available data at each time point.
result Models achieve strong performance and competitive with larger models, mitigating lookahead bias.
CoVAE improves VAEs by reducing training steps and improving quality.
problem Two-stage training overhead and increased sampling times.
method Consistency training of VAEs with progressive latent representations and time-dependent β parameter.
result CoVAE generates high-quality samples in one or few steps.
Paper analyzes consistency between neural networks at different levels.
problem Understanding knowledge consistency between neural networks.
method Generic definition and task-agnostic method to disentangle consistent knowledge.
result Knowledge consistency provides new insights and can improve network performance.
ICT improves semi-supervised learning by making predictions consistent at data interpolations.
problem Improving semi-supervised learning performance with limited labeled data.
method ICT encourages consistent predictions at data interpolations, reducing overfitting.
result ICT achieves state-of-the-art performance on CIFAR-10 and SVHN datasets.
Improved continuous-time consistency models for large-scale image generation.
problem Training instability and discretization errors in existing diffusion models.
method Unified theoretical framework, improved diffusion process, and network architecture.
result Trained continuous-time CMs at 1.5B parameters, achieving state-of-the-art FID scores.
Proposes a new method for rank-consistent ordinal regression without weight-sharing constraints.
problem Ordinal response variables in real-world prediction problems are often ignored by conventional classification losses.
method CORN framework using conditional training sets and the chain rule for conditional probability distributions.
result Improves performance substantially compared to the CORAL reference approach without weight-sharing restrictions.
Consistency models generate high-quality samples fast and without iterative sampling.
problem Slow generation in diffusion models.
method Direct mapping of noise to data, supporting fast one-step generation and multistep sampling.
result Consistency models achieve state-of-the-art FID scores and outperform diffusion models in one-step generation.
Efficiently creates label-consistent backdoor attacks without obvious mislabeling.
problem Vulnerability of deep neural networks to backdoor attacks that can be triggered by a backdoor trigger.
method Uses adversarial perturbations and generative models to inject inputs that are consistent with their labels.
result Demonstrates the feasibility of creating undetectable backdoor attacks by maintaining label-consistency.
Improved TTS style transfer across disjoint datasets with adversarial cycle consistency.
problem Suboptimal TTS style transfer on disjoint datasets with underrepresented styles.
method Adversarial cycle consistency training with paired and unpaired triplets.
result 78% improvement in style transfer with minimal reduction in fidelity and naturalness.
Paper introduces elastic consistency for distributed SGD, enabling convergence analysis.
problem Training large-scale machine learning models in distributed environments.
method Introduces elastic consistency as a general consistency model for distributed SGD.
result Derives convergence bounds for various distributed SGD methods.
Consistency distillation reduces memorization in diffusion models without harming sample quality.
problem Understanding how distillation affects memorization in diffusion models.
method Analysis of consistency distillation in diffusion models using a random feature neural network model.
result Consistency distillation reduces memorization in diffusion models without harming sample quality.
Paper tackles zero-shot translation by encouraging consistent agreement in models.
problem Challenges of generalizing multilingual translation without parallel data.
method Reformulated as probabilistic inference, introduced consistent agreement-based training.
result Agreement-based learning improves zero-shot translation by 2-3 BLEU points.
VAE fails to encode typical samples, new method improves robustness.
problem VAE does not necessarily encode typical samples generated by its decoder.
method Alternative construction of variational approximation distribution, self-consistency approach.
result Encoders trained with self-consistency approach are robust to adversarial attacks.
Improved unsupervised word translation using adversarial autoencoder with cycle consistency and input reconstruction.
problem Challenging language pairs and lack of parallel data for unsupervised word translation.
method Adversarial autoencoder with cycle consistency and input reconstruction regularization.
result More stable and better performance than recent approaches.
Improved neural network surrogates for ICF using manifold and cycle consistency.
problem Modeling and predicting complex physical processes in inertial confinement fusion.
method Training neural network surrogates that are consistent with the physical manifold and cyclically consistent.
result Surrogates are superior in predictive performance, more resilient to sampling artifacts, and more data efficient.
This paper improves adversarial robustness of deep learning models.
problem Vulnerability of machine learning models to adversarial perturbations.
method Analyzes adversarial training for linear regression and neural networks, incorporating L1 penalty.
result Incorporating L1 penalty leads to consistent adversarially robust estimation in high-dimensional settings.
Dissertation tackles zero-shot anomaly detection, focusing on consistent anomalies and proposing CoDeGraph framework.
problem Consistent anomalies bias distance-based zero-shot anomaly detection methods.
method Formalized consistent anomalies, identified similarity scaling and neighbor-burnout phenomena, introduced CoDeGraph framework.
result CoDeGraph effectively suppresses consistent anomalies in zero-shot anomaly detection.
Self-consistency improves the accuracy of model comparison methods.
problem Improving the accuracy of model comparison methods when simulation models are misspecified.
method Supplement traditional simulation-based training with a self-consistency loss on unlabeled real data.
result Self-consistency training improves model comparison accuracy, especially in open-world scenarios.
A new training method for GANs improves image generation quality.
problem Training GANs is challenging and unstable.
method Consistency regularization to stabilize GAN training.
result CR-GAN achieves best FID scores for unconditional image generation.
Single-step samplers generate high-quality samples efficiently.
problem Sampling from unnormalized distributions is computationally expensive.
method Developed consistent diffusion samplers that generate samples in a single step.
result Single-step samplers produce high-fidelity samples with less than 1% of traditional samplers' evaluations.
New method uses consistent gradient estimators for SGD in interconnected graphs.
problem Efficiently computing unbiased gradient estimators in interconnected graph scenarios.
method Uses consistent gradient estimators as an alternative to unbiased ones.
result Consistent estimators lead to the same convergence behavior as unbiased ones in various objective types.
New CycleGAN uses invertible generator for faster, less resource-intensive CT denoising.
problem Efficient unsupervised CT denoising without paired data.
method Single generator with wavelet residual domain, no discriminators, cycle consistency via invertible generator.
result Significantly improved denoising performance with faster training and less parameters.
By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using…
SCR improves GNN training with consistency regularization.
problem Balancing labeled and unlabeled data in GNNs.
method Introduces two consistency regularization strategies: perturbed predictions and Mean Teacher.
result Enhances various GNNs to achieve better performance.
New method interpolates training data and is consistent for various data distributions.
problem Establishing generalization guarantees for ensemble methods in the interpolating regime.
method Developed manifold-Hilbert kernel for Riemannian manifolds and used it in ensemble classification.
result Consistent ensemble classification method for broad data distributions.
Our work presents extensive empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. In particular, larger cosine distances between final an…
A new method for weakly supervised learning that improves model accuracy.
problem Training machine learning models with precise labels is expensive; weak supervision provides a low-cost alternative.
method Data consistent weak supervision algorithm that searches over classifiers to find plausible labelings, considering features of the training data and estimating labels for low/no coverage data.
result Empirically, the method significantly outperforms state-of-the-art weak supervision methods on text and image classification tasks.
Develops methods to create consistent surrogate models for agent-based simulators.
problem High computational costs and misjudgment of interventions in agent-based models.
method Causal abstractions to learn interventionally consistent surrogate models.
result Surrogates trained for interventional consistency closely mimic the agent-based model's behavior under interventions.
This paper describes the participation of Amobee in the shared sentiment analysis task at SemEval 2018. We participated in all the English sub-tasks and the Spanish valence tasks. Our system consists of three parts: training task-specific word embeddings, training a model consisting of gated-recurrent-units (GRU) with …
NCT simplifies one-step generator adaptation to new controls.
problem Adapting one-step generators to new control conditions.
method Noise Consistency Training (NCT) integrates new controls without retraining.
result NCT achieves state-of-the-art controllable generation in a single pass.
Measures consistency of tabular LLM predictions under fine-tuning multiplicity.
problem Conflicting predictions from fine-tuned tabular LLMs.
method Local stability measure in embedding space.
result Probabilistic guarantees on prediction consistency under multiplicity.
Unified framework for certifying LLM reliability without extra supervision.
problem Improving reliability of large language models without additional supervision.
method Unified framework using majority voting and Martingale Majority Certificate (MMC).
result Certifiable inference in LLMs with statistical guarantees and adaptive stopping rules.
New models avoid lookahead bias by training on past data only.
problem Lookahead bias in language models.
method Chronologically consistent training on data before a knowledge-cutoff date.
result Elimination of lookahead bias in predictions.
Improved semi-supervised learning with advanced data augmentation.
problem Enhancing deep learning models with limited labeled data.
method Substitutes simple noising operations with advanced data augmentation methods like RandAugment and back-translation.
result Achieves substantial improvements across multiple tasks and datasets.
Survey and framework for consistent uncertainty quantification in deep learning.
problem Partial uncertainty coverage and inconsistencies in deep learning uncertainty quantification.
method Bayes' theorem and conditional probability densities applied to all major sources of uncertainty.
result Improved robustness and reliability of neural network predictions in real-world scenarios.
Often the challenge associated with tasks like fraud and spam detection[1] is the lack of all likely patterns needed to train suitable supervised learning models. In order to overcome this limitation, such tasks are attempted as outlier or anomaly detection tasks. We also hypothesize that out- liers have behavioral pat…
PINNs struggle with data-to-PDE inconsistencies, limiting their accuracy.
problem Data inconsistency in PINNs affects their accuracy and convergence.
method Systematic analysis of PINNs with varying data fidelity and residual errors.
result PINNs saturate at an error level dictated by data inconsistency.
SC improves robustness in model comparison for misspecified models.
problem Model misspecification challenges in amortized Bayesian inference.
method Parameter posterior-based methods augmented with SC training.
result SC improves robustness under model misspecification.
Paper proposes MUCS for more reliable TDA in diffusion models.
problem Current TDA approaches lack reliability and robustness.
method Mirrored unlearning and noise-consistent skew (MUCS).
result MUCS outperforms existing methods on three datasets.
Diffusion models' consistency across splits explained by random matrix theory.
problem Consistency of diffusion models trained on non-overlapping subsets.
method Random matrix theory framework to quantify dataset effects on denoiser and sampling map.
result The theory explains and predicts cross-split disagreement in diffusion models.
Improved incremental sequence classification with temporal consistency.
problem Updating predictions as new sequence elements are revealed.
method Temporal-difference learning and a temporal-consistency condition for successive predictions.
result Optimizing a novel loss function improves data efficiency and predictive accuracy.
Consistent partial identification of causal effects proved for neural models.
problem Consistency of neural causal partial identification methods.
method Proving consistency for neural models with continuous and categorical variables, considering architecture design and Lipschitz regularization.
result Proven consistency of partial identification via neural causal models in a general setting.
Proposes PSGAN for generating high-res anime images with structural consistency.
problem Lack of high-quality, structurally consistent full-body high-resolution anime images.
method Progressive Structure-conditional Generative Adversarial Networks (PSGAN) with progressive training.
result Demonstrates effectiveness through comparisons and diverse anime character generation.