TFCheck detects training issues in ML programs using TensorFlow.
problem Training programs often contain inconsistencies and bugs.
method Developed a TensorFlow library (TFCheck) with verification routines.
result TFCheck successfully detects training issues in ML code.
This paper explores optimization methods for deep learning.
problem How to effectively train neural networks.
method Discusses gradient explosion/vanishing, initialization, normalization, SGD, adaptive gradient methods, distributed methods, and global issues.
result Provides theoretical insights and practical solutions for neural network training.
DGPs with variational inference suffer from SNR issues that degrade gradient estimates, leading to unreliable training.
problem SNR issues in gradient estimates for DGPs with variational inference.
method Adapted doubly reparameterized gradient estimators for DGP training.
result Fix improves predictive performance of DGP models.
Disagreement-based approaches generate multiple classifiers and exploit the disagreement among them with unlabeled data to improve learning performance. Co-training is a representative paradigm of them, which trains two classifiers separately on two sufficient and redundant views; while for the applications where there…
Paper improves volatility forecasting for new issues and spin-offs.
problem Forecasting volatility with limited historical data.
method Multi-source transfer learning approach.
result Transfer learning approach outperforms alternative models.
High volume of data, perceived as either challenge or opportunity. Deep learning architecture demands high volume of data to effectively back propagate and train the weights without bias. At the same time, large volume of data demands higher capacity of the machine where it could be executed seamlessly. Budding data sc…
GANs analyzed for performance and training issues.
problem Performance and training issues of GANs.
method SDE approximations for training GANs.
result Improved understanding of GANs through analytical perspectives.
Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to solve this issue by pr…
The paper tackles scalarization issues in A2C RL algorithms, proposing methods to avoid gradient overlap and noise.
problem Scalarization issues in A2C RL algorithms leading to gradient overlap and uncontrolled noise.
method Proposes techniques to avoid gradient overlap and noise in A2C RL algorithms.
result Pilot experiments show the proposed method speeds up training in A2C RL algorithms.
Paper proposes faster certified robust training methods with short warmup.
problem Certified robust training methods require long warmup schedules, making training costly.
method Proposes three improvements: new weight initialization, BN, and regularization.
result Achieves 65.03% verified error on CIFAR-10 and 82.36% on TinyImageNet with short warmup.
In domains such as health care and finance, shortage of labeled data and computational resources is a critical issue while developing machine learning algorithms. To address the issue of labeled data scarcity in training and deployment of neural network-based systems, we propose a new technique to train deep neural net…
Improved GAN training stability through clipping and reweighting.
problem Inconsistent GAN training leading to inferior performance.
method Proposes a variational GAN framework with probability ratio clipping and sample reweighting.
result Significantly improved performance across various GAN tasks.
A DRL approach optimizes resource allocation in BFL to reduce latency and energy consumption.
problem Energy and CPU constraints of mobile devices and increased training latency due to blockchain mining.
method Deep Reinforcement Learning (DRL) to derive optimal decisions for MLMO.
result Optimal resource allocation and block generation rate to minimize system latency, energy consumption, and incentive cost.
A new method improves data generation quality by correcting score mismatches.
problem Score mismatch issue in conditional score-based data generation methods.
method Denoising Likelihood Score Matching (DLSM) loss for classifier training.
result The proposed method outperforms previous methods on Cifar-10 and Cifar-100 benchmarks.
Kernel-guided training stabilizes GANs by controlling discrepancies.
problem Stability and interpretability issues in GANs.
method Kernel-based regularization to control discrepancies in GAN loss function.
result Theoretical guarantees on stability of the training dynamics.
Paper identifies objective mismatch in MBRL, affecting control task performance.
problem Objective mismatch in MBRL framework affects control task performance.
method Proposes re-weighting dynamics model training to mitigate mismatch.
result Likelihood of one-step ahead predictions is not always correlated with control performance.
Improved GANs mitigate forgetting and collapse issues.
problem Catastrophic forgetting and mode collapse in GANs.
method Contrastive learning and mutual information maximization.
result Significantly stabilizes GAN training and improves performance.
Modern implicit generative models such as generative adversarial networks (GANs) are generally known to suffer from issues such as instability, uninterpretability, and difficulty in assessing their performance. If we see these implicit models as dynamical systems, some of these issues are caused by being unable to cont…
Recently, deep learning has been applied to many security-sensitive applications, such as facial authentication. The existence of adversarial examples hinders such applications. The state-of-the-art result on defense shows that adversarial training can be applied to train a robust model on MNIST against adversarial exa…
Paper analyzes GANs training difficulties and proposes a control framework.
problem Difficulties in training GANs, especially for financial time series.
method Stochastic control framework for hyper-parameters tuning.
result Explicit forms for optimal adaptive learning rate and batch size derived.
Continuous Speech Keyword Spotting (CSKS) is the problem of spotting keywords in recorded conversations, when a small number of instances of keywords are available in training data. Unlike the more common Keyword Spotting, where an algorithm needs to detect lone keywords or short phrases like "Alexa", "Cortana", "Hi Al…
Improves forecast calibration for extreme events using modified loss functions.
problem Improperly specified models do not issue calibrated forecasts for extreme events.
method Adapting loss functions based on weighted scoring rules and tail miscalibration regularization.
result Calibrated forecasts for extreme wind speeds can be improved by suitable adaptations to the loss function during model training.
Recurrent models can produce infinite sequences, causing bias; new methods prevent this.
problem Inconsistency in decoding infinite-length sequences from recurrent language models.
method Defined and proved inconsistency of common decoding algorithms; proposed remedies.
result Proposed methods prevent inconsistency in practice.
Proposes QDF to improve multi-step time-series forecasting.
problem Ignoring label autocorrelation and unequal task weights in training objectives.
method Quadratic-form weighted training objective and QDF learning algorithm.
result Improves performance of various forecast models, achieving state-of-the-art results.
PCA is a classical statistical technique whose simplicity and maturity has seen it find widespread use as an anomaly detection technique. However, it is limited in this regard by being sensitive to gross perturbations of the input, and by seeking a linear subspace that captures normal behaviour. The first issue has bee…
This paper proposes a method to improve VAEs by extracting latent spaces from pre-trained diffusion models.
problem VAEs struggle with generating high-quality images due to unrealistic Gaussian assumptions.
method Optimizes an encoder to maximize marginal data log-likelihood and derives a decoder analytically.
result The method enhances VAE performance by discarding Gaussian assumptions and training a separate decoder network.
Adversarial examples are carefully perturbed in-puts for fooling machine learning models. A well-acknowledged defense method against such examples is adversarial training, where adversarial examples are injected into training data to increase robustness. In this paper, we propose a new attack to unveil an undesired pro…
HighwayGraph models long-distance node relations in GNNs with improved performance.
problem Limited-layer information propagation in GNNs hinders long-distance node relation modeling.
method Proposes two solutions: implicit and explicit modeling of long-distance node relations using shallow GNN architectures and a self-training framework.
result HighwayGraph achieves consistent and significant improvements over four GNNs on three benchmark datasets.
New theory uses probability sets for data variability, improving machine learning.
problem Variability in data distribution causes learning issues.
method Uses convex sets of probabilities (credal sets) to model data variability.
result Derives bounds for risk of models learned from multiple training sets.
Neural network models and deep models are one of the leading and state of the art models in machine learning. Most successful deep neural models are the ones with many layers which highly increases their number of parameters. Training such models requires a large number of training samples which is not always available…
The impressive success of Generative Adversarial Networks (GANs) is often overshadowed by the difficulties in their training. Despite the continuous efforts and improvements, there are still open issues regarding their convergence properties. In this paper, we propose a simple training variation where suitable weights …
New GAN formulation addresses mode collapse issue.
problem Mode collapse in GANs.
method Randomized decision rules, empirical Bayes, stochastic gradient MCMC.
result Proposed method converges to Nash equilibrium.
K-StoNet improves neural networks by avoiding local minima and assessing uncertainty.
problem Local minima and prediction uncertainty in deep neural networks.
method Combines SVR with latent variable model, using RBF kernel for feature space mapping and IRO algorithm for training.
result The model asymptotically converges to the global optimum and assesses prediction uncertainty easily.
Improved DLG extracts accurate labels from gradients, overcoming DLG's convergence issues.
problem Private training data leakage from shared gradients in distributed learning systems.
method Proposes iDLG, a simple approach to synthesize accurate labels from gradients.
result iDLG reliably extracts ground-truth labels from gradients, unlike DLG.
Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices. In this work, we show that multi-task learning is naturally suited to handle the statistical challenges of this setting, and propose a novel systems-aware optimization method, MOCHA,…
ProSelfLC improves robustness of deep neural networks by automatically deciding trust in predictions.
problem Training robust deep neural networks requires addressing issues like label noise and low entropy predictions.
method ProSelfLC progressively increases trust in predicted labels over time, considering entropy and learning time.
result ProSelfLC demonstrates improved robustness in both clean and noisy settings through empirical validation.
ANNs extrapolate without training data, leading to uncertain predictions.
problem Uncertainty in ANN predictions for unseen data.
method Quantitative approach based on long-standing methods.
result Quantitative uncertainty assessment for ANN predictions.
Paper proposes OKGAN to improve GAN training, addressing mode collapse and cycling.
problem Challenges in GAN training, including mode collapse and cycling.
method Kernel-based non-parametric discriminator for online training.
result OKGAN mitigates training issues and performs better than other GAN formulations.
MiM-StocR combines momentum indicators and adaptive ranking loss for better stock recommendation.
problem Lack of simultaneous short-term trend and ranking prediction in stock recommendation models.
method Integrates momentum indicators and proposes Adaptive-k ApproxNDCG for ranking optimization.
result MiM-StocR outperforms state-of-the-art MTL baselines in stock recommendation.
A new method trains deep neural networks for open set domain adaptation without negative open set difference.
problem Training deep neural networks for open set domain adaptation without negative open set difference.
method Proposes a new upper bound of target-domain risk, including source-domain risk, ε-open set difference (Δε), distributional discrepancy, and constant. Uses gradient descent for source-domain risk and Δε, and adversarial training for distributional discrepancy. Trains DNNs via minimizing the new upper bound. result Shows state-of-the-art performance on benchmark datasets.
Neural networks struggle with long sequences, but a new method improves their performance.
problem Neural networks struggle to generalize to longer sequences and unseen data.
method Proposed a learned conditional masking mechanism and binary encoding for numbers.
result Models can now generalize far outside their training range with near-perfect accuracy.
Sliced kernelized Stein discrepancy improves goodness-of-fit tests and model learning in high dimensions.
problem The curse-of-dimensionality in kernelized Stein discrepancy (KSD).
method Sliced Stein discrepancy and its scalable variants using optimal one-dimensional projections.
result Significantly outperforms KSD and baselines in goodness-of-fit tests and improves model learning.
Generative Adversarial Networks (GANs) have become a popular method to learn a probability model from data. In this paper, we aim to provide an understanding of some of the basic issues surrounding GANs including their formulation, generalization and stability on a simple benchmark where the data has a high-dimensional…
Improved computed tomography reconstruction with deep learning and deep image prior.
problem Low data efficiency in computed tomography reconstruction.
method Combining learned primal-dual methods with deep image prior for improved quality and generalization.
result Proposed methods outperform state-of-the-art in low data regime.
Unsupervised deep learning detects image quality issues without annotations.
problem Anomaly detection in images with quality issues.
method Use of deep generative models, specifically normalizing flows, for anomaly detection.
result Accurate semantic labeling and quality trends observed in images.
MTL method uses unlabeled data with pseudo labels to improve classification with disjoint datasets.
problem Improving classification performance with disjoint labeled datasets using unlabeled data.
method Proposes MTL-SA method to select and augment unlabeled data with confident pseudo labels and close distribution to labeled data.
result Extensive experiments show the effectiveness of MTL-SA method in improving classification performance.
Paper proposes LSVGD to stabilize GAN training via Langevin Stein Variational Gradient Descent.
problem Mode collapse and performance deterioration in GAN training.
method Langevin Stein Variational Gradient Descent (LSVGD) incorporating noise to stabilize training.
result LSVGD improves performance and stability of various GAN models.
As more researchers have become aware of and passionate about algorithmic fairness, there has been an explosion in papers laying out new metrics, suggesting algorithms to address issues, and calling attention to issues in existing applications of machine learning. This research has greatly expanded our understanding of…