Training logs can improve model comparison precision, but careful covariate selection is key.
problem Improving precision in comparing stochastically trained models.
method Use arm-specific covariate adjustment, where each model is adjusted with statistics from its own runs.
result Simple adjustments based on early training logs often reduce uncertainty in model comparisons.
Paper proposes a log-domain training method to reduce neural network complexity.
problem High computational complexity in training deep neural networks limits real-time training.
method End-to-end training and inference scheme using approximate logarithmic operations in the log-domain.
result 16-bit log-based training achieves within 1% accuracy of floating-point baselines.
LogAnMeta detects anomalies from log events using meta learning.
problem Poor performance of current log anomaly detection on new or unseen anomalies.
method Meta-learning-based hybrid few-shot classifier trained in an episodic manner.
result Demonstrates efficacy of LogAnMeta on detecting anomalies with few samples.
Log-Normal Multiplicative Dynamics improves low-precision training of neural networks.
problem Training large neural networks with low precision is unstable.
method Derive a Bayesian learning rule with log-normal posterior distributions and multiplicative updates.
result LMD achieves stable and accurate training for Vision Transformer and GPT-2.
Deep signature/log-signature FBSDE algorithm improves accuracy and training time.
problem Solving FBSDEs with state and path dependent features.
method Incorporates deep signature/log-signature transformation into RNN model.
result Improves accuracy and training time compared to existing methods.
Maximum likelihood training improves the performance of score-based diffusion models.
problem Training score-based diffusion models with maximum likelihood.
method Trained by minimizing a weighted combination of score matching losses, with a specific weighting scheme that bounds negative log-likelihood.
result Maximum likelihood training improves the log-likelihood of score-based diffusion models across multiple datasets.
The study proves a theorem about subword complexity for free group automorphisms.
problem Analyzing subword complexity for attracting fixed points of automorphisms of free groups.
method Combinatorial arguments and train tracks.
result Subword complexity of attracting fixed points is equivalent to n, n log log n, n log n, or n^2.
End-to-end models perform better with learned log-scaled mel-spectrogram features.
problem End-to-end neural network models struggle with performance compared to models using high-level data representations.
method Trained first layers of a CNN model on log-scaled mel-spectrogram transformation and then used these learned features to initialize an end-to-end CNN classifier.
result Convergence and performance on ESC-50 dataset are similar to a model trained on pre-processed log-scaled mel-spectrogram features.
This paper introduces a method to estimate log-likelihood in VAE models.
problem Difficulty in comparing models trained via ELBO due to lack of log-likelihood.
method Introduces a general upper bound to approximate model evidence.
result Efficiently approximates model evidence and compares to other bounds.
End-to-end training improves segmental models for speech recognition.
problem Training segmental models is challenging and lacks understanding under different settings and losses.
method Investigated a model class based on recent successful approaches, trained end-to-end with various losses.
result Marginal log loss provides the most consistent strong performance without requiring ground-truth alignments.
Dividing deep learning models for consistent anomaly detection in changing log data.
problem Anomaly detection methods fail when log data types change, leading to false negatives.
method Divide deep learning models based on log data correlation and extract correlations.
result Continues anomaly detection accuracy even when log data changes.
Derives M2VAE objective from marginal joint log-likelihood.
problem Training Multi-Modal Variational Autoencoders (M2VAEs). method Derives trainable evidence lower bound from marginal joint log-likelihood.
result Derives M2VAE objective from marginal joint log-likelihood. EnLSTM network improves log generation from small datasets.
problem Generating well logs from small datasets with high accuracy.
method Combining ENN and C-LSTM networks with perturbation methods.
result 34% reduction in mean-square-error compared to existing models.
Paper derives a simplified formula for Expected Improvement using log-transformed data.
problem Challenges in enhancing Bayesian optimization with Expected Improvement.
method Derives a closed form of Expected Improvement for Gaussian process trained on log-transformed objective.
result Provides a simplified formula for Expected Improvement.
Quantum annealer speeds up RBM training for image classification.
problem Training RBM with contrastive divergence (CD) is slow and computationally expensive.
method Used D-Wave 2000Q quantum annealer to calculate model expectation of gradient learning for RBM.
result Quantum training yields similar classification performance to CD but faster.
Improved Gaussian process regression with tighter log marginal likelihood bounds.
problem Improving predictive performance in Gaussian process regression models.
method Lower bound on log marginal likelihood using conjugate gradients.
result Improved predictive performance compared to other conjugate gradient based approaches.
Differentially private log-location-scale regression models improve privacy in statistical analysis.
problem Ensuring privacy in statistical regression models while maintaining accuracy.
method Integrates differential privacy into LLS regression using the functional mechanism.
result Proposed DP-LLS models satisfy ε-differential privacy and perform well under various conditions.
This paper improves SNN training by using multiple sample compartments.
problem Training SNNs with single-sample estimators leads to inaccurate log-likelihood estimates.
method Proposes a GEM-based online learning algorithm that uses multiple independent spiking signals.
result Significant improvements in log-likelihood, accuracy, and calibration with multiple compartments.
SUMO provides unbiased log marginal likelihood estimation for latent variable models.
problem Biased estimates of log marginal likelihood in latent variable models.
method Randomized truncation of infinite series for unbiased estimation.
result Models trained with SUMO give better test-set likelihoods than standard methods.
Improved GEC models use scored data from large pretraining to outperform.
problem Addressing data sparsity in Grammatical Error Correction.
method Derive example-level scores from a smaller, higher-quality dataset and incorporate delta-log-perplexity into training schedules.
result Models trained on scored data achieve state-of-the-art results.
Deterministic training improves generative autoencoder performance.
problem Stochastic training limits generative autoencoder performance.
method Invertible layers for deterministic training.
result AEFs outperform VAEs in log-likelihood and sample quality.
PACMAN provides bounds for classification tasks considering accuracy vs. negative log-loss mismatch.
problem Mismatch between accuracy and negative log-loss in classification tasks.
method Point-wise PAC approach over generalization gap, using likelihood ratio and concentration inequalities.
result PACMAN provides point-wise PAC bounds for the generalization problem.
New clustering technique improves RNN event log predictions.
problem Leveraging event attributes for better RNN predictions.
method A novel clustering technique for event attributes.
result Improved prediction accuracy with reduced training time.
Bayesian approach estimates log-determinant with uncertainty quantification.
problem Intractable computation of log-determinant in large kernel matrices.
method Reinterpreting as Bayesian inference with prior bounds and evidence.
result Probabilistic estimates of log-determinant and uncertainty.
A new method normalizes EBM training by introducing a learnable parameter.
problem Training energy-based models with maximum likelihood is challenging due to intractable normalisation constants.
method Proposes a self-normalised log-likelihood (SNL) objective that introduces a learnable parameter representing the normalisation constant.
result The SNL objective is a lower bound of the log-likelihood and can be directly optimised using stochastic gradient techniques.
This paper improves low-precision sampling using SGHMC for deep learning models.
problem Enhancing training efficiency of deep neural networks with low-precision training.
method Investigates low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) for both log-concave and non-log-concave distributions.
result Low-precision SGHMC achieves quadratic improvement in error compared to SGLD for non-log-concave distributions.
DNLL loss improves deep LDA accuracy and consistency.
problem Pathological solutions in unconstrained Deep LDA.
method Introducing Discriminative Negative Log-Likelihood (DNLL) loss.
result Deep LDA trained with DNLL produces clean latent spaces and better calibrated probabilities.
Flexible log file parsing using HMM adapts to evolving content.
problem Dynamic log file processing with evolving content.
method Modeling frequent patterns into HMM for flexible log file parsing.
result High accuracy (over 99%) in parsing different system log files.
New bounds improve training of VAEs on noisy data.
problem Training Variational AutoEncoders on noisy data.
method Proposes Kullback-Leibler and Rényi divergence bounds for log-likelihood.
result Numerically stable training without extra noise.
A new method speeds up training of deep models by avoiding Jacobian determinant computation.
problem Efficiently training deep neural networks with complex log-determinant terms.
method Relative gradients to compute Jacobian updates efficiently.
result Training neural networks with Jacobian log-determinant objectives becomes feasible.
Log-Loss scores expose membership privacy breaches.
problem Privacy leakage from statistical aggregates like Log-Loss scores.
method Proved that Log-Loss scores enable full accuracy membership inference in a single query.
result Complete membership privacy breach is possible with Log-Loss scores.
GAMs combine autoregressive and log-linear components for data-efficient sequence learning.
problem Poor performance of standard autoregressive models under small-data conditions.
method Introduce Global Autoregressive Models (GAMs) combining autoregressive and log-linear components, trained in two steps.
result GAMs show a strong perplexity reduction over standard models in language modelling.
We explore how neural networks train to zero loss, focusing on initial scale.
problem Understanding neural network training dynamics and zero loss.
method Macroscopic limits analysis of gradient descent dynamics.
result Gradient descent can drive deep neural networks to zero loss regardless of initialization.
Residual Flows improve flow-based models for density estimation.
problem Density estimation using flow-based models with biased log-density estimates.
method Proposed a Russian roulette estimator for unbiased log-density estimation and used an alternative infinite series for gradient calculation. Improved invertible residual blocks with activation functions avoiding derivative saturation and generalized Lipschitz condition to induced mixed norms.
result Residual Flows achieve state-of-the-art performance on density estimation and outperform coupling block networks in joint generative and discriminative modeling.
This paper introduces a new learning rule for probabilistic SNNs that improves log-likelihood, accuracy, and calibration.
problem Training and inference of deterministic SNNs are constrained by their inability to generate multiple independent outputs.
method Introduces a generalized expectation-maximization (GEM) learning rule for probabilistic SNNs.
result The GEM-SNN learning rule leads to significant improvements in log-likelihood, accuracy, and calibration.
Proposes a method for training Bayesian neural networks using synthetic data from Raman and CARS spectra.
problem Limited real observations in Raman and CARS spectroscopy.
method Log-Gaussian Gamma Processes and Bayesian Neural Networks.
result Trained Bayesian neural networks provide accurate estimates of Raman and CARS spectra with uncertainty quantification.
Glow uses invertible 1x1 convolutions to improve image generation and manipulation.
problem Efficient and realistic image generation and manipulation.
method Invertible 1x1 convolutions in generative flows.
result Significant improvement in log-likelihood and realistic image synthesis.
Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly with the number of output classes, which can restrict the size of problems we are able to tackle with current hardware. Second, it remains unc…
A fast Gaussian process kernel approximates log marginal likelihood efficiently.
problem Efficiently approximating Gaussian process log marginal likelihood for large datasets.
method GRIEF kernel using Nystrom approximation and algebraic tensor products.
result Fast computation of log marginal likelihood and hyperparameter derivatives.
Proposes a new loss function for deep neural networks.
problem Deep neural networks lack a direct method to discriminate between correct and competing classes.
method Introduces a discriminative loss function based on negative log likelihood ratio.
result Significantly outperforms cross-entropy loss on image classification tasks.
Deep learning compresses and quantizes log-likelihood ratios for fading channels.
problem Efficiently compress and quantize log-likelihood ratios for fading channels.
method Trains a deep autoencoder network to map log-likelihood ratios to a latent space and reconstruct them.
result Achieves a compression factor of nearly three times with minimal performance loss.
Researchers develop variational bounds for perceptron learning from structured data.
problem Learning from structured data with concave utilities and log-concave priors.
method Variational approach combining interpolation method, log-concavity, and concentration estimates.
result Lower and upper minimax variational bounds match, identifying the solution of the model.
Neural networks can interpolate noisy data and still generalize well.
problem Generalization of neural networks trained on noisy data.
method Two-layer neural networks trained to interpolation by gradient descent on corrupted labels.
result Neural networks can achieve zero training error and optimal test error.
Improved diffusion bridge sampling with rKL-LD loss.
problem Improving sampling from unnormalized distributions using diffusion bridges.
method Employing the rKL-LD loss instead of the Log Variance (LV) loss for diffusion bridges.
result rKL-LD consistently outperforms LV loss in diffusion bridges.
Improved speech recognition with language model integration in sequence-to-sequence models.
problem Improving word error rate in speech recognition models.
method Log-linear combination of acoustic and language models with per-token renormalization.
result The proposed method shows good improvements over standard model combination on Librispeech system.
A simple log-transform fixes heavy-tailed data for generative models.
problem Standard generative models struggle with heavy-tailed data.
method Apply the soft-log transform to data before training and exponentiate samples after generation.
result Log-FM outperforms specialized baselines on multivariate benchmarks.
Improves latent variable use in generative models.
problem Poor use of latent variables in variational autoencoders (VAEs).
method Combines data log likelihood with autoencoder reconstruction likelihood.
result Ensures latent variable captures observation information and generates well.
Proposes a new multi-scale architecture for generative flows to improve log-likelihood and sampling quality.
problem Challenges of high-dimensional latent space in flow models.
method Data-dependent dimension factorization based on likelihood contribution heuristic.
result Improvements in log-likelihood score and sampling quality on image benchmarks.