Improved BP on CRBMs yields better structured prediction results.
problem Structured prediction on CRBMs using current methods is suboptimal.
method Matrix-based belief propagation implementation for CRBMs.
result BP on CRBMs provides significantly better results than CD methods.
This paper provides a tutorial on Boltzmann Machines and Deep Belief Networks.
problem Understanding and applying Boltzmann Machines and Deep Belief Networks.
method Explains the structures, conditional distributions, Gibbs sampling, training methods, and deep belief networks of RBMs.
result Comprehensive overview of RBMs and DBNs, useful in various fields.
The paper explores properties and Bayesian fitting of RBMs, addressing their limitations in learning and uncertainty quantification.
problem The generative behavior of RBMs is largely unexplored and typical fitting methodology does not easily allow for uncertainty quantification.
method The paper discusses the relationship between RBM parameter specification, model properties, and difficulties with likelihood-based inference. It also further discusses the potential Bayes fitting of RBMs, especially Gibbs sampling methods.
result The paper highlights the need for better understanding and quantification of uncertainty in RBM fitting.
Reformulates RBM for unified linear and nonlinear dimensionality reduction.
problem Traditional RBM limitations in handling both linear and nonlinear data.
method Reformulates RBM using MAP and EM, introduces deterministic CD algorithm.
result Reformulated RBM can outperform PCA in nonlinear dimensionality reduction.
This paper uses RBM to calculate conditional probabilities for nonlinear system identification.
problem Challenges in obtaining probability distributions for nonlinear system identification.
method Modified RBM to calculate joint, input, and conditional probabilities.
result The method outperforms other black-box models in noisy, complex systems.
A new RBM model handles both linear and log-amplitude spectrograms.
problem Handling amplitude spectra with existing models.
method Proposed gamma-Bernoulli RBM that uses gamma distribution.
result The model can naturally handle positive numbers and log-amplitude spectrograms.
RBM generates complex, graded data features.
problem Extracting complex features from high-dimensional data.
method Characterized structural conditions for RBM to generate compositional representations.
result RBM can operate in a compositional phase under specific conditions.
RBM and TNS are shown to be equivalent, bridging deep learning and quantum physics.
problem Understanding the relationship between RBM and TNS for better model design.
method Developed algorithms to translate between RBM and TNS, and vice versa.
result RBM and TNS have equivalent expressive power and can be transformed into each other.
New method uses RBM flows to find critical temperatures in Ising models.
problem Detecting critical temperatures in RBM flows without model topology information.
method Iterative sampling from RBM mapped on Ising model temperature space using a neural network thermometer.
result Flow of RBM trained on Ising spin configurations approaches critical temperature around kBTc/J≈2.269. New algorithm embeds crime events using RBMs with feature selection.
problem Capturing complex crime data features for event embedding.
method Regularized RBMs with ℓ1 penalty for feature selection. result Improved event embedding with interpretable selected features.
We show how deep learning methods can be applied in the context of crowdsourcing and unsupervised ensemble learning. First, we prove that the popular model of Dawid and Skene, which assumes that all classifiers are conditionally independent, is {\em equivalent} to a Restricted Boltzmann Machine (RBM) with a single hidd…
Lean 2-layer RBMs achieve similar representational power as single-layer RBMs with fewer parameters.
problem Understanding and quantifying the representational power of multi-layer RBMs.
method Inherent Structure Capacity (ISC) and Lean RBMs.
result 2-layer RBMs can achieve the same representational power as single-layer RBMs with fewer parameters.
Proposes p-RBM for retaining memory in RBM models for stock market prediction.
problem Lack of memory in RBM models for dynamic data like time-series analysis.
method Generalizes RBM to p-RBM, capable of retaining p past states, and trains using contrastive divergence.
result p-RBM offers promising prediction potential for stock market direction.
Improved RBM training speed using parallel computing.
problem High computational cost of training RBM models.
method Parallelized training using Horovod framework.
result Training time reduced to less than 12 minutes on 64 CPU nodes.
Quantum machine learns faster by reverse annealing on AQCs.
problem Training RBMs on AQCs is hard due to low qubit connectivity.
method Embedding RBM nodes to virtual qubits, semantic quantum search, reverse annealing schedule.
result Reverse annealing accelerates RBM training and improves reconstruction scores.
New sampling method for leaky RBM using Gaussian annealing.
problem Sampling from leaky RBM with numerical stability issues.
method Anneal leakiness from Gaussian to ReLU units.
result Efficient sampling with improved likelihood estimation and faster mixing.
We consider relations between two families of flat manifolds with holonomy group (Z_2)^k of diagonal type. The family RBM of real Bott manifolds and the family GHW of generalized Hantzsche-Wendt manifolds. In particular, we prove that the intersection GHW∩RBM is not empty. We also …
A new sampler and temperature estimation method enable efficient learning of Boltzmann Machines.
problem Efficient learning of Boltzmann Machines (BMs) is challenging due to high training costs and difficulty in parallelization.
method Proposed a new Boltzmann sampler (Langevin SB, LSB) and an efficient method (Conditional Expectation Matching, CEM) for estimating inverse temperature.
result Established an efficient learning framework (Sampler-Adaptive Learning, SAL) for BMs with greater expressive power than Restricted Boltzmann Machines (RBMs).
Restricted Boltzman Machines (RBMs) have been successfully used in recommender systems. However, as with most of other collaborative filtering techniques, it cannot solve cold start problems for there is no rating for a new item. In this paper, we first apply conditional RBM (CRBM) which could take extra information in…
Restricted Boltzmann Machines (RBMs) are generative models which can learn useful representations from samples of a dataset in an unsupervised fashion. They have been widely employed as an unsupervised pre-training method in machine learning. RBMs have been modified to model time series in two main ways: The Temporal R…
Quantum annealer speeds up RBM training for image classification.
problem Training RBM with contrastive divergence (CD) is slow and computationally expensive.
method Used D-Wave 2000Q quantum annealer to calculate model expectation of gradient learning for RBM.
result Quantum training yields similar classification performance to CD but faster.
Generative machine learning models reveal latent travel behavior characteristics.
problem Understanding complex travel behavior through latent variables.
method Developed a joint tri-partite Bayesian graphical network model using RBM.
result Significant improvement in model likelihood compared to traditional models.
Compact model uses RBMs for sequence classification with fewer parameters.
problem Sequence classification with dynamic models and complex neural networks.
method Rolling RBMs over time for representation learning and temporal inference.
result Outperforms state-of-the-art models in melody and character recognition.
This paper shows RBMs can maintain strong performance even after extreme pruning, but only if done early in training.
problem The computational and environmental costs of large neural networks.
method Investigating the performance of RBMs under extreme pruning conditions, inspired by the Lottery Ticket Hypothesis.
result RBMs can achieve high-quality generative performance even after 80% pruning, but performance degrades sharply above a critical point.
BEAMs use adversarial training to improve RBM performance.
problem Likelihood-based training fails to penalize high-probability regions.
method Adversarial training against RBM hidden layer activations.
result BEAMs outperform RBMs and GANs on benchmarks.
New algorithm for learning RBMs with sparse latent variables.
problem Learning RBMs with sparse latent variables efficiently.
method Algorithm with time complexity O(n^(2^s+1)) for sparse RBMs.
result Improves learning time for RBMs with sparse latent variables.
Hardware-accelerated RBM solves large combinatorial problems and integer factorization.
problem Solving large combinatorial optimization and integer factorization problems.
method Logically synthesized RBM architecture, hardware acceleration, and efficient training methods.
result Hardware-accelerated RBM factorizes 16-bit numbers with 10000x speed and 32x power improvements.
Online learning algorithm for RBM adapts network architecture to streaming data.
problem Adapting RBM architecture to streaming data for better feature representation and classification.
method Two-phase training: online generative phase for unsupervised feature learning, discriminative phase for classification.
result Online RBM converges to a stable, concise network architecture with inherently discriminative neurons.
Study proposes an RBM with multivalued hidden variables to improve generalization.
problem Improving generalization in RBMs to prevent overfitting.
method Introduced an RBM with multivalued hidden variables as a simple extension of conventional RBMs.
result The proposed model outperforms conventional RBMs in contrastive divergence learning and MNIST classification.
Study extends deep learning theory to long-range spin systems.
problem Exploring deep learning in long-range spin systems.
method MCMC simulations and RBM training for critical temperature and scaling dimensions.
result RBM flow for long-range models does not converge to correct scaling dimensions.
RBM learns in high dimensions via AMP and GD, reaching optimal weak recovery.
problem Learning from high-dimensional data with RBM.
method AMP and GD analysis for RBM training in high dimensions.
result RBM reaches optimal weak recovery threshold in spiked covariance model.
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences.…
Infinite RBMs use Frank-Wolfe for efficient training and initialization.
problem Training infinite RBMs with sparse solutions.
method Frank-Wolfe algorithm for constrained convex optimization.
result Infinite RBMs can be trained efficiently and initialized effectively.
This paper generalizes SSD for Gaussian RBMs, improving convergence for continuous data.
problem Improving convergence of RBMs with Gaussian inputs using SSD.
method Deriving upper bounds of logarithmic partition function for RBMs via Schatten-infinity norm.
result Empirical improvement of SSD over SGD for Gaussian RBMs.
This work analyzes centered binary Restricted Boltzmann Machines (RBMs) and binary Deep Boltzmann Machines (DBMs), where centering is done by subtracting offset values from visible and hidden variables. We show analytically that (i) centering results in a different but equivalent parameterization for artificial neural …
Improved sampling quality with RBM-Flow and D-Flow models.
problem Efficient sampling of complex data distributions using invertible flows.
method Implement RBM-Flow and D-Flow models with discrete latent variables.
result Significant improvement in sampling quality over baseline models.
D-Wave quantum annealing fails to improve sampling quality from RBMs compared to Gibbs sampling.
problem Improving sampling quality from RBMs using D-Wave quantum annealing.
method Comparison of D-Wave quantum annealing and Gibbs sampling for RBM sampling.
result D-Wave sampling does not significantly improve the number of local valleys compared to Gibbs sampling.
Conditional Restricted Boltzmann Machines (CRBMs) are rich probabilistic models that have recently been applied to a wide range of problems, including collaborative filtering, classification, and modeling motion capture data. While much progress has been made in training non-conditional RBMs, these algorithms are not a…
Approximate Message Passing (AMP) has been shown to be an excellent statistical approach to signal inference and compressed sensing problem. The AMP framework provides modularity in the choice of signal prior; here we propose a hierarchical form of the Gauss-Bernouilli prior which utilizes a Restricted Boltzmann Machin…
RBM and DBM are represented as 2D tensor networks, revealing their expressive power and efficiency.
problem Understanding and optimizing RBM and DBM models.
method Representing RBM and DBM as 2D tensor networks and developing an efficient tensor network contraction algorithm.
result The proposed algorithm for computing partition functions is more accurate than state-of-the-art methods.
The Restricted Boltzmann Machines (RBM) can be used either as classifiers or as generative models. The quality of the generative RBM is measured through the average log-likelihood on test data. Due to the high computational complexity of evaluating the partition function, exact calculation of test log-likelihood is ver…
Neural network feature extraction is compared to statistical physics renormalization group flow.
problem Understanding how deep neural networks extract hierarchical features from input images.
method Used RBM to model Ising model and analyze weight matrices of trained RBM.
result RBM trained on spin configurations flows towards critical temperature Tc instead of typical RG flow. Improved RBM training using MCLV-K outperforms CD-K on MNIST.
problem Training RBMs efficiently and with statistical guarantees.
method Markov Chain Las Vegas (MCLV-K) with stopping sets.
result MCLV-K significantly outperforms CD-K on MNIST.
New method transfers instances between domains using Gibbs Sampling and RBM.
problem Transfer Learning between domains with limited target data.
method Gibbs Sampling and RBM for instance transition.
result Significant improvement in target classification.
Restricted Boltzmann Machines (RBMs) are one of the fundamental building blocks of deep learning. Approximate maximum likelihood training of RBMs typically necessitates sampling from these models. In many training scenarios, computationally efficient Gibbs sampling procedures are crippled by poor mixing. In this work w…
Deep networks generalize well due to hidden mechanisms like renormalization.
problem Generalization in deep learning networks.
method Analyzing RBMs and autoencoders, applying renormalization group principles.
result Deep networks can generalize well with fewer parameters than expected.
Restricted Boltzmann machines (RBMs) are powerful machine learning models, but learning and some kinds of inference in the model require sampling-based approximations, which, in classical digital computers, are implemented using expensive MCMC. Physical computation offers the opportunity to reduce the cost of sampling …
RBMs model binary interactions with hidden node activation effects.
problem Understanding how RBM hidden node activation affects binary variable distributions.
method Investigated RBM marginal distributions with different hidden node activation functions.
result Found exact expressions for RBM marginals as interacting binary variables.