Paper proposes a new estimator for generic discrete distributions.
problem Estimating gradients for stochastic nodes in deep generative models.
method Generalized Gumbel-Softmax estimator using truncation, Gumbel-Softmax trick, and linear transformation.
result Efficacy and practical value demonstrated in synthetic examples and topic models.
Paper improves Gumbel-Softmax estimator variance reduction.
problem Challenges in gradient estimation for models with discrete latent variables.
method Rao-Blackwellization applied to straight-through Gumbel-Softmax estimator.
result Reduces mean squared error and variance of Gumbel-Softmax estimator.
Generative Adversarial Networks (GAN) have limitations when the goal is to generate sequences of discrete elements. The reason for this is that samples from a distribution on discrete objects such as the multinomial are not differentiable with respect to the distribution parameters. This problem can be avoided by using…
Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the inability to backpropagate through samples. In this work, we present an efficient gradient estimator that replaces the non-differentiable sa…
A new method reparameterizes Gaussian noise for better flexibility and performance.
problem Improving the Gumbel-Softmax for better flexibility and performance.
method Invertible Gaussian Reparameterization (IGR) using modified softmax and transformations.
result IGR outperforms Gumbel-Softmax in various experiments.
Many problems in real life can be converted to combinatorial optimization problems (COPs) on graphs, that is to find a best node state configuration or a network structure such that the designed objective function is optimized under some constraints. However, these problems are notorious for their hardness to solve bec…
Improved CAEs reduce training time and enhance generalization.
problem Stability issues in Concrete Autoencoders (CAEs) for feature selection.
method Indirectly Parameterized Concrete Autoencoders (IP-CAEs) learn parameters of Gumbel-Softmax distributions.
result IP-CAEs achieve significant improvements in generalization and training time.
For network architecture search (NAS), it is crucial but challenging to simultaneously guarantee both effectiveness and efficiency. Towards achieving this goal, we develop a differentiable NAS solution, where the search space includes arbitrary feed-forward network consisting of the predefined number of connections. Be…
This paper improves MADDPG's performance in discrete grid-world scenarios.
problem MADDPG struggles with discrete action spaces in grid-world tasks.
method Integrates alternative Gumbel-Softmax estimators into MADDPG.
result One alternative estimator significantly improves performance in grid-world tasks.
Neural jump model improves option pricing accuracy.
problem Jump risk in option pricing.
method Neural jump stochastic differential equation model with Gumbel-Softmax gradient learning.
result Neural jump components significantly improve option pricing accuracy.
Deep learning improves community detection in graph datasets.
problem Community detection in graph datasets using deep learning.
method Proposes a deep learning approach using Gumbel Softmax for clustering graph nodes.
result The new approach significantly outperforms traditional clustering methods.
A new algorithm optimizes graph problems faster and more accurately.
problem Hard optimization problems on graphs.
method Gumbel-softmax technique with gradient descent and evolution strategy.
result High-quality solutions obtained with less time.
Paper proposes a new method for predicting drug interactions using adversarial autoencoders.
problem Predicting drug interactions to prevent adverse events.
method Introduces adversarial autoencoders based on Wasserstein distances and Gumbel-Softmax relaxation to generate high-quality negative samples.
result Significant improvements in link prediction and DDI classification tasks.
Investigates statistical properties of perturb-softmax and perturb-argmax distributions.
problem Underexplored statistical properties of Gumbel-Softmax and Gumbel-Argmax distributions.
method Investigates convexity and differentiability to determine completeness and minimality of these distributions.
result Identifies parameters that admit complete and minimal representation of probability distributions.
Automated multi-task learning algorithm that optimizes network topology.
problem Over-sharing in multi-task learning leads to over-generalization and suboptimal performance.
method Tree-structured design space with gumbel-softmax sampling for differentiable network splitting.
result End-to-end trainable algorithm that optimizes network topology for multiple objectives across tasks.
Modified EAT method improves Poisson gradient estimation.
problem Challenging differentiation through Poisson-distributed latent variables.
method Exponential Arrival Time (EAT) simulation with modifications and Gumbel-SoftMax relaxation.
result Modified EAT method provides unbiased first moment and reduced second-moment bias.
New method resolves inconsistency in learning directed acyclic graphs using penalized likelihood.
problem Inconsistency of ℓ1-penalized likelihood in learning directed acyclic graphs. method Developed a hybrid differentiable structure learning method based on ℓ0-penalized likelihood with hard acyclicity constraint. result Demonstrated and explained why ℓ1-penalized likelihood is fundamentally inconsistent in identifying true structure up to Markov equivalence classes. Extends feature selection to GNNs, improving accuracy and feature ranking.
problem Improving feature selection in Graph Neural Networks (GNNs).
method Implemented a feature selection algorithm using Gumbel Softmax for ranking and selecting features in GNNs.
result Selected 225 features out of 1433 for the Cora dataset, improving classification accuracy.
We propose a method to train generative adversarial networks on mutivariate feature vectors representing multiple categorical values. In contrast to the continuous domain, where GAN-based methods have delivered considerable results, GANs struggle to perform equally well on discrete data. We propose and compare several …
Concrete distribution properties examined on simplex.
problem Properties of Concrete distribution on simplex.
method Reflection and location-scale transformation of uniform distribution; explicit parameterization to Poincaré half-space.
result Fisher information and information metric are hyperbolic space; Fisher-Rao geodesic distance computed.
In many applications we seek to maximize an expectation with respect to a distribution over discrete variables. Estimating gradients of such objectives with respect to the distribution parameters is a challenging problem. We analyze existing solutions including finite-difference (FD) estimators and continuous relaxatio…
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary lat…
Deep k-means learns clustering and features from unlabeled data.
problem Clustering and feature learning from unlabeled data.
method Gradient-estimator for non-differentiable k-means objective via Gumbel-Softmax reparameterisation.
result Concrete k-means model optimised for canonical k-means objective, end-to-end trainable.
Unified framework for gradient estimation in combinatorial spaces.
problem Scaling relaxed gradient estimators to large combinatorial distributions.
method Introducing stochastic softmax tricks within the perturbation model framework.
result Stochastic softmax tricks improve model performance and discover more latent structure.
Many machine learning tasks require sampling a subset of items from a collection based on a parameterized distribution. The Gumbel-softmax trick can be used to sample a single item, and allows for low-variance reparameterized gradients with respect to the parameters of the underlying distribution. However, stochastic o…
Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data. Learning in such models is difficult, however, because exact marginalization over these combinatorial objects is intractable. In response, this paper introduces a collectio…
A new method for categorical variational inference using discrete normalizing flows.
problem Challenges in optimizing variational approximations for discrete latent variables.
method Differentiable reparameterization using a mixture of discrete normalizing flows.
result Improves optimization of evidence lower bound and reduces sensitivity to hyperparameters.
A new method for multi-task learning by allocating parameters.
problem Sharing parameters between unrelated tasks can hurt performance.
method Learned binary variables to allocate components to tasks, encouraging sharing between related tasks.
result Achieves a 17% relative reduction of the error rate on Omniglot benchmark.
HMQ improves quantization for edge devices with mixed precision.
problem Efficient quantization for edge devices with uniform, power-of-two thresholds.
method Introduces HMQ, a mixed precision quantization block that repurposes Gumbel-Softmax for searching over quantization schemes.
result Achieves competitive and state-of-the-art results on ImageNet despite restrictions.
New method optimizes MRI sampling patterns for faster scans.
problem Accelerate MRI scans without sacrificing image quality.
method Joint learning of adaptive sampling patterns and model-based recovery.
result Improved MR image quality compared to other methods.
Proposes a new method for clustering compressed data.
problem Limited communication bandwidth and low-power consumption.
method Joint Variational Autoencoders with Bernoulli mixture models (VAB).
result The model can perform clustering in the compressed data domain.
Paper proposes GANs for generating business process suffixes and remaining times.
problem Generating accurate suffixes and remaining times for business process models.
method Encoder-decoder GAN architecture with Gumbel-Softmax distribution for categorical sequences.
result The approach outperforms baselines in suffix and remaining time prediction accuracy.
Recently a variety of methods have been developed to encode graphs into low-dimensional vectors that can be easily exploited by machine learning algorithms. The majority of these methods start by embedding the graph nodes into a low-dimensional vector space, followed by using some scheme to aggregate the node embedding…
Paper learns causal structures from data using gradient-based optimization.
problem Learning causal structures from observational data.
method Reformulates SEM with binary adjacency matrix and uses gradient-based optimization.
result Gradient-based method efficiently identifies causal structures from data.
A body of recent work has focused on constructing a variational family of filtered distributions using Sequential Monte Carlo (SMC). Inspired by this work, we introduce Particle Smoothing Variational Objectives (SVO), a novel backward simulation technique and smoothed approximate posterior defined through a subsampling…
The deployment of deep neural networks in real-world applications is mostly restricted by their high inference costs. Extensive efforts have been made to improve the accuracy with expert-designed or algorithm-searched architectures. However, the incremental improvement is typically achieved with increasingly more expen…
QA-Token improves tokenization for noisy data, boosting model performance.
problem Tokenization ignores data quality, limiting model effectiveness on noisy corpora.
method QA-Token combines signal quality with vocabulary construction through bilevel optimization and reinforcement learning.
result QA-Token achieves state-of-the-art performance on genomic and financial datasets.
New method for efficient marginalization of discrete latent variables in neural networks.
problem Computational challenges in training models with discrete latent variables.
method Parameterizing discrete distributions using sparse mappings (sparsemax and structured variants) to reduce support and enable efficient marginalization.
result Achieved good performance in various tasks with efficient and practical training.
Differentiable losses for combinatorial optimization problems in sequence modeling.
problem Mismatch between training and inference objectives in sequence models.
method Gradient descent over linear programs representing combinatorial optimization problems.
result Gradient descent can be applied to combinatorial optimization problems efficiently.
Boltzmann exploration is widely used in reinforcement learning to provide a trade-off between exploration and exploitation. Recently, in (Cesa-Bianchi et al., 2017) it has been shown that pure Boltzmann exploration does not perform well from a regret perspective, even in the simplest setting of stochastic multi-armed b…
Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable…
This work uses variational inference to estimate parameters of opinion dynamics models.
problem Challenges in parameter estimation for ABMs of social phenomena.
method Transformed ABM parameter estimation into an optimization problem using variational inference.
result Estimates parameters more accurately than simulation-based and MCMC methods.
DDLK uses deep learning to find important features in models.
problem Discovering important features in black box models like deep neural networks.
method DDLK directly minimizes KL divergence to generate knockoffs that obey the swap property.
result DDLK outperforms baselines in discovering important features while controlling false discovery rate.
Neural language models have been widely used in various NLP tasks, including machine translation, next word prediction and conversational agents. However, it is challenging to deploy these models on mobile devices due to their slow prediction speed, where the bottleneck is to compute top candidates in the softmax layer…
asp2vec learns dynamic node aspect distributions for better network embedding.
problem Lack of multi-aspect node representations in network embedding.
method Dynamic aspect assignment via Gumbel-Softmax and aspect regularization.
result Improved network embedding quality through dynamic aspect modeling.
SINN combines social science and deep learning for predicting opinion dynamics.
problem Predicting opinion dynamics in social networks using traditional models requires extensive calibration with real data.
method SINN integrates theoretical models and social media data using physics-informed neural networks (PINNs) and matrix factorization.
result SINN outperforms six baseline methods in predicting opinion dynamics on real-world and synthetic datasets.
Graph neural network optimizes energy-efficient precoding for massive MIMO systems.
problem Energy bottleneck in massive MIMO systems due to high DAC complexity and power consumption.
method Proposes a graph neural network to directly output precoded quantized vectors from channel matrix and transmit symbols.
result Significant increase in achievable sum rate with reduced DAC power consumption.