New method trains sparse deep networks efficiently.
problem Training deep neural networks with limited parameters.
method Dynamic sparse reparameterization.
result Our method outperforms previous techniques in accuracy for a fixed parameter budget.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,181 papers · 148 categories
Trend · papers per month
48 results for “dynamic reparameterization”
Efficiently infers switching nonlinear systems with collapsed amortized variational inference.
problem Inference in switching nonlinear dynamical systems with discrete latent variables.
method Learn an inference network as a proposal for continuous latent variables, performing exact marginalization of discrete variables.
result Successfully segments time series data into meaningful regimes using piece-wise nonlinear dynamics.
DiMS sampler explores neural network loss minima via dissipative dynamics.
problem Sampling reparameterization invariant solutions in neural networks.
method Dynamical system based on kinetic energy with dissipative friction.
result DiMS sampler samples exactly from minimum level sets.
New method for analyzing learning dynamics in singular models.
problem Challenges in analyzing learning of singular models with no one-to-one parameter space.
method Relative reparameterization technique to extract regular sub-models.
result Demonstrated differences in convergence behavior due to algorithmic and intrinsic aspects.
The reparameterization gradient has become a widely used method to obtain Monte Carlo gradients to optimize the variational objective. However, this technique does not easily apply to commonly used distributions such as beta or gamma without further approximations, and most practical applications of the reparameterizat…
Paper generalizes reparameterization trick for broader applicability.
problem Limited applicability of reparameterization trick to specific distributions.
method Introduces a generalized transformation-based gradient method.
result Proposed model combines advantages of control variates and generalized reparameterization.
Gradient descent implicitly favors group sparsity in neural networks.
problem Understanding implicit regularization in neural networks for structured sparsity.
method Novel neural reparameterization for diagonally grouped linear networks.
result Gradient descent without explicit regularization biases towards group sparsity.
Reparameterizes mirror descent as gradient descent for efficient sparse learning.
problem Efficiently training small sparse networks with mirror descent.
method Develops a framework to convert mirror descent updates into gradient descent updates on different parameters.
result Mirror descent can be reparameterized as gradient descent on modified parameters, facilitating standard backpropagation.
Reparameterization trick yields more accurate gradient estimates in variational inference.
problem Improving gradient estimates in variational inference.
method Idealized analysis of mean-field Gaussian approximations and quadratic log densities.
result Marginal variances of reparameterization gradient are smaller than score function gradient.
Geometric approach improves motion alignment accuracy and efficiency.
problem Temporal alignment of human motion data for various applications.
method Geometric point of view, principal fiber bundle, reparameterization invariant projection, dynamic programming, keyframe correspondences.
result Temporal alignment procedures are more accurate and computationally efficient.
REP-GAN improves GANs by reparameterizing proposals for better sample quality and efficiency.
problem Poor sample efficiency in GANs due to independent proposal sampling.
method REParameterizing Markov chains into the latent space of the generator to create dependent proposals.
result Empirically shows significant improvement in sample efficiency and quality.
Variational inference using the reparameterization trick has enabled large-scale approximate Bayesian inference in complex probabilistic models, leveraging stochastic optimization to sidestep intractable expectations. The reparameterization trick is applicable when we can simulate a random variable by applying a differ…
Introduces a new method for computing gradients of continuous distributions.
problem Inability of classic reparameterization trick to compute gradients for certain distributions.
method Implicit differentiation for computing reparameterization gradients.
result Proposed method is faster and more accurate for Gamma, Beta, Dirichlet, and von Mises distributions.
New flatness measure for neural nets is invariant to reparameterizations.
problem Lack of invariance in existing flatness measures to reparameterizations.
method Proposed a reparameterization-invariant flatness measure.
result The new flatness measure correlates with generalization error.
Introduces a neural network-based method for efficient state and parameter estimation in complex systems.
problem Efficiently estimating state paths and parameters from noisy measurements in high-dimensional nonlinear systems.
method Bayesian Information Field Theory with neural network parameterization and optimization algorithms.
result Proposes a method to simplify and enrich state path parameterizations using neural networks, improving inference accuracy.
A new method for optimizing models with categorical variables using diffusion.
problem Optimizing models with categorical variables, especially in discrete distributions.
method Introducing ReDGE, a diffusion-based soft reparameterization method for categorical distributions.
result ReDGE consistently matches or outperforms existing gradient-based methods in experiments.
The reparameterization trick simplifies optimization of acquisition functions in parallel Bayesian optimization.
problem Optimizing acquisition functions for parallel Bayesian optimization is challenging due to non-convexity, high dimensionality, and intractability.
method Formulate popular acquisition functions as Gaussian integrals and apply the reparameterization trick for gradient-based optimization.
result An efficient Monte Carlo estimator for the upper confidence bound acquisition function is derived.
Two new estimators improve VAE training for hierarchical and prior parameters.
problem Efficient gradient estimation for VAEs with hierarchical and prior parameters.
method Developed two generalizations of Doubly-Reparameterized Gradient Estimators (DReGs) for VAEs.
result Improved training of conditional and hierarchical VAEs on image modeling tasks.
A new method reparameterizes Gaussian noise for better flexibility and performance.
problem Improving the Gumbel-Softmax for better flexibility and performance.
method Invertible Gaussian Reparameterization (IGR) using modified softmax and transformations.
result IGR outperforms Gumbel-Softmax in various experiments.
Embedding-reparameterization procedure for manifold-valued latent variables in generative modelscs.LG
Proposes a new technique for VAEs using manifold-valued variables.
problem Effect of prior distribution choice on VAE learning capacity.
method Embedding-reparameterization procedure (ER) for manifold-valued latent variables.
result ER technique outperforms conventional VAE on a toy benchmark.
New method computes pathwise gradients for non-reparameterizable distributions.
problem Computing gradients for complex distributions not directly amenable to the reparameterization trick.
method Using optimal transport theory, compute gradients for Gamma, Beta, and Dirichlet distributions.
result Optimal gradients have reduced variance and are competitive with other methods.
Low-variance gradient estimation is crucial for learning directed graphical models parameterized by neural networks, where the reparameterization trick is widely used for those with continuous variables. While this technique gives low-variance gradient estimates, it has not been directly applicable to discrete variable…
Researchers extend reparameterization to Lie groups for better probability modeling.
problem Lack of reparameterization for distributions on Lie groups.
method Developed a general framework for creating reparameterizable densities on Lie groups.
result Demonstrated complex and multimodal distributions on SO(3) for pose estimation.
A new method for stochastic optimal control improves accuracy over existing techniques.
problem Improving the accuracy of stochastic optimal control for noisy systems.
method Stochastic Optimal Control Matching (SOCM) using Iterative Diffusion Optimization (IDO) with path-wise reparameterization trick.
result SOCM achieves lower error than existing techniques for three out of four control problems, sometimes by an order of magnitude.
New method trains neural networks with binary or ternary weights efficiently.
problem Training high-precision, real-time neural networks on limited hardware.
method Local reparameterization trick modification for discrete weights.
result State-of-the-art results on binary and ternary models on various benchmarks.
Improves performance in various machine learning tasks by reparameterizing subset sampling.
problem Stochastic optimization involving subset sampling is not reparameterizable.
method Continuous relaxation of subset sampling to provide reparameterization gradients.
result Improves performance in instance-wise feature selection, deep stochastic k-nearest neighbors, and parametric t-SNE.
Geometric correspondence links flow metrics to reparameterizations.
problem Linking flow metrics to reparameterizations of geodesic flows.
method Analysis of Mineyev's flow space and Green metrics.
result First examples of continuous reparameterizations on negatively curved manifolds.
New algorithm for variational inference on non-differentiable models.
problem Challenges in stochastic variational inference for non-differentiable models.
method Generalizes reparameterization trick for non-differentiable models, splitting latent variables into differentiable and non-differentiable regions.
result Our algorithm reduces variance and remains unbiased for non-differentiable models.
New measure of maximal entropy found for a class of geometrically finite groups.
problem Finding a measure of maximal entropy for relatively Anosov groups.
method Constructing reparameterizations and using exponential expansion along unstable foliations.
result The Bowen-Margulis-Sullivan measure is finite and unique for relatively Anosov groups.
The paper addresses the invariance issue in Bayesian neural networks using linearized Laplace approximation.
problem Bayesian neural networks fail to maintain invariance under reparameterization, leading to different posterior densities for identical functions.
method Developed a geometric view of reparameterizations and a Riemannian diffusion process to extend reparameterization invariance to neural network predictive.
result Empirically improved posterior fit through approximate posterior sampling.
Paper presents a reparameterized DP-DLGMM for clustering.
problem Non-parametric DP priors in DLGMM are hard to couple with variational inference.
method Closed-form updates for DP-DLGMM's variational posterior.
result Model generates realistic samples and performs competitively in semi-supervised settings.
Continuous latent time series models are prevalent in Bayesian modeling; examples include the Kalman filter, dynamic collaborative filtering, or dynamic topic models. These models often benefit from structured, non mean field variational approximations that capture correlations between time steps. Black box variational…
The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propag…
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise th…
This research simplifies verification of machine learning systems using reparameterization.
problem Reduce or eliminate serious bugs in machine learning systems.
method Use proof assistants to construct machine-checked proofs of correctness, leveraging reparameterization to handle probabilistic claims.
result Demonstrates broad applicability of reparameterization to verify different types of machine learning systems.
STR reparameterizes DNN weights with soft thresholds for better sparsity and accuracy.
problem Improving sparsity in DNNs for better accuracy and lower inference cost.
method Soft Threshold Reparameterization (STR) using the soft-threshold operator on DNN weights.
result STR achieves state-of-the-art accuracy and reduces FLOPs by up to 50%.
PIPPS solves deep learning's exploding gradient problem by reparameterization gradients.
problem Exploding gradients in deep learning and model-based RL.
method Develops PIPPS framework, a flexible policy search method robust to chaos-like gradients.
result PIPPS improves over reparameterization gradients by up to 10^6 times.
We stabilize the Kumaraswamy distribution for efficient sampling and differentiation.
problem Numerical instabilities in the Kumaraswamy distribution's inverse CDF and log-pdf.
method Identified and resolved numerical issues, introduced a stabilized KS distribution.
result Stabilized Kumaraswamy distribution supports efficient sampling and differentiation.
Optimization with noisy gradients has become ubiquitous in statistics and machine learning. Reparameterization gradients, or gradient estimates computed via the "reparameterization trick," represent a class of noisy gradients often used in Monte Carlo variational inference (MCVI). However, when these gradient estimator…
JKO-iFlow uses neural ODEs to improve generative models with reduced memory and training complexity.
problem Efficiently training deep generative models in high dimensions with reduced memory and training complexity.
method JKO scheme inspired neural ODE flow network with adaptive time reparameterization.
result JKO-iFlow achieves competitive performance compared to existing models at reduced computational and memory cost.
New method enhances model fine-tuning with minimal data.
problem Improving model performance on new tasks with limited data.
method Introducing -LoRA, a reparameterization method for fine-tuning.
result Enhanced generalization ability of fine-tuned models.
Unified view of LR and RP gradients with improved importance sampling.
problem Understanding and optimizing gradient estimators in machine learning.
method First principles explanation of LR and RP, divergence theorem, optimal importance sampling schemes.
result Optimal importance sampling schemes derived with analytic probability densities.
A simple method compresses neural network weights using entropy penalties.
problem Neural network weight compression for scalability and accuracy.
method Reparameterization with learned entropy penalty for compression.
result Maximized classification accuracy and model compressibility.
Unified view of LR and RP gradients explained via divergence theorem.
problem Explaining the nature and relationship of LR and RP gradients.
method First principles approach using divergence theorem.
result Characterization of all possible estimators combining LR and RP.
Paper tackles performative prediction without convexity assumptions.
problem Performative prediction where data distribution changes with model deployment.
method Reparameterization framework to transform non-convex objective into convex one.
result Provably sublinear regret guarantees for learnable model.
Enhances sample diversity in SGMCMC for better uncertainty estimation in BNNs.
problem Limited sample diversity in SGMCMC affects uncertainty estimation and model performance.
method Reparameterizes neural network weights to produce a more diverse set of samples.
result The proposed approach achieves superior performance in image classification tasks, including OOD robustness.
Investigates how flatness of loss curve relates to generalization in machine learning models.
problem Understanding why flatness correlates with generalization in machine learning models.
method Relates flatness to interpolation from representative data, derives notions of representativeness and feature robustness.
result Derives a novel relative flatness measure that correlates with generalization and solves reparameterization issues.
EXPO framework eliminates need for reward model, achieving better optimization.
problem Optimizing large language model responses without a separate reward model.
method Introduces EXPO framework that avoids reparameterization, directly optimizing preferences.
result Demonstrates better regularization and intuitive interpolation behaviors.