New insights show stochastic initialization prevents token clustering in deep Transformers.
problem Understanding token dynamics in deep stochastic Transformers.
method Analysis of deep Transformers with random initialization noise, proving convergence to an interacting-particle system on the sphere.
result Initialization noise prevents token clustering, leading to antipodal formations.
Study shows deterministic equivalent for neural network kernel convergence.
problem Understanding convergence of neural network kernels.
method Analyzes empirical spectral distribution of Conjugate Kernel, proving convergence to a deterministic limit.
result Obtains a deterministic equivalent for the Stieltjes transform and resolvent of the Conjugate Kernel.
A new method for discrete data normalizing flows using latent transformations.
problem Challenges in parameterizing bijective transformations for discrete data.
method Predict a distribution over latent transformations to make the marginal likelihood differentiable.
result Discrete-data normalizing flows can be trained using gradient-based learning with unbiased score function estimation.
Traditional GANs use a deterministic generator function (typically a neural network) to transform a random noise input z to a sample x that the discriminator seeks to distinguish. We propose a new GAN called Bayesian Conditional Generative Adversarial Networks (BC-GANs) that use a random generator function…
MINIROCKET speeds up time series classification by 75x.
problem High computational complexity in time series classification.
method Random convolutional kernels and linear classifier.
result Up to 75 times faster on larger datasets.
The reparameterization trick has become one of the most useful tools in the field of variational inference. However, the reparameterization trick is based on the standardization transformation which restricts the scope of application of this method to distributions that have tractable inverse cumulative distribution fu…
Unified analysis for deterministic samplers in diffusion models.
problem Challenges in analyzing deterministic samplers for diffusion models.
method Unified convergence analysis framework.
result Achieved polynomial iteration complexity for DDIM-type samplers.
A new method trains discrete EBMs without sampling.
problem Training EBMs on discrete spaces is hard.
method Energy Discrepancy (ED), a contrastive loss.
result ED offers theoretical guarantees for various perturbation types.
Derives bounds for deterministic predictors using smooth loss functions.
problem Generalizing probabilistic predictors to deterministic ones.
method Exploits smoothness properties of loss and predictor classes, controlling the Jensen gap class through Rademacher complexity.
result Derives bounds for deterministic predictors involving flatness quantities from Jacobians and Hessians.
Transformation Equivariant Representations (TERs) aim to capture the intrinsic visual structures that equivary to various transformations by expanding the notion of {\em translation} equivariance underlying the success of Convolutional Neural Networks (CNNs). For this purpose, we present both deterministic AutoEncoding…
Factor graphs have recently gained increasing attention as a unified framework for representing and constructing algorithms for signal processing, estimation, and control. One capability that does not seem to be well explored within the factor graph tool kit is the ability to handle deterministic nonlinear transformati…
Probabilistic STNs improve image classification and robustness.
problem Training and robustness issues in STNs.
method Probabilistic extension of STNs that estimates stochastic transformations.
result Improved classification performance, robustness, and model calibration.
Study improves probabilistic circuits using transformations for better predictions.
problem Predictive limitations of probabilistic circuits in robotic scenarios.
method Integrates transformations into joint probability trees, extending their capabilities.
result Achieves higher likelihoods with fewer parameters on various data sets.
Novel approach simplifies VI problems with faster performance.
problem Black-box VI optimization problems.
method Sample Average Approximation (SAA) combined with quasi-Newton methods and line search.
result Achieves faster performance than existing methods.
This paper uses deep reinforcement learning to optimize stock portfolios considering transaction costs and risks.
problem Optimizing stock portfolios with transaction costs and risks.
method Formulated stock portfolio optimization as a reinforcement learning problem, applied DDPG, GDPG, and PPO algorithms, and used Wavelet Transform.
result DDPG and GDPG algorithms outperformed PPO in continuous action space.
SurVAE Flows combine VAEs and flows using surjective transformations.
problem Combining the strengths of VAEs and flows to model complex densities.
method Modular framework of composable deterministic and stochastic transformations.
result Exact likelihood computation and lower bound on likelihood.
Data augmentation is a ubiquitous technique for increasing the size of labeled training sets by leveraging task-specific data transformations that preserve class labels. While it is often easy for domain experts to specify individual transformations, constructing and tuning the more sophisticated compositions typically…
The paper studies how noise synchronizes tokens in deep transformer models.
problem Understanding synchronization in deep learning models with noise.
method Proves convergence to a stochastic particle system and identifies the limiting SDE.
result The limiting model displays synchronization by noise and exponential dissipation of interaction energy.
Deep learning model estimates uncertainty in complex regression tasks.
problem Uncertainty quantification in probabilistic regression predictions.
method Combines statistical and deep learning transformation models using gradient descent.
result State-of-the-art performance on small datasets and complex image data.
Derandomizing PAC-Bayes bounds for smooth loss functions
problem Derandomizing PAC-Bayes bounds for smooth loss functions
method Exploiting smoothness properties of both the loss and the predictor class
result Bounds for deterministic predictors that involve flatness quantities
Proposes using diffusion models for probabilistic stock market predictions.
problem Uncertainties in financial data make deterministic models ineffective for stock market predictions.
method Utilizes Denoising Diffusion Probabilistic Models (DDPM) and Masked Relational Transformer (MRT).
result Achieves state-of-the-art performance in stock movement prediction and portfolio management.
The paper presents a method to assign accurate and calibrated uncertainties to deterministic model predictions.
problem Assigning uncertainties to deterministic model predictions.
method Transforming deterministic predictions into probabilistic ones using a cost function that balances accuracy and reliability.
result The method improves the reliability of probabilistic predictions without sacrificing accuracy.
Diffusion models' sampling paths lie in a low-dimensional subspace, resembling boomerangs.
problem Understanding the geometric structure of diffusion-based generative models.
method Characterization of deterministic sampling trajectories using low-dimensional subspace and kernel-estimated data modeling.
result Sampling trajectories in diffusion models are confined to a low-dimensional subspace and exhibit a boomerang shape.
Enformer and GEnformer use Transformers with stochastic learning to forecast multivariate and spatiotemporal data with uncertainty.
problem Uncertainty quantification in multivariate time series and spatiotemporal forecasting.
method Synthesizing Transformer's expressive power with stochastic learning to model conditional distributions directly.
result Enformer and GEnformer yield calibrated probabilistic forecasts and outperform state-of-the-art baselines.
The paper prices weather contracts using a complex temperature model.
problem Accurate pricing of weather contracts under temperature dynamics.
method Time-changed Levy model with mean-reverting dynamics, Fourier expansion, Esscher transform.
result An accurate approximation of weather contract prices.
Recent studies identified that sequential Recommendation is improved by the attention mechanism. By following this development, we propose Relation-Aware Kernelized Self-Attention (RKSA) adopting a self-attention mechanism of the Transformer with augmentation of a probabilistic model. The original self-attention of Tra…
Bayesian Transformer improves probabilistic load forecasting with calibrated uncertainty estimates.
problem Overconfident point predictions from deep learning models fail under extreme weather distributional shifts.
method Integrates three uncertainty mechanisms: MC Dropout, variational layers, and stochastic attention.
result Achieves state-of-the-art performance with CRPS of 0.0289 and 90% PICP across various horizons.
A new method prunes neural networks efficiently without losing effectiveness.
problem Efficient pruning of neural networks without sacrificing performance.
method Deterministic approximation of binary gates and L0 regularization. result Pruning neural networks significantly without loss in effectiveness.
Study aggregation of statistical evidence under unknown dependence using group-invariance.
problem Aggregating statistical evidence under unknown and complex dependence structures.
method Develops a framework using group-invariance and permutation-based constructions to aggregate evidence across transformed datasets.
result Shows uniform improvement in critical values for single-batch aggregation over deterministic calibrations, adapting to unknown dependence structures.
Refined BN-S model improves crude oil hedging with machine learning.
problem Finding optimal hedging strategy for commodity markets.
method Implemented a refined Barndorff-Nielsen and Shephard model with machine learning algorithms.
result The refined model performs better than the classical BN-S model.
Recent research on accelerated gradient methods of use in optimization has demonstrated that these methods can be derived as discretizations of dynamical systems. This, in turn, has provided a basis for more systematic investigations, especially into the geometric structure of those dynamical systems and their structur…
New concept of epiplexity quantifies useful information from data.
problem Understanding useful information content from data without unlimited computational capacity.
method Introducing epiplexity, a measure of information computationally bounded observers can learn.
result Epiplexity captures useful information content, not just randomness.
Self-supervised VAEs improve data compression and generation.
problem Efficient data compression and generation.
method Introducing self-supervised Variational Auto-Encoders with deterministic and discrete variational posteriors.
result Self-supervised VAEs simplify the objective function and improve data reconstruction.
Minimalistic unsupervised learning with sparse manifold transform achieves SOTA performance.
problem Achieving state-of-the-art unsupervised learning performance without complex engineering.
method Sparse manifold transform, leveraging sparse coding, manifold learning, and slow feature analysis.
result 99.3% KNN top-1 accuracy on MNIST, 81.1% on CIFAR-10, and 53.2% on CIFAR-100.
This paper develops tools for nonreversible MCMC with convergence guarantees.
problem Designing nonreversible MCMC kernels with convergence guarantees.
method Develops tools for nonreversible Markov kernels using conditional invertible transforms.
result Ensures nonreversible kernels have the desired invariance property and lead to convergent algorithms.
New model optimizes portfolios with realistic transaction costs.
problem Real-world transaction costs impact portfolio profitability.
method DPGRGT model with 2D relative-attentional Gated Transformer.
result Model outperforms baseline models in U.S. stock market data.
Study benchmarks uncertainty quantification in chest X-ray classification.
problem Reliable uncertainty quantification for medical AI models.
method Evaluation of 13 uncertainty quantification methods on MIMIC-CXR-JPG dataset.
result Insights into effectiveness and disentanglement of epistemic and aleatoric uncertainties.
We introduce a new algorithm for approximate inference that combines reparametrization, Markov chain Monte Carlo and variational methods. We construct a very flexible implicit variational distribution synthesized by an arbitrary Markov chain Monte Carlo operation and a deterministic transformation that can be optimized…
Deep learning has become an area of interest in most scientific areas, including physical sciences. Modern networks apply real-valued transformations on the data. Particularly, convolutions in convolutional neural networks discard phase information entirely. Many deterministic signals, such as seismic data or electrica…
EVI-MMD approximates target distributions via MMD minimization with adaptive kernel.
problem Approximating target distributions using kernel discrepancy methods.
method EVI-MMD uses Maximum Mean Discrepancy (MMD) to minimize kernel discrepancy, solving ODEs with implicit Euler scheme and L-BFGS optimization.
result EVI-MMD with adaptive bandwidth selection significantly improves performance in sampling problems.
New method enforces orthogonality in convolutional layers for improved robustness.
problem Improving adversarial robustness in deep learning models.
method Applying the Cayley transform to skew-symmetric convolutions in the Fourier domain.
result The proposed method preserves orthogonality and enhances adversarial robustness compared to existing techniques.
An extension of the regularized least-squares in which the estimation parameters are stretchable is introduced and studied in this paper. The solution of this ridge regression with stretchable parameters is given in primal and dual spaces and in closed-form. Essentially, the proposed solution stretches the covariance c…
We investigate the low-dimensional structure of deterministic transformations between random variables, i.e., transport maps between probability measures. In the context of statistics and machine learning, these transformations can be used to couple a tractable "reference" measure (e.g., a standard Gaussian) with a tar…
Algorithm recovers factors of rank-1 matrices from noisy measurements.
problem Estimating factors of a rank-1 matrix from nonlinearly transformed and noisy measurements.
method Alternating minimization with random initialization and analysis of empirical error recursion.
result Algorithm converges geometrically fast from random initialization, with sharp guarantees.
Developed mlf-core for deterministic machine learning.
problem Ensuring machine learning models are deterministic for verification.
method Formulated requirements, developed mlf-core ecosystem, tested various models.
result Demonstrated deterministic models in biomedical fields.
New method finds exact Pareto front for MO-MDPs efficiently.
problem Finding the exact Pareto front for MO-MDPs is challenging.
method Investigates geometric structure, develops efficient algorithm.
result Pareto front is on boundary of convex polytope of deterministic policies.
New method estimates SDE parameters efficiently using WCE and SGD.
problem Parameter estimation for stochastic differential equations.
method Wiener Chaos Expansion and Stochastic Gradient Descent.
result Accurate parameter recovery from noisy observations.
Study on regret minimization in deterministic MDPs.
problem Minimizing regret in deterministic reinforcement learning.
method Logarithmic regret lower bounds, leveraging graph theory and cycles.
result Explicitly quantifies the fundamental limit of performance achievable by any learning algorithm.