This paper proposes a new RV prediction model using neural distributional transformation and co-training.
problem Predicting skewed and fat-tailed realized volatility (RV) is challenging.
method The paper uses a neural distributional transformation and co-training to predict RV. It jointly trains the transformation and prediction model using a maximum-likelihood objective function.
result The proposed method significantly outperforms other methods on a dataset of 100 stocks.
Unified method for deriving ridgelet transforms for various neural network architectures.
problem Deriving closed-form expressions for ridgelet transforms in modern neural network architectures.
method Unified Fourier slice method to derive ridgelet transforms for diverse neural network types.
result Systematic method to derive ridgelet transforms for various neural network architectures.
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
Detects out-of-distribution sentences in Neural Machine Translation.
problem Identifying sentences from a different language than the training data.
method Developed a new uncertainty measure for long sequences of words in Transformers.
result Shows ability to identify Dutch sentences as German input.
Connections between integration along hypersufaces, Radon transforms, and neural networks are exploited to highlight an integral geometric mathematical interpretation of neural networks. By analyzing the properties of neural networks as operators on probability distributions for observed data, we show that the distribu…
Study shows neural network parameters converge to ridgelet spectrum.
problem Characterization of local minima in over-parametrized neural networks.
method Developed ridgelet transform to analyze neural network parameters.
result Distribution of parameters converges to ridgelet spectrum.
This paper analyzes deep and wide transformer training dynamics.
problem Understanding the training dynamics of infinitely deep and wide transformers.
method Develops a mean-field framework for gradient-based training of transformers, controlling a neural PDE.
result Establishes a rigorous foundation for gradient-based transformer training, proving convergence to global minima.
TTF improves performance of normalizing flows for heavy-tailed distributions.
problem Improving performance of normalizing flows for heavy-tailed distributions.
method Uses a Gaussian base distribution and a final transformation layer to produce heavy tails.
result Experimental results show TTF outperforms current methods, especially in high-dimensional or heavy-tailed scenarios.
Sparse transformer architecture improves accuracy and speed in generative modeling and inverse problems.
problem Improving accuracy and speed in generative modeling and inverse problems.
method Proposes a sparse transformer architecture using regularized Wasserstein proximal operator with L1 prior. result Sparse transformer achieves higher accuracy and faster convergence than classical methods.
VAEs and GANs use simple distributions and neural networks to implicitly approximate complex data distributions.
problem Approximating high-dimensional complex distributions explicitly is often intractable.
method VAEs and GANs use simple base distributions and neural networks to implicitly approximate complex distributions.
result Implicit approximation of complex distributions is crucial but introduces limitations, especially in VAEs with fixed Gaussian priors.
Algorithm learns polynomial transformations of Gaussian distributions.
problem Learning high-dimensional polynomial transformations of Gaussian distributions.
method Polynomial-time algorithms for smoothed settings, tensor ring decomposition.
result First end-to-end guarantees for learning pushforwards under neural networks.
Canonical transformation plays a fundamental role in simplifying and solving classical Hamiltonian systems. We construct flexible and powerful canonical transformations as generative models using symplectic neural networks. The model transforms physical variables towards a latent representation with an independent harm…
Study on neural scaling laws for solving linear systems in-context.
problem Theoretical guarantees for solving linear systems using a linear transformer architecture.
method Neural scaling laws and task diversity for in-domain and out-of-domain generalization.
result Novel notion of task diversity for necessary and sufficient condition of generalization under task shifts.
Deep conditional transformation models unify interpretable and complex predictors.
problem Challenging to learn conditional CDFs in high-dimensional settings.
method Unified deep learning framework for interpretable and complex predictors.
result Efficacy demonstrated through numerical experiments and applications.
Novel power transform unifies various mathematical functions.
problem Normalizing and standardizing datasets.
method Presented a novel power transform.
result Unified various mathematical functions.
Researchers analyze neural process architectures and their representational capacities.
problem Understanding what functions can be represented by different neural process architectures.
method Analyzing four types of neural process architectures: CNPs, ANPs, TNPs, and their latent variants.
result Prove these architectures form a strict hierarchy and characterize their representational capabilities.
Transformer improves sequence generation with insertion and deletion phases.
problem Sequence generation challenges in machine translation.
method Insertion-Deletion Transformer with iterative insertion and deletion phases.
result Significant BLEU score improvement over insertion-only models.
Flexible copula model using implicit generative neural networks.
problem Limited flexibility of parametric copulas and curse of dimensionality in non-parametric methods.
method Implicit generative neural networks to model high-dimensional copula distributions with unspecified marginals.
result Demonstrated flexibility and performance on various datasets.
Transformers learn functionals from distributions without losing information.
problem Lack of rigorous mathematical theory supporting Transformer performance.
method Proposed a Transformer learning framework, attention operator, and distribution regression.
result Transformers can compress distributions into function representations without loss of information.
REGS samples from unnormalized distributions using gradient flow and neural networks.
problem Sampling from unnormalized distributions with high accuracy and efficiency.
method REGS is a particle method that iteratively transforms samples from a reference distribution to match an unnormalized target distribution using Wasserstein gradient flow and neural networks.
result REGS outperforms state-of-the-art methods in sampling from challenging multimodal distributions and real datasets.
DNFS trains efficient samplers for discrete distributions using locally equivariant Transformers.
problem Sampling from unnormalised discrete distributions.
method DNFS learns a rate matrix to satisfy the Kolmogorov equation, using control variates and locally equivariant Transformers.
result DNFS achieves efficient and effective sampling across various applications.
Improved singular value approximation for convolutional layers.
problem Improving accuracy of singular value approximation for linear convolutional layers.
method Developed a new spectral density matrix method for singular value approximation with improved accuracy and reduced computational complexity.
result Obtained moderate improvement in singular value distribution compared to circular approximation.
NDMs enable non-linear transformations in diffusion models for better generative tasks.
problem Limited to linear transformations, diffusion models struggle with generative tasks.
method Presented NDMs that allow time-dependent non-linear transformations.
result NDMs outperform conventional diffusion models in likelihood and sample quality.
Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify a…
The paper explores properties of the Radon transform in relation to neural networks and ridges.
problem Understanding the Radon transform and its application to neural networks and ridges.
method Investigates properties of the Radon transform, introduces new subspaces, and characterizes ridges for any distributional profile.
result Clarifies and simplifies results on the optimality of ReLU networks using the Radon transform.
The paper extends entropy maximization to multiscale settings and applies it to neural networks.
problem Achieving optimal risk bounds in neural networks using multiscale entropy.
method Generalizing maximum entropy to multiscale settings and applying it to neural networks.
result The multiscale Gibbs posterior can achieve a smaller excess risk than the single-scale Gibbs posterior in a teacher-student scenario.
The paper uses Tukey g-and-h neural networks for non-Gaussian data regression.
problem Regression with non-Gaussian data.
method Training neural networks to predict Tukey g-and-h distribution parameters via negative log-likelihood minimization.
result Efficiency demonstrated in simulated and real-world datasets.
Neural networks struggle with extrapolation, but a new framework allows them to learn counterfactual invariances.
problem Neural networks' inability to extrapolate beyond training data distribution.
method Introduces a learning framework that allows neural networks to extrapolate over group transformations based on counterfactual invariances.
result Neural networks can learn counterfactual invariances from a single environment, overcoming their limitations in extrapolation.
Monotonic neural networks have recently been proposed as a way to define invertible transformations. These transformations can be combined into powerful autoregressive flows that have been shown to be universal approximators of continuous probability distributions. Architectures that ensure monotonicity typically enfor…
We study the invariance characteristics of pre-trained predictive models by empirically learning transformations on the input that leave the prediction function approximately unchanged. To learn invariant transformations, we minimize the Wasserstein distance between the predictive distribution conditioned on the data i…
New bounds on NTK's smallest eigenvalue for arbitrary data without distributional assumptions.
problem Existing bounds on NTK's smallest eigenvalue require distributional assumptions and high-dimensional data.
method Novel application of the hemisphere transform.
result Bounds on NTK's smallest eigenvalue hold with high probability even for constant input dimension.
Neural networks compress and sample WDN contamination dynamics efficiently.
problem Infrastructure monitoring of complex, networked systems like water distribution networks is expensive and challenging.
method Developed Graph Fourier Transform (GFT) operators and neural networks (NN) for efficient data collection and inference.
result High accuracy reconstruction of contamination dynamics using only 5-10% of the sample set.
A method for efficient approximate inference on discrete distributions.
problem Applying SVGD to discrete distributions.
method Transforming discrete distributions to piecewise continuous distributions for SVGD application.
result Outperforms traditional algorithms and ensemble methods on discrete graphical models.
Generative model uses SDEs to transform data distributions.
problem Creating data from complex distributions.
method Stochastic differential equations (SDEs) for data transformation.
result Achieved record-breaking performance in image generation.
Transformers can learn optimal regression mixtures efficiently.
problem Limited adoption of tailored regression methods due to their model-specific nature.
method Constructed a generative process for a mixture of linear regressions and used transformers to learn optimal predictors.
result Transformers achieve low mean-squared error and make predictions close to the optimal procedure.
DRIFT uses neural flows to replace distributional regression models.
problem Lack of neural network representations for distributional regression models.
method Inverse flow transformations (DRIFT) for distributional regression.
result Neural representations in DRIFT match classical statistical methods in performance.
Transformer model pretrains on synthetic graphs for AD detection.
problem Limited labeled data and class imbalance in AD diagnosis.
method Diffusion-generated synthetic graphs, Graph Transformers, transfer learning.
result Framework outperforms baselines in AD diagnosis metrics.
Transformers learn linear models in-context without updates.
problem Understanding how transformers mimic linear models in-context.
method Gradient flow on linear regression tasks with random initialization.
result Transformers achieve prediction error competitive with best linear predictors.
Deep neural networks can generate any 2D distribution with high accuracy.
problem Generating accurate high-dimensional distributions from random noise.
method A deep neural network with a space-filling property of sawtooth functions.
result The network can approximate any 2D Lipschitz-continuous distribution arbitrarily closely.
Deep neural networks forecast financial return distributions accurately.
problem Forecasting probability distributions of financial returns.
method Used 1D CNN and LSTM architectures with custom loss functions to optimize distribution parameters.
result LSTM with skewed Student's t distribution outperformed classical models in multiple evaluation metrics.
DMT enhances deep neural networks to better preserve data structures.
problem Preserving geometric, topological, and distributional structures of data in NLDR.
method Deep manifold transformation (DMT) using cross-layer LGP constraints.
result DMT networks outperform existing NLDR methods in preserving data structures.
Neural networks use their hidden layers to transform input data into linearly separable data clusters, with a linear or a perceptron type output layer making the final projection on the line perpendicular to the discriminating hyperplane. For complex data with multimodal distributions this transformation is difficult t…
Deep learning model estimates uncertainty in complex regression tasks.
problem Uncertainty quantification in probabilistic regression predictions.
method Combines statistical and deep learning transformation models using gradient descent.
result State-of-the-art performance on small datasets and complex image data.
Framework analyzes neural network dynamics for better understanding and optimization.
problem Understanding the fundamental mechanisms of deep neural networks.
method Dynamical systems theory, transformation units, attraction basins.
result Different transformation modes lead to distinct learning phases and network performance.
REST improves robustness of black-box models to geometric transformations.
problem Overconfident incorrect predictions on out-of-distribution samples.
method REinforcement Spatial Transform learner (REST) that transforms input data into in-distribution samples.
result Improves robustness to geometric transformations and sample efficiency.
Unified neural network for linear and nonlinear dimension reduction.
problem Efficiently perform linear and nonlinear sufficient dimension reduction.
method Belted and Ensembled Neural Network (BENN) framework.
result Unified framework for both linear and nonlinear dimension reduction.
DeepMaxent uses neural networks to improve species distribution models.
problem Sampling biases and lack of absence data in presence-only observations.
method DeepMaxent employs neural networks to learn shared features among species using the maximum entropy principle.
result DeepMaxent outperforms traditional methods in predicting species distributions, especially in unevenly sampled regions.
This paper improves test-time adaptation for distribution shifts using confidence maximization and input transformation.
problem Improving deep networks' performance on data shifted from the training distribution.
method Proposes a novel loss function combining confidence maximization and batch-wise entropy maximization with an input transformation module.
result Significantly improves robustness of pretrained networks to corruptions on benchmarks like ImageNet-C.