New neural networks learn distribution functions using quantiles and moments.
problem Approximating functions of distributions in probability spaces.
method Quantile and moment neural networks, mixing quantile and moment features.
result Moment neural network outperforms others for bivariate distributions.
Paper provides Edgeworth expansions for network moments, improving accuracy of sampling distributions.
problem Accurate descriptions of sampling distributions of network moment statistics.
method Edgeworth expansion applied to studentized network moment statistics.
result Higher-order accurate approximation to sampling CDF of network moment statistics.
Moment Pooling reduces latent space dimensions in machine learning models.
problem High-dimensional latent spaces in machine learning models are hard to interpret.
method Moment Pooling extends Deep Sets networks to arbitrary multivariate moments.
result Latent dimensions as small as 1 can achieve similar performance to higher dimensions.
We propose a method of moments (MoM) algorithm for training large-scale implicit generative models. Moment estimation in this setting encounters two problems: it is often difficult to define the millions of moments needed to learn the model parameters, and it is hard to determine which properties are useful when specif…
Empower efficient representation of distributions through moment-preserving methods.
problem Representing high-dimensional probability measures efficiently and accurately.
method Empower efficient representation of distributions through moment-preserving methods.
result Empowers efficient and accurate representation of high-dimensional probability measures.
New method improves estimation of complex models from conditional moment restrictions.
problem Estimation of complex models from conditional moment restrictions.
method Functional Generalized Empirical Likelihood (GEL) with a practical method.
result The method achieves state-of-the-art performance on two problems.
Derives moments of PL networks for robust DNNs.
problem Analyzing robustness of DNNs to adversarial attacks.
method Derives exact analytic expressions for moments of PL networks, generalizes variance expression, and constructs sparse and smooth adversarial attacks.
result New variance expressions are tighter and can be efficiently approximated.
We tackle causal inference under conditional moment restrictions using importance weighting.
problem Challenges in causal inference under conditional moment restrictions, especially in high-dimensional settings.
method Transform conditional moment restrictions to unconditional moment restrictions through importance weighting.
result Successfully estimate nonparametric functions defined under conditional moment restrictions.
GANs learn distributions by matching low-degree moments.
problem Understanding when GANs learn the target distribution efficiently.
method Theoretical analysis and empirical observation of GAN training process.
result GANs can learn notable distributions by matching polynomially many low-degree moments.
Machine learning models accurately predict molecular magnetic anisotropy tensors.
problem Accurately modeling molecular magnetic anisotropy tensors.
method Gaussian-moment neural-network approach for machine learning.
result Achieved accuracy of 0.3--0.4 cm−1 for magnetic anisotropy tensor predictions. We introduce a novel approach, requiring only mild assumptions, for the characterization of deep neural networks at initialization. Our approach applies both to fully-connected and convolutional networks and easily incorporates batch normalization and skip-connections. Our key insight is to consider the evolution with …
Framework generates causal probabilities from observational data.
problem Generating causal probabilities from observational data.
method Moment-matching graph-networks for causal inference.
result Automated sampling of latent space conditional probability distributions.
Estimates high-dimensional posterior densities by marginal distributions and neural networks.
problem High-dimensional probability density estimation for inference is difficult.
method Direct estimation of lower-dimensional marginal distributions, using Moment Networks for fast computation of moments.
result Demonstrates estimation of gravitational wave time series and applications in cosmology.
New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.
problem Solving mean-field control problems in continuous time reinforcement learning.
method Gradient-based policy and value function learning with moment neural networks on the Wasserstein space.
result Effective solution for diverse mean-field control problems, including multi-dimensional and nonlinear settings.
Improved GAN performance using higher-order Wasserstein moments.
problem Stabilizing and enhancing GANs for better mode coverage and stability.
method Deriving and training a GAN with a modified Wasserstein distance using higher-order moments.
result Training a GAN with higher-order Wasserstein moments improves performance, even with increased computational cost.
New method unfolds distribution moments directly from data without binning.
problem Deconvolving detector distortions in particle physics.
method Uses machine learning, inspired by GANs, to unfold moments directly.
result More precise than bin-based approaches and comparable to unbinned methods.
Graph spectra have been successfully used to classify network types, compute the similarity between graphs, and determine the number of communities in a network. For large graphs, where an eigen-decomposition is infeasible, iterative moment matched approximations to the spectra and kernel smoothing are typically used. …
Proposes Moment Exchange to use moments in image recognition models, improving generalization.
problem Discarding moments in image recognition models reduces stability and training time.
method Moment Exchange: replaces moments of learned features with another image's moments and interpolates labels.
result Improves generalization of recognition models across multiple datasets.
The learning of domain-invariant representations in the context of domain adaptation with neural networks is considered. We propose a new regularization method that minimizes the discrepancy between domain-specific latent feature representations directly in the hidden activation space. Although some standard distributi…
TGNN combines GNN and SMM for better trading network predictions.
problem Predicting asset prices in trading networks with structural impact factors.
method Combines GNN and SMM for asset price prediction.
result TGNN outperforms existing methods in prediction accuracy.
Paper provides unbiased spectral moment estimates from finite data.
problem Challenges in estimating spectral moments from limited data.
method Dynamic programming approach to estimate spectral moments of kernel integral operator.
result Demonstrates consistency with theoretical spectra and practical utility in neural networks.
New initialization schemes preserve fractional moments of weights in deep networks, improving training and test performance.
problem Heavy-tailed distribution of stochastic gradients in DNNs during training.
method Developed initialization schemes that preserve any given fractional moment of order s < 2 over layers for various activations.
result The network output admits a heavy-tailed distribution with finite moments, improving training and test performance.
New method uses geometric moments for accurate machine learning potentials.
problem Creating high-dimensional potential energy surfaces efficiently.
method Feed-forward neural networks with invariant local molecular descriptors based on geometric moments.
result Accuracy comparable to established models, high efficiency.
We provide an approach for learning deep neural net representations of models described via conditional moment restrictions. Conditional moment restrictions are widely used, as they are the language by which social scientists describe the assumptions they make to enable causal inference. We formulate the problem of est…
To deepen our understanding of graph neural networks, we investigate the representation power of Graph Convolutional Networks (GCN) through the looking glass of graph moments, a key property of graph topology encoding path of various lengths. We find that GCNs are rather restrictive in learning graph moments. Without c…
Adaptive gradient methods such as Adam have been shown to be very effective for training deep neural networks (DNNs) by tracking the second moment of gradients to compute the individual learning rates. Differently from existing methods, we make use of the most recent first moment of gradients to compute the individual …
Develops MENT for interpreting and detecting changes in network trajectories.
problem Distortion of network geometry and invalidation of temporal comparisons in dynamic network analysis.
method Develops Multiscale Euclidean Network Trajectories (MENT) framework based on second-moment geometry.
result Validates and interprets network trajectories through isotropic normalization and orthogonal transformations.
New method improves robustness of neural network-based debiasing.
problem Improving robustness of neural network-based debiasing.
method Moment-constrained learning for neural networks.
result Improved performance compared to state-of-the-art benchmarks.
The article introduces inferential moments for analyzing uncertain multivariable systems.
problem Handling inference tasks in uncertain multivariable systems.
method Bayesian inference and quantification of inferential moments.
result Quantification of inferential moments and their connection to mutual information.
Improved neural network models predict molecular and material properties efficiently.
problem Training neural networks for accurate interatomic potentials is computationally expensive.
method Gaussian moment-based neural networks with improved architecture and active learning.
result The new models achieve high accuracy and reduced training times.
A new framework for lightweight BNNs learns heteroscedastic uncertainties efficiently.
problem Learning heteroscedastic uncertainties from BNNs for lightweight networks.
method Embedding heteroscedastic variances into BNN parameters and using moment propagation for inference.
result Improves predictive performance for lightweight BNNs without increasing parameter count.
The paper examines how the angle between inputs in ReLU networks decreases with depth, impacting training.
problem Depth degeneracy in neural networks, leading to constant function behavior on initialization.
method Combinatorial expansions and Monte Carlo experiments to analyze the angle between inputs in ReLU networks of increasing depth.
result The angle between inputs in ReLU networks decreases exponentially with depth, leading to constant function behavior on initialization.
A new method for uncertainty estimation in neural networks using existing optimization steps.
problem Uncertainty quantification in deep neural networks.
method L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation.
result L2M method yields reasonable results without requiring changes in models or extra computational steps.
Motivated by the prediction of cell loads in cellular networks, we formulate the following new, fundamental problem of statistical learning of geometric marks of point processes: An unknown marking function, depending on the geometry of point patterns, produces characteristics (marks) of the points. One aims at learnin…
Maxout networks study gradients and propose initialization strategies.
problem Complexity in input-output Jacobian distribution complicates stable parameter initialization.
method Obtained bounds on moments of gradients and formulated initialization strategies.
result Parameter initialization strategies improve training of deep maxout networks.
New algorithm learns ReLU networks efficiently using Schur polynomials.
problem PAC learning a linear combination of ReLU activations under Gaussian distribution.
method Uses tensor decomposition and Schur polynomials to identify and analyze higher-order moments.
result Near-optimal sample and computational complexity for learning ReLU networks.
Novel neural network predicts electricity prices with higher moments.
problem Probabilistic forecasting of volatile electricity prices.
method Distributional neural network with a probability layer.
result Significantly outperforms benchmarks in forecasting.
The paper extends RDPG model to handle weighted graphs, enabling better analysis of network data.
problem Modeling networks with weighted edges to capture heterogeneous weight distributions.
method Proposes a nonparametric W-RDPG model with latent positions and moment-generating functions.
result Establishes statistical guarantees for estimating nodal latent positions and sampling graphs.
Paper develops methods for inference on time series data using neural networks and sieves.
problem Inference on time series data with nonparametric conditional moment restrictions.
method GN-QLR based inference using general nonlinear sieves and multilayer neural networks.
result Optimally weighted GN-QLR statistic is asymptotically Chi-square distributed.
Polynomial-time algorithm learns ReLU networks without assumptions.
problem Learning linear combinations of ReLU activations with Gaussian inputs.
method Random contractions of moment tensors and multi-scale analysis.
result First polynomial-time algorithm without additional assumptions.
Study how neural networks learn from non-Gaussian data models.
problem Understanding neural network learning dynamics with non-Gaussian data.
method Developed a two-layer neural network with Hermite polynomial activations to control high-order cumulants.
result Neural networks progressively learn high-order cumulants after capturing low-order statistics.
New model detects communities in network data from edge nominations.
problem Noise and bias in network data from edge nominations.
method General model for network sampling, spectral clustering, method of moments.
result Community detection improved for network data collected via edge nominations.
A new filter reduces density fitting to a linear solve, improving performance on nonlinear systems.
problem Nonlinear Bayesian filtering challenges in representing belief distributions.
method Combines score matching with Stein's identity to avoid partition function evaluation.
result The Score Kalman Filter (SKF) outperforms existing methods on nonlinear systems.
This paper presents sampling-based speech parameter generation using moment-matching networks for Deep Neural Network (DNN)-based speech synthesis. Although people never produce exactly the same speech even if we try to express the same linguistic and para-linguistic information, typical statistical speech synthesis pr…
Develops a new method for estimating models with conditional moment restrictions.
problem Estimating models with conditional moment restrictions, especially non-parametric instrumental variable regression.
method Introduces a min-max criterion function to solve a zero-sum game between modeler and adversary, analyzing estimation rates for various hypothesis spaces.
result Shows that with regularization and rich test function spaces, estimation rates scale with the critical radius of hypothesis and test function spaces.
A new neural network initialization method is proposed for faster and more accurate training.
problem Efficient initialization for training multi-layer feedforward neural networks.
method Initialization based on Stein's identity, using eigenvectors of cross-moment matrix.
result The SteinGLM method is faster and more accurate than other initialization methods.
New bounds for neural networks without loss boundedness assumption.
problem Generalization error bounds for two-layer neural networks.
method Wasserstein distance estimates and moment bounds for stochastic gradient method.
result Dimension-free rate of order O(n−1/2) for independent test data. Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive learning rates based on squared past gradients, e.g., AdaGrad, AdaDelta, and Adam, eases the job slightly. However, such methods have also …