Automatically designs normalization and activation layers together.
problem Designing normalization and activation layers separately.
method Unified tensor-to-tensor computation graph, low-level mathematical functions, rejection protocols, multi-objective evolution.
result Discovery of EvoNorms with novel structures.
Causal normalizing flows recover causal models from observational data.
problem Recovering causal models from observational data.
method Use autoregressive normalizing flows and analyze design choices.
result Causal normalizing flows can capture causal data-generating processes.
We propose a theoretical framework for thinking about score normalization, which confirms that normalization is not needed under (admittedly fragile) ideal conditions. If, however, these conditions are not met, e.g. under data-set shift between training and runtime, our theory reveals dependencies between scores that c…
This paper reviews normalization techniques for DNNs.
problem Improving training speed and generalization of DNNs.
method Taxonomy of normalization methods, decomposition of activation methods.
result Insight for designing new normalization techniques.
The Lasso method is analyzed for high-dimensional regression with Gaussian designs, leading to new insights on its performance.
problem Analyzing the Lasso method for high-dimensional regression with Gaussian designs.
method Generalizing the Lasso characterization to Gaussian correlated designs with non-singular covariance structure.
result Establishing non-asymptotic bounds on the distance between the distribution of various quantities in the two models.
In this work, we investigate Batch Normalization technique and propose its probabilistic interpretation. We propose a probabilistic model and show that Batch Normalization maximazes the lower bound of its marginalized log-likelihood. Then, according to the new probabilistic model, we design an algorithm which acts cons…
Study shows sample complexity for logistic regression with normal covariates.
problem Estimating parameters of logistic regression with normal design.
method Analyzes sample complexity in terms of dimension and inverse temperature.
result Shows two change-points in sample complexity curve based on inverse temperature.
The study examines the limitations of bi-Lipschitz Normalizing Flows in approximating certain distributions.
problem The expressivity of bi-Lipschitz Normalizing Flows in approximating specific target distributions.
method Characterization of expressivity through lower bounds on Total Variation distance and discussion of potential remedies.
result Several target distributions are difficult to approximate using bi-Lipschitz Normalizing Flows, and lower bounds on their approximation are provided.
We provide a unified treatment of a broad class of noisy structure recovery problems, known as structured normal means problems. In this setting, the goal is to identify, from a finite collection of Gaussian distributions with different means, the distribution that produced some observed data. Recent work has studied s…
Fractal Flow enhances normalizing flows with interpretable latent space and hierarchical modeling.
problem High-dimensional density estimation and generative modeling challenges.
method Integrates topic modeling (LDA) and fractal strategy into normalizing flows.
result Achieves latent clustering, controllable generation, and superior estimation accuracy.
Layer normalization placement affects training stability and warm-up stage necessity.
problem Training instability and the necessity of a learning rate warm-up stage in Transformers.
method Theoretical analysis and mean field theory to prove gradient behavior at initialization.
result Removing the warm-up stage for Pre-LN Transformers can achieve comparable results with less time and tuning.
vOED-NFs uses normalizing flows to improve Bayesian OED without likelihood evaluations.
problem Optimizing experiments to maximize information gain in model parameters.
method vOED-NFs combines variational approximations with normalizing flows for efficient EIG estimation.
result vOED-NFs achieves lower EIG estimation bias compared to previous methods.
EvoMSN tackles time series forecasting under distribution shifts by evolving multi-scale normalization.
problem Accurate long-term time series forecasting under complex distribution shifts.
method EvoMSN framework with multi-scale statistics prediction and adaptive ensembling for collaborative updating.
result Improves forecasting performance of five mainstream methods on benchmark datasets.
FPCA optimizes fairness in target vectors' span.
problem Fairness in principal component analysis for multiple target vectors.
method Non-concave maximization of worst projected target norm using sub-gradient descent.
result Optimization landscape is benign with globally optimal local minima.
CAFLOW uses auto-regressive flows to translate images efficiently.
problem Image-to-image translation tasks.
method Transforms conditioning image into latent encodings using normalizing flows, models conditional distribution with auto-regressive distributions.
result Outperforms former conditional flow designs.
New method trains deep ResNets without normalization, achieving state-of-the-art performance.
problem Training deep ResNets without normalization layers leads to instability and lower accuracy.
method Adaptive gradient clipping and Normalizer-Free ResNets design.
result Normalizer-Free ResNets achieve 86.5% top-1 accuracy on ImageNet, matching EfficientNet-B7.
vsOED optimizes experiment design with reinforcement learning for Bayesian models.
problem Optimizing the sequence of experiments in Bayesian models for efficient data collection.
method Reinforcement learning with variational posterior approximations to optimize design policy.
result vsOED achieves superior sample efficiency compared to existing methods.
New method trains any neural network as a generative model.
problem Constrained design of normalizing flows due to analytical invertibility.
method Efficient gradient estimator for non-analytically invertible networks.
result Any dimension-preserving neural network can be used as a generative model.
Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy.…
A new method for adaptive experiments improves inference.
problem Current inference methods for adaptive experiments are weak and asymptotic.
method Simulation-based inference using optimistic simulations.
result Our method achieves better coverage and narrower intervals.
Log-Normal Multiplicative Dynamics improves low-precision training of neural networks.
problem Training large neural networks with low precision is unstable.
method Derive a Bayesian learning rule with log-normal posterior distributions and multiplicative updates.
result LMD achieves stable and accurate training for Vision Transformer and GPT-2.
The paper uses transformed ANOVA to identify important fire detection variables.
problem Identifying key variables for forest fire detection.
method Developed a complete orthonormal system for standard normal distribution, applied Z-score transformation, and used ANOVA approximation.
result Attribute ranking reveals important variables for fire detection.
A new method improves generative models by learning lower-dimensional representations.
problem Normalizing flows cannot learn lower-dimensional representations of data.
method Noisy injective flows (NIF) that map latent space to a learnable manifold in high-dimensional data space using injective transformations and an additive noise model.
result Simple application of NIF to existing flow architectures significantly improves sample quality and yields separable data embeddings.
A new Heckman selection model uses a bivariate contaminated normal distribution for more accurate data analysis.
problem Sample selection biases in econometric data analysis.
method Introduces a Heckman selection model using a bivariate contaminated normal distribution and presents an efficient ECM algorithm for parameter estimation.
result The proposed model outperforms normal and Student's t counterparts in real data analysis and simulation studies.
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find th…
We propose two novel samplers to generate high-quality samples from a given (un-normalized) probability density. Motivated by the success of generative adversarial networks, we construct our samplers using deep neural networks that transform a reference distribution to the target distribution. Training schemes are deve…
Normalizing flows optimize Jacobian determinant for unique likelihood objective.
problem Optimizing normalizing flows for unique likelihood.
method Showed Jacobian determinant is unique for given distributions, leading to a unique global optimum. Used eigenvalues of auto-correlation matrix for explicit likelihood expression.
result Explicit expression of likelihood for flows, independent of neural network parameterization, with theoretical optimal value.
Improved phylogenetic inference with normalizing flows.
problem Limitations of current diagonal Lognormal branch length approximation in VBPI.
method Proposes VBPI-NF using normalizing flows to handle non-Euclidean branch length space.
result Significantly improves phylogenetic posterior estimation on real data.
Estimates funding impact from an algorithmic relief rule, finding little effect on hospital activities.
problem Evaluating the impact of algorithmic policy decisions.
method Developed a treatment-effect estimator using algorithmic decisions as instruments.
result Funding from an algorithmic relief rule had little effect on COVID-19-related hospital activities.
Normalizing flows can now estimate densities on unknown manifolds.
problem Normalizing flows struggle with data on unknown low-dimensional manifolds.
method Conformal Embedding Flows, which combine standard flows with trainable conformal embeddings.
result Tractable density estimation on manifold-supported data is possible.
Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expa…
The statistical analysis of discrete data has been the subject of extensive statistical research dating back to the work of Pearson. In this survey we review some recently developed methods for testing hypotheses about high-dimensional multinomials. Traditional tests like the χ2 test and the likelihood ratio test ca…
Capitalizing on the need for addressing the existing challenges associated with gesture recognition via sparse multichannel surface Electromyography (sEMG) signals, the paper proposes a novel deep learning model, referred to as the XceptionTime architecture. The proposed innovative XceptionTime is designed by integrati…
Neural Manifold ODEs improve manifold data modeling.
problem Adapting deep generative models to non-Euclidean spaces.
method Introducing Neural Manifold ODEs for manifold generalization and continuous probability computation.
result Improves density estimation and downstream tasks on arbitrary manifolds.
Utilizing recently introduced concepts from statistics and quantitative risk management, we present a general variant of Batch Normalization (BN) that offers accelerated convergence of Neural Network training compared to conventional BN. In general, we show that mean and standard deviation are not always the most appro…
Derives TAP approximation for Bayesian linear regression.
problem Log-normalizing constant of posterior distribution in high-dimensional linear regression.
method Variational representation and Thouless-Anderson-Palmer approximation.
result Proves TAP approximation for spherical prior in proportional asymptotic regime.
A new filter design improves system identification accuracy.
problem Improving system identification accuracy for various system types.
method Generalized proportionate-type normalized subband adaptive filter (GPtNSAF) using least squares on subband errors with a sparsity penalty.
result GPtNSAF benefits from increasing subbands more than sparsity for quasi-sparse or dispersive systems, and both aspects are complementary for sparse systems.
Paper proposes using CNN for stock trading with data normalization.
problem Improving stock trading accuracy in volatile markets.
method Developed CNN-based trading framework with novel data normalization.
result CNN-based framework outperforms other methods on 29 stocks.
This work analyzes Batch Normalization through convex optimization, providing insights and improved training methods.
problem Understanding and improving the effectiveness of Batch Normalization in deep neural networks.
method Introducing convex duality to model weight-decay regularized ReLU networks with BN, and designing an explicit regularization approach.
result Gradient Descent provides an algorithmic bias effect on BN networks, which can be explicitly encoded into the convex objective.
New method designs fairer transport plans with uncertainty.
problem Designing fair and balanced mass transport plans.
method Hierarchical fully probabilistic design (HFPD) for transport plans.
result Optimal hyperprior for transport plans with uncertain marginals.
Four new methods for computing generalized chi-square distribution.
problem Computing the generalized chi-square distribution accurately and efficiently.
method Two exact and two approximate methods, with software for cdf, pdf, and inverse cdf.
result Comparison of methods' accuracy and speed, identifying best for different cases.
This work introduces a new model for complex stochastic processes.
problem Difficulties in representing non-stationary distributions with conventional models.
method Recurrent Autoregressive Flows using normalizing flows with recurrent neural connections.
result Demonstrates the effectiveness of the proposed model through experiments.
Paper develops an efficient algorithm for experiment design using synthetic controls.
problem Designing optimal experiments for treatment effect estimation.
method Solves a phase synchronization problem via a normalized generalized power method.
result First global optimality guarantee for experiment design with pre-treatment data.
New methods improve Monte Carlo estimation of partition functions.
problem Estimating the normalization constant of complex distributions.
method Annealing through paths of distributions to estimate partition functions.
result Optimal path for estimation is arithmetic, improving efficiency.
Mitigates anomaly score imbalance in long-tailed distributions.
problem Class imbalance in normal data leads to skewed anomaly detection performance.
method Proposes an importance-weighted loss function to balance anomaly scores.
result Improves anomaly detection performance by 0.043 on real-world datasets.
Antithetic noise improves diffusion models' uncertainty quantification.
problem Improving uncertainty quantification in diffusion models.
method Pairing each noise sample with its negation, leading to strong negative correlation.
result Substantially more reliable uncertainty quantification with up to 90% narrower confidence intervals.
This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.
problem Exploding inverses in INNs cause numerical non-invertibility, leading to failures in various tasks.
method Derived bi-Lipschitz properties of INN building blocks, proposed regularizers for local invertibility, and stable INN designs for global invertibility.
result Bi-Lipschitz properties and stable INN designs are crucial for addressing numerical non-invertibility.
Researchers relax the CVF's smoothness requirement to create more flexible flow models.
problem Challenges in constructing flexible density models due to the CVF's smoothness requirement.
method Introduce L-diffeomorphisms as generalized transformations that may violate smoothness on zero Lebesgue-measure sets. result The relaxation allows for the use of non-smooth activation functions like ReLU in residual flows.