We improve neural network explainability by bypassing batch normalization.
problem Lack of transparency in neural networks.
method Layer-wise Relevance Propagation with a method to include normalization layers.
result Heatmaps are more accurate for convolutional layers with our method.
The article constructs Feynman propagators for normally hyperbolic operators on curved spacetimes.
problem Constructing Feynman propagators for non-scalar geometric operators on curved spacetimes.
method Global microlocalisation constructions for normally hyperbolic operators on globally hyperbolic spacetimes.
result Feynman propagators can be constructed to satisfy a positivity property for selfadjoint normally hyperbolic operators.
Proposes a faster normalization technique to address internal covariate shift in deep networks.
problem Internal covariate shift during deep network training and validation.
method Normalization Propagation: a non-adaptive technique using parametric estimates of mean and standard deviation.
result Faster and more accurate normalization without batch statistics, applicable with batch-size 1.
Proposes a method to estimate neural network statistics analytically for normalization.
problem Estimating statistics of hidden units in neural networks for better initialization and normalization.
method Analytic moment propagation of mean and variance through the network structure.
result Analytic estimates of statistics are useful for initialization and normalization, independent of batch input.
ILM-Norm normalizes instances individually for better performance.
problem Learning to normalize parameters for improved model performance.
method ILM-Norm learns normalization parameters via feature feed-forward and gradient back-propagation.
result ILM-Norm consistently improves model performance across different architectures and tasks.
Proposes a new normalization technique to address issues with Batch Normalization.
problem Inaccurate batch statistics and inability to use with batch-size 1.
method Normalization Propagation: uses data-independent parametric estimates of mean and standard deviation.
result Computational efficiency and faster convergence compared to BN.
A method for converting NIW parameters for better estimation.
problem Estimating parameters of multivariate normal distribution.
method Convergent procedure for converting mean parameters to natural parameters in NIW family.
result Maximum likelihood estimation of natural parameters from observed statistics.
New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
Efficient EP algorithm improves smoothing distribution inference in financial models.
problem Computational intractability of smoothing distribution in high dimensions.
method Adapted expectation propagation (EP) algorithms for the unified skew-normal family.
result Accuracy gains in financial illustrations over existing approximate algorithms.
Paper proposes MABN to stabilize BN in small batch sizes.
problem Weakness of BN in small batch sizes limits its use in tasks with small batch sizes.
method Identifies and addresses two extra batch statistics in BN's backward propagation.
result MABN completely restores BN's performance in small batch sizes without additional nonlinear operations.
Analysis shows BN prevents gradient vanishing/explosion in residual networks.
problem Gradient vanishing/explosion problem in residual networks.
method Mathematical analysis of BN and residual network training.
result BN confounds gradient variance, preventing vanishing/explosion.
New method improves ResNet performance without batch normalization.
problem Improving ResNet performance without batch normalization.
method Adapted Weight Standardization to maintain signal propagation.
result Highly performant ResNets achieve state-of-the-art performance on ImageNet.
An analytico-geometric reflection principle is established by means of normal deformations of analytic discs.
EP method speeds up Bayesian probit regression in high dimensions.
problem Computational challenges in high-dimensional Bayesian probit regression.
method Adapting EP approximation to multivariate Gaussian prior and skew-normal distribution.
result EP routine is computationally feasible in high-dimensional settings.
Propagates adversarial robustness in federated learning.
problem Heterogeneous federated users with non-iid data and limited resources.
method Adversarial robustness propagation through batch-normalization.
result Federated models achieve remarkable robustness even with limited adversarial training.
Research aims to explain how ResNets' stability improves image classification performance.
problem Understanding why ResNets enhance image classification performance.
method Examines batch normalization and the dynamical systems view of ResNets to understand stability and smoothness.
result Stability of inter-layer propagation in ResNets contributes to enhanced performance.
High frequency limit for most of wave phenomena is known as quasiclassical limit or ray optics limit. Propagation of waves in this limit is described in terms of wave fronts and rays. Wave front is a surface of constant phase whose points are moving along rays. As it appears, their motion can be described by Hamilton e…
Paper classifies heart sound recordings as normal or abnormal.
problem Classifying normal/abnormal heart sound recordings.
method Four steps: preprocessing, feature extraction, training, validation. Back propagation neural network used.
result Optimal threshold determined for distinguishing normal and abnormal.
Belief Propagation outperforms other algorithms in reconstructing binary symmetric channel trees.
problem Reconstructing binary symmetric channel trees with bounded memory.
method Combining recursive reconstruction, information theory, and optimal transport.
result Any recursive algorithm with bounded memory for the reconstruction problem on binary symmetric channel trees has a phase transition strictly below the Belief Propagation threshold.
Proposes a feed-forward method for uncertainty propagation in neural networks.
problem Uncertainty in neural network inputs and activations.
method Feed-forward inference method for belief and neural networks, considering both mean and variance.
result Propagates uncertainty through neural network layers and parameters.
A new method for target propagation using iterative approximations converges fast and is more biologically plausible.
problem Improving target propagation methods for neural networks.
method Iterative approximate inverses and local auto-encoders.
result The method converges exponentially fast under certain conditions.
The paper proves the existence of a folded annulus with multiple creases.
problem Existence of a folded annulus with multiple creases.
method Analyze developable surfaces, use normal curvature and relative torsion, compute geometric descriptors, prove propagation of folds.
result Proves the existence of a folded annulus with multiple creases.
NeVI-Cut uses neural networks to efficiently propagate uncertainty without feedback.
problem Efficiently propagating uncertainty in downstream Bayesian analysis without feedback.
method NeVI-Cut combines neural networks and normalizing flows for variational inference.
result NeVI-Cut achieves significant computational gains and higher accuracy than traditional methods.
Meta-embeddings improve PLDA model's accuracy for speaker recognition.
problem Improving PLDA model's accuracy for speaker recognition.
method Proposed Gaussian meta-embeddings (GMEs) for heavy-tailed PLDA models.
result GMEs with variable precisions propagate uncertainty, leading to up to 20% more accurate results.
EBU improves deep RL with episodic backward updates, achieving human-level performance with less data.
problem Efficiently learn from sparse and delayed rewards in reinforcement learning.
method Episodic Backward Update (EBU) algorithm that recursively propagates value updates.
result EBU achieves human-level performance in Atari games with significantly fewer samples.
Decoupled GCN is shown to be equivalent to label propagation.
problem Improving semi-supervised node classification in graph learning.
method The paper proves the equivalence of decoupled GCN and label propagation, and proposes a new method named PTA.
result Decoupled GCN is equivalent to two-step label propagation and can automatically assign weights to pseudo-labels.
Many inference problems involving questions of optimality ask for the maximum or the minimum of a finite set of unknown quantities. This technical report derives the first two posterior moments of the maximum of two correlated Gaussian variables and the first two posterior moments of the two generating variables (corre…
Solves inverse problem for Calderón using microlocal normal forms.
problem Recovering unknown coefficient from boundary measurements.
method Microlocal normal forms and propagation of singularities.
result Recovery of integrals of unknown coefficient over good bicharacteristic leaves.
Identifies root causes of outliers in unknown cyclic graphs.
problem Outliers in unknown cyclic graphs with linear structural equations.
method Identifies a short list of potential root causes based on strong perturbation and structural equations.
result The shortlist includes true root causes and their parents on the cycle.
New insights into belief propagation and Bethe approximation for factor graphs.
problem Understanding the correctness and efficiency of belief propagation and its relation to partition functions.
method Viewing factor graphs through the lens of polynomials and reformulating Bethe approximation as a polynomial optimization problem.
result For bipartite normal factor graphs, the Bethe approximation is a lower bound to the partition function under certain analytic conditions.
Paper analyzes misclassification in binary stochastic block model with correlated degrees.
problem Sharp recovery thresholds for cluster recovery in degree-correlated stochastic block model.
method Connects cluster recovery to tree reconstruction problems, analyzes density evolution of belief propagation on Gaussian approximated trees.
result Minimum fraction of misclassified vertices on average is given by Q ( v ∗ ) Q(\sqrt{v^*}) Q ( v ∗ ) where v ∗ v^* v ∗ is a fixed point of a specific equation. GHNet improves graph learning by balancing homogeneity and heterogeneity.
problem Over-smoothing in GCN leads to similar node representations.
method GHNet uses gating units to balance homogeneity and heterogeneity in feature propagation.
result GHNet achieves larger receptive fields without over-smoothing.
A new neural layer normalizes time series data for improved DL performance.
problem Degradation of DL model performance due to inappropriate normalization of time series data.
method Adaptive normalization neural layer trained end-to-end using back-propagation.
result Significant performance improvements over other normalization schemes.
EDAIN layer normalizes time series data for neural networks, improving model performance.
problem Irregularities in time series data degrade model performance in neural networks.
method EDAIN layer learns adaptive normalization parameters during end-to-end training.
result EDAIN layer outperforms conventional normalization methods and adaptive layers.
Paper proposes faster certified robust training methods with short warmup.
problem Certified robust training methods require long warmup schedules, making training costly.
method Proposes three improvements: new weight initialization, BN, and regularization.
result Achieves 65.03% verified error on CIFAR-10 and 82.36% on TinyImageNet with short warmup.
The paper improves RBP for stable and efficient training of recurrent neural networks.
problem Stability issues in RBP for training recurrent neural networks.
method Proposed two variants of RBP: CG-RBP and Neumann-RBP, and compared them with BPTT and TBPTT.
result Neumann-RBP is more efficient in terms of memory usage compared to TBPTT.
Flowification enriches neural networks with an inverse pass and likelihood monitoring.
problem Neural networks lack an inverse pass and likelihood monitoring, limiting their generative capabilities.
method Introduce flowification, enriching neural networks with a stochastic inverse pass and likelihood monitoring.
result Certain neural network architectures can be enriched to fall under the generalized notion of a normalizing flow.
A new adversarial DBN framework for deep learning.
problem Training deep belief networks (DBNs) efficiently and with scalability.
method Replacing generator in GANs with DBN and developing a parallelizable training algorithm.
result Framework can be applied to general DBNs without backpropagation.
Study geometric isomorphisms between spacetime solutions using paracausal metrics.
problem Geometric isomorphisms between solutions of normally hyperbolic operators over different spacetimes.
method Introduce paracausal relation to define isomorphisms between spacetime metrics and use Møller operators.
result Møller operators preserve causal propagators and natural symplectic forms on initial data.
Deep gradient boosting normalizes neural network inputs layer by layer.
problem Improving neural network performance on image recognition tasks.
method Introducing deep gradient boosting with layer-wise input normalization.
result Layer-wise input normalization improves image recognition performance.
A new aggregation strategy improves GNN performance and learning dynamics.
problem Improving expressivity and learning dynamics of GNNs.
method Proposes a variance-preserving aggregation function (VPA) for GNNs.
result VPA leads to increased predictive performance and improved learning dynamics.
Bidirectional learning improves neural network robustness to noise and attacks.
problem Improving neural network robustness to adversarial and static noise.
method Bidirectional learning (BL) techniques using error propagation and hybrid adversarial networks (HAN).
result Both methods improve robustness and accuracy, with HAN showing state-of-the-art performance.
It is known that fixed points of loopy belief propagation (BP) correspond to stationary points of the Bethe variational problem, where we minimize the Bethe free energy subject to normalization and marginalization constraints. Unfortunately, this does not entirely explain BP because BP is a dual rather than primal algo…
New equations connect unit Killing vectors to initial data.
problem Characterizing initial data for Einstein vacuum with unit Killing vectors.
method Developed new equations (uKID) by eliminating scaling and using propagation identity.
result Found equations that are finite type and characterize unit normalized Killing vectors.
Proves well-posedness for Einstein equations with totally geodesic timelike boundary condition.
problem Initial boundary value problem for Einstein equations with specific geometric boundary condition.
method ADM system, parallelly propagated orthonormal frame, modified evolution equations, hyperbolic systems, constraints propagation.
result First well-posedness result for Einstein equations with totally geodesic timelike boundary condition.
It is of increasing importance to develop learning methods for ranking. In contrast to many learning objectives, however, the ranking problem presents difficulties due to the fact that the space of permutations is not smooth. In this paper, we examine the class of rank-linear objective functions, which includes popular…
AdaFlow adapts DNN density estimators to new distributions for anomaly detection and cross-domain translation.
problem Unsupervised anomaly detection and cross-domain translation with limited paired data.
method AdaFlow combines Normalizing Flows and Adaptive Batch-Normalizations for domain adaptation.
result AdaFlow efficiently adapts to new distributions with minimal computational resources.
Iterative BP-CNN improves channel decoding under correlated noise.
problem Channel decoding under correlated noise.
method Concatenates CNN with BP decoder, iteratively improving SNR.
result Iterative BP-CNN achieves better BER with lower complexity.