A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Four improvements to Batch Normalization improve deep learning performance.
A new metric GNQ audits LLMs for privacy risks during training.
Background: Deep learning models are typically trained using stochastic gradient descent or one of its variants. These methods update the weights using their gradient, estimated from a small fraction of the training data. It has been observed that when using large batch sizes there is a persistent degradation in genera…
Computational ghost imaging is an imaging technique in which an object is imaged from light collected using a single-pixel detector with no spatial resolution. Recently, ghost cytometry has been proposed for a high-speed cell-classification method that involves ghost imaging and machine learning in flow cytometry. Ghos…
Ghost points affect stability in finite difference schemes for diffusion equations.
A new method normalizes activations to match batch normalization without batch dependence.
Formula connects length and correlation functions via ghost polygons and Poisson bracket.
In this paper we define and study the "ghost loop orbifold" of an orbifold consisting of those loops that remain constant in the coarse moduli space of . We construct a configuration space model for the ghost loop orbifold using an idea of G. Segal. From this we exhibit the relation between the Hochschild and cy…
CBN improves batch normalization for small mini-batch sizes.
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
Online Normalization is a new technique for normalizing the hidden activations of a neural network. Like Batch Normalization, it normalizes the sample dimension. While Online Normalization does not use batches, it is as accurate as Batch Normalization. We resolve a theoretical limitation of Batch Normalization by intro…
TaskNorm improves meta-learning performance by rethinking batch normalization.
The study counts periodic orbits on smooth manifolds, adding ghost orbits for completeness.
Paper proposes MABN to stabilize BN in small batch sizes.
Batch normalization makes deep residual networks train faster.
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute…
Ghost mechanism explains abrupt learning in RNNs, revealing constraints on optimization landscapes.
In this work, we investigate Batch Normalization technique and propose its probabilistic interpretation. We propose a probabilistic model and show that Batch Normalization maximazes the lower bound of its marginalized log-likelihood. Then, according to the new probabilistic model, we design an algorithm which acts cons…
Batch uniformization improves anomaly detection in sound data.
A BV algebra is a formal framework within which the BV quantization algorithm is implemented. In addition to the gauge symmetry, encoded in the BV master equation, the master action often exhibits further global symmetries, which may be in turn gauged. We show how to carry this out in a BV algebraic set up. Depending o…
Zero-shot anomaly detection method using batch normalization.
New method uses batch normalization to improve OoD detection.
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly im…
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks. Yet, despite its apparent empirical benefits, the reasons behind the success of Batch Normalization are mostly hypothetical. We here aim to provide a more thorough theoretical understanding from a clas…
Nyquist ghost artifacts in EPI are originated from phase mismatch between the even and odd echoes. However, conventional correction methods using reference scans often produce erroneous results especially in high-field MRI due to the non-linear and time-varying local magnetic field changes. Recently, it was shown that …
This research investigates if deep neural networks can be trained without batch normalization.
Batch normalization in the last layer reduces sharpness in wide neural networks.
We propose a novel unsupervised domain adaptation framework based on domain-specific batch normalization in deep neural networks. We aim to adapt to both domains by specializing batch normalization layers in convolutional neural networks while allowing them to share all other model parameters, which is realized by a tw…
As an indispensable component, Batch Normalization (BN) has successfully improved the training of deep neural networks (DNNs) with mini-batches, by normalizing the distribution of the internal representation for each hidden layer. However, the effectiveness of BN would diminish with scenario of micro-batch (e.g., less …
IEBN normalizes noise by enhancing instance-specific information, improving deep learning performance.
Develops methods for reliable inference on batched bandit data.
New method trains deep ResNets without normalization, achieving state-of-the-art performance.
Finet uses FBN for efficient, lightweight neural networks.
Proposes group whitening to enhance deep learning models' performance.
A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with o…
BYOL learns useful representations without batch statistics.
Gradient control plays an important role in feed-forward networks applied to various computer vision tasks. Previous work has shown that Recurrent Highway Networks minimize the problem of vanishing or exploding gradients. They achieve this by setting the eigenvalues of the temporal Jacobian to 1 across the time steps. …
We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, a…
We show that the (4,5)-torus knot admits exactly one ghost character. We then show that this ghost character provides the following two important results. (1) It is known that for any knot every (meridionally) trace-free $\SL_2(\C)$-representation of the knot group yields an $\SL_2(\C)$-representat…
New Convolutional Unit improves Batch Whitening performance.
Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural n…
Deep Neural Networks (DNNs) thrive in recent years in which Batch Normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the reduction operations. In this paper, we propose alleviating this problem through sampling only a small fraction of data for normalization at each i…
This paper explains how batch normalization auto-tunes the regularization parameter based on data statistics.
Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by d…
Batch normalization prevents rank collapse in deep networks, improving training stability.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …
The study explains delayed spikes in batch-normalized models.