New Convolutional Unit improves Batch Whitening performance.
problem Improving the efficiency and effectiveness of Batch Whitening.
method Proposes a new Convolutional Unit that aligns with Batch Whitening theory and empirically analyzes the original Convolutional Unit.
result Significantly improved performance on multiple image classification datasets.
A new SSL method using whitening of latent-space features.
problem Efficiency and effectiveness of self-supervised representation learning.
method Proposes a new loss function based on whitening of latent-space features, avoiding the need for negatives and asymmetric networks.
result Improves efficiency and effectiveness of SSL by avoiding the need for negatives and asymmetric networks.
Improved convergence in deep learning networks via gradient transformation.
problem Reduced convergence speed in deep learning networks.
method Gradient transformation to weight gradients and recursive condition number reduction.
result Improved convergence speed and accuracy in image classification tasks.
Proposes group whitening to enhance deep learning models' performance.
problem Improving learning efficiency and representational capacity in deep learning models.
method Group Whitening (GW) combines whitening and group normalization to balance these aspects.
result Group Whitening consistently improves model performance across different architectures and benchmarks.
Reservoir subspace injection improves online ICA by preserving injected features.
problem Discarding injected features in top-n whitening can degrade performance. method Formalized reservoir subspace injection (RSI) and developed diagnostics (IER, SSO, ρ_x) to identify and mitigate the failure mode.
result RSI controller preserves passthrough retention, improving performance by up to 2.2 dB.
Batch Normalization (BN) is a common technique used to speed-up and stabilize training. On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing class-specific information using conditional Batch Normalization (cBN). In this paper we pro…
Whitening, or sphering, is a common preprocessing step in statistical analysis to transform random variables to orthogonality. However, due to rotational freedom there are infinitely many possible whitening procedures. Consequently, there is a diverse range of sphering methods in use, for example based on principal com…
Corrected whitening restores orthogonality in high-dimensional spherical Gaussian mixtures.
problem In high-dimensional data, standard whitening fails to preserve orthogonality of mixture means.
method Derived exact limits for whitened means dot products using random matrix theory, constructed a corrected whitening matrix.
result Corrected whitening allows for improved estimation of spherical Gaussian mixtures in the large-dimensional regime.
One primary focus in multimodal feature extraction is to find the representations of individual modalities that are maximally correlated. As a well-known measure of dependence, the Hirschfeld-Gebelein-Rényi (HGR) maximal correlation becomes an appealing objective because of its operational meaning and desirable propert…
We present here a new model and algorithm which performs an efficient Natural gradient descent for Multilayer Perceptrons. Natural gradient descent was originally proposed from a point of view of information geometry, and it performs the steepest descent updates on manifolds in a Riemannian space. In particular, we ext…
CW-Gen models improve probabilistic time series forecasting by incorporating prior information.
problem Challenges in probabilistic forecasting of multivariate time series due to non-stationarity, inter-variable dependencies, and distribution shifts.
method CW-Gen framework that incorporates prior information through conditional whitening. JMCE learns conditional mean and covariance, improving sample quality.
result CW-Gen consistently enhances predictive performance, capturing non-stationary dynamics and inter-variable correlations more effectively than prior-free approaches.
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find th…
Adaptive classifier optimizes high-dimensional data with spiked covariance structure.
problem Classification of high-dimensional data with spiked covariance structure.
method Adaptive classifier that whitens data, screens features, and applies Fisher linear discriminant.
result The classifier is Bayes optimal under certain conditions and performs well on real and synthetic data.
TAROT improves data selection for complex multimodal distributions.
problem Suboptimal data selection in multimodal distributions.
method TAROT uses optimal transport theory to select data, addressing greedy heuristics' limitations.
result TAROT outperforms state-of-the-art methods across various deep learning tasks.
We propose Power Slow Feature Analysis, a gradient-based method to extract temporally slow features from a high-dimensional input stream that varies on a faster time-scale, as a variant of Slow Feature Analysis (SFA) that allows end-to-end training of arbitrary differentiable architectures and thereby significantly ext…
Data whitening and second order optimization harm generalization by reducing access to dataset information.
problem Harmful effects of data whitening and second order optimization on generalization in machine learning.
method Analysis of fully connected models and experimental verification.
result Data whitening and second order optimization reduce or prevent generalization by limiting access to dataset information.
Unified framework for scale-invariant representation learning using MAPCA.
problem Learning invariant representations in data.
method Metric-Aware Principal Component Analysis (MAPCA) based on generalized eigenproblem.
result MAPCA provides a unified geometric language for various self-supervised learning objectives.
What does a neural network encode about a concept as we traverse through the layers? Interpretability in machine learning is undoubtedly important, but the calculations of neural networks are very challenging to understand. Attempts to see inside their hidden layers can either be misleading, unusable, or rely on the la…
Likelihood-free methods are an established approach for performing approximate Bayesian inference for models with intractable likelihood functions. However, they can be computationally demanding. Bayesian synthetic likelihood (BSL) is a popular such method that approximates the likelihood function of the summary statis…
New method extracts biological concepts from cell microscopy images.
problem Extracting meaningful concepts from vision foundation models trained on cell microscopy images.
method Sparse dictionary learning (DL) combined with PCA whitening pre-processing.
result Successfully retrieved biologically meaningful concepts like cell types and genetic perturbations.
WS diffusion models handle anisotropic Gaussian noise better than conventional methods.
problem Handling anisotropic Gaussian noise in imaging inverse problems.
method Whitened Score (WS) diffusion models based on stochastic differential equations.
result WS DMs outperform conventional DMs on anisotropic Gaussian noise.
Non-Gaussian component analysis (NGCA) is an unsupervised linear dimension reduction method that extracts low-dimensional non-Gaussian "signals" from high-dimensional data contaminated with Gaussian noise. NGCA can be regarded as a generalization of projection pursuit (PP) and independent component analysis (ICA) to mu…
PCA whitening weighted by Zipfian word frequencies improves task performance.
problem Skewed word embedding spaces in neural models.
method PCA whitening weighted by empirical word frequencies following Zipf's law.
result Significantly improves task performance, surpassing baselines.
Real-world optimization problems often have expensive objective functions in terms of cost and time. It is desirable to find near-optimal solutions with very few function evaluations. Surrogate-assisted optimizers tend to reduce the required number of function evaluations by replacing the real function with an efficien…
New proof shows coupling-based flows converge linearly to diagonalize data covariance.
problem Understanding convergence of coupling-based normalizing flows to arbitrary data distributions.
method Proved linear convergence rate for whitening of data distribution.
result Coupling-based flows achieve linear convergence to diagonalize data covariance.
Neural networks can learn kernel machines with a data-dependent kernel.
problem Can neural networks in the rich feature learning regime learn a kernel machine?
method Demonstrated silent alignment effect in neural networks, showing they can learn a kernel machine with a data-dependent kernel.
result Neural networks in the rich feature learning regime can learn a kernel machine with a data-dependent kernel due to silent alignment.
CVAE detects weak complex signals in maritime radar, improving detection over classical methods.
problem Detecting weak complex-valued signals in non-Gaussian, range-varying interference.
method Complex-valued Variational AutoEncoder (CVAE) trained on clutter-plus-noise, whitening, ANMF fusion.
result CVAE yields higher detection probability Pd at matched false-alarm rate Pfa, especially with whitening.
The natural gradient allows for more efficient gradient descent by removing dependencies and biases inherent in a function's parameterization. Several papers present the topic thoroughly and precisely. It remains a very difficult idea to get your head around however. The intent of this note is to provide simple intuiti…
Machine learning classifies gravitational wave signals to test General Relativity.
problem Testing General Relativity with gravitational wave signals from binary black hole mergers.
method Convolutional Neural Networks (CNNs) trained on whitened waveforms and response function type observables.
result CNNs improve classification sensitivity by a factor of approximately 33 compared to whitened waveforms.
Paper introduces a new cost function to improve deep learning model generalization.
problem Overfitting and poor extrapolation of deep learning models.
method Introduces a 'whitening' cost function based on the Ljung-Box statistic.
result Significant improvement in generalization for RNNs and image autoencoders.
Proposes σ-PCA to learn identifiable linear transformations without whitening.
problem Cannot identify axes with equal variances in PCA.
method Unified model for linear and nonlinear PCA, introducing a missing piece to eliminate rotational indeterminacy.
result Eliminates subspace rotational indeterminacy in PCA.
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
Low rank matrix factorization is a fundamental building block in machine learning, used for instance to summarize gene expression profile data or word-document counts. To be robust to outliers and differences in scale across features, a matrix factorization step is usually preceded by ad-hoc feature normalization steps…
There are a multitude of methods to perform multi-set correlated component analysis (MCCA), including some that require iterative solutions. The methods differ on the criterion they optimize and the constraints placed on the solutions. This note focuses perhaps on the simplest version, which can be solved in a single s…
Sharp-SSL uses random projections to identify important variables for semi-supervised learning.
problem High-dimensional semi-supervised learning problems.
method Careful aggregation of low-dimensional results from many axis-aligned random projections.
result Sharp-SSL algorithm can recover signal coordinates with high probability.
Novel framework synthesizes stochastic trajectories with anticipated structural breaks.
problem Synthesizing forward-looking, time-evolving stochastic trajectories with anticipated structural breaks.
method Anticipatory Neural Jump-Diffusion (ANJD) flow, AVNSG for dynamic spectral whitening.
result The framework effectively captures non-commutative moments and high-order stochastic texture.
After deep generative models were successfully applied to image generation tasks, learning disentangled latent variables of data has become a crucial part of deep generative model research. Many models have been proposed to learn an interpretable and factorized representation of latent variable by modifying their objec…
Eigendecomposition (ED) is widely used in deep networks. However, the backpropagation of its results tends to be numerically unstable, whether using ED directly or approximating it with the Power Iteration method, particularly when dealing with large matrices. While this can be mitigated by partitioning the data in sma…
Proposes a flexible feature allocation model for sparse factor analysis.
problem Sparse data and rigid assumptions in traditional exploratory tools.
method Adaptive latent feature sharing with control over feature sparsity.
result Derives a novel adaptive Factor analysis (aFA) and aPPCA for flexible dimensionality reduction.
A new model for sequential memory using temporal predictive coding.
problem Forming accurate memory of sequential stimuli in the brain.
method Proposes a novel PC-based model called temporal predictive coding (tPC).
result Shows that tPC models can accurately memorize and retrieve sequential inputs.
HIP-GP improves GP inference for inter-domain observations with millions of inducing points.
problem Inference for Gaussian Processes across different domains.
method Hierarchical inducing point Gaussian process with grid structure and stationary kernel assumption.
result Improved approximation accuracy through increased number of inducing points.
New GLS estimator handles high-dimensional data with autocorrelated errors.
problem High-dimensional regressions with autocorrelated errors.
method LASSO regression, autoregressive model fitting, and whitening.
result The method outperforms unadjusted LASSO in estimating errors driven by autoregressive processes.
We analyze deep neural networks using convex duality to reveal hidden layer structures.
problem Understanding the structure of deep neural networks.
method Introducing a convex analytic framework to characterize hidden layer weights.
result Optimal hidden layer weights align with previous layers via duality.
The study explains delayed spikes in batch-normalized models.
problem Delayed spikes in batch-normalized models.
method Analyzing batch-normalized linear models, deriving conditions for delayed onset and waiting time.
result Explicit conditions for delayed-onset and waiting time in whitened square-loss linear regression.
Boosts linear neurons for faster convergence in neural networks.
problem Optimizing neural networks' neurons in function space.
method Reparameterizing network to whiten neuron features using Boosted Backpropagation.
result Equivalent to preconditioned gradient descent, leading to faster convergence.
The study finds a trade-off between model size, test loss, and training loss for linear predictors.
problem Finding the optimal balance between model size, test loss, and training loss for linear predictors.
method Established an algorithm and distribution-independent trade-off using non-asymptotic analysis.
result Models with low test loss are either classical (close to noise level training loss) or modern (large number of parameters).
This paper addresses the problem of blind demixing of instantaneous mixtures in a multiple-input multiple-output communication system. The main objective is to present efficient blind source separation (BSS) algorithms dedicated to moderate or high-order QAM constellations. Four new iterative batch BSS algorithms are p…
MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.
problem Training matrix-valued parameters with orthogonalized-update optimizers like Muon.
method MuonEq introduces three lightweight pre-orthogonalization equilibration schemes: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).
result Row/column normalization acts as a zeroth-order surrogate for whitening and improves the geometry seen by orthogonalization.