A new framework extracts informative features from multiple data types.
problem Finding maximally correlated feature representations from multimodal data.
method Soft-HGR framework, which relaxes whitening constraints while preserving feature geometry.
result Soft-HGR learns more informative feature mappings and is more efficient to optimize.
New Convolutional Unit improves Batch Whitening performance.
problem Improving the efficiency and effectiveness of Batch Whitening.
method Proposes a new Convolutional Unit that aligns with Batch Whitening theory and empirically analyzes the original Convolutional Unit.
result Significantly improved performance on multiple image classification datasets.
Whitening, or sphering, is a common preprocessing step in statistical analysis to transform random variables to orthogonality. However, due to rotational freedom there are infinitely many possible whitening procedures. Consequently, there is a diverse range of sphering methods in use, for example based on principal com…
Corrected whitening restores orthogonality in high-dimensional spherical Gaussian mixtures.
problem In high-dimensional data, standard whitening fails to preserve orthogonality of mixture means.
method Derived exact limits for whitened means dot products using random matrix theory, constructed a corrected whitening matrix.
result Corrected whitening allows for improved estimation of spherical Gaussian mixtures in the large-dimensional regime.
We present here a new model and algorithm which performs an efficient Natural gradient descent for Multilayer Perceptrons. Natural gradient descent was originally proposed from a point of view of information geometry, and it performs the steepest descent updates on manifolds in a Riemannian space. In particular, we ext…
Proposes group whitening to enhance deep learning models' performance.
problem Improving learning efficiency and representational capacity in deep learning models.
method Group Whitening (GW) combines whitening and group normalization to balance these aspects.
result Group Whitening consistently improves model performance across different architectures and benchmarks.
wBSL uses whitening transformations to speed up BSL for intractable likelihood models.
problem Computational demands of Bayesian synthetic likelihood with growing summary statistics.
method Whitening transformations to decorrelate summary statistics.
result Significant reduction in model simulations required for accurate inference.
CW-Gen models improve probabilistic time series forecasting by incorporating prior information.
problem Challenges in probabilistic forecasting of multivariate time series due to non-stationarity, inter-variable dependencies, and distribution shifts.
method CW-Gen framework that incorporates prior information through conditional whitening. JMCE learns conditional mean and covariance, improving sample quality.
result CW-Gen consistently enhances predictive performance, capturing non-stationary dynamics and inter-variable correlations more effectively than prior-free approaches.
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find th…
A new SSL method using whitening of latent-space features.
problem Efficiency and effectiveness of self-supervised representation learning.
method Proposes a new loss function based on whitening of latent-space features, avoiding the need for negatives and asymmetric networks.
result Improves efficiency and effectiveness of SSL by avoiding the need for negatives and asymmetric networks.
Data whitening and second order optimization harm generalization by reducing access to dataset information.
problem Harmful effects of data whitening and second order optimization on generalization in machine learning.
method Analysis of fully connected models and experimental verification.
result Data whitening and second order optimization reduce or prevent generalization by limiting access to dataset information.
Unified framework for scale-invariant representation learning using MAPCA.
problem Learning invariant representations in data.
method Metric-Aware Principal Component Analysis (MAPCA) based on generalized eigenproblem.
result MAPCA provides a unified geometric language for various self-supervised learning objectives.
New method improves SACOBRA's performance on high-conditioning optimization problems.
problem High-conditioning optimization problems with expensive objective functions.
method Online whitening applied to SACOBRA in the black-box optimization paradigm.
result Online whitening reduces optimization error by a factor of 10 to 1e12 compared to plain SACOBRA.
A simple whitening method improves disentanglement in VAEs without sacrificing reconstruction quality.
problem Learning disentangled latent variables in VAEs.
method Proposes a whitening-based approach to disentangle latent variables in VAEs.
result The method finds interpretable latent factors without compromising reconstruction quality.
WS diffusion models handle anisotropic Gaussian noise better than conventional methods.
problem Handling anisotropic Gaussian noise in imaging inverse problems.
method Whitened Score (WS) diffusion models based on stochastic differential equations.
result WS DMs outperform conventional DMs on anisotropic Gaussian noise.
CW normalizes and decorrelates neural network layers for better concept understanding.
problem Challenges in understanding neural network hidden layers.
method Introducing concept whitening (CW) to alter neural network layers.
result CW provides clearer understanding of how networks learn concepts over layers.
Non-Gaussian component analysis (NGCA) is an unsupervised linear dimension reduction method that extracts low-dimensional non-Gaussian "signals" from high-dimensional data contaminated with Gaussian noise. NGCA can be regarded as a generalization of projection pursuit (PP) and independent component analysis (ICA) to mu…
PCA whitening weighted by Zipfian word frequencies improves task performance.
problem Skewed word embedding spaces in neural models.
method PCA whitening weighted by empirical word frequencies following Zipf's law.
result Significantly improves task performance, surpassing baselines.
New proof shows coupling-based flows converge linearly to diagonalize data covariance.
problem Understanding convergence of coupling-based normalizing flows to arbitrary data distributions.
method Proved linear convergence rate for whitening of data distribution.
result Coupling-based flows achieve linear convergence to diagonalize data covariance.
Improved convergence in deep learning networks via gradient transformation.
problem Reduced convergence speed in deep learning networks.
method Gradient transformation to weight gradients and recursive condition number reduction.
result Improved convergence speed and accuracy in image classification tasks.
Batch Normalization (BN) is a common technique used to speed-up and stabilize training. On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing class-specific information using conditional Batch Normalization (cBN). In this paper we pro…
Reservoir subspace injection improves online ICA by preserving injected features.
problem Discarding injected features in top-n whitening can degrade performance. method Formalized reservoir subspace injection (RSI) and developed diagnostics (IER, SSO, ρ_x) to identify and mitigate the failure mode.
result RSI controller preserves passthrough retention, improving performance by up to 2.2 dB.
CVAE detects weak complex signals in maritime radar, improving detection over classical methods.
problem Detecting weak complex-valued signals in non-Gaussian, range-varying interference.
method Complex-valued Variational AutoEncoder (CVAE) trained on clutter-plus-noise, whitening, ANMF fusion.
result CVAE yields higher detection probability Pd at matched false-alarm rate Pfa, especially with whitening.
The natural gradient allows for more efficient gradient descent by removing dependencies and biases inherent in a function's parameterization. Several papers present the topic thoroughly and precisely. It remains a very difficult idea to get your head around however. The intent of this note is to provide simple intuiti…
Paper introduces a new cost function to improve deep learning model generalization.
problem Overfitting and poor extrapolation of deep learning models.
method Introduces a 'whitening' cost function based on the Ljung-Box statistic.
result Significant improvement in generalization for RNNs and image autoencoders.
Proposes σ-PCA to learn identifiable linear transformations without whitening.
problem Cannot identify axes with equal variances in PCA.
method Unified model for linear and nonlinear PCA, introducing a missing piece to eliminate rotational indeterminacy.
result Eliminates subspace rotational indeterminacy in PCA.
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
There are a multitude of methods to perform multi-set correlated component analysis (MCCA), including some that require iterative solutions. The methods differ on the criterion they optimize and the constraints placed on the solutions. This note focuses perhaps on the simplest version, which can be solved in a single s…
Sharp-SSL uses random projections to identify important variables for semi-supervised learning.
problem High-dimensional semi-supervised learning problems.
method Careful aggregation of low-dimensional results from many axis-aligned random projections.
result Sharp-SSL algorithm can recover signal coordinates with high probability.
Novel framework synthesizes stochastic trajectories with anticipated structural breaks.
problem Synthesizing forward-looking, time-evolving stochastic trajectories with anticipated structural breaks.
method Anticipatory Neural Jump-Diffusion (ANJD) flow, AVNSG for dynamic spectral whitening.
result The framework effectively captures non-commutative moments and high-order stochastic texture.
Employers actively look for talents having not only specific hard skills but also various soft skills. To analyze the soft skill demands on the job market, it is important to be able to detect soft skill phrases from job advertisements automatically. However, a naive matching of soft skill phrases can lead to false pos…
New method extracts biological concepts from cell microscopy images.
problem Extracting meaningful concepts from vision foundation models trained on cell microscopy images.
method Sparse dictionary learning (DL) combined with PCA whitening pre-processing.
result Successfully retrieved biologically meaningful concepts like cell types and genetic perturbations.
Soft labeling impacts OOD detection in neural networks.
problem Impact of soft labeling on OOD detection in deep neural networks.
method Empirical analysis of how soft labeling affects OOD detection performance.
result Soft labeling can either improve or deteriorate OOD detection performance.
Developed a new thresholding method that connects soft and hard thresholding.
problem Connecting soft and hard thresholding methods in data analysis.
method Scaled soft thresholding method with empirical scaling values.
result Found two sources of over-fitting in the scaled soft thresholding method.
Soft cells fill space without gaps, derived from minimal surfaces and deformed using edge bending.
problem Creating space-filling shapes without sharp corners.
method Edge bending algorithm to deform polyhedral tilings into soft tilings.
result Soft tilings derived from minimal surfaces can be continuously transformed into one another.
A new method for RL with continuous actions improves stability and scalability.
problem Stability and scalability issues in existing RL methods.
method Soft policy gradient with entropy regularization, combined with double sampling for soft Bellman equation.
result Outperforms off-policy prior methods in continuous action RL tasks.
ASBART accelerates Soft BART for faster Bayesian regression.
problem Slow computation in Soft BART.
method Proposed ASBART, a variant of Soft BART.
result ASBART is about 10 times faster than Soft BART with similar accuracy.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
A new model for sequential memory using temporal predictive coding.
problem Forming accurate memory of sequential stimuli in the brain.
method Proposes a novel PC-based model called temporal predictive coding (tPC).
result Shows that tPC models can accurately memorize and retrieve sequential inputs.
Generalizes soft noncommutative schemes to flag varieties.
problem Applying soft noncommutative schemes to flag varieties.
method Generalization via toric geometry and distinguished affine charts.
result Soft noncommutative schemes can be applied to flag varieties.
HIP-GP improves GP inference for inter-domain observations with millions of inducing points.
problem Inference for Gaussian Processes across different domains.
method Hierarchical inducing point Gaussian process with grid structure and stationary kernel assumption.
result Improved approximation accuracy through increased number of inducing points.
Neural networks can learn kernel machines with a data-dependent kernel.
problem Can neural networks in the rich feature learning regime learn a kernel machine?
method Demonstrated silent alignment effect in neural networks, showing they can learn a kernel machine with a data-dependent kernel.
result Neural networks in the rich feature learning regime can learn a kernel machine with a data-dependent kernel due to silent alignment.
Gradient descent converges linearly for deep linear networks under specific conditions.
problem Speed of convergence in gradient descent for deep linear neural networks.
method Analysis of gradient descent training for deep linear neural networks minimizing ℓ2 loss. result Gradient descent converges linearly under specific conditions on layer dimensions, initialization, and initial loss.
New Soft Actor-Critic for discrete actions.
problem Applying reinforcement learning to games with discrete actions.
method Derived an alternative Soft Actor-Critic for discrete actions.
result Competitive with tuned model-free state-of-the-art on Atari games.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
Soft-AE interprets autoencoders with adaptable wavelet units.
problem Lack of interpretability in autoencoders.
method Proposes Soft-AE with adaptable wavelet units and GenLU.
result Soft-AE offers interpretability and competitive performance.
Introduces Soft-SVM for binary classification bridging logistic and SVM.
problem Data separability issues in binary classification.
method Soft-SVM regression using convex relaxation of hinge loss with softness and class-separation parameters.
result Soft-SVM performs well in classification and prediction errors.
Deep model learns from labeled and unlabeled data for industrial soft sensing.
problem Lack of data limits soft sensor development in industrial processes.
method Hierarchical, generative deep latent variable model for semi-supervised multi-unit soft sensing.
result Model outperforms current methods for soft sensing in industrial processes.