Novel method decorrelates neurons for better deep learning model generalization.
problem High correlations between neurons limit deep learning model generalization.
method Regularization terms from minimum spanning tree of neuron cliques, using correlation dissimilarities.
result Our regularizers outperform existing methods and minimize neuron redundancies.
Deep neural networks are widely used in various domains. However, the nature of computations at each layer of the deep networks is far from being well understood. Increasing the interpretability of deep neural networks is thus important. Here, we construct a mean-field framework to understand how compact representation…
In this paper, we propose novel strategies for neutral vector variable decorrelation. Two fundamental invertible transformations, namely serial nonlinear transformation and parallel nonlinear transformation, are proposed to carry out the decorrelation. For a neutral vector variable, which is not multivariate Gaussian d…
Improves causal graph learning on dependent binary data.
problem Challenges in learning causal graphical models from dependent binary data.
method Decorrelation-based approach using latent utility model and EM-like algorithm.
result Significant improvement in accuracy of causal graph learning.
This work explores how feature decorrelation improves self-supervised learning.
problem Complete and dimensional collapse issues in self-supervised learning.
method Study of a concise framework and connection to feature decorrelation.
result Feature decorrelation improves self-supervised learning.
Optimizes signal detection in particle physics by decorrelating classifiers.
problem Systematic errors in background models can mislead signal detection.
method Use optimal transport to decorrelate classifiers from protected variables, then apply semiparametric mixture model.
result Decorrelation and signal enrichment improve the stability, robustness, and power of signal detection tests.
A new distributed algorithm for fitting sparse additive models with feature division and decorrelation.
problem Fitting high-dimensional sparse additive models efficiently and accurately.
method Divide, decorrelate, and conquer approach.
result Effective and efficient recovery of sparsity patterns and statistical inference for each component.
PFDL improves deep learning models' OOD generalization by decorrelating feature embeddings.
problem Out-of-distribution generalization in deep learning models.
method PFDL algorithm that optimizes feature decomposition network and image classification model.
result PFDL improves the accuracy of image classification models on OOD datasets.
Proposes a method to improve Lasso stability via variable decorrelation.
problem Stability of Lasso in high-dimensional settings with correlated variables.
method Variable decorrelation before applying the Lasso.
result Improves stability of variable selection regardless of predictor correlation.
New theory shows how multi-head attention reduces variance and decorrelates outputs.
problem Understanding and optimizing multi-head attention in neural networks.
method Developed a statistical theory linking multi-head attention to ensemble Nadaraya-Watson estimators.
result MHA variance reduction depends on head decorrelation, not just head count.
Improves prediction stability with model misspecification and distribution shift.
problem Inaccurate parameter estimation and instability of prediction in real-world applications.
method Proposes Decorrelated Weighting Regression (DWR) algorithm to optimize weights for samples and variables.
result Significantly improves accuracy of parameter estimation and prediction stability.
Novel method decorrelates batches of triplets for active metric learning.
problem Correlation among triplets degrades active learning performance.
method Proposes a novel method to decorrelate batches of triplets, balancing informativeness and diversity.
result Method outperforms state-of-the-art in active metric learning.
We describe a strategy for constructing a neural network jet substructure tagger which powerfully discriminates boosted decay signals while remaining largely uncorrelated with the jet mass. This reduces the impact of systematic uncertainties in background modeling while enhancing signal purity, resulting in improved di…
SGD reduces test error by decorrelating updates.
problem Improving generalization error in machine learning models.
method Derive a formula for generalization gap change due to SGD updates, compare to GD, and show decorrelation effect.
result SGD implicitly regularizes generalization error by decorrelating updates.
Learning an effective representation for high-dimensional data is a challenging problem in reinforcement learning (RL). Deep reinforcement learning (DRL) such as Deep Q networks (DQN) achieves remarkable success in computer games by learning deeply encoded representation from convolution networks. In this paper, we pro…
A new algorithm removes unexpected correlations in biased data for better clustering.
problem Clustering with selection bias in data.
method Decorrelation regularized K-Means (DCKM) algorithm.
result DCKM achieves significant performance gains on real-world datasets.
Paper proposes methods to improve graph domain adaptation by decorrelating node features.
problem Challenges in transferring knowledge from one graph to another.
method Proposes decorrelating node features using GCN and graph transformer layers.
result Significant performance enhancements and clear visualizations of learned representations.
Unified framework for information-theoretic bounds on learning algorithms.
problem Deriving generalization bounds for learning algorithms.
method Probabilistic decorrelation lemma, symmetrization, couplings, chaining, Young's inequality.
result New upper bounds on generalization error in expectation and high probability.
We introduce the blind subspace deconvolution (BSSD) problem, which is the extension of both the blind source deconvolution (BSD) and the independent subspace analysis (ISA) tasks. We examine the case of the undercomplete BSSD (uBSSD). Applying temporal concatenation we reduce this problem to ISA. The associated `high …
VAE improves anomaly detection for jet tagging at the LHC.
problem Anomaly detection in jet tagging at the LHC.
method Variational Autoencoder (VAE) trained on background QCD jets, with latent space learning for anomaly detection.
result Outlier Exposed VAE (OE-VAE) achieves excellent results in both sensitivity and decorrelation of jet mass.
This paper proposes a decorrelation-based approach to test hypotheses and construct confidence intervals for the low dimensional component of high dimensional proportional hazards models. Motivated by the geometric projection principle, we propose new decorrelated score, Wald and partial likelihood ratio statistics. Wi…
Introduces a new feature importance measure using Gram-Schmidt decorrelation.
problem Determining feature influence strength and nature in datasets.
method Gram-Schmidt decorrelation and random forest regression.
result Empirical comparison of new estimators with established methods.
FreST Loss decorrelates spatio-temporal dependencies in graph signals.
problem Complex spatio-temporal dependencies in graph-structured signals are not well captured by standard forecasting models.
method FreST Loss extends supervision to the joint spatio-temporal spectrum using Joint Fourier Transform (JFT).
result FreST Loss reduces estimation bias and improves forecasting accuracy on real-world datasets.
Two methods use simulation to improve anomaly detection in particle physics.
problem Artificial bumps in invariant mass spectra from machine learning classifiers.
method Simulation-assisted decorrelation techniques.
result Both methods are robust to correlations in the data and improve anomaly detection.
Proposes SVI for covariate-shift generalization with sparse variable independence.
problem Covariate-shift generalization with limited data and unstable variables.
method Introduces sparsity constraint and combines reweighting and selection in an iterative way.
result Improves covariate-shift generalization performance on synthetic and real-world datasets.
CW normalizes and decorrelates neural network layers for better concept understanding.
problem Challenges in understanding neural network hidden layers.
method Introducing concept whitening (CW) to alter neural network layers.
result CW provides clearer understanding of how networks learn concepts over layers.
This study shows how social insects and machine learning methods share a common mathematical framework.
problem Understanding how decentralized systems achieve optimal decision-making.
method Developed a rigorous mathematical framework to show isomorphism between ant colonies and ensemble machine learning.
result Demonstrated that ant colony decision-making and random forest learning implement identical variance reduction strategies through decorrelation of identical units.
Fitting statistical models is computationally challenging when the sample size or the dimension of the dataset is huge. An attractive approach for down-scaling the problem size is to first partition the dataset into subsets and then fit using distributed algorithms. The dataset can be partitioned either horizontally (i…
We consider the problem of uncertainty assessment for low dimensional components in high dimensional models. Specifically, we propose a decorrelated score function to handle the impact of high dimensional nuisance parameters. We consider both hypothesis tests and confidence regions for generic penalized M-estimators. U…
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find th…
Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of non-convergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide new insight into the non-convergence issue of Adam …
Sparse codes improve optimal control tasks with correlated inputs.
problem Optimal control tasks with correlated feature inputs.
method Used a sparse code to represent natural images in an optimal control task solved with neuro-dynamic programming.
result An over-complete sparse code increases memory capacity and learning speed beyond a complete code.
Simplified non-contrastive learning avoids representation collapse.
problem Training failure modes in self-supervised learning.
method Hyperdimensional computing and inductive bias.
result The approach avoids representation collapses.
A crucial problem in learning disentangled image representations is controlling the degree of disentanglement during image editing, while preserving the identity of objects. In this work, we propose a simple yet effective model with the encoder-decoder architecture to address this challenge. To encourage disentanglemen…
Estimators computed from adaptively collected data do not behave like their non-adaptive brethren. Rather, the sequential dependence of the collection policy can lead to severe distributional biases that persist even in the infinite data limit. We develop a general method -- W-decorrelation -- for transformi…
Generalizes PCA and ICA for continuous-time signals using neural networks.
problem Low-rank decomposition of continuous-time vector-valued signals.
method Implicit neural network framework to learn numerical approximations of PCA and ICA.
result Unified approach to PCA and ICA in continuous domain, enforcing decorrelation and independence.
Machine learning models predict EUR/USD currency direction with 58.52% accuracy.
problem Predicting the directional movement of EUR/USD in the Foreign Exchange market.
method Comparative analysis of machine learning models, including decorrelated and non-decorrelated feature sets, and meta-estimators.
result 58.52% accuracy for one-day ahead forecasts.
The paper develops methods for high-dimensional inference in Markov random fields.
problem Statistical inference for high-dimensional Markov random fields.
method Markov Chain Monte Carlo Maximum Likelihood Estimation (MCMC-MLE) with Elastic-net regularization.
result The proposed methods achieve ℓ1-consistency and false discovery rate control. Learning good representations is a long standing problem in reinforcement learning (RL). One of the conventional ways to achieve this goal in the supervised setting is through regularization of the parameters. Extending some of these ideas to the RL setting has not yielded similar improvements in learning. In this pape…
Neuron Shapley identifies key neurons in deep networks, improving model accuracy and fairness.
problem Identifying responsible neurons in deep networks for better model performance and fairness.
method Neuron Shapley framework quantifies neuron contributions, accounting for interactions.
result Removing just 30 critical filters can destroy model accuracy, revealing network function.
Describes explaining neurons in deep representations using compositional logical concepts.
problem Interpreting neuron behavior in deep neural networks.
method Identifying compositional logical concepts that closely approximate neuron behavior.
result Compositional explanations provide insights into model performance and allow for adversarial example creation.
SeReNe prunes neurons with low sensitivity to reduce network size.
problem Large neural networks consume too many resources on resource-constrained devices.
method Exploits neural sensitivity as a regularizer to prune neurons with low sensitivity.
result Pruning neurons with low sensitivity achieves competitive compression ratios.
Under-parameterized networks can either copy or average teacher weights, leading to universal optimal solutions.
problem Approximating a teacher network with an under-parameterized student network.
method Analyzing shallow neural networks with erf activation function and unitary teacher weights, proving copy-average configurations are critical points and finding the optimal solution.
result The optimal solution for under-parameterized networks has a universal structure, whether copying or averaging teacher neurons.
Modeling hidden neurons in SNNs using mesoscopic approximations.
problem Underconstrained problem of modeling unobserved neurons in SNNs.
method Coarse-graining and mean-field approximations to derive neuLVM.
result neuLVM can efficiently model large SNNs and recover connectivity parameters.
Topological methods improve neuron analysis and tracer injection summary.
problem Traditional methods fail to capture the tree-like structure of neurons.
method Discrete Morse (DM) Theory for neuron skeletonization and consensus tree summarization.
result Significant performance improvements over non-topological methods.
Based on a recent theorem due to the authors, it is shown how the extreme tail dependence between an asset and a factor or index or between two assets can be easily calibrated. Portfolios constructed with stocks with minimal tail dependence with the market exhibit a remarkable degree of decorrelation with the market at…
Inspired by complexity and diversity of biological neurons, our group proposed quadratic neurons by replacing the inner product in current artificial neurons with a quadratic operation on input data, thereby enhancing the capability of an individual neuron. Along this direction, we are motivated to evaluate the power o…
Losaw improves FI scores by decorrelating features in ML models.
problem Feature correlation distorts feature importance scores in ML models.
method Losaw uses local sample weighting to decorrelate features.
result Losaw consistently improves feature importance scores and prediction accuracy.