Bayesian deep learning method using subnetwork inference.
problem Improving deep neural networks' calibration and efficiency.
method Perform inference over a subset of model weights, keeping others as point estimates.
result Subnetwork inference enables accurate predictive posteriors without full network approximations.
SupSup model learns thousands of tasks without forgetting, using randomly initialized subnetworks.
problem Sequentially learning many tasks without forgetting.
method Randomly initialized base network with task-specific subnetworks (supermasks).
result Gradient-based optimization can identify the correct subnetwork for new tasks.
We discuss two views on extending existing methods for complex network modeling which we dub the communities first and the networks first view, respectively. Inspired by the networks first view that we attribute to White, Boorman, and Breiger (1976)[1], we formulate the multiple-networks stochastic blockmodel (MNSBM), …
Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket h…
New algorithm finds unbiased subnetworks in biased datasets.
problem Finding unbiased subnetworks in biased neural networks.
method Debiased Contrastive Weight Pruning (DCWP) algorithm.
result DCWP significantly outperforms state-of-the-art debiasing methods.
New method finds unbiased subnetworks in biased models for better OOD performance.
problem How to improve out-of-distribution generalization in deep models.
method Functional modular probing method and Modular Risk Minimization.
result Even in biased models, there are unbiased subnetworks that can achieve better OOD performance.
We propose a novel perspective to understand deep neural networks in an interpretable disentanglement form. For each semantic class, we extract a class-specific functional subnetwork from the original full model, with compressed structure while maintaining comparable prediction performance. The structure representation…
Greedy selection finds smaller, more accurate subnetworks.
problem Finding smaller, accurate subnetworks in large neural networks.
method Greedy forward selection starting from an empty network.
result Theoretical guarantee of finding subnetworks with lower loss.
Energy disaggregation, known in the literature as Non-Intrusive Load Monitoring (NILM), is the task of inferring the power demand of the individual appliances given the aggregate power demand recorded by a single smart meter which monitors multiple appliances. In this paper, we propose a deep neural network that combin…
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
Single model predicts multiple subnetworks for robustness.
problem Efficiently ensemble neural networks for robustness and uncertainty.
method Multi-input multi-output (MIMO) configuration to train independent subnetworks.
result Significant improvement in negative log-likelihood, accuracy, and calibration error.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
SGD tends to favor simpler subnetworks, improving generalization.
problem SGD's tendency to favor simpler subnetworks over complex ones.
method Identifying invariant sets and analyzing SGD's behavior around them.
result SGD collapses networks to simpler subnetworks, improving generalization.
Proposes a new method for ensembling neural subnetworks.
problem Computational expense and limited flexibility of traditional deep ensembles.
method Sequential Bayesian neural subnetwork ensembling.
result Outperforms traditional ensembles in various metrics.
Logarithmic pruning simplifies lottery ticket hypothesis.
problem Finding efficient subnetworks in large neural networks.
method Logarithmic pruning approach to identify subnetworks.
result Randomly initialized subnetworks achieve comparable performance.
Develops a deep survival model for causal inference in longitudinal studies.
problem Estimating treatment effects on time-to-event outcomes in observational studies with time-dependent covariates.
method TCS model using potential outcomes framework and ensemble of recurrent subnetworks.
result Identifies conditional average treatment effects and individual treatment effect heterogeneity over time.
Improved pruning method finds winning neural network subnetworks.
problem Finding a small subnetwork that performs as well as a full neural network.
method Data-dependent pruning criterion using gradient of training loss.
result Data-dependent pruning improves existing pruning algorithms.
New proof shows random neural networks contain sparse subnetworks.
problem Finding sparse subnetworks in random neural networks.
method Proof of the Strong Lottery Ticket Hypothesis using the Random Fixed-Size Subset Sum Problem.
result Guaranteed sparsity of subnetworks in random neural networks.
HeteroFL trains diverse clients with varying capabilities efficiently.
problem Training models on clients with different computational and communication capacities.
method Proposes HeteroFL framework to adaptively distribute subnetworks based on client capabilities.
result Adaptive subnetwork distribution leads to efficient computation and communication.
Randomly initialized networks contain subnetworks that perform similarly to target networks.
problem Proving the lottery ticket hypothesis for neural networks.
method Pruning over-parameterized neural networks to find subnetworks.
result Randomly initialized networks contain subnetworks with similar performance to target networks without additional training.
New classical algorithm outperforms quantum in neural network subnetwork selection.
problem Selecting sparse subnetworks from large neural networks efficiently.
method Quantum-inspired classical algorithm using ridgelet transform sampling.
result Runs in polynomial time, outperforming naive classical methods.
Meta-ticket finds optimal sparse subnetworks for few-shot learning in randomly initialized neural networks.
problem Avoiding overfitting in few-shot learning for over-parameterized neural networks.
method Meta-learning approach to find optimal sparse subnetworks.
result Meta-ticket discovers sparse subnetworks that adapt to each task, achieving superior meta-generalization.
Pruning improves DNNs against MIA while reducing model size and computation.
problem Vulnerability of DNNs to membership inference attacks (MIA).
method Proposes a pruning algorithm to reduce model size and computational operations.
result Pruned subnetwork prevents privacy leakage from MIA with competitive accuracy.
This work proposes a method to optimize hyperparameters without validation data.
problem Optimizing hyperparameters for neural networks with limited data.
method Neural network partitioning and subnetwork optimization.
result The method optimizes hyperparameters efficiently and without validation data.
New method makes neural networks transparent, revealing learning modes.
problem Lack of interpretability in neural networks.
method Weight pathway analysis (WPA) to decompose neural networks into subnetworks.
result Neural networks store and utilize information holographically, with linear and nonlinear learning modes.
Machine learning solves Einstein equations without symmetry assumptions.
problem Solving the Euclidean vacuum Einstein equations with a cosmological constant.
method Semi-supervised machine learning with patch-based architecture and coordinate consistency loss.
result Hints against the existence of Ricci-flat metrics on spheres in dimensions 4 and 5.
This research uncovers high-performing subnetworks in deep GNNs without training.
problem Challenges in applying SLTH to deeper GNNs with high memory requirements.
method Introduces Multicoated Supermasks (M-Sup) and Multi-Stage Folding for GNNs.
result Uncovered untrained recurrent networks with performance similar to trained models.
Framework predicts clinical severity from rs-fMRI data using network optimization.
problem Predicting clinical severity from rs-fMRI data.
method Joint network optimization framework combining sparse subnetworks and linear regression.
result Framework outperforms standard methods and identifies clinically relevant ASD networks.
To infer multilayer deep representations of high-dimensional discrete and nonnegative real vectors, we propose an augmentable gamma belief network (GBN) that factorizes each of its hidden layers into the product of a sparse connection weight matrix and the nonnegative real hidden units of the next layer. The GBN's hidd…
This paper introduces a generic method which enables to use conventional deep neural networks as end-to-end one-class classifiers. The method is based on splitting given data from one class into two subsets. In one-class classification, only samples of one normal class are available for training. During inference, a cl…
BetaExplainer improves GNN interpretability by masking unimportant edges.
problem Interpreting GNNs' predictions is difficult due to black-box behavior and lack of uncertainty quantification.
method BetaExplainer uses a sparsity-inducing prior to mask unimportant edges during training.
result BetaExplainer provides uncertainty in edge importance and improves predictive accuracy on challenging datasets.
The abstract proves the existence of universal lottery tickets without needing further training.
problem The existence of universal sparse subnetworks in large neural networks.
method Theoretical proofs and technical innovations in pruning and subset sum results.
result Universal tickets exist and do not require further training.
Randomly initialized networks can perform as well as pruned networks.
problem Understanding and improving network pruning methods.
method Sanity checks on recent pruning methods, proposing random tickets.
result Randomly initialized networks can perform as well as pruned networks.
Study how neural networks optimize to stable linearly connected regions.
problem Understanding how neural networks converge to stable solutions under different training conditions.
method Investigate the stability of neural networks to SGD noise and apply it to iterative magnitude pruning.
result Subnetworks that reach full accuracy must be stable to SGD noise, either at initialization or early in training.
Benefitting from large-scale training datasets and the complex training network, Convolutional Neural Networks (CNNs) are widely applied in various fields with high accuracy. However, the training process of CNNs is very time-consuming, where large amounts of training samples and iterative operations are required to ob…
The thesis explores IMP, a process that identifies winning tickets in DNNs, and its universality.
problem Understanding how winning subnetworks (tickets) perform across different problems.
method Iterative Magnitude Pruning (IMP) and comparison with Renormalisation Group (RG) theory.
result IMP identifies winning subnetworks that can perform similarly across various problems.
We study the problem of learning Bayesian network structures from data. Koivisto and Sood (2004) and Koivisto (2006) presented algorithms that can compute the exact marginal posterior probability of a subnetwork, e.g., a single edge, in O(n2n) time and the posterior probabilities for all n(n-1) potential edges in O(n2n…
New method filters large networks from financial data to reveal key subnetworks.
problem Filtering large dimensional networks to isolate key constituents.
method Exploits spectral properties of high-dimensional data networks, tuning for sparsity and consistency.
result Shows method can interpolate between zero and maximal filtering, preserving spectral properties.
TSSM splits neural networks for parallel training with minimal accuracy loss.
problem Accuracy degradation in parallel training of deep neural networks.
method TSSM reformulates alternating minimization to achieve parallelism with minimal accuracy loss.
result TSSM achieves significant speedup without accuracy loss on multiple datasets.
Develops a Causal Transformer for estimating counterfactual outcomes from longitudinal data.
problem Estimating counterfactual outcomes over time from observational data is challenging due to complex, long-range dependencies.
method Combines three transformer subnetworks with separate inputs for time-varying covariates, previous treatments, and previous outcomes into a joint network with in-between cross-attentions. Uses a custom, end-to-end training procedure with a counterfactual domain confusion loss to address confounding bias.
result Achieves superior performance over current baselines in synthetic and real-world datasets.
SDAMI enhances interpretable high-dimensional regression with sparse deep learning and footprint principle.
problem Personalized models for small samples and high-dimensional features with interpretability.
method Sparse Deep Additive Model with Interactions (SDAMI) combining sparsity-driven feature selection and deep subnetworks.
result SDAMI successfully identifies pure interactions with near-zero false positive rates.
The paper tests if LLMs' capabilities are executed by small subnetworks (circuits).
problem Understanding how LLMs execute their capabilities.
method Formalized criteria for circuits, developed hypothesis tests, applied to six circuits.
result Synthetic circuits align with idealized properties, while Transformer circuits vary in their alignment.
Recent advances in neuroscience and in the technology of functional magnetic resonance imaging (fMRI) and electro-encephalography (EEG) have propelled a growing interest in brain-network clustering via time-series analysis. Notwithstanding, most of the brain-network clustering methods revolve around state clustering an…
New method finds efficient low-rank neural networks during training.
problem High memory and computational demands of neural networks.
method Restricts weight matrices to a low-rank manifold and updates low-rank factors.
result Significantly reduced time and memory resources required for training and evaluation.
IMP finds sparse subnetworks that match full networks, revealing geometric insights.
problem Finding sparse subnetworks that match full, overparameterized networks.
method Iterative magnitude pruning (IMP) algorithm, analyzing error landscape geometry.
result IMP masks found at end of training convey useful information for rewound networks.
GAMI-Net improves neural network interpretability while maintaining accuracy.
problem Lack of interpretability in neural network models.
method GAMI-Net is a disentangled feedforward network with multiple additive subnetworks designed for capturing main effects and pairwise interactions, considering sparsity, heredity, and marginal clarity.
result GAMI-Net achieves superior interpretability and competitive prediction accuracy compared to explainable boosting machine and other models.
While noise is commonly considered a nuisance in computing systems, a number of studies in neuroscience have shown several benefits of noise in the nervous system from enabling the brain to carry out computations such as probabilistic inference as well as carrying additional information about the stimuli. Similarly, no…
Quantum Ridgelet Transform speeds up neural network learning.
problem Efficiently finding sparse trainable subnetworks in neural networks.
method Developed a quantum ridgelet transform (QRT) for linear runtime.
result Quantum Ridgelet Transform efficiently finds sparse trainable subnetworks.