The study finds lottery tickets with positive expected returns are not worth buying.
problem Determining which lottery tickets have positive expected returns.
method Elementary portfolio analysis to evaluate investment strategies.
result Optimal investment strategy includes a very small allocation for lottery tickets with positive expected returns.
Study finds differences in LTs across tasks and architectures, proposing a consensus-based method for generating refined lottery tickets.
problem Understanding the variability and uniqueness of Lottery Tickets across different image classification tasks and architectures.
method 28 combinations of image classification tasks and architectures, iterative pruning techniques, consensus-based method for generating refined lottery tickets.
result Disproves the uniqueness of Lottery Tickets and connects emergent mask structure to the choice of pruning.
Study preferences over uncertain time payments, finds growth-optimality better than expected utility theory.
problem Understanding how people make decisions with uncertain timing of payments.
method Normative model of growth-optimality, revisiting experimental evidence on time lotteries.
result Growth-optimality better explains experimental data on time lotteries than expected discounted utility theory.
The abstract proves the existence of universal lottery tickets without needing further training.
problem The existence of universal sparse subnetworks in large neural networks.
method Theoretical proofs and technical innovations in pruning and subset sum results.
result Universal tickets exist and do not require further training.
LotteryFL improves federated learning efficiency and personalization.
problem Statistical heterogeneity and communication efficiency in federated learning.
method Exploiting Lottery Ticket hypothesis to learn compact lottery networks.
result LotteryFL achieves better personalization and reduces communication cost.
This paper improves lottery ticketing by calibrating network confidence.
problem Uncalibrated confidence in lottery tickets leads to overconfidence and poor performance.
method The paper introduces various calibration strategies and explores their impact on lottery tickets.
result Calibration mechanisms consistently improve lottery ticket performance, even under distribution shifts.
DPLTM uses private winners to improve privacy and utility.
problem Balancing privacy and utility in machine learning models.
method DPLTM applies differential privacy to the lottery ticket method.
result DPLTM converges faster and uses less privacy budget.
We show that, in a resource allocation problem, the ex ante aggregate utility of players with cumulative-prospect-theoretic preferences can be increased over deterministic allocations by implementing lotteries. We formulate an optimization problem, called the system problem, to find the optimal lottery allocation. The …
New neural scaling law found for simple quadratic function.
problem Neural scaling laws and their predictions for model performance.
method Analysis of neural networks, lottery ticket ensembling, statistical interpretation.
result Found a new scaling law (α=1) for a simple quadratic function, contradicting previous theories. Improved pruning method finds winning neural network subnetworks.
problem Finding a small subnetwork that performs as well as a full neural network.
method Data-dependent pruning criterion using gradient of training loss.
result Data-dependent pruning improves existing pruning algorithms.
The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process (Frankle & Carbin, 2019). Intriguingly, this phenomenon suggests that initial…
The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2019) suggests that the performance of over-parameterized deep networks is due to the random initialization seeding the network with a small fraction of favorable weights. These weights retain their dominant status throughout training -- in a very r…
This work improves the lottery ticket hypothesis by reducing over-parameterization requirement.
problem Approximating a neural network by pruning a randomly over-parameterized network.
method Connecting pruning ReLU networks to extsc{SubsetSum} problem, showing logarithmic over-parameterization sufficiency.
result Logarithmic over-parameterization is sufficient for approximating any target neural network.
We find faster-converging sub-networks that significantly reduce adversarial training time.
problem Finding optimal sub-networks for adversarial training is costly and time-consuming.
method We identify a subset of sub-networks that converge faster during training.
result Sub-networks can reduce adversarial training time by up to 49%.
Sparse Transformers degrade semantic information first, with early layers encoding more.
problem Understanding how sparse Transformers affect learned representations and semantic information.
method Probed Transformers with progressively pruned weights to observe changes in semantic information and model behavior.
result Complex semantic information is first to degrade in sparse Transformers, with early layers encoding more.
The lottery ticket hypothesis finds multiple winning sub-networks in neural networks.
problem Finding a single winning sub-network in neural networks.
method Analyzing neural networks trained in isolation and on different tasks.
result Neural networks contain multiple sub-networks that match the accuracy of the original network, not just one.
Logarithmic pruning simplifies lottery ticket hypothesis.
problem Finding efficient subnetworks in large neural networks.
method Logarithmic pruning approach to identify subnetworks.
result Randomly initialized subnetworks achieve comparable performance.
E-LTH finds winning tickets scalable across different network architectures.
problem Finding winning tickets in different network architectures efficiently.
method Tweaking winning tickets from one network to another.
result Winning tickets from one network can be stretched or squeezed into another network's subnetwork.
Pruning method removes less important features in linear models.
problem Removing less important features in linear models trained by gradient flow.
method Iterative Magnitude Pruning (IMP) applied to linear models trained by gradient flow.
result IMP prunes features with smallest projection onto the data.
The thesis explores IMP, a process that identifies winning tickets in DNNs, and its universality.
problem Understanding how winning subnetworks (tickets) perform across different problems.
method Iterative Magnitude Pruning (IMP) and comparison with Renormalisation Group (RG) theory.
result IMP identifies winning subnetworks that can perform similarly across various problems.
New method finds compact strong lottery tickets in partially frozen networks.
problem Finding compact strong lottery tickets in randomly pruned networks.
method Freezing a random subset of initial weights in a partially frozen network.
result Better accuracy and compression achieved with frozen networks.
The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network. We prove an even stronger hypothesis (as was also conjectured in Ramanujan et al., 2019), showi…
The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance …
Convolutional networks struggle to learn Game of Life, even with lottery ticket weights.
problem Training convolutional networks to predict Conway's Game of Life is challenging.
method Examined small convolutional networks trained on Game of Life, focusing on weight initializations and network sizes.
result Minimal networks require significantly more parameters to converge, and their performance is sensitive to small changes in weights.
Sparser Random Feature Models via IMP (ShRIMP) efficiently learns sparse models for high-dimensional data.
problem Learning sparse models for high-dimensional data with sparse variable dependencies.
method Iterative Magnitude Pruning applied to Random Feature Models.
result ShRIMP achieves better or competitive test accuracy compared to state-of-the-art methods.
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural…
By specifying model free preferences towards simple nested classes of lottery pairs, we develop the dual story to stand on equal footing with that of (primal) risk apportionment. The dual story provides an intuitive interpretation, and full characterization, of dual counterparts of such concepts as prudence and tempera…
Develops framework to analyze pruning of neural networks.
problem Understanding pruning of neural networks in non-imaging data.
method Develops framework to plant and hide winning tickets in neural networks.
result Similar trends in ticket sparsity observed across different tasks.
Pruning FCNs reveals sub-networks that match CNNs' performance.
problem Understanding the inductive bias of pruning in neural networks.
method Iterative magnitude pruning of a simple FCN followed by analysis of the resulting architecture.
result Pruned FCNs exhibit key features of CNNs, suggesting new architectural biases.
Unified framework sparsifies GNNs for faster inference on large graphs.
problem Space and computational bottlenecks in GNNs due to graph size and connectivity.
method Unified GNN sparsification (UGS) framework that prunes graph adjacency matrix and model weights.
result Graph lottery tickets (GLTs) can be trained in isolation to match full model performance.
Quantum Ridgelet Transform speeds up neural network learning.
problem Efficiently finding sparse trainable subnetworks in neural networks.
method Developed a quantum ridgelet transform (QRT) for linear runtime.
result Quantum Ridgelet Transform efficiently finds sparse trainable subnetworks.
Paper proposes a method to create smaller, more efficient deep generative audio models.
problem High computation cost and complexity of deep generative models in audio applications.
method Developed a method for structured trimming of deep generative audio models.
result 95% of model weights can be removed without significant degradation in accuracy.
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
This work proves that large models can be compressed significantly without losing performance.
problem Achieving comparable performance with smaller models and less data.
method Developed a universal compression theory for neural networks and datasets.
result Proved that a generic permutation-invariant function can be compressed into a function of polylogarithmic size with vanishing error.
Paper prunes deep MIR models to ultra-light versions.
problem Massive complexity of deep learning models in MIR.
method Lottery ticket hypothesis-based model pruning.
result Up to 90% of model parameters can be removed without loss of accuracy.
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a …
Paper confirms winning tickets have sharp minima, useful for generalization.
problem Explaining why over-parameterized models generalize well despite having sharp minima.
method PAC-Bayesian theory applied to analyze winning tickets and their generalization behavior.
result PAC-Bayesian theory confirms winning tickets have relatively sharp minima, a disadvantage for generalization.
Adversarial robustness improved by sparsity in network weights.
problem How sparsity affects adversarial robustness in neural networks.
method Theoretical proof and experimental validation of adversarial pruning methods.
result Weights sparsity improves adversarial robustness, especially through inheritance from smaller networks.
Explains neural network properties with a simple operation.
problem Understanding neural network performance and generalization.
method Analyze a single operation: expand-and-sparsify.
result Clarifies phenomena like Lottery Ticket Hypothesis and overparameterized models.
New proof shows random neural networks contain sparse subnetworks.
problem Finding sparse subnetworks in random neural networks.
method Proof of the Strong Lottery Ticket Hypothesis using the Random Fixed-Size Subset Sum Problem.
result Guaranteed sparsity of subnetworks in random neural networks.
Pruning at initialization fails to find sparse subnetworks, revealing information-theoretic barriers.
problem Difficulty in finding sparse subnetworks without training the full model.
method Analysis of effective parameter count and mutual information between sparsity mask and data.
result Pruning at initialization cannot find sparse subnetworks due to high mutual information.
A new method to prune neural networks with iterative randomization improves efficiency.
problem Efficiency in pruning randomly initialized neural networks.
method Iteratively randomizing weight values to reduce parameter requirements.
result The method achieves remarkable performance with fewer parameters.
Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket h…
Formalizes vNM utility theorem using Lean 4, proving existence and uniqueness.
problem Formalizing and proving the von Neumann-Morgenstern utility theorem.
method Implement classical axioms in Lean 4, formalizing preference relations over lotteries.
result Machine-verified proofs of existence and uniqueness of utility representations.
There are some statistical anomalies in the Chinese stock market, i.e., positive return skewness, anti-leverage effect (positive returns induce higher volatility than negative returns); and reverse volatility asymmetry (contemporaneous return-volatility correlation is positive). In this paper, we first confirm the exis…
Overparameterisation can limit the benefits of curriculum learning in neural networks.
problem The ineffectiveness of curriculum learning in deep learning applications.
method Analytical study connecting curriculum learning and overparameterisation in an online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem.
result High degree of overparameterisation can limit the benefit from curricula.
Fewer degrees of freedom can train deep networks, showing a sharp phase transition.
problem Training deep networks with fewer degrees of freedom than parameters.
method Examined success probability of hitting training loss sub-level sets within random subspaces.
result Threshold training dimension increases as desired final loss decreases.
A resolution of the St. Petersburg paradox is presented. In contrast to the standard resolution, utility is not required. Instead, the time-average performance of the lottery is computed. The final result can be phrased mathematically identically to Daniel Bernoulli's resolution, which uses logarithmic utility, but is …