The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same wi…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “initial tickets”
Randomly initialized networks can perform as well as pruned networks.
problem Understanding and improving network pruning methods.
method Sanity checks on recent pruning methods, proposing random tickets.
result Randomly initialized networks can perform as well as pruned networks.
Paper confirms winning tickets have sharp minima, useful for generalization.
problem Explaining why over-parameterized models generalize well despite having sharp minima.
method PAC-Bayesian theory applied to analyze winning tickets and their generalization behavior.
result PAC-Bayesian theory confirms winning tickets have relatively sharp minima, a disadvantage for generalization.
The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process (Frankle & Carbin, 2019). Intriguingly, this phenomenon suggests that initial…
The lottery ticket hypothesis finds multiple winning sub-networks in neural networks.
problem Finding a single winning sub-network in neural networks.
method Analyzing neural networks trained in isolation and on different tasks.
result Neural networks contain multiple sub-networks that match the accuracy of the original network, not just one.
The abstract proves the existence of universal lottery tickets without needing further training.
problem The existence of universal sparse subnetworks in large neural networks.
method Theoretical proofs and technical innovations in pruning and subset sum results.
result Universal tickets exist and do not require further training.
Meta-ticket finds optimal sparse subnetworks for few-shot learning in randomly initialized neural networks.
problem Avoiding overfitting in few-shot learning for over-parameterized neural networks.
method Meta-learning approach to find optimal sparse subnetworks.
result Meta-ticket discovers sparse subnetworks that adapt to each task, achieving superior meta-generalization.
Develops framework to analyze pruning of neural networks.
problem Understanding pruning of neural networks in non-imaging data.
method Develops framework to plant and hide winning tickets in neural networks.
result Similar trends in ticket sparsity observed across different tasks.
The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2019) suggests that the performance of over-parameterized deep networks is due to the random initialization seeding the network with a small fraction of favorable weights. These weights retain their dominant status throughout training -- in a very r…
(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the c…
E-LTH finds winning tickets scalable across different network architectures.
problem Finding winning tickets in different network architectures efficiently.
method Tweaking winning tickets from one network to another.
result Winning tickets from one network can be stretched or squeezed into another network's subnetwork.
Improved pruning method finds winning neural network subnetworks.
problem Finding a small subnetwork that performs as well as a full neural network.
method Data-dependent pruning criterion using gradient of training loss.
result Data-dependent pruning improves existing pruning algorithms.
Logarithmic pruning simplifies lottery ticket hypothesis.
problem Finding efficient subnetworks in large neural networks.
method Logarithmic pruning approach to identify subnetworks.
result Randomly initialized subnetworks achieve comparable performance.
The hypothesis that sub-network initializations (lottery) exist within the initializations of over-parameterized networks, which when trained in isolation produce highly generalizable models, has led to crucial insights into network initialization and has enabled efficient inferencing. Supervised models with uncalibrat…
Convolutional networks struggle to learn Game of Life, even with lottery ticket weights.
problem Training convolutional networks to predict Conway's Game of Life is challenging.
method Examined small convolutional networks trained on Game of Life, focusing on weight initializations and network sizes.
result Minimal networks require significantly more parameters to converge, and their performance is sensitive to small changes in weights.
The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance …
Pruning at initialization fails to find sparse subnetworks, revealing information-theoretic barriers.
problem Difficulty in finding sparse subnetworks without training the full model.
method Analysis of effective parameter count and mutual information between sparsity mask and data.
result Pruning at initialization cannot find sparse subnetworks due to high mutual information.
New method finds compact strong lottery tickets in partially frozen networks.
problem Finding compact strong lottery tickets in randomly pruned networks.
method Freezing a random subset of initial weights in a partially frozen network.
result Better accuracy and compression achieved with frozen networks.
The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network. We prove an even stronger hypothesis (as was also conjectured in Ramanujan et al., 2019), showi…
A new method to prune neural networks with iterative randomization improves efficiency.
problem Efficiency in pruning randomly initialized neural networks.
method Iteratively randomizing weight values to reduce parameter requirements.
result The method achieves remarkable performance with fewer parameters.
Explains neural network properties with a simple operation.
problem Understanding neural network performance and generalization.
method Analyze a single operation: expand-and-sparsify.
result Clarifies phenomena like Lottery Ticket Hypothesis and overparameterized models.
Lottery tickets find good initializations for IMP with sparse training.
problem Finding good initializations for iterative magnitude pruning (IMP) in sparse networks.
method Empirical study of IMP performance with varying pre-training data and iterations.
result Training on a small fraction of data suffices to obtain good initializations for IMP.
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a …
The thesis explores IMP, a process that identifies winning tickets in DNNs, and its universality.
problem Understanding how winning subnetworks (tickets) perform across different problems.
method Iterative Magnitude Pruning (IMP) and comparison with Renormalisation Group (RG) theory.
result IMP identifies winning subnetworks that can perform similarly across various problems.
Study finds differences in LTs across tasks and architectures, proposing a consensus-based method for generating refined lottery tickets.
problem Understanding the variability and uniqueness of Lottery Tickets across different image classification tasks and architectures.
method 28 combinations of image classification tasks and architectures, iterative pruning techniques, consensus-based method for generating refined lottery tickets.
result Disproves the uniqueness of Lottery Tickets and connects emergent mask structure to the choice of pruning.
We propose the differentially private lottery ticket mechanism (DPLTM). An end-to-end differentially private training paradigm based on the lottery ticket hypothesis. Using "high-quality winners", selected via our custom score function, DPLTM significantly improves the privacy-utility trade-off over the state-of-the-ar…
Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket h…
Unified framework sparsifies GNNs for faster inference on large graphs.
problem Space and computational bottlenecks in GNNs due to graph size and connectivity.
method Unified GNN sparsification (UGS) framework that prunes graph adjacency matrix and model weights.
result Graph lottery tickets (GLTs) can be trained in isolation to match full model performance.
A new model for learning and unlearning predictors efficiently.
problem Learning and unlearning predictors from subsets of data.
method Proposes a ticketed model for learning-unlearning, allowing examples to present tickets for unlearning.
result Space-efficient schemes for various concept classes, including thresholds and parities.
The study finds lottery tickets with positive expected returns are not worth buying.
problem Determining which lottery tickets have positive expected returns.
method Elementary portfolio analysis to evaluate investment strategies.
result Optimal investment strategy includes a very small allocation for lottery tickets with positive expected returns.
Study analyzes how discounts affect train ticket purchases and rescheduling in Switzerland.
problem Understanding how discounts influence train ticket buying and rescheduling behavior.
method Machine learning techniques, including causal machine learning, to analyze survey data.
result Increasing a discount rate by 1% increases the rescheduled trip share by 0.16% among always buyers.
This paper shows RBMs can maintain strong performance even after extreme pruning, but only if done early in training.
problem The computational and environmental costs of large neural networks.
method Investigating the performance of RBMs under extreme pruning conditions, inspired by the Lottery Ticket Hypothesis.
result RBMs can achieve high-quality generative performance even after 80% pruning, but performance degrades sharply above a critical point.
This research uncovers high-performing subnetworks in deep GNNs without training.
problem Challenges in applying SLTH to deeper GNNs with high memory requirements.
method Introduces Multicoated Supermasks (M-Sup) and Multi-Stage Folding for GNNs.
result Uncovered untrained recurrent networks with performance similar to trained models.
Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weigh…
New neural scaling law found for simple quadratic function.
problem Neural scaling laws and their predictions for model performance.
method Analysis of neural networks, lottery ticket ensembling, statistical interpretation.
result Found a new scaling law () for a simple quadratic function, contradicting previous theories.
Fewer degrees of freedom can train deep networks, showing a sharp phase transition.
problem Training deep networks with fewer degrees of freedom than parameters.
method Examined success probability of hitting training loss sub-level sets within random subspaces.
result Threshold training dimension increases as desired final loss decreases.
Event ticket price prediction is important to marketing strategy for any sports team or musical ensemble. An accurate prediction model can help the marketing team to make promotion plan more effectively and efficiently. However, given all the historical transaction records, it is challenging to predict the sale price o…
This work improves the lottery ticket hypothesis by reducing over-parameterization requirement.
problem Approximating a neural network by pruning a randomly over-parameterized network.
method Connecting pruning ReLU networks to extsc{SubsetSum} problem, showing logarithmic over-parameterization sufficiency.
result Logarithmic over-parameterization is sufficient for approximating any target neural network.
LotteryFL improves federated learning efficiency and personalization.
problem Statistical heterogeneity and communication efficiency in federated learning.
method Exploiting Lottery Ticket hypothesis to learn compact lottery networks.
result LotteryFL achieves better personalization and reduces communication cost.
We find faster-converging sub-networks that significantly reduce adversarial training time.
problem Finding optimal sub-networks for adversarial training is costly and time-consuming.
method We identify a subset of sub-networks that converge faster during training.
result Sub-networks can reduce adversarial training time by up to 49%.
Model shows how Ethereum can capture MEV from block construction, but centralization remains a concern.
problem Ethereum's ability to capture MEV from block construction.
method Economic model of Execution Tickets to study MEV extraction.
result MEV capture decreases with risk aversion and capital costs, and can be low with heterogeneous buyers.
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural…
Sparse Transformers degrade semantic information first, with early layers encoding more.
problem Understanding how sparse Transformers affect learned representations and semantic information.
method Probed Transformers with progressively pruned weights to observe changes in semantic information and model behavior.
result Complex semantic information is first to degrade in sparse Transformers, with early layers encoding more.
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokkingcs.LG
We explain how neural networks learn to solve modular addition tasks.
problem How two-layer neural networks learn to solve modular addition tasks.
method Formalized a diversification condition during training, proving it allows the network to approximate the correct logic for modular addition.
result Neural networks can robustly identify the correct sum through phase symmetry and frequency diversification.
When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including …
Pruning method removes less important features in linear models.
problem Removing less important features in linear models trained by gradient flow.
method Iterative Magnitude Pruning (IMP) applied to linear models trained by gradient flow.
result IMP prunes features with smallest projection onto the data.
Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expen…