(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the c…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “train tickets”
The lottery ticket hypothesis finds multiple winning sub-networks in neural networks.
problem Finding a single winning sub-network in neural networks.
method Analyzing neural networks trained in isolation and on different tasks.
result Neural networks contain multiple sub-networks that match the accuracy of the original network, not just one.
The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same wi…
The abstract proves the existence of universal lottery tickets without needing further training.
problem The existence of universal sparse subnetworks in large neural networks.
method Theoretical proofs and technical innovations in pruning and subset sum results.
result Universal tickets exist and do not require further training.
E-LTH finds winning tickets scalable across different network architectures.
problem Finding winning tickets in different network architectures efficiently.
method Tweaking winning tickets from one network to another.
result Winning tickets from one network can be stretched or squeezed into another network's subnetwork.
Randomly initialized networks can perform as well as pruned networks.
problem Understanding and improving network pruning methods.
method Sanity checks on recent pruning methods, proposing random tickets.
result Randomly initialized networks can perform as well as pruned networks.
Study analyzes how discounts affect train ticket purchases and rescheduling in Switzerland.
problem Understanding how discounts influence train ticket buying and rescheduling behavior.
method Machine learning techniques, including causal machine learning, to analyze survey data.
result Increasing a discount rate by 1% increases the rescheduled trip share by 0.16% among always buyers.
DPLTM uses private winners to improve privacy and utility.
problem Balancing privacy and utility in machine learning models.
method DPLTM applies differential privacy to the lottery ticket method.
result DPLTM converges faster and uses less privacy budget.
Improved pruning method finds winning neural network subnetworks.
problem Finding a small subnetwork that performs as well as a full neural network.
method Data-dependent pruning criterion using gradient of training loss.
result Data-dependent pruning improves existing pruning algorithms.
Develops framework to analyze pruning of neural networks.
problem Understanding pruning of neural networks in non-imaging data.
method Develops framework to plant and hide winning tickets in neural networks.
result Similar trends in ticket sparsity observed across different tasks.
Paper confirms winning tickets have sharp minima, useful for generalization.
problem Explaining why over-parameterized models generalize well despite having sharp minima.
method PAC-Bayesian theory applied to analyze winning tickets and their generalization behavior.
result PAC-Bayesian theory confirms winning tickets have relatively sharp minima, a disadvantage for generalization.
We find faster-converging sub-networks that significantly reduce adversarial training time.
problem Finding optimal sub-networks for adversarial training is costly and time-consuming.
method We identify a subset of sub-networks that converge faster during training.
result Sub-networks can reduce adversarial training time by up to 49%.
A new model for learning and unlearning predictors efficiently.
problem Learning and unlearning predictors from subsets of data.
method Proposes a ticketed model for learning-unlearning, allowing examples to present tickets for unlearning.
result Space-efficient schemes for various concept classes, including thresholds and parities.
Study finds differences in LTs across tasks and architectures, proposing a consensus-based method for generating refined lottery tickets.
problem Understanding the variability and uniqueness of Lottery Tickets across different image classification tasks and architectures.
method 28 combinations of image classification tasks and architectures, iterative pruning techniques, consensus-based method for generating refined lottery tickets.
result Disproves the uniqueness of Lottery Tickets and connects emergent mask structure to the choice of pruning.
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural…
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
Randomly initialized networks contain subnetworks that perform similarly to target networks.
problem Proving the lottery ticket hypothesis for neural networks.
method Pruning over-parameterized neural networks to find subnetworks.
result Randomly initialized networks contain subnetworks with similar performance to target networks without additional training.
The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process (Frankle & Carbin, 2019). Intriguingly, this phenomenon suggests that initial…
The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2019) suggests that the performance of over-parameterized deep networks is due to the random initialization seeding the network with a small fraction of favorable weights. These weights retain their dominant status throughout training -- in a very r…
Logarithmic pruning simplifies lottery ticket hypothesis.
problem Finding efficient subnetworks in large neural networks.
method Logarithmic pruning approach to identify subnetworks.
result Randomly initialized subnetworks achieve comparable performance.
IMP finds sparse subnetworks that match full networks, revealing geometric insights.
problem Finding sparse subnetworks that match full, overparameterized networks.
method Iterative magnitude pruning (IMP) algorithm, analyzing error landscape geometry.
result IMP masks found at end of training convey useful information for rewound networks.
This paper improves lottery ticketing by calibrating network confidence.
problem Uncalibrated confidence in lottery tickets leads to overconfidence and poor performance.
method The paper introduces various calibration strategies and explores their impact on lottery tickets.
result Calibration mechanisms consistently improve lottery ticket performance, even under distribution shifts.
Meta-ticket finds optimal sparse subnetworks for few-shot learning in randomly initialized neural networks.
problem Avoiding overfitting in few-shot learning for over-parameterized neural networks.
method Meta-learning approach to find optimal sparse subnetworks.
result Meta-ticket discovers sparse subnetworks that adapt to each task, achieving superior meta-generalization.
Convolutional networks struggle to learn Game of Life, even with lottery ticket weights.
problem Training convolutional networks to predict Conway's Game of Life is challenging.
method Examined small convolutional networks trained on Game of Life, focusing on weight initializations and network sizes.
result Minimal networks require significantly more parameters to converge, and their performance is sensitive to small changes in weights.
The thesis explores IMP, a process that identifies winning tickets in DNNs, and its universality.
problem Understanding how winning subnetworks (tickets) perform across different problems.
method Iterative Magnitude Pruning (IMP) and comparison with Renormalisation Group (RG) theory.
result IMP identifies winning subnetworks that can perform similarly across various problems.
The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance …
LotteryFL improves federated learning efficiency and personalization.
problem Statistical heterogeneity and communication efficiency in federated learning.
method Exploiting Lottery Ticket hypothesis to learn compact lottery networks.
result LotteryFL achieves better personalization and reduces communication cost.
Pruning method removes less important features in linear models.
problem Removing less important features in linear models trained by gradient flow.
method Iterative Magnitude Pruning (IMP) applied to linear models trained by gradient flow.
result IMP prunes features with smallest projection onto the data.
Unified framework sparsifies GNNs for faster inference on large graphs.
problem Space and computational bottlenecks in GNNs due to graph size and connectivity.
method Unified GNN sparsification (UGS) framework that prunes graph adjacency matrix and model weights.
result Graph lottery tickets (GLTs) can be trained in isolation to match full model performance.
We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a …
Pruning at initialization fails to find sparse subnetworks, revealing information-theoretic barriers.
problem Difficulty in finding sparse subnetworks without training the full model.
method Analysis of effective parameter count and mutual information between sparsity mask and data.
result Pruning at initialization cannot find sparse subnetworks due to high mutual information.
Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket h…
The study finds lottery tickets with positive expected returns are not worth buying.
problem Determining which lottery tickets have positive expected returns.
method Elementary portfolio analysis to evaluate investment strategies.
result Optimal investment strategy includes a very small allocation for lottery tickets with positive expected returns.
This research uncovers high-performing subnetworks in deep GNNs without training.
problem Challenges in applying SLTH to deeper GNNs with high memory requirements.
method Introduces Multicoated Supermasks (M-Sup) and Multi-Stage Folding for GNNs.
result Uncovered untrained recurrent networks with performance similar to trained models.
Adversarial robustness improved by sparsity in network weights.
problem How sparsity affects adversarial robustness in neural networks.
method Theoretical proof and experimental validation of adversarial pruning methods.
result Weights sparsity improves adversarial robustness, especially through inheritance from smaller networks.
New neural scaling law found for simple quadratic function.
problem Neural scaling laws and their predictions for model performance.
method Analysis of neural networks, lottery ticket ensembling, statistical interpretation.
result Found a new scaling law () for a simple quadratic function, contradicting previous theories.
New proof shows random neural networks contain sparse subnetworks.
problem Finding sparse subnetworks in random neural networks.
method Proof of the Strong Lottery Ticket Hypothesis using the Random Fixed-Size Subset Sum Problem.
result Guaranteed sparsity of subnetworks in random neural networks.
Pruning FCNs reveals sub-networks that match CNNs' performance.
problem Understanding the inductive bias of pruning in neural networks.
method Iterative magnitude pruning of a simple FCN followed by analysis of the resulting architecture.
result Pruned FCNs exhibit key features of CNNs, suggesting new architectural biases.
Event ticket price prediction is important to marketing strategy for any sports team or musical ensemble. An accurate prediction model can help the marketing team to make promotion plan more effectively and efficiently. However, given all the historical transaction records, it is challenging to predict the sale price o…
This work improves the lottery ticket hypothesis by reducing over-parameterization requirement.
problem Approximating a neural network by pruning a randomly over-parameterized network.
method Connecting pruning ReLU networks to extsc{SubsetSum} problem, showing logarithmic over-parameterization sufficiency.
result Logarithmic over-parameterization is sufficient for approximating any target neural network.
This paper shows RBMs can maintain strong performance even after extreme pruning, but only if done early in training.
problem The computational and environmental costs of large neural networks.
method Investigating the performance of RBMs under extreme pruning conditions, inspired by the Lottery Ticket Hypothesis.
result RBMs can achieve high-quality generative performance even after 80% pruning, but performance degrades sharply above a critical point.
This work proves that large models can be compressed significantly without losing performance.
problem Achieving comparable performance with smaller models and less data.
method Developed a universal compression theory for neural networks and datasets.
result Proved that a generic permutation-invariant function can be compressed into a function of polylogarithmic size with vanishing error.
Lottery tickets find good initializations for IMP with sparse training.
problem Finding good initializations for iterative magnitude pruning (IMP) in sparse networks.
method Empirical study of IMP performance with varying pre-training data and iterations.
result Training on a small fraction of data suffices to obtain good initializations for IMP.
Model shows how Ethereum can capture MEV from block construction, but centralization remains a concern.
problem Ethereum's ability to capture MEV from block construction.
method Economic model of Execution Tickets to study MEV extraction.
result MEV capture decreases with risk aversion and capital costs, and can be low with heterogeneous buyers.
Explains neural network properties with a simple operation.
problem Understanding neural network performance and generalization.
method Analyze a single operation: expand-and-sparsify.
result Clarifies phenomena like Lottery Ticket Hypothesis and overparameterized models.
Sparse Transformers degrade semantic information first, with early layers encoding more.
problem Understanding how sparse Transformers affect learned representations and semantic information.
method Probed Transformers with progressively pruned weights to observe changes in semantic information and model behavior.
result Complex semantic information is first to degrade in sparse Transformers, with early layers encoding more.
When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including …
New method finds compact strong lottery tickets in partially frozen networks.
problem Finding compact strong lottery tickets in randomly pruned networks.
method Freezing a random subset of initial weights in a partially frozen network.
result Better accuracy and compression achieved with frozen networks.