Novel metric space magnitude and weighting vectors improve machine learning tasks.
problem Improving machine learning algorithms using novel metric space concepts.
method Metric space magnitude and weighting vectors for better machine learning.
result The weighting vector effectively detects boundaries and improves classic machine learning tasks.
This paper introduces new invariants for time series analysis.
problem Analyzing the diversity and invariants of time series data.
method Introduces new invariants derived from the continuity of magnitude and maximum diversity.
result Demonstrates improved performance in machine learning experiments with real-world data.
New insights into MAE show it treats examples unequally and IMAE improves this.
problem Noise-robust learning in deep learning models.
method Analysis of MAE's noise-robustness and proposing IMAE to improve it.
result IMAE improves MAE's fitting ability while preserving its noise-robustness.
Defines magnitude for length spaces with measures, agreeing with finite spaces' magnitude.
problem Defining magnitude for non-finite metric spaces with measures.
method Integrals over geodesics, using counting and weight measures.
result Magnitude agrees with finite spaces' magnitude and volume under specific conditions.
This work explores the importance of model weights and Hessian bias in pruning.
problem Understanding the relative importance of model weights for efficient pruning.
method A principled exploration of pruning, focusing on linear models and neural networks.
result Asymptotic formulas reveal the performance of different pruning methods.
A new energy-efficient pruning method for federated learning.
problem Energy inefficiency in gradient sparsification for federated learning.
method Formalized energy-constrained projection problem and proposed Cost-Weighted Magnitude Pruning (CWMP).
result CWMP optimally balances performance and energy efficiency in federated learning.
Study examines effects of pruning techniques on deep learning models.
problem Understanding the impact of pruning methods on deep learning model structure and dynamics.
method Investigated differences in connectivity and learning dynamics of pruned models using various iterative pruning techniques.
result Emergence of structure in pruned models through magnitude-based unstructured pruning and weight rewinding.
RSO uses random weight perturbations to train deep networks without gradients.
problem Training deep neural networks efficiently and without gradient information.
method RSO is a gradient-free Markov Chain Monte Carlo approach that updates weights based on mini-batch loss reduction.
result RSO achieves high accuracy (99.1% on MNIST) with significantly fewer updates than traditional methods.
NS-GAN mode collapse due to sample weighting inversion, solved with MM-nsat.
problem Mode collapse in GANs due to sample weighting inversion.
method Preserves MM-GAN sample weighting while avoiding saturation by rescaling gradients.
result MM-nsat improves mode coverage, stability, and FID on MNIST and CIFAR-10.
SLERP interpolation optimizes dynamic weight rebalancing in AMMs.
problem Optimizing dynamic weight rebalancing in automated market makers (AMMs).
method Riemannian geometry and SLERP interpolation.
result SLERP interpolation minimizes the KL divergence loss in dynamic weight rebalancing.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
SWAT reduces training efficiency and speed for CNNs.
problem Efficient training of neural networks on sparse hardware.
method SWAT modifies backpropagation to eliminate small weights and activations.
result SWAT reduces FLOPS by 80% on ResNet-50 for ImageNet, improving training speed by 3.3x.
A new pruning criterion reduces model size and improves performance.
problem Overparameterized neural networks are computationally and memory intensive, leading to overfitting.
method Introduces a magnitude and uncertainty (M&U) pruning criterion inspired by statistical Wald test.
result Our M&U pruning criterion leads to more compressed models with less loss in predictive power.
Sparse learning speeds up neural network training without sacrificing accuracy.
problem Training deep neural networks efficiently while maintaining performance.
method Sparse momentum algorithm that redistributes and grows weights based on momentum magnitude.
result State-of-the-art sparse performance on various datasets with up to 5.61x faster training.
New pruning method retains model expressiveness for NLP tasks.
problem Pruning large pretrained transformer models for real-world deployment.
method Mixture Gaussian Prior Pruning (MGPP) algorithm.
result MGPP outperforms existing pruning methods in high sparsity settings.
Improved sample complexity bounds for neural networks with weight magnitude constraints.
problem Bounding the sample complexity of neural networks with weight magnitude constraints.
method Developed a new technique to analyze sample complexity of function classes, using approximate description length.
result Optimal sample complexity bound of $O\left(\frac{dR^2}{ε^2}
ight)$ for neural networks with weight magnitude constraints.
Improved diffusion models for image synthesis with better training dynamics.
problem Uneven and ineffective training in diffusion models.
method Redesigned network layers to preserve activation, weight, and update magnitudes.
result Significantly better networks at equal computational complexity, improving FID to 1.81.
The thesis explores IMP, a process that identifies winning tickets in DNNs, and its universality.
problem Understanding how winning subnetworks (tickets) perform across different problems.
method Iterative Magnitude Pruning (IMP) and comparison with Renormalisation Group (RG) theory.
result IMP identifies winning subnetworks that can perform similarly across various problems.
Lipschitz regularization improves neural network robustness by coupling weights across layers.
problem Improving neural network robustness under random input uncertainties.
method Regularization of neural networks by their Lipschitz constant, highlighting the coupling effect on weights across layers.
result Lipschitz regularization introduces a tradeoff between robustness and expressiveness, suggesting careful implementation.
Significant success has been reported recently using deep neural networks for classification. Such large networks can be computationally intensive, even after training is over. Implementing these trained networks in hardware chips with a limited precision of synaptic weights may improve their speed and energy efficienc…
The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an esti…
DM improves deep model robustness for noisy, imbalanced datasets.
problem Noisy labels and imbalanced datasets in real-world large-scale datasets.
method Derivative Manipulation (DM) approach to example weighting.
result DM enhances robustness of deep models under adverse conditions.
New neural networks without weight transport are more robust to adversarial attacks.
problem Existing neural networks are easily fooled by adversarial attacks.
method Trained neural networks using feedback alignment instead of weight transport.
result Neural networks without weight transport are significantly harder to fool (98% adversarial accuracy vs 0.03%).
Guarantees sparse recovery for neural networks with iterative hard thresholding.
problem Recovering sparse network weights in neural networks.
method Structural properties of sparse network weights and iterative hard thresholding algorithm.
result Simple iterative hard thresholding algorithm recovers sparse network weights exactly using linear memory.
Sharp analysis of out-of-distribution error in overparameterized models with importance weights.
problem Understanding and quantifying the degradation of performance in overparameterized models when faced with underrepresented data.
method Sharp analysis of an overparameterized Gaussian mixture model with spurious features and cost-sensitive interpolating solutions incorporating importance weights.
result Characterization of a novel tradeoff between worst-case robustness and average accuracy as a function of importance weight magnitude.
Softmax is found ineffective for NL block, leading to improved performance.
problem Inefficiency of softmax in NL block for global context modeling.
method Empirical analysis and replacement of softmax with scaling factor.
result Improved performance on various datasets with reduced computational cost.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
In this paper, we propose a perturbation framework to measure the robustness of graph properties. Although there are already perturbation methods proposed to tackle this problem, they are limited by the fact that the strength of the perturbation cannot be well controlled. We firstly provide a perturbation framework on …
S-SGD adds symmetrical noise to weights to avoid sharp minima in deep learning.
problem SGD does not always converge to a flat minimum, leading to poor generalization.
method Symmetrical weight noise injection in SGD.
result S-SGD outperforms conventional SGD and weight-noise injection methods in large batch training.
New pruning methods improve dynamic sparse training performance.
problem Improving dynamic sparse training performance.
method Design and empirical analysis of pruning criteria.
result Most pruning methods yield similar results, but magnitude-based pruning performs best in low-density regimes.
Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatica…
New methods train neural networks without changing weights, achieving similar or higher performance.
problem Training neural networks efficiently with randomly initialized weights.
method Switching connections on and off, flipping weights' signs, minimizing changed connections.
result Achieves similar or higher performance with less computational cost than training all weights.
Study shows refugee matching gains are robust to different evaluation methods.
problem Stability of refugee matching gains under various evaluation methods.
method Used multiple off-policy evaluation methods including IPW and AIPW.
result Impact estimates remain consistent in magnitude and statistically significant.
We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model. This enables memory-limited execution of several statistical analyses over streams, including online featu…
IMP finds sparse subnetworks that match full networks, revealing geometric insights.
problem Finding sparse subnetworks that match full, overparameterized networks.
method Iterative magnitude pruning (IMP) algorithm, analyzing error landscape geometry.
result IMP masks found at end of training convey useful information for rewound networks.
Constraints improve deep neural network training by stabilizing and enhancing robustness.
problem Vanishing/exploding gradients and poor weight magnitudes in deep neural networks.
method Weight-constrained stochastic dynamics using Langevin dynamics framework.
result Enhanced exploration of the loss landscape and improved generalization.
Random feature maps improve forecasting with cheaper computation.
problem Improving forecasting accuracy with random feature maps.
method Developed a hit-and-run algorithm to select optimal internal weights.
result Optimal internal weights lead to superior forecasting skill.
MTAdam optimizes multiple loss terms in neural models, balancing gradients dynamically.
problem Balancing multiple loss terms in neural model training is challenging and computationally demanding.
method Generalized Adam algorithm that computes separate derivatives and balances gradients across layers dynamically.
result Training with MTAdam leads to faster recovery from suboptimal initial loss weighting and matches conventional training outcomes.
Improved DeepONets for PDE solution operators with adaptive re-weighting and new architecture.
problem Training DeepONets for PDE solution operators without paired data.
method Adaptive re-weighting of training examples and novel network architecture.
result Consistently improved predictive accuracy by a factor of 10-50x.
Ongoing studies have identified similarities between neural representations in biological networks and in deep artificial neural networks. This has led to renewed interest in developing analogies between the backpropagation learning algorithm used to train artificial networks and the synaptic plasticity rules operative…
Proposes methods to add constraints to neural networks to improve stability and generalization.
problem Improving stability and generalization of neural networks.
method Constraint-based regularization using stochastic gradient Langevin dynamics.
result Constraints help stabilize and improve the robustness of deep neural networks.
Machine learning approximates Calabi-Yau Hodge numbers from weight systems.
problem Approximating Hodge numbers of Calabi-Yau manifolds from weight systems.
method Neural networks learned Hodge numbers from weight systems, symbolic regression inspired truncation, and machine learning generated new datasets.
result Approximation provides tight lower bounds and dramatically faster computation.
New method uses weighting vectors for efficient boundary and outlier detection.
problem Boundary and outlier detection in machine learning.
method Recast metric space magnitude as weighting vector, solve kernelized SVM, apply nearest neighbor methods.
result Weighting vector can be efficiently approximated in linear time, outperforming state-of-the-art techniques.
DRE combines DNN with random feature regression for efficient neural network design.
problem Designing and training deep neural networks (DNN) efficiently and effectively.
method DRE architecture with two-layer neural networks, randomly drawn input and output weights trained with linear ridge regression.
result DRE outperforms state-of-the-art DNN in many data sets with lower computational cost.
The rise of deep learning in recent years has brought with it increasingly clever optimization methods to deal with complex, non-linear loss functions. These methods are often designed with convex optimization in mind, but have been shown to work well in practice even for the highly non-convex optimization associated w…
Magnitude homology reveals that graphs can have torsion subgroups.
problem Understanding torsion in magnitude homology of graphs.
method Analysis of magnitude homology defined by Hepworth and Willerton.
result Torsion of any prime order can appear in graphs' magnitude homology.
New training method for ReLU networks achieves optimal weight size for memorization.
problem Approximate memorization of arbitrary real labels with neural networks.
method Complex recombination training procedure for ReLU networks.
result Approximate memorization with nearly optimal weight size and neuron count.
Generalization bounds derived for neural ODEs and deep residual networks.
problem Understanding the generalization capability of neural ODEs and deep residual networks.
method Lipschitz-based argument and analogy with deep residual networks.
result A generalization bound involving the magnitude of weight matrix differences.