On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

5 results for “HT-SR”

Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcs.LG

TempBalance improves neural network training by balancing learning rates across layers.

problem Improving neural network training performance.

method Layer-wise learning rate scheduling based on HT-SR Theory.

result TempBalance significantly outperforms ordinary SGD and state-of-the-art optimizers.

neural networks learning rate temperature balancing HT-SR model training

2023-12-01abs ↗pdf ↗

Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricscs.LG

Study analyzes neural network models to understand generalization performance.

problem Understanding good generalization in neural networks.

method Analyzed a corpus of models from a public contest, breaking ALPHAHAT into scale and shape metrics.

result Identified a Simpson's paradox in metric performance across different model depths and regularization hyperparameters.

Simpson's paradox neural networks generalization ALPHAHAT Heavy-Tailed Self-Regularization

2021-06-01abs ↗pdf ↗

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscs.LG

AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.

problem Improving pruning of large language models to reduce size without sacrificing performance.

method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.

result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.

LLM pruning Heavy-Tailed Self-Regularization Layer-wise Pruning Large Language Models Perplexity

2024-10-14abs ↗pdf ↗

Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscs.LG

Given two or more Deep Neural Networks (DNNs) with the same or similar architectures, and trained on the same dataset, but trained with different solvers, parameters, hyper-parameters, regularization, etc., can we predict which DNN will have the best test accuracy, and can we do so without peeking at the test data? In …

2019-01-24abs ↗pdf ↗

Model Balancing Helps Low-data Training and Fine-tuningcs.LG

TempBalance boosts model performance with low data.

problem Low-data training and fine-tuning in model alignment.

method Inspired by HT-SR theory, TempBalance balances training quality across layers.

result TempBalance improves model performance as data decreases.

Low-data training Fine-tuning Model balancing Heavy-Tailed Self-Regularization Layer-wise learning rate scheduler

2024-10-16abs ↗pdf ↗