A new method improves few-shot image classification by updating top layers.
problem Few-shot image classification with limited data.
method Layer-wise adaptive updating (LWAU) for meta-learning.
result LWAU outperforms existing methods with a clear margin and learns more efficiently.
Drop-Muon updates only some layers, speeding up training.
problem Conventional deep learning optimizers update all layers at once, which can be inefficient.
method Drop-Muon updates only a subset of layers per step, with randomized schedules.
result Drop-Muon achieves up to 1.4x faster training time with similar accuracy.
We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…
Study on rich regime training in deep learning, finding active parameters in bottom layers.
problem Understanding the practical success of deep learning models.
method Empirical study on rich regime training with benchmark datasets, re-initialization analysis, and probabilistic Layer-Wise Sparse SGD.
result Probabilistic Layer-Wise Sparse SGD matches vanilla SGD's generalization performance with improved efficiency.
Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep neural networks is a challenging task. Many alternatives have been proposed in place of end-to-end back-propagation. Layer-wise training is one of them, which trains a single …
Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network …
Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.
problem Growth of quantization errors across layers degrades performance, especially in low-bit regimes.
method Quantization Error Propagation (QEP) framework that explicitly propagates and compensates for quantization errors.
result QEP-enhanced layer-wise PTQ achieves substantially higher accuracy, especially in low-bit regimes.
In the analysis of machine learning models, it is often convenient to assume that the parameters are IID. This assumption is not satisfied when the parameters are updated through training processes such as SGD. A relaxation of the IID condition is a probabilistic symmetry known as exchangeability. We show the sense in …
Layer-wise preconditioning methods improve neural network optimization and feature learning.
problem Suboptimal feature learning in standard optimization algorithms.
method Layer-wise preconditioning methods that introduce preconditioners per axis of each layer's weight tensors.
result Layer-wise preconditioning is necessary for provable feature learning in linear and single-index models.
Backpropagation-free RL method trains layers using local signals.
problem Vanishing or exploding gradients in backpropagation-based RL.
method Local pairwise distance matching for layer-wise training without backpropagation.
result Backpropagation-free method achieves competitive performance and stability.
Unified framework LPCD optimizes quantization of complex submodules.
problem Quantization of complex submodules in neural networks.
method Layer-Projected Coordinate Descent (LPCD) for quantizing arbitrary submodules.
result LPCD enhances both layer-wise PTQ methods and existing submodule approaches.
Arguably the biggest challenge in applying neural networks is tuning the hyperparameters, in particular the learning rate. The sensitivity to the learning rate is due to the reliance on backpropagation to train the network. In this paper we present the first application of Implicit Stochastic Gradient Descent (ISGD) to…
Layer-wise networks have a closed-form solution and a stopping criterion.
problem Training networks one layer at a time without backpropagation.
method Proved the Kernel Mean Embedding as the closed-form solution and developed a stopping criterion.
result Layer-wise networks converge to a highly desirable kernel for classification.
Layer-wise networks have a closed-form solution and a stopping criterion.
problem Training networks one layer at a time without backpropagation.
method Proved the closed-form solution using the kernel Mean Embedding and Neural Indicator Kernel.
result Layer-wise networks have a closed-form solution and a stopping criterion.
In this paper, we tackle the problem of explanations in a deep-learning based model for recommendations by leveraging the technique of layer-wise relevance propagation. We use a Deep Convolutional Neural Network to extract relevant features from the input images before identifying similarity between the images in featu…
LEWIS merges LLMs without training, improving performance on specific tasks.
problem Limited performance improvement of merged models on specific benchmarks.
method Guided model merging using layer-wise sparsity and task-vector pruning.
result Improved model performance by up to 11.3% on math-solving tasks.
A fundamental question in deep learning concerns the role played by individual layers in a deep neural network (DNN) and the transferable properties of the data representations which they learn. To the extent that layers have clear roles, one should be able to optimize them separately using layer-wise loss functions. S…
Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.
problem Reduction of model size and computational resources in neural networks.
method Conducted baseline experiments, developed RL-based search algorithm for finding transferable layer-wise pruning profiles.
result RL-based layer-wise pruning profiles are as good or better than best profiles found on the original dataset via exhaustive search.
GAIT-prop derives a biologically plausible learning rule from backpropagation.
problem Biological implausibility in traditional backpropagation for neural networks.
method GAIT-prop uses a top-down model to convert output error into plausible targets for weight updates.
result GAIT-prop and backpropagation give identical weight updates under certain conditions.
Analyzes layer-wise quantization effects in neural networks.
problem Identifying and fixing degradation in quantized neural networks.
method Layer-wise quantization analysis framework.
result Local fixes can significantly reduce quantization degradation.
The rise of deep learning in recent years has brought with it increasingly clever optimization methods to deal with complex, non-linear loss functions. These methods are often designed with convex optimization in mind, but have been shown to work well in practice even for the highly non-convex optimization associated w…
Automatically learns flexible symmetry constraints in neural networks using gradients.
problem Fixed hard constraints on neural network functions that cannot be adapted.
method Improves parameterisations of soft equivariance and optimizes marginal likelihood using differentiable Laplace approximations.
result Achieves equivalent or improved performance on image classification tasks compared to baselines with hard-coded symmetry.
Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…
Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices. DNN pruning is an approach for deep model compression, which aims at eliminating some parameters with tolerable performance degradation. In this paper, w…
Speeds up deep neural networks training by 10x using GPU concurrency.
problem Training deep residual neural networks efficiently.
method Layer-wise parallel training with GPU concurrency and Nonlinear Multigrid.
result 10.2x speedup over traditional techniques.
Hyperfitting improves LLM generation quality by enhancing diversity, contrary to simple temperature scaling.
problem Improving open-ended generation quality of LLMs with minimal fine-tuning effort.
method Demonstrates that hyperfitting, a phenomenon where LLMs are fine-tuned to near-zero training loss, enhances generation quality and mitigates repetition.
result Hyperfitting is distinct from temperature scaling and involves a dynamic, context-dependent rank reordering mechanism in the final transformer block.
Study on shortcuts in deep networks, revealing their layer-wise distribution and impact.
problem Understudied impact of shortcuts on feature representations in deep networks.
method Layer-wise localization through counterfactual training on clean and skewed datasets.
result Shortcuts are distributed throughout the network, not localized in specific layers.
LR-EDNN reduces PDE solver complexity by limiting network weights to low-rank subspace.
problem Efficiently solving time-dependent PDEs with deep neural networks.
method Low-rank constraint on network weights using SVD for efficient parameter updates.
result LR-EDNN achieves comparable accuracy to full EDNN with fewer parameters and lower cost.
Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well on unseen data even when the neural network has millions of parameters…
QuantEase optimizes LLMs with CD-based quantization, achieving state-of-the-art performance.
problem Efficiently quantize large language models for deployment.
method Layer-wise quantization using CD-based algorithms with matrix and vector operations.
result State-of-the-art performance in perplexity and zero-shot accuracy.
Training the deep convolutional neural network for computer vision problems is slow and inefficient, especially when it is large and distributed across multiple devices. The inefficiency is caused by the backpropagation algorithm's forward locking, backward locking, and update locking problems. Existing solutions for a…
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with mom…
Standard optimizers perform as well as LARS and LAMB at large batch sizes.
problem Comparing optimizers for neural network training at large batch sizes.
method Used standard optimizers like Nesterov momentum and Adam to match or exceed LARS and LAMB results.
result Standard optimizers can match or exceed LARS and LAMB at large batch sizes.
This work analyzes how different layers in deep neural networks contribute to generalization error.
problem Understanding the role of each layer in deep neural networks for generalization.
method Spectral analysis, Neural Tangent Kernel, Hermite polynomials, Spherical Harmonics.
result Initial layers in deep neural networks have a larger bias towards high-frequency functions.
Model compression has been widely adopted to obtain light-weighted deep neural networks. Most prevalent methods, however, require fine-tuning with sufficient training data to ensure accuracy, which could be challenged by privacy and security issues. As a compromise between privacy and performance, in this paper we inve…
This work improves OOD detection using deep generative models by approximating Fisher information metrics.
problem Deep generative models often incorrectly infer higher likelihoods for out-of-distribution data.
method Approximating Fisher information metrics using gradient norms of data points.
result The method outperforms existing OOD detection techniques.
InteractionNet models noncovalent protein-ligand interactions with GNNs and explains predictions.
problem Modeling noncovalent protein-ligand interactions with graph neural networks.
method InteractionNet uses a GNN architecture with separated covalent and noncovalent convolution layers and layer-wise relevance propagation for explainability.
result InteractionNet successfully predicts noncovalent protein-ligand interactions with chemical relevance.
When using deep, multi-layered architectures to build generative models of data, it is difficult to train all layers at once. We propose a layer-wise training procedure admitting a performance guarantee compared to the global optimum. It is based on an optimistic proxy of future performance, the best latent marginal. W…
Improved CNN training speed with K-FAC on large datasets.
problem Challenges in training large-scale neural networks with distributed processors.
method Scalable K-FAC design, layer-wise distribution, inverse-free second-order gradient evaluation, dynamic K-FAC update decoupling.
result Distributed K-FAC implementation converges to 75.9% MLPerf baseline in 18-25% less time than SGD.
A new method prevents forgetting during knowledge transfer.
problem Catastrophic forgetting in transfer learning.
method Transfer without Forgetting (TwF) using a fixed pretrained network.
result TwF outperforms other CL methods by 4.81% in Class-Incremental accuracy.
New learning rate approach reveals phase transitions in SGD performance.
problem Understanding feature learning dynamics in neural networks.
method Characterizing the relationship between learning rate(s) and sample complexity for gradient-based algorithms.
result Phase transition from information exponent to generative exponent regime with different learning rates.
Improved bounds on neural network regions using activation histograms.
problem Bounding the number of affine regions in ReLU networks.
method Analysis of algebraic topology problem, extension of framework to subnetwork composition.
result Slightly tighter bounds and insights into parameter initialization.
To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple workers is limited by the communication overhead. Two approaches, namely pipelining and gradient spar…
Efficiently trains GCNs with reduced time and memory usage.
problem Hard training of GCNs over large graph datasets.
method Layer-wise and learned efficient training framework (L2-GCN). result Significantly reduces training time and memory usage.
New adversarial attack method based on deep feature distributions.
problem Adversarial attacks on CNN classifiers using output layer information.
method Modeling and exploiting class-wise and layer-wise deep feature distributions.
result Achieves state-of-the-art transfer-based attack results for undefended ImageNet models.
XAI identifies key time steps for early crop classification.
problem Early crop classification with high accuracy and timeliness.
method Training a baseline model with LRP to identify important time steps.
result Identified a 21st April 2019 to 9th August 2019 timeframe with 0.75% accuracy loss.
Gluon optimizes LMO-based methods for large-scale tasks, improving performance and theory-practice gap.
problem LMO-based methods lack theoretical support for practical implementation and smoothness assumptions.
method Introduces Gluon, a new LMO-based method with refined smoothness model.
result Gluon's theoretical stepsizes match fine-tuned values, closing the theory-practice gap.
Physics-inspired methods optimize SVD compression of LLMs.
problem Efficiently compressing large language models (LLMs) using SVD.
method FermiGrad for globally optimal rank selection and PivGa for lossless compression.
result Global optimization of SVD ranks and lossless compression of low-rank factors.