arXiv research

A new method improves few-shot image classification by updating top layers.

Drop-Muon updates only some layers, speeding up training.

We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…

Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.

We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with mom…

Study on rich regime training in deep learning, finding active parameters in bottom layers.

LAGS-SGD optimizes deep learning training by sparsifying gradients layer-wise.

Layer-wise training for deep linear networks achieves faster convergence with optimal learning rate.

Automatically learns flexible symmetry constraints in neural networks using gradients.

LR-EDNN reduces PDE solver complexity by limiting network weights to low-rank subspace.

We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each lay…

The rise of deep learning in recent years has brought with it increasingly clever optimization methods to deal with complex, non-linear loss functions. These methods are often designed with convex optimization in mind, but have been shown to work well in practice even for the highly non-convex optimization associated w…

Stochastic gradient descent (SGD) has achieved great success in training deep neural network, where the gradient is computed through back-propagation. However, the back-propagated values of different layers vary dramatically. This inconsistence of gradient magnitude across different layers renders optimization of deep …

Improves synthetic-to-real generalization without real data.

Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network …

Large batch training improves deep learning performance without needing warmup.

In the analysis of machine learning models, it is often convenient to assume that the parameters are IID. This assumption is not satisfied when the parameters are updated through training processes such as SGD. A relaxation of the IID condition is a probabilistic symmetry known as exchangeability. We show the sense in …

Layer-wise preconditioning methods improve neural network optimization and feature learning.

Backpropagation-free RL method trains layers using local signals.

TempBalance boosts model performance with low data.

Unified framework LPCD optimizes quantization of complex submodules.

Arguably the biggest challenge in applying neural networks is tuning the hyperparameters, in particular the learning rate. The sensitivity to the learning rate is due to the reliance on backpropagation to train the network. In this paper we present the first application of Implicit Stochastic Gradient Descent (ISGD) to…

Layer-wise networks have a closed-form solution and a stopping criterion.

Layer-wise networks have a closed-form solution and a stopping criterion.

Study reveals XAI methods fail in neuroimaging, suggesting domain-specific adaptation.

A new method for efficient neural network fine-tuning using queryable low-rank update atoms.

We show LLMs can be locally linear, enabling better control of activations.

Local adaptive methods in FL can accelerate convergence but introduce bias, which is corrected.

This research explores new optimization methods for training large neural networks.

In this paper, we tackle the problem of explanations in a deep-learning based model for recommendations by leveraging the technique of layer-wise relevance propagation. We use a Deep Convolutional Neural Network to extract relevant features from the input images before identifying similarity between the images in featu…

LEWIS merges LLMs without training, improving performance on specific tasks.

A fundamental question in deep learning concerns the role played by individual layers in a deep neural network (DNN) and the transferable properties of the data representations which they learn. To the extent that layers have clear roles, one should be able to optimize them separately using layer-wise loss functions. S…

Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.

Adaptive sparse GP model for non-stationary data.

Test-time training adapts a pretrained model to each prompt via parameter updates, improving accuracy under pretraining-to-test distribution shifts.

While neural networks have acted as a strong unifying force in the design of modern AI systems, the neural network architectures themselves remain highly heterogeneous due to the variety of tasks to be solved. In this chapter, we explore how to adapt the Layer-wise Relevance Propagation (LRP) technique used for explain…

MAME models a separate exploration policy for faster adaptation.

GAIT-prop derives a biologically plausible learning rule from backpropagation.

Various forms of representations may arise in the many layers embedded in deep neural networks (DNNs). Of these, where can we find the most compact representation? We propose to use a pruning framework to answer this question: How compact can each layer be compressed, without losing performance? Most of the existing DN…

Analyzes layer-wise quantization effects in neural networks.

Two algorithms achieve optimal regret with limited adaptivity in multinomial logistic bandits.

New algorithm optimizes MCMC sampling for structural dynamic models.

AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.

For large scale on-line inference problems the update strategy is critical for performance. We derive an adaptive scan Gibbs sampler that optimizes the update frequency by selecting an optimum mini-batch size. We demonstrate performance of our adaptive batch-size Gibbs sampler by comparing it against the collapsed Gibb…

Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…

AMUSE uses reinforcement learning to predict optimal model updates.

We consider 1-qubit mixed quantum state estimation by adaptively updating measurements according to previously obtained outcomes and measurement settings. Updates are determined by the average-variance-optimality (A-optimality) criterion, known in the classical theory of experimental design and applied here to quantum …

Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices. DNN pruning is an approach for deep model compression, which aims at eliminating some parameters with tolerable performance degradation. In this paper, w…