We consider a general framework for reducing the number of trainable model parameters in deep learning networks by decomposing linear operators as a product of sums of simpler linear operators. Recently proposed deep learning architectures such as CNN, KFC, Dilated CNN, etc. are all subsumed in this framework and we il…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “trainable parameters”
Investigates ways to train larger models with fewer resources, finding that test loss depends only on the actual number of trainable parameters.
problem Training larger models for cheaper under hardware constraints.
method Emulates an increase in effective parameters using frozen random parameters or fast structured transforms.
result Scaling laws cannot be deceived by spurious parameters; test loss depends only on the actual number of trainable parameters.
We propose reinforcement learning on simple networks consisting of random connections of spiking neurons (both recurrent and feed-forward) that can learn complex tasks with very little trainable parameters. Such sparse and randomly interconnected recurrent spiking networks exhibit highly non-linear dynamics that transf…
New proof links initial class bias to DNN trainability, challenging traditional understanding.
problem Understanding the initial class bias in DNNs and its impact on trainability.
method Theoretical proof linking initial class bias to mean field theories of DNNs.
result Efficient learning is connected to a network's prejudice towards a specific class, contradicting traditional understanding.
Study finds differences in LTs across tasks and architectures, proposing a consensus-based method for generating refined lottery tickets.
problem Understanding the variability and uniqueness of Lottery Tickets across different image classification tasks and architectures.
method 28 combinations of image classification tasks and architectures, iterative pruning techniques, consensus-based method for generating refined lottery tickets.
result Disproves the uniqueness of Lottery Tickets and connects emergent mask structure to the choice of pruning.
In this work, we leverage advances in sparse coding techniques to reduce the number of trainable parameters in a fully connected neural network. While most of the works in literature impose regularization, DropOut or DropConnect techniques to induce sparsity, our scheme considers feature importance as a criter…
Architecture optimization, which is a technique for finding an efficient neural network that meets certain requirements, generally reduces to a set of multiple-choice selection problems among alternative sub-structures or parameters. The discrete nature of the selection problem, however, makes this optimization difficu…
This research improves neural network performance with adaptive activation functions in sparse data settings.
problem Limited data availability in scientific and engineering problems.
method Investigation of two types of adaptive activation functions with individual trainable parameters.
result Adaptive activation functions, especially with individual trainable parameters, enhance prediction accuracy and confidence in sparse data settings.
Predicts trainability of deep neural networks using reconstruction entropy.
problem Predicting the initial conditions for trainability of deep neural networks.
method Cascade of auxiliary networks to reconstruct input from activation layers, computing relative entropy.
result Predicts trainability of deep feedforward networks on various datasets with a single epoch.
Hard constraint learning approaches with trainable influence functions for evolutionary equationscs.LG
Paper improves deep learning for solving evolutionary equations with trainable hard constraints.
problem Low computational accuracy of standard PINNs in large temporal domains.
method Sequential learning strategies and trainable influence functions for hard constraints.
result Significantly improved computational accuracy and universality of the method.
Modeling financial time series with LSTM and trainable initial states.
problem Extracting patterns and information from financial time series.
method Long Short-Term Memory (LSTM) network with trainable initial hidden states.
result Model captures relative similarity and predicts future stock trends.
A quantum circuit designed for efficient statistical model preparation and training.
problem Challenges in preparing and learning statistical models on quantum processors.
method Utilizes the maximum entropy principle to design a statistics-informed parameterized quantum circuit (SI-PQC).
result Improves trainability and interpretability for learning quantum states and classical model parameters.
PatchGT uses non-trainable graph patches to improve graph representation learning.
problem Learning high-level information in graph tasks with direct Transformer models.
method PatchGT segments graphs into non-trainable patches, uses GNN for patch-level learning, and Transformer for graph-level learning.
result PatchGT achieves higher expressiveness and competitive performance on benchmark datasets.
CLOPS improves deep learning for continuous physiological data.
problem Deep learning struggles with streaming, multi-sensor clinical data.
method CLOPS is a replay-based continual learning strategy.
result CLOPS outperforms state-of-the-art methods in continual learning scenarios.
Quantum machine learning faces challenges similar to variational quantum algorithms in training.
problem Challenges in training quantum machine learning models.
method Bridge between variational quantum algorithms and quantum machine learning, applying gradient scaling results.
result Gradient scaling results for variational quantum algorithms can also be applied to quantum machine learning models, revealing new trainability issues.
QCNNs avoid barren plateaus, making them trainable.
problem Exponentially vanishing gradients in QNNs.
method Graph-based method to analyze Haar-distributed unitaries.
result QCNNs do not exhibit barren plateaus, implying trainability.
In this paper, we study the trainability of rectified linear unit (ReLU) networks. A ReLU neuron is said to be dead if it only outputs a constant for any input. Two death states of neurons are introduced; tentative and permanent death. A network is then said to be trainable if the number of permanently dead neurons is …
AGBoost uses attention weights to improve GBM for regression problems.
problem Improving gradient boosting machine for regression tasks.
method Attention-based modification of GBM with trainable attention weights.
result AGBoost achieves better performance on regression datasets.
This paper compares hypernetworks and embedding methods for function approximation.
problem Learning to map inputs to functions efficiently for each instance.
method Compared embedding-based and hypernetwork methods using expressivity and complexity.
result Hypernetworks can be more efficient in terms of trainable parameters for certain functions.
Survey of trainable activation functions in neural networks.
problem Improving neural network performance through trainable activation functions.
method Taxonomy and comparison of recent and past models of trainable activation functions.
result Many trainable activation functions are equivalent to adding neuron layers with fixed activation functions and simple constraints.
Despite their successes in the field of self-learning AI, Convolutional Neural Networks (CNNs) suffer from having too many trainable parameters, impacting computational performance. Several approaches have been proposed to reduce the number of parameters in the visual domain, the Inception architecture [Szegedy et al.,…
VQC-MLPNet combines quantum and classical elements for scalable quantum machine learning.
problem Challenges in expressivity, trainability, and noise resilience of VQCs.
method Hybrid architecture with a VQC generating weights for a classical MLP during training.
result Improved expressivity, trainability, and robustness compared to standalone quantum or hybrid approaches.
Many neural speech enhancement and source separation systems operate in the time-frequency domain. Such models often benefit from making their Short-Time Fourier Transform (STFT) front-ends trainable. In current literature, these are implemented as large Discrete Fourier Transform matrices; which are prohibitively inef…
ABRF uses attention weights to improve RF performance.
problem Improving the performance of random forest models.
method Attention mechanism applied to random forest with optimization and gradient-based methods.
result The proposed ABRF models outperform standard RF models on various datasets.
On the relation between trainability and dequantization of variational quantum learning modelsquant-ph
This work explores the relation between trainability and dequantization in variational QML models.
problem Understanding the interplay between trainability and dequantization in variational QML models.
method Provide precise definitions of trainability and dequantization, study their relation, and introduce recipes for building PQC-based QML models.
result Identify conditions under which trainability and non-dequantization are not mutually exclusive.
New framework improves interpretability of trainable prompts.
problem Improving task-specific LLM performance with soft prompts remains a black-box method.
method Developed a theoretical framework for evaluating interpretability of trainable prompts, inspired new objective functions.
result Found a fundamental trade-off between interpretability and task performance in trainable prompts.
Improving sparsity of deep neural networks (DNNs) is essential for network compression and has drawn much attention. In this work, we disclose a harmful sparsifying process called filter collapse, which is common in DNNs with batch normalization (BN) and rectified linear activation functions (e.g. ReLU, Leaky ReLU). It…
New algorithms accelerate SVGD convergence using deep unfolding.
problem Improving the speed of SVGD convergence.
method Integrating deep unfolding into SVGD for parameter learning.
result Proposed algorithms achieve faster convergence in various tasks.
EVA adapts LoRA for faster, more efficient fine-tuning.
problem Fast and efficient fine-tuning of large models for specific tasks.
method EVA uses directions capturing most activation variance for initialization, maximizing gradient signal and reducing parameters.
result EVA achieves faster convergence and higher average scores across tasks, reducing parameters.
We introduce causal pieces to improve spiking neural networks.
problem Improving the expressiveness and trainability of spiking neural networks.
method We decompose the input domain of SNNs into causal regions, proving that the number of these regions is a measure of SNNs' approximation capabilities.
result Parameter initialisations yielding a high number of causal pieces correlate with SNN training success.
Proposes a new method to prevent overfitting in deep neural networks.
problem Overfitting in deep neural networks with many trainable parameters.
method Randomly replaces elements in feature maps with specific values during training.
result Improves the testing performance of deep neural networks on benchmark datasets.
Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we int…
A new text classification method using separable convolution reduces memory consumption.
problem Manual categorization of documents is inefficient and resource-intensive.
method Introducing a new architecture based on separable convolution for text classification.
result Achieved a drastic reduction in trainable parameters without compromising accuracy.
TopoFisher learns topological summaries by maximizing Fisher information, improving parameter efficiency and inference quality.
problem Simulation-based inference misses key information in low-order statistics, especially for non-Gaussian fields.
method TopoFisher uses a differentiable persistent-homology pipeline that learns topological summaries by maximizing local Gaussian Fisher information.
result TopoFisher recovers much of the available information and outperforms fixed topological vectorizations in weak gravitational lensing.
We propose a novel quantum model for the restricted Boltzmann machine (RBM), in which the visible units remain classical whereas the hidden units are quantized as noninteracting fermions. The free motion of the fermions is parametrically coupled to the classical signal of the visible units. This model possesses a quant…
A new VAE approach solves inverse problems without explicit inverse mapping.
problem Solving inverse problems without explicit inverse mapping.
method Discarding the encoder in VAE architecture, directly optimizing latent variables.
result The latent variables can exhibit mutually independent properties without an encoding process.
Composite Bayesian Optimization In Function Spaces Using NEON -- Neural Epistemic Operator Networkscs.LG
NEON uses neural networks to optimize functions in infinite-dimensional spaces.
problem Optimizing composite functions in function spaces.
method NEON (Neural Epistemic Operator Networks) for sequential decision-making.
result NEON achieves state-of-the-art performance with fewer parameters.
Structural RBM reduces parameters for image denoising and classification.
problem High parameter count in RBMs limits their applicability to large datasets.
method Introduces SRBM with constrained connections to reduce parameters.
result SRBM achieves better performance and faster training than vanilla RBM.
Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizesstat.ML
In this paper, we theoretically prove that gradient descent can find a global minimum of non-convex optimization of all layers for nonlinear deep neural networks of sizes commonly encountered in practice. The theory developed in this paper only requires the practical degrees of over-parameterization unlike previous the…
Paper presents a new VMBQC model with fewer parameters for better generative modeling.
problem Limited generative power of VMBQC due to more parameters than unitary models.
method Introduces a restricted VMBQC model with a single additional trainable parameter.
result Minimal extension of VMBQC model generates distributions not learnable by unitary models.
Study evaluates capacity and trainability of parametrized quantum circuits.
problem Finding the best type of circuits for hybrid quantum-classical algorithms.
method Geometric structure of parameter space, effective quantum dimension, and circuit expressiveness.
result Identifies a transition in quantum geometry leading to decay of quantum natural gradient for deep circuits.
Connections between nodes of fully connected neural networks are usually represented by weight matrices. In this article, functional transfer matrices are introduced as alternatives to the weight matrices: Instead of using real weights, a functional transfer matrix uses real functions with trainable parameters to repre…
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscs.LG
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
Tensor neural network improves human pose classification from 3D skeleton data.
problem Efficiently processing spatiotemporal data for human pose classification.
method Proposes a tensor-based neural network with three components: spatiotemporal feature construction, tensor fusion, and tensor-based neural network processing.
result Achieves state-of-the-art performance in human pose classification.
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extens…
Graph processes model a number of important problems such as identifying the epicenter of an earthquake or predicting weather. In this paper, we propose a Graph Convolutional Recurrent Neural Network (GCRNN) architecture specifically tailored to deal with these problems. GCRNNs use convolutional filter banks to keep th…
Quantum ML promises faster data analysis but faces trainability challenges.
problem Challenges in training quantum machine learning models.
method Review of current methods and applications of quantum neural networks and quantum deep learning.
result Opportunities for quantum advantage in quantum machine learning.
NeuroMAS treats multi-agent systems as neural networks for scalable, trainable coordination.
problem Designing multi-agent systems as hand-designed workflows is inefficient and inflexible.
method NeuroMAS treats multi-agent systems as a neural network architecture with reinforcement learning for scalable coordination.
result NeuroMAS improves significantly over multi-agent baselines and can be scaled progressively.