A novel quantum model improves RBM performance and is efficiently trainable.
problem Improving the performance of RBM models.
method Quantum model with parametrically coupled fermions to classical signals.
result The model outperforms classical RBM with the same number of hidden units.
ABUs learn and adapt activation functions for deep neural networks.
problem Lack of a unified theory connecting task and network properties with activation functions.
method Introduce Adaptive Blending Units (ABUs) as a trainable linear combination of activation functions.
result Advantages of ABUs over common activation functions across various network specifications.
Study on trainability of ReLU networks and proposes data-dependent initialization method.
problem Understanding and quantifying the trainability of ReLU networks.
method Introduced death states of neurons, studied probability distribution of active neurons at initialization, proposed data-dependent initialization method.
result Trainability is a necessary condition for successful training and over-parameterization is both necessary and sufficient for minimizing training loss.
Bayesian units improve speech recognition with minimal parameters.
problem Improving speech recognition models with fewer parameters.
method Derived Bayesian recurrent units integrated into deep learning frameworks.
result Adding Bayesian units improves speech recognition performance.
Structural RBM reduces parameters for image denoising and classification.
problem High parameter count in RBMs limits their applicability to large datasets.
method Introduces SRBM with constrained connections to reduce parameters.
result SRBM achieves better performance and faster training than vanilla RBM.
This research improves neural network performance with adaptive activation functions in sparse data settings.
problem Limited data availability in scientific and engineering problems.
method Investigation of two types of adaptive activation functions with individual trainable parameters.
result Adaptive activation functions, especially with individual trainable parameters, enhance prediction accuracy and confidence in sparse data settings.
A new RNN architecture GVFN improves RNN trainability by constraining states to future predictions.
problem Improving RNN trainability in complex environments.
method Developed a novel RNN architecture called General Value Function Network (GVFN) that constrains states to future predictions.
result GVFNs are more robust to truncation levels, requiring only one-step gradient updates.
New MBL hidden Born machine learns various tasks.
problem Learning from quantum many-body systems.
method MBL dynamics and hidden units for training.
result Enhanced trainability and stability in learning.
Efficient trainable front-end for neural speech enhancement.
problem Inefficient STFT front-ends in neural speech enhancement models.
method Butterfly mechanism for Fast Fourier Transform, trainable STFT window.
result Accuracy and efficiency improvements for low-compute systems.
Unified perspective on Hopfield networks with attention module.
problem Understanding and optimizing Hopfield networks with attention mechanisms.
method Study of BM counterparts of modern Hopfield networks and their salient properties.
result Introduction of AttnBM with tractable likelihood and gradient.
Survey of trainable activation functions in neural networks.
problem Improving neural network performance through trainable activation functions.
method Taxonomy and comparison of recent and past models of trainable activation functions.
result Many trainable activation functions are equivalent to adding neuron layers with fixed activation functions and simple constraints.
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and pe…
We propose semi-random features for nonlinear function approximation. The flexibility of semi-random feature lies between the fully adjustable units in deep learning and the random features used in kernel methods. For one hidden layer models with semi-random features, we prove with no unrealistic assumptions that the m…
This work explores the relation between trainability and dequantization in variational QML models.
problem Understanding the interplay between trainability and dequantization in variational QML models.
method Provide precise definitions of trainability and dequantization, study their relation, and introduce recipes for building PQC-based QML models.
result Identify conditions under which trainability and non-dequantization are not mutually exclusive.
New framework improves interpretability of trainable prompts.
problem Improving task-specific LLM performance with soft prompts remains a black-box method.
method Developed a theoretical framework for evaluating interpretability of trainable prompts, inspired new objective functions.
result Found a fundamental trade-off between interpretability and task performance in trainable prompts.
A trainable gate optimizes neural network selection and pruning.
problem Optimizing neural networks for specific tasks.
method Introduces a trainable gate function to make discrete selection problems differentiable.
result Efficiently optimizes arbitrary neural networks across various tasks.
New proof links initial class bias to DNN trainability, challenging traditional understanding.
problem Understanding the initial class bias in DNNs and its impact on trainability.
method Theoretical proof linking initial class bias to mean field theories of DNNs.
result Efficient learning is connected to a network's prejudice towards a specific class, contradicting traditional understanding.
ProdSumNet reduces deep neural network parameters via matrix decompositions.
problem Reducing model parameters in deep neural networks.
method Decomposing linear operators into products of sums of simpler operators.
result Achieved 98.44% accuracy with only 3554 trainable parameters.
Predicts trainability of deep neural networks using reconstruction entropy.
problem Predicting the initial conditions for trainability of deep neural networks.
method Cascade of auxiliary networks to reconstruct input from activation layers, computing relative entropy.
result Predicts trainability of deep feedforward networks on various datasets with a single epoch.
PatchGT uses non-trainable graph patches to improve graph representation learning.
problem Learning high-level information in graph tasks with direct Transformer models.
method PatchGT segments graphs into non-trainable patches, uses GNN for patch-level learning, and Transformer for graph-level learning.
result PatchGT achieves higher expressiveness and competitive performance on benchmark datasets.
CARRNN tackles deep learning for sporadic data, improving prediction errors in healthcare.
problem Challenges in learning temporal patterns from sporadic multivariate longitudinal data.
method Developed a novel deep learning architecture combining RNN and CAR models, using a generalized discrete-time autoregressive model.
result CARRNN achieves the lowest prediction errors in multivariate time-series regression tasks.
A big challenge in algorithmic composition is to devise a model that is both easily trainable and able to reproduce the long-range temporal dependencies typical of music. Here we investigate how artificial neural networks can be trained on a large corpus of melodies and turned into automated music composers able to gen…
Quantum machine learning faces challenges similar to variational quantum algorithms in training.
problem Challenges in training quantum machine learning models.
method Bridge between variational quantum algorithms and quantum machine learning, applying gradient scaling results.
result Gradient scaling results for variational quantum algorithms can also be applied to quantum machine learning models, revealing new trainability issues.
The study characterizes conditions for trainability and generalization in deep neural networks.
problem Understanding the conditions for deep neural networks to be trainable and generalize well.
method Analysis of Neural Tangent Kernel (NTK) for wide and deep networks.
result Large regions of hyperparameter space exist where networks can memorize training data but fail to generalize.
Quantum ML promises faster data analysis but faces trainability challenges.
problem Challenges in training quantum machine learning models.
method Review of current methods and applications of quantum neural networks and quantum deep learning.
result Opportunities for quantum advantage in quantum machine learning.
Modeling financial time series with LSTM and trainable initial states.
problem Extracting patterns and information from financial time series.
method Long Short-Term Memory (LSTM) network with trainable initial hidden states.
result Model captures relative similarity and predicts future stock trends.
New method infers nonlinear Granger causality from time series data.
problem Inferring nonlinear Granger causality from time series data.
method Statistical Recurrent Units (SRUs) for modeling nonlinear interactions.
result The proposed economy-SRU model outperforms existing models in inferring Granger causality.
New method uses trainable activations to make BNNs behave like GPs.
problem Making Bayesian Neural Networks (BNNs) behave like Gaussian Processes (GPs).
method Introduced trainable activations and periodic activations to map GP priors to BNNs. Used 2-Wasserstein distance for optimization.
result Method consistently outperforms existing approaches or matches heuristic methods.
Investigates ways to train larger models with fewer resources, finding that test loss depends only on the actual number of trainable parameters.
problem Training larger models for cheaper under hardware constraints.
method Emulates an increase in effective parameters using frozen random parameters or fast structured transforms.
result Scaling laws cannot be deceived by spurious parameters; test loss depends only on the actual number of trainable parameters.
Quantum models face barren plateaus, but specific losses can be trainable.
problem Barren plateaus and loss concentration in quantum generative models.
method Investigated explicit and implicit losses, and their interplay.
result Explicit losses lead to new barren plateaus, while implicit losses can be trainable.
Study finds differences in LTs across tasks and architectures, proposing a consensus-based method for generating refined lottery tickets.
problem Understanding the variability and uniqueness of Lottery Tickets across different image classification tasks and architectures.
method 28 combinations of image classification tasks and architectures, iterative pruning techniques, consensus-based method for generating refined lottery tickets.
result Disproves the uniqueness of Lottery Tickets and connects emergent mask structure to the choice of pruning.
Low-complexity spiking networks learn complex tasks with minimal trainable parameters.
problem Training complex reinforcement learning tasks with minimal resources.
method Reinforcement learning on simple networks of spiking neurons with random connections.
result Small random spiking networks achieve learning efficiency similar to humans on complex tasks.
QCNNs avoid barren plateaus, making them trainable.
problem Exponentially vanishing gradients in QNNs.
method Graph-based method to analyze Haar-distributed unitaries.
result QCNNs do not exhibit barren plateaus, implying trainability.
Improved stable RNNs trained faster with less expressibility trade-off.
problem Stable recurrent neural networks are hard to train without sacrificing expressibility.
method Implicit model structure with contraction analysis for stable models.
result Significant increase in training speed and model performance.
PePR scores assess DL model performance per resource unit, promoting smaller, more efficient models.
problem Limited access to large-scale resources hinders medical image analysis research.
method Introduced PePR score to measure DL model performance per resource unit.
result Small-scale, specialized models outperform large-scale models in resource-constrained settings.
Paper improves deep learning for solving evolutionary equations with trainable hard constraints.
problem Low computational accuracy of standard PINNs in large temporal domains.
method Sequential learning strategies and trainable influence functions for hard constraints.
result Significantly improved computational accuracy and universality of the method.
New approach connects quantum phases to VQA trainability, enabling better scaling.
problem Scalability issues in VQAs, especially barren plateaus.
method Analog VQA ansätze composed of quenches of a disordered Ising chain, tuning disorder strength.
result Thermalized and MBL phases reach maximal expressivity at large M, but barren plateaus emerge at smaller M in the thermalized phase. Neural Kernel Network (NKN) learns flexible kernels from neural networks.
problem Choosing the right kernel for Gaussian processes is challenging.
method NKN uses a neural network to represent kernels based on composition rules, making them differentiable and trainable.
result NKN can approximate complex kernel structures and is universal for stationary kernels.
Although there has been substantial research in software analytics for effort estimation in traditional software projects, little work has been done for estimation in agile projects, especially estimating user stories or issues. Story points are the most common unit of measure used for estimating the effort involved in…
New ReLU initialization improves network performance and dynamical isometry.
problem Improving the initialization of ReLU units for better network performance.
method Derive exact joint signal output distribution for fully-connected networks with Gaussian weights and biases, and propose a new initialization scheme for ReLU units.
result Proposed initialization scheme achieves dynamical isometry, improving network performance.
New method finds sparse networks without labels, improving performance.
problem Sparse connectivity in neural networks to reduce memory and energy demands.
method Neural Tangent Transfer method to find sparse networks without labels.
result Sparse networks achieve higher classification performance and faster convergence.
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
LGCL uses learnable filters to apply CNNs on graphs.
problem Applying CNNs to generic graphs like networks.
method Learnable graph convolutional layer (LGCL) and sub-graph training.
result LGCL achieves better performance on various graph datasets.
We introduce MinimalRNN, a new recurrent neural network architecture that achieves comparable performance as the popular gated RNNs with a simplified structure. It employs minimal updates within RNN, which not only leads to efficient learning and testing but more importantly better interpretability and trainability. We…
A quantum circuit designed for efficient statistical model preparation and training.
problem Challenges in preparing and learning statistical models on quantum processors.
method Utilizes the maximum entropy principle to design a statistics-informed parameterized quantum circuit (SI-PQC).
result Improves trainability and interpretability for learning quantum states and classical model parameters.
Efficient deep learning for CT images reduces memory and training time.
problem Training deep neural networks for CT images is computationally expensive.
method Unrolled proximal gradient descent, replaced penalty terms with CNNs, used greedy learning with deep UNet and surrogate.
result Achieved comparable image quality to state-of-the-art methods on CT image reconstruction challenges.
Gating units in GRUs and LSTMs create slow modes and control phase-space complexity.
problem Training challenges in RNNs due to exploding or vanishing gradients.
method Random matrix theory and mean-field theory applied to GRUs and LSTMs.
result Gates in GRUs and LSTMs lead to accumulation of slow modes and control phase-space complexity.
Study shows depth improves trainability of neural networks by improving kernel conditioning.
problem Improving trainability of neural networks with random initialization and overparameterization.
method Analyzes the role of depth in training neural networks, proving that depth improves conditioning of kernel matrices.
result General result showing depth improves trainability of neural networks by improving the conditioning of kernel matrices.