Sparse Meta Networks adapt deep neural networks incrementally for fast learning.
problem Training deep neural networks is slow and impractical for complex, changing environments.
method Sparse Meta Networks use a memory layer to learn online sequential adaptation, accumulating fast-weights incrementally.
result Sparse Meta Networks achieve strong performance in various sequential adaptation scenarios.
Unintended effects from scaling neural network outputs with adaptive learning rates.
problem Adaptive learning rate optimization's behavior is altered by output scaling, leading to misinterpretation.
method Presented a modified optimization algorithm to mitigate unintended effects.
result Adaptive learning rate's effectiveness is significantly impacted by output scaling, especially for small scaling factors.
Consider the problem: given the data pair (x,y) drawn from a population with f∗(x)=E[y∣x=x], specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does ft, the function computed by the neural network…
Linearized neural networks provide a fast and interpretable way to adapt models to new settings.
problem Difficulty in understanding and adapting inductive biases of trained neural networks.
method Linearization of neural networks and embedding these biases into Gaussian processes through a kernel designed from the Jacobian.
result Domain adaptation becomes interpretable posterior inference with analytic and scalable computational speed-ups.
Adaptive weights improve physics-informed neural networks and deep operator networks.
problem Training physics-informed neural networks and deep operator networks can be challenging, leading to unsatisfactory accuracy and efficiency.
method Proposes a pointwise adaptive weighting method that balances the residual decay rate across different training points.
result Our proposed approach of balanced residual decay rates offers advantages including bounded weights, high prediction accuracy, fast convergence rate, low training uncertainty, low computational cost, and ease of hyperparameter tuning.
Adaptive Group Lasso selects important features in neural networks.
problem Lack of interpretability in neural networks.
method Adaptive Group Lasso for feature selection.
result Consistent feature selection for neural networks with theoretical guarantee.
This paper examines the convergence of adaptive sampling methods for Bayesian neural networks.
problem Uncertainty quantification in deep neural networks, especially for medical applications.
method Locally adaptive and scalable diffusion-based sampling methods.
result These methods can have a substantial bias in the distribution they sample, even in the limit of vanishing step sizes.
Enhances physics-informed neural networks with adaptive sampling and weighting.
problem Challenges in training physics-informed neural networks on complex problems.
method Hybrid adaptive sampling and weighting method.
result Consistently improves prediction accuracy and training efficiency.
This paper improves neural network generalization by dynamically learning kernel parameters.
problem Improving neural network generalization and adaptability.
method Diagonal adaptive kernel model that learns kernel eigenvalues and output coefficients during training.
result The diagonal adaptive kernel model significantly improves generalization over fixed-kernel methods.
learn2mix trains neural nets faster by adjusting class proportions dynamically.
problem Training neural nets efficiently with limited resources and imbalanced classes.
method Adaptive class proportion adjustment during training.
result Neural nets trained with learn2mix converge faster than static methods.
This paper explores adaptive neural activation in RNNs for better learning.
problem Fixed neural activation functions limit the performance and adaptability of RNNs.
method Developed a novel parametric family of nonlinear activation functions inspired by biological neurons.
result Adaptive neural activation improves learning speed and performance in RNNs.
Adaptive gradient methods like AdaGrad are widely used in optimizing neural networks. Yet, existing convergence guarantees for adaptive gradient methods require either convexity or smoothness, and, in the smooth setting, only guarantee convergence to a stationary point. We propose an adaptive gradient method and show t…
Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the gene…
UDN adapts depth to data complexity, outperforming standard neural networks.
problem Adapting neural network depth to data complexity.
method Variational inference for infinitely deep neural networks with a novel algorithm.
result UDN outperforms standard neural networks and other infinite-depth approaches.
Deep neural network with l_1-regularization achieves nearly optimal risk bounds.
problem Achieving optimal risk bounds in deep learning.
method Empirical risk minimization with l_1-regularization.
result Adaptively nearly-minimax risk bound across various function classes.
New algorithm reduces misclassification costs in neural networks.
problem Reduces costs of misclassified instances in neural networks.
method Adaptive Cost-Sensitive Learning (AdaCSL) adjusts loss function to bridge class distribution mismatches.
result Deep neural networks with AdaCSL outperform other methods on cost-sensitive binary classification tasks.
New algorithm improves Bayesian neural networks using adaptive importance sampling.
problem High computational cost in training Bayesian neural networks.
method Adaptive Importance Sampling (AIS) integrated into a novel algorithm (PMCnet).
result Improved performance and exploration capabilities for both shallow and deep neural networks.
Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt…
Nash integrates covariate-specific side info into sparse regression via neural networks.
problem Sparse linear regression struggles with covariates exhibiting structure or coming from heterogeneous sources.
method Neural Adaptive Shrinkage (Nash) framework that integrates side information into sparse regression via neural networks. Uses split variational empirical Bayes algorithm.
result Nash improves accuracy and adaptability over existing methods in real data experiments.
New method proves neural networks can select features consistently.
problem Feature selection for deep neural networks is challenging.
method Adaptive Group Lasso selection procedure with Group Lasso as the base estimator.
result Adaptive Group Lasso is selection-consistent for a wide class of neural networks.
Self-adaptive PINNs improve accuracy in stiff PDEs.
problem Accurate numerical solution of stiff PDEs using neural networks.
method Adaptive weights applied to each training point individually, using a soft attention mechanism.
result SA-PINNs outperform other PINN algorithms in L2 error with fewer training epochs.
Probabilistic vehicle trajectory prediction is essential for robust safety of autonomous driving. Current methods for long-term trajectory prediction cannot guarantee the physical feasibility of predicted distribution. Moreover, their models cannot adapt to the driving policy of the predicted target human driver. In th…
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
New method adapts neural networks without losing prior knowledge.
problem Understanding and enabling flexible adaptation of neural networks.
method Differential geometry framework, functionally invariant paths (FIP).
result Achieves comparable state-of-the-art performance on continual learning and sparsification tasks.
Adaptive stepsizing improves sampling in Bayesian neural networks.
problem Scalable sampling of posterior distributions in Bayesian neural networks.
method SA-SGLD, employing time rescaling to adapt stepsize dynamically.
result SA-SGLD achieves more accurate posterior sampling than SGLD.
EDAIN layer normalizes time series data for neural networks, improving model performance.
problem Irregularities in time series data degrade model performance in neural networks.
method EDAIN layer learns adaptive normalization parameters during end-to-end training.
result EDAIN layer outperforms conventional normalization methods and adaptive layers.
Deep neural networks (DNNs) have been proven to have many redundancies. Hence, many efforts have been made to compress DNNs. However, the existing model compression methods treat all the input samples equally while ignoring the fact that the difficulties of various input samples being correctly classified are different…
Adaptive kernels from neural networks improve model performance.
problem Improving neural network performance through adaptive kernels.
method Deriving adaptive kernels from infinite-width neural networks using feature learning and gradient flow training.
result Adaptive kernels achieve lower test loss compared to traditional kernels.
Deep networks adapt to function regularity and data distribution.
problem Understanding deep learning's adaptability to function regularity and data distribution.
method Developed nonparametric approximation and estimation theories for a broad class of functions using deep ReLU networks.
result Deep neural networks are adaptive to different regularity of functions and nonuniform data distributions.
We consider unsupervised domain adaptation: given labelled examples from a source domain and unlabelled examples from a related target domain, the goal is to infer the labels of target examples. Under the assumption that features from pre-trained deep neural networks are transferable across related domains, domain adap…
Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…
SpeqNets improve graph neural networks by scaling and adapting to graph sparsity.
problem Graph neural networks struggle with permutation-equivariant functions and scalability to large graphs.
method Introducing sparsity-aware, permutation-equivariant graph networks with heuristics for graph isomorphism.
result Significantly improved predictive performance and reduced computation times compared to existing methods.
Supporting evidence for adaptive feature program across diverse models.
problem Analyzing feature learning in neural networks.
method Over-parameterized sequence models and feature error measure (FEM).
result FEM is decreasing during training of adaptive feature models.
HALO learns to prune neural networks by adaptively shrinking weights.
problem Sparsity and model size in deep neural networks.
method Bayesian hierarchical models and trainable parameters for adaptive sparsification.
result HALO learns to create highly sparse networks with significant performance gains.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
Adapting robust statistics to neural networks, researchers found neural networks can be more robust with certain loss functions.
problem The robustness of neural networks in complex learning tasks.
method Adapting the regression breakdown point from robust statistics to neural networks and comparing different configurations and contamination settings.
result Neural networks can benefit from robust loss functions, as demonstrated in extensive simulations.
POLA adapts learning rates for online time series prediction.
problem Adapting to changing data distributions in dynamic environments.
method Adaptive learning rate regulation for recurrent neural networks.
result POLA outperforms other online prediction methods in real-world datasets.
Transfer Learning (TL) in Deep Neural Networks is gaining importance because in most of the applications, the labeling of data is costly and time-consuming. Additionally, TL also provides an effective weight initialization strategy for Deep Neural Networks . This paper introduces the idea of Adaptive Transfer Learning …
This research improves neural network performance with adaptive activation functions in sparse data settings.
problem Limited data availability in scientific and engineering problems.
method Investigation of two types of adaptive activation functions with individual trainable parameters.
result Adaptive activation functions, especially with individual trainable parameters, enhance prediction accuracy and confidence in sparse data settings.
Chameleon optimizes neural network compilation for faster execution and shorter time.
problem Faster execution and shorter compilation time for neural networks.
method Adaptive code optimization using reinforcement learning and adaptive sampling.
result 4.45x speed up in optimization time over AutoTVM, 5.6% improvement in inference time.
AdamCB optimizes neural network training by adaptively selecting samples.
problem Inefficient convergence due to unequal influence of different data samples.
method Integrates combinatorial bandit techniques into Adam to adaptively select samples.
result AdamCB achieves faster convergence and better performance than existing methods.
Adaptive optimization methods bias neural network trajectories towards regions of lower local geometry.
problem The success of adaptive optimization methods in neural networks is not fully explained by traditional second-order methods.
method Local trajectory analysis and introduction of a new statistic RextmedextOPT. result Adaptive methods like Adam bias trajectories towards regions of lower local geometry, leading to faster convergence.
Adaptive dropout and regularization are shown to be dual in linear networks.
problem Sparsifying deep neural networks.
method Examining dropout in the linear case, revealing a duality with regularization.
result Adaptive dropout methods lead to sparse solutions with effective penalties similar to classical sparse optimization penalties.
Paper proposes efficient pruning method for neural networks.
problem Compressing deep neural networks for resource-constrained devices.
method Adaptive sparsity loss for budget-aware optimization during training.
result Demonstrated effectiveness on various architectures and datasets.
Evolutionary algorithm finds optimal pixel perturbations to improve neural network generalization.
problem Minimal data corruption by pixel modifications causes overfitting in neural networks.
method Evolutionary algorithm with a novel cost function to maximize generalization gap and domain divergence.
result Method outperforms previous pixel-based data distribution shift methods on CNNs.
Bayesian neural networks improve likelihood-free inference efficiency.
problem Efficient parameter inference from simulation models with uncertainty.
method Bayesian neural networks for summary statistics, adaptive sampling.
result More robust and efficient posterior estimation.
Oracle inequality for sparse neural nets adapts to unknown structure.
problem Sparse deep neural nets in nonparametric regression.
method Gibbs posterior distribution with Metropolis-adjusted Langevin algorithms and mixture of uniform priors.
result Oracle inequality showing adaptation to unknown regularity and structure, achieving minimax-optimal rate of convergence.
Ada-LISTA adapts neural solvers for varying models.
problem Adapting neural solvers for varying models.
method Ada-LISTA receives pairs of signals and dictionaries, learns a universal architecture, and solves sparse coding in linear rate.
result Ada-LISTA solves sparse coding in linear rate for varying models.