Investigates optimal parameter allocation in Transformers for efficiency and expressivity.
problem Balancing expressivity and efficiency in Transformer model parameters.
method Mathematical analysis and theoretical characterization of attention heads and head dimensions.
result Later layers can operate more efficiently with reduced parameters due to saturation of softmax activations.
KM method reduces ConvNet parameters to 9% higher accuracy with minimal additional memory.
problem Expensive memory usage for training ConvNets on embedded devices.
method Kernel Modulation (KM) method that adapts all network parameters for each task.
result KM delivers up to 9% higher accuracy than other parameter-efficient methods.
FiT combines transfer and meta-learning for efficient few-shot image classification.
problem Few-shot image classification in personalized and federated learning settings.
method Combines transfer learning and meta-learning with fixed pretrained backbones and fine-tuned FiLM adapter layers.
result Achieves state-of-the-art accuracy on VTAB-1k benchmark with fewer than 1% of updateable parameters.
TensorGuide improves LoRA efficiency and expressivity through joint tensor-train optimization.
problem Limited expressivity and generalization of standard LoRA.
method TensorGuide uses a unified tensor-train structure with controlled Gaussian noise to generate correlated low-rank matrices.
result TensorGuide achieves superior accuracy and scalability with fewer parameters compared to standard LoRA and TT-LoRA.
Efficient estimators for smooth Hilbert-valued parameters with theoretical guarantees.
problem Estimating smooth Hilbert-valued parameters with theoretical guarantees.
method Pathwise differentiable Hilbert-valued parameters, efficient influence functions, regularized one-step estimators.
result Theoretical guarantees for efficient estimators even when nuisance functions are arbitrary.
NanoFlow reduces parameter complexity in normalizing flows.
problem Efficient parameter complexity in flow-based models.
method Single neural density estimator with flow indication embedding.
result Sublinear parameter complexity achieved.
CAM-GAN improves GANs for continual learning with efficient feature map transformations.
problem Efficient continual learning for GANs with reduced parameter growth.
method Designing and leveraging parameter-efficient feature map transformations, including global and task-specific parameters, residual bias, and Fisher information matrix.
result Significantly improved model performance and high-quality samples with fewer parameters.
Develops a new asymptotic efficiency theory for non-Euclidean parameter spaces.
problem Lack of a unified efficiency theory for non-Euclidean parameter spaces.
method Introduces a new theory for Riemannian manifolds with regularity conditions.
result Establishes efficiency bounds for non-Euclidean parameter spaces.
Scout-Nd optimizes parameters of stochastic simulators efficiently.
problem Optimizing parameters of stochastic, computationally expensive simulators.
method Scout-Nd algorithm, reducing gradient noise, multi-fidelity schemes.
result Demonstrates better performance compared to existing methods.
Quaternion Conformer GAN (QC-GAN) is a parameter-efficient speech enhancement framework that combines a Quaternion Conformer generator with MetricGAN-based training.
problem Speech Enhancement
method Quaternion Conformer GAN
result Achieved a PESQ score of 3.48 with 0.89M parameters, comparable to state-of-the-art models at less than half their size.
Recent work on policy learning from observational data has highlighted the importance of efficient policy evaluation and has proposed reductions to weighted (cost-sensitive) classification. But, efficient policy evaluation need not yield efficient estimation of policy parameters. We consider the estimation problem give…
In this paper we explore the parameter efficiency of BERT arXiv:1810.04805 on version 2.0 of the Stanford Question Answering dataset (SQuAD2.0). We evaluate the parameter efficiency of BERT while freezing a varying number of final transformer layers as well as including the adapter layers proposed in arXiv:1902.00751. …
To keep up with increasing dataset sizes and model complexity, distributed training has become a necessity for large machine learning tasks. Parameter servers ease the implementation of distributed parameter management---a key concern in distributed training---, but can induce severe communication overhead. To reduce c…
A new FL algorithm reduces communication overhead by selectively updating model parameters.
problem Data heterogeneity and communication overhead in federated learning.
method Uses age of information metric to selectively update model parameters and group clients with similar data.
result Our method can expedite training and surpass other communication-efficient strategies in efficiency.
The study assesses how financial markets' efficiency changed during the COVID-19 crisis.
problem The impact of COVID-19 on financial market efficiency.
method Dynamic estimation method for Hurst exponent and memory parameter using alpha-stable distribution and dependence structure.
result Financial markets' efficiency varied during the COVID-19 crisis, with some indices showing less impact than others.
We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and …
Simplifies efficient estimation via automatic differentiation and probabilistic programming.
problem Constructing efficient estimators for complex statistical models.
method Automatic differentiation applied to statistical functionals, avoiding the need to derive efficient influence functions.
result Users can generate efficient estimators with minimal code, simplifying the process for non-experts.
Likelihood-free inference for simulator-based statistical models has developed rapidly from its infancy to a useful tool for practitioners. However, models with more than a handful of parameters still generally remain a challenge for the Approximate Bayesian Computation (ABC) based inference. To advance the possibiliti…
We introduce a new embarrassingly parallel parameter learning algorithm for Markov random fields with untied parameters which is efficient for a large class of practical models. Our algorithm parallelizes naturally over cliques and, for graphs of bounded degree, its complexity is linear in the number of cliques. Unlike…
Efficiently learns exponential family distributions with i.i.d. samples.
problem Learning natural parameters of truncated exponential families efficiently.
method Proposes a novel loss function and computationally efficient estimator.
result Achieves optimal sample complexity and asymptotic normality.
Efficiently estimate Boolean product distribution parameters from truncated samples.
problem Estimating parameters of Boolean product distributions from truncated samples.
method Introducing fatness of truncation set, using membership queries, and adapting Stochastic Gradient Descent.
result Efficiently learn Boolean product distributions from truncated samples with small sample complexity.
Neural network architectures found by sophistic search algorithms achieve strikingly good test performance, surpassing most human-crafted network models by significant margins. Although computationally efficient, their design is often very complex, impairing execution speed. Additionally, finding models outside of the …
New method for efficient inference over complex parameter spaces.
problem Challenges in Bayesian inference for high-dimensional, intractable likelihoods.
method Arbitrary Marginal Neural Ratio Estimation (AMNRE) for simulation-based inference.
result Efficient inference over arbitrary subsets of parameters without numerical integration.
A new method to prune neural networks with iterative randomization improves efficiency.
problem Efficiency in pruning randomly initialized neural networks.
method Iteratively randomizing weight values to reduce parameter requirements.
result The method achieves remarkable performance with fewer parameters.
Transformers can efficiently approximate nonparametric regression with minimal parameters and sequences.
problem Efficiently approximating nonparametric regression functions with transformers.
method Kernel-weighted polynomial basis and gradient descent.
result Achieves minimax optimal rate of convergence with fewer parameters and sequences.
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
Invertible DenseNets improve model efficiency and performance.
problem Improving model efficiency and performance in neural networks.
method Enforcing invertibility in DenseNets by satisfying the Lipschitz constraint and proposing a learnable concatenation.
result i-DenseNets outperform Residual Flows in negative log-likelihood on various datasets.
This technical report describes an efficient technique for computing the norm of the gradient of the loss function for a neural network with respect to its parameters. This gradient norm can be computed efficiently for every example.
Adaptive multi-domain learning reduces parameter count for efficient deep learning.
problem Different domains have varying complexity, leading to inefficient model training.
method Proposes adaptive parameterization to reduce model complexity without sacrificing performance.
result Efficient multi-domain learning solutions with far fewer parameters.
This paper proposes an efficient autoHPO method based on data-to-hyper-parameter mapping.
problem Manual hyper-parameter tuning is costly and dependent.
method The approach is based on mapping from data to hyper-parameters using a sophisticated network structure and effective construction algorithms.
result The proposed approach significantly outperforms state-of-the-art methods.
Novel covariance function improves Bayesian optimization efficiency.
problem Efficient global optimization of expensive black-box functions.
method Additive tree-structured covariance function and parallel optimization algorithm.
result Significantly outperforms state-of-the-art methods in conditional parameter optimization.
Transformer learns to estimate negative binomial parameters efficiently.
problem Parameter estimation for over-dispersed count data in large screens.
method Pre-trained transformer trained on synthetic data generation to invert parameter to count transformation.
result Method of moments provides faster, more efficient, and better-calibrated estimates.
Efficiently learns deep factor graphs using Gaussian belief propagation.
problem Learning in deep factor graphs with efficient inference.
method Treats all relevant quantities as random variables, uses belief propagation for inference.
result Efficiently solves training and prediction problems in deep factor graphs with belief propagation.
TENP prunes experts and neurons in Mixture-of-Experts models for efficient deployment.
problem Efficient deployment of large language models constrained by static parameter footprint.
method Structured Trapezoidal ExpertNeuron Pruning (TENP) identifies and retains important experts and neurons.
result DeepSeek model achieves 10% better performance on code generation tasks with 40% expert sparsity.
Efficiently models learning curves using Gaussian processes with latent Kronecker structure.
problem Joint modeling of machine learning model performance across hyper-parameters and training progress.
method Imposes latent Kronecker structure to leverage efficient product kernels and handle missing values.
result Matches the performance of a Transformer on a learning curve prediction task.
Unified Bayesian Optimization framework for model selection balancing effectiveness and training efficiency.
problem Balancing model effectiveness and training efficiency in machine learning model selection.
method Proposes a unified Bayesian Optimization framework to jointly optimize model effectiveness and training efficiency.
result Models selected using the proposed framework significantly improve training efficiency while maintaining strong effectiveness.
New method estimates SDE parameters efficiently using WCE and SGD.
problem Parameter estimation for stochastic differential equations.
method Wiener Chaos Expansion and Stochastic Gradient Descent.
result Accurate parameter recovery from noisy observations.
Examines WENDy-IRLS algorithm's noise robustness and efficiency in various differential equations.
problem Noise robustness and efficiency of WENDy-IRLS algorithm.
method Studied coverage and bias properties of WENDy-IRLS algorithm's estimators in various differential equations and noise distributions.
result WENDy-IRLS algorithm shows notable noise robustness and computational efficiency.
Proposes efficient model for continual learning that grows model over task-specific parameters.
problem Limited transfer learning ability and forgetting of earlier knowledge in existing methods.
method Filter and channel expansion method that grows model over previous task parameters.
result Better knowledge transfer and improved performance in task incremental learning.
i-DenseNets improve parameter efficiency and performance in density estimation.
problem Improving parameter efficiency and performance in density estimation models.
method Invertible Dense Networks (i-DenseNets) with learnable weighted concatenation and Concatenated LipSwish activation function.
result i-DenseNets outperform Residual Flows and other flow-based models in bits per dimension.
MuRiT efficiently computes multi-parameter persistence barcodes.
problem Efficient computation of multi-parameter persistent homology.
method Vietoris-Rips transformation to reduce multi-parameter to single-parameter computation.
result MuRiT computes pathwise persistence barcodes for multi-filtered flag complexes.
Bayesian model tackles high-dimensional inverse problems efficiently.
problem Estimating spatially-varying parameters in expensive models.
method Multiscale Bayesian inference with deep generative models and MCMC.
result Efficient estimation of global and local parameter features.
This paper improves parameter estimation in cardiac models using Gaussian process-based MH sampling.
problem Uncertainty in estimating patient-specific model parameters from sparse and noisy clinical data.
method Integrates surrogate modeling into Metropolis-Hastings sampling to improve computational efficiency and accuracy.
result Significant gain in computational efficiency without compromising accuracy, and insights into tissue heterogeneity.
Efficient algorithms for sparse parameter recovery in mixture models.
problem Support recovery of high-dimensional sparse latent vectors in mixture models.
method Efficient algorithms with logarithmic sample complexity dependence on dimensionality.
result First guarantees on support recovery for various mixture models.
We propose reinforcement learning on simple networks consisting of random connections of spiking neurons (both recurrent and feed-forward) that can learn complex tasks with very little trainable parameters. Such sparse and randomly interconnected recurrent spiking networks exhibit highly non-linear dynamics that transf…
Proposes a new method for efficient model reconstruction with uncertain parameters.
problem Reconstructing models with latent variables or parameters of unknown distribution.
method Local squared Wasserstein-2 (W_2) method.
result Efficiently reconstructs output distributions from observation data.
Efficiently quantifies uncertainty in subsurface flow using neural networks guided by theory.
problem Uncertainty in dynamic subsurface flow predictions.
method Theory-guided Neural Network (TgNN) for efficient uncertainty quantification.
result TgNN surrogate improves efficiency of uncertainty quantification compared to MC method.