TensorGuide improves LoRA efficiency and expressivity through joint tensor-train optimization.
problem Limited expressivity and generalization of standard LoRA.
method TensorGuide uses a unified tensor-train structure with controlled Gaussian noise to generate correlated low-rank matrices.
result TensorGuide achieves superior accuracy and scalability with fewer parameters compared to standard LoRA and TT-LoRA.
i-DenseNets improve parameter efficiency and performance in density estimation.
problem Improving parameter efficiency and performance in density estimation models.
method Invertible Dense Networks (i-DenseNets) with learnable weighted concatenation and Concatenated LipSwish activation function.
result i-DenseNets outperform Residual Flows and other flow-based models in bits per dimension.
Invertible DenseNets improve model efficiency and performance.
problem Improving model efficiency and performance in neural networks.
method Enforcing invertibility in DenseNets by satisfying the Lipschitz constraint and proposing a learnable concatenation.
result i-DenseNets outperform Residual Flows in negative log-likelihood on various datasets.
Improved neural network approximates analytic and L^p functions efficiently.
problem Efficiently approximating analytic and L^p functions using neural networks.
method Three-dimensional ReLU network architecture for sawtooth functions, improving approximation rates.
result Substantially improved exponential approximation rates for analytic functions and general L^p functions.
Slice-based Learning improves model performance on critical data subsets.
problem Low performance on critical data subsets in machine learning models.
method Proposes a new programming model (Slice-based Learning) that uses slicing functions to specify critical data subsets and combines these with an attention mechanism.
result Improves model performance by up to 19.0 F1 on slices and 4.6 F1 overall.
KM method reduces ConvNet parameters to 9% higher accuracy with minimal additional memory.
problem Expensive memory usage for training ConvNets on embedded devices.
method Kernel Modulation (KM) method that adapts all network parameters for each task.
result KM delivers up to 9% higher accuracy than other parameter-efficient methods.
FiT combines transfer and meta-learning for efficient few-shot image classification.
problem Few-shot image classification in personalized and federated learning settings.
method Combines transfer learning and meta-learning with fixed pretrained backbones and fine-tuned FiLM adapter layers.
result Achieves state-of-the-art accuracy on VTAB-1k benchmark with fewer than 1% of updateable parameters.
New method learns interaction-aware orderbook representation for better intraday electricity price forecasting.
problem Challenges in probabilistic intraday electricity price forecasting due to dynamic orderbook microstructure.
method OrderFusion: an end-to-end and parameter-efficient probabilistic forecasting model that learns interaction-aware representation of buy-sell dynamics.
result Consistent improvements over conventional baselines in probabilistic forecasting of CID price indices.
Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.
problem Efficient training and deployment of large-scale deep learning models.
method Complementary theoretical perspectives on low-rank structures during training and convergence, and practical applications of LoRA and masked training.
result Understanding and exploiting low-rank structures can improve efficiency and effectiveness of training and fine-tuning.
Paper explores BERT's efficiency on SQuAD2.0, freezing layers and using adapters.
problem Improving BERT's efficiency for SQuAD2.0 while maintaining performance.
method Freezing transformer layers, using adapters, and context-aware convolutional filters.
result Context-aware convolutional filters do not improve practical efficiency.
SG-NTF completes HDI tensors with spectral mapping and spatio-temporal gating.
problem High-dimensional and incomplete tensor completion.
method Spectra-Guided Neural Tucker Factorization (SG-NTF) with Spatio-Temporal Co-Gating (STCG).
result Maintains competitive completion accuracy with parameter efficiency.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
Quaternion Conformer GAN (QC-GAN) is a parameter-efficient speech enhancement framework that combines a Quaternion Conformer generator with MetricGAN-based training.
problem Speech Enhancement
method Quaternion Conformer GAN
result Achieved a PESQ score of 3.48 with 0.89M parameters, comparable to state-of-the-art models at less than half their size.
New method selects neural network architectures without needing data.
problem Choosing efficient deep neural network architectures.
method Developed the deep frame potential to quantify network capacity.
result Deep frame potential correlates with generalization error.
Transformers learn functionals from distributions without losing information.
problem Lack of rigorous mathematical theory supporting Transformer performance.
method Proposed a Transformer learning framework, attention operator, and distribution regression.
result Transformers can compress distributions into function representations without loss of information.
A new framework for generative modeling using controlled vector fields.
problem Expressive modeling with limited parameters.
method Continuous-time modeling with modulated fixed vector fields and learned scalar controls.
result Expressive transport achieved with a small number of learned control channels.
SPIRE enables efficient federated learning for diffusion models by separating client-specific embeddings from a shared backbone.
problem Large diffusion models are impractical for federated learning due to their size.
method SPIRE separates the network into a global backbone and client-specific embeddings, enabling efficient finetuning.
result SPIRE achieves parameter-efficient finetuning, updating only a small fraction of weights.
Efficiently scales continuous kernels with sparse Fourier domain learning.
problem High computational and memory demands, spectral bias in continuous kernels.
method Sparse learning in the Fourier domain.
result Efficient scaling of continuous kernels, reduced computational and memory requirements, mitigated spectral bias.
Enhances anomaly detection in high dimensions with pretrained networks.
problem Difficult to characterize anomaly in high-dimensional data.
method Residual adaptation to adjust pretrained networks for anomaly detection.
result Significantly outperforms existing methods on anomaly detection benchmarks.
Kernelised flows improve density estimation and generation with fewer parameters.
problem Limited expressiveness of flow-based models due to invertibility constraints.
method Integrates kernels into normalising flows to enhance expressiveness and efficiency.
result Kernelised flows outperform neural network-based flows in parameter efficiency and low-data scenarios.
Paper proposes CLAIR for efficient LLM fine-tuning across clients.
problem Fine-tuning large language models (LLMs) efficiently and collaboratively.
method Federated LoRA fine-tuning with Collaborative Low-rank Alignment and Identifiable Recovery (CLAIR).
result CLAIR achieves better performance and contamination detection compared to local fine-tuning.
GraphToken encodes structured data for LLMs, improving graph reasoning tasks.
problem Efficiently encoding structured data for large language models.
method GraphToken learns an encoding function to extend prompts with explicit structured information.
result Significant improvements across node, edge, and graph-level tasks on the GraphQA benchmark.
We introduce a new parameterization method for deep learning layers using spectral tensor train decomposition.
problem Efficiency and stability in deep learning models with weight matrix compression.
method Spectral Tensor Train Parameterization (STTP) of weight matrices.
result Improved compression and training stability in neural networks.
Paper fine-tunes LLMs for financial tasks using data fusion.
problem Improving LLMs for financial analysis tasks.
method Fine-tuned Llama3-8B and Mistral-7B using PEFT and LoRA, combined datasets for data fusion.
result Enhanced model performance across financial tasks.
Pea-KD improves BERT student models by 4.4% on average in GLUE tasks.
problem Efficiently compressing BERT while maintaining performance.
method Parameter-efficient and accurate Knowledge Distillation (Pea-KD) with Shuffled Parameter Sharing (SPS) and Pretraining with Teacher's Predictions (PTP).
result Pea-KD improves student model's performance by 4.4% on average in four GLUE tasks.
A new method to prune neural networks with iterative randomization improves efficiency.
problem Efficiency in pruning randomly initialized neural networks.
method Iteratively randomizing weight values to reduce parameter requirements.
result The method achieves remarkable performance with fewer parameters.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that l…
Improved financial VA intent classification accuracy.
problem Determining user intents for unseen open intents.
method Supervised pre-training of intent representations using prefix-tuning and fine-tuning.
result 1.63% - 2.07% higher accuracy on banking77 benchmark.
LoRA enhances model adaptability without increasing parameters.
problem Theoretical understanding of LoRA's adaptability.
method Theoretical analysis of LoRA's expressive power.
result LoRA can adapt any model to a smaller target model with a specific rank.
Spatial Adapter adds structured spatial representation to frozen predictors.
problem Efficiently adding spatial structure to pre-trained models.
method Structured spatial decomposition and closed-form covariance for residual fields.
result Adapter improves spatial prediction and uncertainty quantification.
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
GP-KAN uses Gaussian Processes in KANs for robust, parameter-efficient non-linear modeling.
problem Non-linear modeling with limited parameters and uncertainty estimates.
method Integrates Gaussian Processes into Kolmogorov Arnold Networks (KANs) for robust non-linear modeling.
result GP-KAN achieves 98.5% accuracy on MNIST with 80k parameters compared to 1.5M for state-of-the-art models.
LaLoRA prevents forgetting in LoRA fine-tuning.
problem Catastrophic forgetting in fine-tuned models.
method LaLoRA applies Laplace approximation to LoRA weights for regularization.
result Improved learning-forgetting trade-off with controllable regularization strength.
AdaCat improves density estimation and planning in autoregressive models.
problem Efficiently modeling sharp density changes in continuous data.
method Adaptive Categorical Discretization (AdaCat) for autoregressive models.
result Improves density estimation and planning in various data types.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
The paper reformulates U-Nets as wavelet-based models and applies this to hierarchical VAEs.
problem Theoretical understanding and regularization properties of U-Nets and their relationship to wavelets.
method Formulating a multi-resolution framework to identify U-Nets as finite-dimensional truncations of infinite-dimensional models, proving average pooling corresponds to projection, and identifying HVAEs as discretizations of multi-resolution diffusion processes.
result HVAEs learn a time representation allowing for improved parameter efficiency through weight-sharing.
DiBS learns Bayesian network structure and parameters efficiently.
problem Bayesian structure learning with uncertainty reasoning.
method Differentiable framework for continuous latent graph representation, agnostic to local conditional distributions.
result Significantly outperforms related approaches in posterior inference.
Translating or rotating an input image should not affect the results of many computer vision tasks. Convolutional neural networks (CNNs) are already translation equivariant: input image translations produce proportionate feature map translations. This is not the case for rotations. Global rotation equivariance is typic…
Framework reuses pre-trained models for data-free transfer learning.
problem Challenges in retrieving source data for model training.
method Model Recycling Framework for parameter-efficient training.
result Makes multi-source data-free supervised transfer learning possible.
ADAPT improves robustness of Vision Transformers without full model fine-tuning.
problem Vulnerability of Vision Transformers to adversarial attacks.
method Parameter-efficient prompt tuning with ADAPT framework for adaptive adversarial training.
result ADAPT achieves robust accuracy of ~40% w.r.t. SOTA methods using only ~1% of the parameters.
Autotune optimizes Lasso tuning parameters efficiently and accurately.
problem Efficiently selecting Lasso tuning parameters in high-dimensional models.
method Alternates optimizing regression coefficients and noise standard deviation.
result Autotune outperforms existing methods in low signal-to-noise regimes.
DenseNets improve accuracy and efficiency in convolutional networks.
problem Improving accuracy and efficiency in deep convolutional networks.
method Introducing Dense Convolutional Networks (DenseNet) with direct connections between all layers.
result DenseNets achieve significant improvements over state-of-the-art networks on object recognition benchmarks.
CoLoRA leverages task similarity to boost fine-tuning efficiency.
problem Efficiently fine-tuning large foundation models with scarce labeled data.
method CoLoRA trains a shared adapter for task similarity and personalized adapters for user-specific tasks.
result CoLoRA significantly boosts fine-tuning performance when tasks are similar.
Model projection transfers convolutional network properties to feedforward networks.
problem Transferring properties between feedforward and convolutional networks.
method Unified node-level framework with tensor-valued activations, model projection.
result Projected CNN nodes inherit GFFN-style trainable structure.
The ability to discover approximately optimal policies in domains with sparse rewards is crucial to applying reinforcement learning (RL) in many real-world scenarios. Approaches such as neural density models and continuous exploration (e.g., Go-Explore) have been proposed to maintain the high exploration rate necessary…
New neural network learns relevant transformations in data, improving object recognition.
problem Current equivariant architectures consider all possible transformations, ignoring relevant ones.
method Co-attentive equivariant neural networks that focus on co-occurring transformations.
result Outperforms conventional equivariant networks on rotated MNIST and CIFAR-10.
Kolmogorov-Arnold network improves GW catalog posterior construction.
problem Efficiently constructing posterior distributions for GW catalogs.
method Using the Kolmogorov-Arnold network to create lightweight neural density estimators.
result Kolmogorov-Arnold network achieves superior interpretability and accuracy in posterior construction.