FLoE adapts LLMs by selectively deploying LoRA adapters based on layer importance and task requirements.
problem Uniform LoRA deployment across all layers leads to inefficient and redundant parameter allocation.
method FLoE uses Fisher information to dynamically identify task-critical layers and optimizes LoRA ranks.
result FLoE achieves significant efficiency-accuracy trade-offs, especially in resource-constrained environments.
LOFT separates subspace rotation and transformation for orthogonal fine-tuning.
problem Conflating subspace rotation and transformation in orthogonal fine-tuning.
method LOFT explicitly separates subspace rotation and transformation, using task-aware support selection.
result LOFT recovers principal-subspace orthogonal adaptation and improves efficiency-performance trade-off.
Develops a method to ensure accuracy of few-shot transfer learning models.
problem Lack of generalization guarantees for low-data transfer learning.
method Trains a distribution over PEFT parameters using upstream tasks and samples plausible PEFTs for downstream tasks.
result Demonstrates non-vacuous generalization guarantees compared to existing methods in the low-shot regime.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
LORENZA improves LLM fine-tuning efficiency and generalization.
problem Improving robustness and generalization of LLMs under hardware constraints.
method AdaZo-SAM and LORENZA, combining Adam and SAM with zeroth-order estimation and randomized SVD.
result LORENZA achieves better generalization and reduced memory consumption compared to existing methods.
This work introduces 'Artificial Entanglement' to understand LLMs' fine-tuning effectiveness.
problem Understanding the effectiveness of parameter-efficient fine-tuning methods for large language models.
method Adopting a quantum-information-inspired perspective, the study measures 'Artificial Entanglement' in neural networks.
result LoRA and FFT induce distinct internal entanglement signatures but not external ones, suggesting a 'no-hair' property.
New method calibrates LLMs for safety-critical tasks with scalable Bayesian inference.
problem Overconfidence in LLMs after fine-tuning for specific tasks.
method Orthogonalized Low-Rank Adapters (PoLAR) with variational Bayesian inference.
result Scalable and well-calibrated uncertainty estimation for LLMs.
Paper fine-tunes LLMs for financial tasks using data fusion.
problem Improving LLMs for financial analysis tasks.
method Fine-tuned Llama3-8B and Mistral-7B using PEFT and LoRA, combined datasets for data fusion.
result Enhanced model performance across financial tasks.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
TFB simplifies Bayesian LLM uncertainty estimation without extra training.
problem Estimating uncertainty in LLM responses remains challenging.
method Training-Free Bayesianization (TFB) that transforms low-rank adapters into Bayesian ones without additional training.
result TFB achieves superior uncertainty estimation and generalization compared to existing methods.
Study shows different initialization schemes for LoRA finetuning impact performance.
problem The impact of initialization schemes on LoRA finetuning performance.
method Compared two initialization schemes: B=0, A=random vs. A=0, B=random.
result First initialization scheme yields better performance on average.
Model projection transfers convolutional network properties to feedforward networks.
problem Transferring properties between feedforward and convolutional networks.
method Unified node-level framework with tensor-valued activations, model projection.
result Projected CNN nodes inherit GFFN-style trainable structure.
Study shows LLMs can remove half of layers without significant performance drop.
problem Understanding knowledge storage in LLMs' weights.
method Layer pruning and finetuning to identify and remove unnecessary parameters.
result Minimal degradation of performance after removing up to half of layers.
Paper analyzes asymmetry in LoRA initialization for foundation models.
problem Asymmetry in LoRA initialization affects generalization of foundation models.
method Theoretical analysis of asymmetric LoRA with frozen random factors.
result Upper bound on sample complexity of $ ilde{\mathcal{O}}\left(\frac{\sqrt{r}}{\sqrt{N}}
ight)$ with high probability.
DoRA improves adaptation efficiency for large models by factoring norms and fusing kernels.
problem High-rank DoRA is computationally expensive and infeasible on common GPUs.
method Factored norms and fused Triton kernels to reduce memory and speed up computation.
result Fused implementation is up to 2.0x faster for inference and 1.9x faster for gradient computation.