Improved LoRA+ adapts large models more efficiently.
problem Suboptimal feature learning in LoRA for large models.
method Different learning rates for LoRA adapter matrices A and B.
result Improves performance and speed by 1-2% and up to 2X, respectively.
LoRA enhances model adaptability without increasing parameters.
problem Theoretical understanding of LoRA's adaptability.
method Theoretical analysis of LoRA's expressive power.
result LoRA can adapt any model to a smaller target model with a specific rank.
PLoP optimizes LoRA placement for efficient large model finetuning.
problem Improving efficiency of LoRA adaptation for large models.
method Intuitive theoretical analysis for automatic adapter placement.
result PLoP consistently outperforms existing placement strategies.
LoRA-One uses one-step full gradient to align adapters for efficient large model fine-tuning.
problem Fine-tuning large language models efficiently and accurately.
method Properly initializing LoRA adapters using the one-step full gradient and incorporating preconditioners.
result LoRA-One achieves significant empirical improvements over existing methods.
TensorGuide improves LoRA efficiency and expressivity through joint tensor-train optimization.
problem Limited expressivity and generalization of standard LoRA.
method TensorGuide uses a unified tensor-train structure with controlled Gaussian noise to generate correlated low-rank matrices.
result TensorGuide achieves superior accuracy and scalability with fewer parameters compared to standard LoRA and TT-LoRA.
We analyze the computational limits of LoRA for transformer models using fine-grained complexity theory.
problem Computational efficiency of LoRA fine-tuning for transformer models.
method Fine-grained complexity theory, identifying phase transitions, almost linear algorithms.
result Existence of almost linear algorithms for LoRA adaptation based on specific norms.
LoRA fine-tuning on CPUs without GPUs achieves comparable performance to GPU-based methods.
problem Limited computational resources for fine-tuning LLMs.
method Developed a CPU-efficient method to learn meta-operators for LoRA weights.
result LoRA adapters trained on CPUs outperform base models on downstream tasks.
GeLoRA optimizes LoRA fine-tuning by dynamically adjusting ranks based on intrinsic dimensionality.
problem Efficient fine-tuning of large language models with limited computational resources.
method GeLoRA computes intrinsic dimensionality to adaptively select LoRA ranks, balancing expressivity and efficiency.
result GeLoRA consistently outperforms recent baselines within the same parameter budget on multiple tasks.
LoRAs enable efficient adaptation of large models; this paper explores processing LoRA weights with machine learning.
problem Efficient processing of low-rank weight decompositions in large finetuned models.
method Developed symmetry-aware invariant and equivariant LoL models to process LoRA weights.
result LoL models can predict CLIP scores, finetuning data attributes, and accuracy on downstream tasks.
A new Riemannian framework optimizes LoRA for faster convergence and better performance.
problem Optimizing low-rank adapters in neural networks to improve convergence and performance.
method Integrates Riemannion optimizer, LoRA initialization, and efficient implementation for geometrically treating low-rank adapters.
result Consistent and noticeable improvements in convergence speed and final task performance over standard LoRA and its modifications.
FLoE adapts LLMs by selectively deploying LoRA adapters based on layer importance and task requirements.
problem Uniform LoRA deployment across all layers leads to inefficient and redundant parameter allocation.
method FLoE uses Fisher information to dynamically identify task-critical layers and optimizes LoRA ranks.
result FLoE achieves significant efficiency-accuracy trade-offs, especially in resource-constrained environments.
StatLoRA uses statistical inference to allocate ranks in LoRA fine-tuning, improving performance.
problem Balancing efficiency, expressiveness, and generalization in LoRA rank allocation.
method Formulates LoRA rank allocation as a statistical hypothesis testing problem, using estimated p-values to determine component retention or pruning.
result StatLoRA achieves comparable or better performance than existing methods under matched rank budgets.
Paper analyzes asymmetry in LoRA initialization for foundation models.
problem Asymmetry in LoRA initialization affects generalization of foundation models.
method Theoretical analysis of asymmetric LoRA with frozen random factors.
result Upper bound on sample complexity of $ ilde{\mathcal{O}}\left(\frac{\sqrt{r}}{\sqrt{N}}
ight)$ with high probability.
Improved IVON boosts LoRA model accuracy and calibration.
problem Improving the accuracy and calibration of large language models.
method Replaced AdamW with IVON for finetuning Llama-2.
result IVON improves Llama-2 accuracy by 2.8% and calibration error by 4.6%.
Pre-trained LLM adapted with LoRA improves offline RL for quantitative trading.
problem Challenges in offline RL for quantitative trading due to complex temporal dependencies and overfitting.
method Integrates pre-trained GPT-2 weights and LoRA for efficient fine-tuning of a Decision Transformer.
result Outperforms existing offline RL methods in certain trading scenarios.
LoRA-Curve connects independent LoRA optima through continuous low-loss valleys, improving Bayesian model averaging.
problem Challenges in estimating epistemic uncertainty in LoRA-based Bayesian inference.
method Introduces LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space, with free and anchored configurations.
result Empirically shows that connecting independent LoRA optima through continuous low-loss valleys improves mutual information of the predictive distribution.
New method quantifies uncertainty in fine-tuned LLMs using LoRA ensembles.
problem Uncertainty in fine-tuned LLMs and how to trust their predictions.
method Posterior approximations using low-rank adaptation ensembles.
result Unexpected retention of acquired knowledge during fine-tuning in overfitting regime.
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
EVA adapts LoRA for faster, more efficient fine-tuning.
problem Fast and efficient fine-tuning of large models for specific tasks.
method EVA uses directions capturing most activation variance for initialization, maximizing gradient signal and reducing parameters.
result EVA achieves faster convergence and higher average scores across tasks, reducing parameters.
Paper proposes CLAIR for efficient LLM fine-tuning across clients.
problem Fine-tuning large language models (LLMs) efficiently and collaboratively.
method Federated LoRA fine-tuning with Collaborative Low-rank Alignment and Identifiable Recovery (CLAIR).
result CLAIR achieves better performance and contamination detection compared to local fine-tuning.
Flora uses random projections to achieve high-rank updates with low memory usage.
problem Excessive memory usage in large neural networks during training.
method Flora approximates LoRA using random projections to enable high-rank updates with sublinear space complexity.
result Flora achieves high-rank updates with significantly reduced memory usage compared to LoRA.
ABMLL adapts LLMs to multiple datasets efficiently.
problem Limited generalization across multiple datasets in LLMs.
method Amortized Bayesian Meta-Learning for LoRA (ABMLL).
result ABMLL supports effective generalization across datasets and scales to large models.
LaLoRA prevents forgetting in LoRA fine-tuning.
problem Catastrophic forgetting in fine-tuned models.
method LaLoRA applies Laplace approximation to LoRA weights for regularization.
result Improved learning-forgetting trade-off with controllable regularization strength.
LoRA-MCL improves language models by generating diverse sentence continuations.
problem Language models struggle with generating diverse, plausible sentence continuations.
method Low-Rank Adaptation combined with Multiple Choice Learning (MCL) to handle ambiguity.
result LoRA-MCL generates high-diversity and relevant outputs in various tasks.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
LoRA- improves EFCL by stabilizing feature drifts.
problem Catastrophic forgetting in exemplar-free continual learning.
method LoRA-subtraction to adaptively subtract old task weights.
result LoRA- achieves state-of-the-art performance in EFCL.
New approach uses compressible dynamics to train deep models efficiently.
problem Efficient training of deep overparameterized models with low-rank structures.
method Leveraging low-dimensional structures and compressible dynamics within model parameters.
result Improved training efficiency and reduced overfitting in language models.
Study shows different initialization schemes for LoRA finetuning impact performance.
problem The impact of initialization schemes on LoRA finetuning performance.
method Compared two initialization schemes: B=0, A=random vs. A=0, B=random.
result First initialization scheme yields better performance on average.
New method enhances model fine-tuning with minimal data.
problem Improving model performance on new tasks with limited data.
method Introducing α-LoRA, a reparameterization method for fine-tuning. result Enhanced generalization ability of fine-tuned models.
A study shows that a fine-tuned model's directional accuracy in financial forecasting is largely due to chance, not skill.
problem Misleading directional accuracy in financial forecasting models.
method A reproducible, frozen-data benchmark with paired significance tests to separate skill from base-rate artifact.
result Fine-tuned models do not show significant directional skill over a base rate of 70% in financial forecasting.
Paper fine-tunes LLaMA-3-8B for financial NER using instruction and LoRA.
problem LLMs struggle with financial NER, especially differentiating entities and amounts.
method Instruction fine-tuning combined with LoRA for parameter-efficient learning.
result Micro-F1 score of 0.894 on financial NER tasks, outperforming other models.
This work introduces 'Artificial Entanglement' to understand LLMs' fine-tuning effectiveness.
problem Understanding the effectiveness of parameter-efficient fine-tuning methods for large language models.
method Adopting a quantum-information-inspired perspective, the study measures 'Artificial Entanglement' in neural networks.
result LoRA and FFT induce distinct internal entanglement signatures but not external ones, suggesting a 'no-hair' property.
Study improves LLMs for PPI analysis by addressing uncertainty.
problem Uncertainty in LLM predictions for PPIs.
method Fine-tuned LLaMA-3 and BioMedGPT models, LoRA ensembles, Bayesian LoRA for UQ.
result Competitive PPI identification performance across diverse disease contexts.
The Synthetic Minority Oversampling TEchnique (SMOTE) is widely-used for the analysis of imbalanced datasets. It is known that SMOTE frequently over-generalizes the minority class, leading to misclassifications for the majority class, and effecting the overall balance of the model. In this article, we present an approa…
IVON improves LoRA finetuning with minimal overhead and significant accuracy gains.
problem Improving LoRA finetuning with Bayesian methods.
method IVON, a variational algorithm, with posterior pruning.
result Significant accuracy improvements over AdamW and other Bayesian methods.
Chronicals boosts LLM fine-tuning speed by 3.51x over Unsloth.
problem Memory bottleneck in fine-tuning large language models.
method Four optimizations: fused kernels, cross-entropy reduction, LoRA+, and sequence packing.
result 3.51x speedup on A100-40GB with Qwen2.5-0.5B.
Paper introduces NumLLM for better financial text understanding with numeric variables.
problem Poor performance of existing financial large language models in numeric financial text.
method Constructed financial corpus, fine-tuned with LoRA modules, merged into foundation model.
result NumLLM achieves best performance on financial question-answering benchmark, especially with numeric questions.
Proposes a new framework for uncertainty-aware LLM post-training.
problem Heterogeneous, conflicting data in large language models.
method α-Rényi variational framework for learning distributions over post-training parameters.
result Enables training examples to be softly routed across ensemble members, promoting model specialisation and providing uncertainty estimates.
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
LoRA and privacy: Random projections help but not always.
problem Ensuring differential privacy in LoRA fine-tuning.
method Wishart projection mechanism and noisy variants.
result LoRA is not inherently private, but low-rank fine-tuning can be more private.
Fine-tunes LLMs to correct bias in predictions.
problem LLMs exhibit bias in predictions from data.
method Supervised fine-tuning with Low-Rank Adaptation (LoRA).
result Fine-tuning corrects bias in both controlled and real-world settings.
CoLoRA leverages task similarity to boost fine-tuning efficiency.
problem Efficiently fine-tuning large foundation models with scarce labeled data.
method CoLoRA trains a shared adapter for task similarity and personalized adapters for user-specific tasks.
result CoLoRA significantly boosts fine-tuning performance when tasks are similar.
BoRA finetunes multi-task LLMs by sharing information through hierarchical priors.
problem Limited data for some tasks in multi-task LLMs.
method Bayesian hierarchical low-rank adaption.
result BoRA outperforms individual and unified model approaches.
Paper fine-tunes LLMs for financial tasks using data fusion.
problem Improving LLMs for financial analysis tasks.
method Fine-tuned Llama3-8B and Mistral-7B using PEFT and LoRA, combined datasets for data fusion.
result Enhanced model performance across financial tasks.
DataInf efficiently approximates data influence in large models, improving transparency and identifying mislabeled data.
problem Efficiently estimating data influence in large-scale models like LoRA-tuned LLMs and diffusion models.
method DataInf uses a closed-form expression to approximate influence scores efficiently.
result DataInf outperforms existing methods in computational and memory efficiency, accurately identifying influential data points.
KodeXv0.1 improves financial question answering over GPT-4.
problem Lack of specialized financial language models.
method Custom training on financial documents, RAG-aware 4bit LoRA tuning.
result KodeX-8Bv0.1 outperforms GPT-4 by up to 9.24% in financial tasks.
Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.
problem Efficient training and deployment of large-scale deep learning models.
method Complementary theoretical perspectives on low-rank structures during training and convergence, and practical applications of LoRA and masked training.
result Understanding and exploiting low-rank structures can improve efficiency and effectiveness of training and fine-tuning.
LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.
problem Understanding why gradient descent converges to useful low-rank perturbations in LoRA fine-tuning.
method Generalized student-teacher setting with i.i.d. samples and online gradient descent.
result Gradient descent converges to the teacher model in dkO(1) iterations under certain conditions.