Vanishing gradients hinder reinforcement finetuning of language models.
problem Vanishing gradients impede the optimization of language models using reinforcement finetuning.
method The study identifies vanishing gradients as a fundamental optimization obstacle in reinforcement finetuning and proposes an initial supervised finetuning phase to mitigate this issue.
result An initial supervised finetuning phase is crucial for successful reinforcement finetuning of language models, as it helps prevent vanishing gradients and maximizes rewards.
PLoP optimizes LoRA placement for efficient large model finetuning.
problem Improving efficiency of LoRA adaptation for large models.
method Intuitive theoretical analysis for automatic adapter placement.
result PLoP consistently outperforms existing placement strategies.
This paper bridges offline and online RL by studying policy finetuning with a reference policy.
problem Sample-efficient reinforcement learning in online and offline settings.
method Design of policy finetuning algorithms and analysis of sample complexity.
result Theoretical analysis shows that the optimal policy finetuning algorithm is either offline reduction or purely online RL.
The paper proposes a learning algorithm that improves adaptability and generalization.
problem Improving adaptability and generalization in learning models.
method Learning to meta-learn by meta-finetuning on related tasks before adapting to specific tasks.
result Learning to meta-learn improves adaptability and generalization across various tasks.
Algorithm improves reinforcement learning policies using offline data.
problem Improving reinforcement learning policies with limited online data.
method Designs a single non-reactive policy using offline data with provable guarantees.
result Algorithm achieves better policy quality with less online data.
Transforming sparse outcomes into dense process rewards for efficient reinforcement learning.
problem Training RL policies to maximize sparse outcomes.
method Incentivizing policy matching state-action visitations of successful episodes.
result Significantly faster RL finetuning performance.
Improves drug properties using a novel LLM and reinforcement learning.
problem Optimizing drug properties while retaining chemical stability.
method Structured Policy Optimization (SPO) for fine-tuning a large language model.
result Enhanced drug properties across multiple target objectives.
WebGUM learns web navigation from multimodal data, outperforming previous methods.
problem Limited generalization from domain-specific models in web navigation.
method Instruction-following multimodal agent trained on vision-language foundation models.
result Significant improvement in web navigation performance on benchmarks.
FinFlowRL combines imitation and reinforcement learning for better financial control.
problem Traditional stochastic control methods fail in real-world finance due to changing market conditions.
method FinFlowRL uses imitation learning to pretrain an adaptive meta policy, then finetunes it with reinforcement learning.
result FinFlowRL consistently outperforms individual strategies across various market conditions.
SkMM selects data for finetuning by balancing bias and variance.
problem Balancing bias and variance in high-dimensional finetuning.
method Gradient sketching for bias reduction and moment matching for variance reduction.
result Gradient sketching selects samples efficiently and accurately.
FinFlowRL learns from experts to optimize financial control in changing markets.
problem Traditional finance control methods fail in real-world, non-stationary markets.
method Imitation-Reinforcement Learning framework that pretrains on expert strategies and finetunes in noise space.
result Consistently outperforms individually optimized experts across diverse market conditions.
Paper introduces a multi-stage influence function to track model predictions.
problem Improving natural language processing and computer vision performance.
method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.
result Identifies pretraining examples contributing most to finetuning task predictions.
RAFT fine-tunes models using high-quality samples to align them with human preferences.
problem Aligning generative models with human ethics and preferences.
method RAFT selects high-quality samples, discards undesired behavior, and fine-tunes the model on filtered samples.
result RAFT improves model performance in reward learning and automated metrics.
In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degen…
This dissertation automates deep learning pipelines and uses meta-learning for better model selection and data augmentation.
problem Challenges in selecting and fine-tuning deep learning pipelines for new datasets.
method Meta-learning for DL pipeline selection and data augmentation, using synthetic data.
result Meta-learned approaches outperform traditional methods in automated DL pipeline selection and data augmentation.
Unified finetuning of all quantization degrees of freedom achieves state-of-the-art 4-bit quantization.
problem Achieving high accuracy in quantized neural networks while maintaining speed and resource constraints.
method Quantization-aware finetuning (QFT) that jointly optimizes all quantization degrees of freedom.
result 4-bit weight quantization results on-par with state-of-the-art (SoTA) within PTQ constraints.
Linguistic calibration improves long-form text confidence.
problem LMs hallucinate, leading to suboptimal decisions.
method Defining linguistic calibration, training framework, reinforcement learning.
result Llama 2 7B is significantly more calibrated than baselines.
Study shows pretraining and finetuning can effectively tackle covariate shift in linear regression.
problem Linear regression under covariate shift where source and target distributions differ but conditional distribution remains similar.
method Pretraining on source data and finetuning on target data using online SGD.
result Transfer learning with O(N2) source data is as effective as supervised learning with N target data. LoRAs enable efficient adaptation of large models; this paper explores processing LoRA weights with machine learning.
problem Efficient processing of low-rank weight decompositions in large finetuned models.
method Developed symmetry-aware invariant and equivariant LoL models to process LoRA weights.
result LoL models can predict CLIP scores, finetuning data attributes, and accuracy on downstream tasks.
G1 uses RL to enhance LLMs' graph reasoning, improving performance on diverse tasks.
problem Limited graph reasoning abilities of LLMs, especially in synthetic graph-theoretic tasks.
method Curated synthetic graph dataset, RL training on LLMs.
result Significant improvements in graph reasoning, zero-shot generalization to unseen tasks.
DPA aligns LLMs with multi-objective rewards for diverse user preferences.
problem Fine-grained control over LLMs for diverse user needs.
method Integrates multi-objective reward modeling and directional preference control.
result DPA offers better performance trade-offs and intuitive user control over LLM generation.
New framework reduces LLM complexity by directly finetuning in Boolean domain.
problem Reducing the complexity of large language models (LLMs) while maintaining performance.
method Proposes a novel framework using multi-kernel Boolean parameters for direct finetuning in the Boolean domain.
result Significantly reduces complexity during both finetuning and inference, outperforming recent techniques.
Meta-learning techniques like MAML and Reptile fail to generalize well to out-of-distribution tasks compared to simple finetuning.
problem Generalization of meta-learning techniques in low-data scenarios.
method Investigation of MAML, Reptile, and finetuning on various tasks.
result MAML and Reptile specialize for fast adaptation but fail to generalize to out-of-distribution tasks.
Study shows different initialization schemes for LoRA finetuning impact performance.
problem The impact of initialization schemes on LoRA finetuning performance.
method Compared two initialization schemes: B=0, A=random vs. A=0, B=random.
result First initialization scheme yields better performance on average.
A pretrained LLM and its finetuned version can detect OOD data effectively.
problem Detecting out-of-distribution data in language models.
method Using the likelihood ratio between a pretrained and finetuned LLM.
result The likelihood ratio is an effective criterion for OOD detection.
RLVR maintains safety while improving reasoning capabilities in LLMs.
problem Safety-capability tradeoff in fine-tuning LLMs.
method Reinforcement Learning with Verifiable Rewards (RLVR) and theoretical analysis.
result RLVR can enhance reasoning while maintaining safety guardrails.
ConMeZO speeds up zeroth-order optimization for large language models.
problem Slow convergence in high-dimensional parameter spaces of large language models.
method Adaptive directional sampling in a cone centered around a momentum estimate.
result Achieves the same convergence rate as MeZO but up to 2X faster.
Vision transformers benefit from non-smooth components in adaptation.
problem Understanding the role of non-smoothness in vision transformer adaptation.
method Theoretical analysis and extensive experiments on large-scale vision transformers.
result High plasticity of attention modules and feedforward layers leads to better finetuning performance.
We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learni…
Paper introduces RiskEmbed, a finetuned model for financial risk management.
problem Improving retrieval accuracy in financial question-answering systems.
method Curated dataset and finetuned BERT model for financial domain.
result RiskEmbed significantly outperforms general-purpose and financial embedding models.
BioFinBERT analyzes sentiment of biotech press releases and financial text around inflection points.
problem Analyzing sentiment of biotech press releases and financial text around inflection points.
method Finetuning BioBERT on financial datasets to create BioFinBERT for sentiment analysis.
result BioFinBERT accurately analyzes sentiment of biotech press releases and financial text around inflection points.
IVON improves LoRA finetuning with minimal overhead and significant accuracy gains.
problem Improving LoRA finetuning with Bayesian methods.
method IVON, a variational algorithm, with posterior pruning.
result Significant accuracy improvements over AdamW and other Bayesian methods.
BoRA finetunes multi-task LLMs by sharing information through hierarchical priors.
problem Limited data for some tasks in multi-task LLMs.
method Bayesian hierarchical low-rank adaption.
result BoRA outperforms individual and unified model approaches.
New Bayesian method improves Pareto front estimation in multitask finetuning.
problem Efficiently estimating Pareto fronts for multitask finetuning.
method Variational Model Merging using non-Gaussian posteriors.
result More flexible posteriors lead to better Pareto front estimates.
This thesis proposes a derivatives hedging framework using deep learning and reinforcement learning.
problem Traditional hedging models fail in complex, uncertain markets due to assumptions like continuous trading and zero transaction costs.
method Integrates deep learning and reinforcement learning, using a spatiotemporal attention-based Transformer for probabilistic forecasting and hedging.
result The proposed method significantly outperforms traditional approaches in U.S. and Chinese financial markets.
Study reveals attributes help in few-shot classification generalization.
problem Understanding what makes some novel classes easier to learn.
method Defined attributes to quantify concept relatedness, used supervised and self-supervised learning.
result Combining self-supervised pretraining with supervised finetuning improves generalization.
Improves retrieval accuracy for hierarchical documents, especially for distant matches.
problem Limited expressive power of dual encoder models in hierarchical retrieval.
method Proves feasibility of DEs for HR, introduces pretrain-finetune recipe to improve long-distance retrieval.
result Pretrain-finetune boosts recall on long-distance pairs from 19% to 76%.
Study improves cryptocurrency price prediction using unlabeled text data.
problem Predicting cryptocurrency returns from unlabelled text data.
method Introduced weak learning approach to finetune BERT on unlabeled text data.
result Finetuning pretrained NLP models with weak labels enhances forecast accuracy.
This paper uses CausalGANs and RL with LLM to predict bond yields.
problem Challenges in financial bond yield forecasting due to data scarcity and market conditions.
method Proposes a novel framework combining CausalGANs, RL, and LLM for synthetic data generation and trading signals.
result Improves forecasting performance over existing methods with low Mean Absolute Error.
Meta learning optimizes neural network quantization for efficient inference.
problem Uniform bitwidth quantization is sub-optimal for neural network compression.
method Meta learning to automatically generate hybrid quantization policies.
result Meta learning outperforms uniform quantization and RL approaches.
MPP trains a transformer to predict multiple physical systems, improving accuracy across various tasks.
problem Training models for specific physical systems is inefficient and requires fine-tuning.
method MPP trains a shared transformer on multiple heterogeneous physical systems, projecting fields into a shared embedding space.
result A single MPP-pretrained transformer outperforms task-specific models on all pretraining sub-tasks and downstream tasks.
Proposes a new algorithm for efficient online model selection of LLMs considering the increasing-then-converging trend.
problem Balancing cost and performance in choosing the best LLM among a diverse set of models.
method Introduces a time-increasing bandit algorithm (TI-UCB) that predicts model performance increases and balances exploration and exploitation.
result Achieves a logarithmic regret upper bound, indicating a fast convergence rate in model selection.
FinZero improves financial time series forecasting accuracy with multimodal modeling.
problem Lack of interpretability, uncertainty, and scalability in financial time series forecasting.
method Developed a multimodal pre-trained model FinZero using UARPO method for reasoning, prediction, and uncertainty analysis.
result FinZero achieves an approximate 13.48% improvement in prediction accuracy over GPT-4o in high-confidence group.
CFA improves model's ability to generalize across unseen domain-class combinations.
problem Challenges in real-world machine learning applications due to data distribution shifts and limited training data.
method Developed Compositional Feature Alignment (CFA) technique to improve CG ability of pretrained models.
result CFA outperforms common finetuning techniques in compositional generalization.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
Simplifies transfer learning with deep neural networks using ridge regression.
problem High computational cost of finetuning deep models for transfer learning.
method Leverage the low-rank property of deep neural networks' feature vectors in kernel ridge regression.
result Successful on supervised and semi-supervised transfer learning tasks.
Many sleep studies suffer from the problem of insufficient data to fully utilize deep neural networks as different labs use different recordings set ups, leading to the need of training automated algorithms on rather small databases, whereas large annotated databases are around but cannot be directly included into thes…
CollaQ improves multi-agent performance in StarCraft by 40% with fewer samples.
problem Limited generalization and high training rounds in multi-agent reinforcement learning.
method Formulates multi-agent collaboration as joint optimization on reward assignment, decomposes Q-function into self and interactive terms, and uses MARA loss.
result Improves win rate by 40% in StarCraft maps with same number of samples compared to state-of-the-art techniques.