New measure FTC quantifies how much a ReLU network can fine-tune.
problem Analyzing memorization capacity in fine-tuned neural networks.
method Defined Fine-Tuning Capacity (FTC) for additive fine-tuning of ReLU networks.
result Upper and lower bounds on FTC for 2 and 3-layer ReLU networks.
Fine-tuning large language models requires minimal data, making them efficient.
problem Achieving state-of-the-art performance with large language models.
method Using BERT as an example, fine-tuning only the most critical layers of the pre-trained model.
result Fine-tuned models are close in parameter space to the pre-trained model, with many good solutions found in sparsified versions.
As entity type systems become richer and more fine-grained, we expect the number of types assigned to a given entity to increase. However, most fine-grained typing work has focused on datasets that exhibit a low degree of type multiplicity. In this paper, we consider the high-multiplicity regime inherent in data source…
EVA adapts LoRA for faster, more efficient fine-tuning.
problem Fast and efficient fine-tuning of large models for specific tasks.
method EVA uses directions capturing most activation variance for initialization, maximizing gradient signal and reducing parameters.
result EVA achieves faster convergence and higher average scores across tasks, reducing parameters.
The condition number predicts efficient information encoding in neural units, aiding model fine-tuning.
problem Efficient information encoding in neural units for various tasks and input modalities.
method Linking the condition number to the log-volume scaling factor and entropy of the output distribution.
result High condition number indicates efficient encoding, reducing overall information transfer.
Efficient algorithms learn from coarse labels instead of fine grained ones.
problem Learning from coarse labels when fine labels are unavailable.
method Formalized coarse label settings, used a reduction for SQs, and provided efficient algorithms.
result Any problem learnable from fine labels can be learned efficiently from coarse labels.
Machine learning predicts minimal surfaces for knots, supporting a conjecture.
problem Predicting minimal surfaces for knots in hyperbolic space.
method Physics-Informed Neural Networks (PINNs) to solve minimal surface equation.
result Computational minimal surfaces align with Fine's Conjecture.
A new method constrains deep networks during fine-tuning to improve generalization.
problem Improving generalization of fine-tuned deep networks.
method A neural network generalisation bound based on distance from initial weights constrains the hypothesis class to a small sphere.
result Empirical evaluation shows superior generalization performance compared to existing methods.
Improved robustness of BERT through orthogonal classifier heads.
problem Improving robustness of language models for smaller datasets.
method Enforces orthogonal classifier heads and adaptively eliminates weaker ones.
result Improves accuracy by +9% on average across datasets.
Our goal is to identify the type and number of static equilibrium points of solids arising from fine, equidistant n-discretrizations of smooth, convex surfaces. We assume uniform gravity and a frictionless, horizontal, planar support. We show that as n approaches infinity these numbers fluctuate around specific val…
The paper studies combinatorics of injective words in the context of Temperley-Lieb algebras.
problem Combinatorial properties of injective words in the context of Temperley-Lieb algebras.
method Investigation of a chain complex of modules over the Temperley-Lieb algebra, focusing on Euler characteristic, homology modules, and Jacobsthal numbers.
result The Euler characteristic of the complex is the n-th Fine number, and the top-dimensional homology module is decomposed in terms of standard Young tableaux.
Fine-tuning with pre-training data improves performance.
problem Limited training data for tasks.
method Theoretical analysis of excess risk bound and selection of pre-training data subset.
result Improvement in generalization performance with pre-training data.
In this study, we propose the integration of competitive learning into convolutional neural networks (CNNs) to improve the representation learning and efficiency of fine-tuning. Conventional CNNs use back propagation learning, and it enables powerful representation learning by a discrimination task. However, it require…
Paper presents UrbanFM and UrbanPy models for inferring fine-grained urban flows.
problem Reduce cost of urban flow monitoring while maintaining data accuracy and granularity.
method Develops UrbanFM and UrbanPy models to infer fine-grained urban flows from coarse-grained observations.
result UrbanPy model demonstrates favorable performance for larger-scale inference tasks.
Pruning + fine-tuning reduces model complexity and generalizes well for matrix sensing.
problem Reducing model complexity for matrix sensing problems.
method Group Lasso regularization and greedy pruning.
result Pruning results in a solution with minimum columns close to the ground truth.
In an earlier work we identified the types and numbers of static equilibrium points of solids arising from fine, equidistant n-discretrizations of smooth, convex surfaces. We showed that such discretizations carry equilibrium points on two scales: the local scale corresponds to the discretization, the global scale to…
Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a sep…
Transformer models improve arithmetic accuracy with number decomposition.
problem Transformer models struggle with arithmetic operations without decomposition.
method Fine-tuning models with a pipeline that decomposes numbers into units, tens, etc.
result Accuracy increased by 63% in five-digit addition tasks.
In2Core selects a coreset for efficient LLM fine-tuning with reduced data.
problem Costly fine-tuning of large language models due to extensive parameters and data requirements.
method Analyzes model gradients to estimate training sample influence, optimizing for efficiency.
result Achieves similar performance with 50% of training data using In2Core.
Paper develops fine-grain spatiotemporal risk scores using high-resolution mobility data.
problem Developing reliable spatiotemporal risk scores for safe economic reopening.
method Hawkes process-based technique leveraging high-resolution cell-phone location signals.
result Fine-grain spatiotemporal risk scores based on high-resolution mobility data provide useful insights for safe re-opening.
Study improves breast lesion segmentation with limited in vivo data using simulated and natural images.
problem Challenges in automatic breast lesion segmentation due to limited annotated data.
method Pre-training a segmentation network on simulated and natural images, followed by fine-tuning with limited in vivo data.
result Fine-tuning improves dice score by 21% with as little as 19 in vivo images.
LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.
problem Understanding why gradient descent converges to useful low-rank perturbations in LoRA fine-tuning.
method Generalized student-teacher setting with i.i.d. samples and online gradient descent.
result Gradient descent converges to the teacher model in dkO(1) iterations under certain conditions. Solves dual imbalance in detecting sparse anomalies in MIL.
problem Detecting scarce and sparse anomalous samples in MIL.
method Reformulates MIL as a fine-grained PU learning problem, addressing imbalance at both macro and micro levels.
result Demonstrates effectiveness of BFGPU framework on synthetic and real-world datasets.
The purpose of this paper is to construct confidence intervals for the regression coefficients in the Fine-Gray model for competing risks data with random censoring, where the number of covariates can be larger than the sample size. Despite strong motivation from biomedical applications, a high-dimensional Fine-Gray mo…
Develops active learning for scale-bridging simulations.
problem Quantitative predictions in nanoporous media and inertial confinement fusion.
method Active learning approach to optimize fine-scale simulations for coarse-scale hydrodynamics.
result Optimizes use of fine-scale simulations for coarse-scale predictions.
This paper re-evaluates hyperparameters for fine-tuning pre-trained models.
problem Current hyperparameter settings for fine-tuning are often ad-hoc and fixed.
method Empirical evaluation of learning rate, batch size, and momentum for fine-tuning.
result Optimal hyperparameters are not only dataset-dependent but also sensitive to domain similarity.
RAFT fine-tunes models using high-quality samples to align them with human preferences.
problem Aligning generative models with human ethics and preferences.
method RAFT selects high-quality samples, discards undesired behavior, and fine-tunes the model on filtered samples.
result RAFT improves model performance in reward learning and automated metrics.
New method quantifies uncertainty in fine-tuned LLMs using LoRA ensembles.
problem Uncertainty in fine-tuned LLMs and how to trust their predictions.
method Posterior approximations using low-rank adaptation ensembles.
result Unexpected retention of acquired knowledge during fine-tuning in overfitting regime.
Fine-tuning a deep network trained with the standard cross-entropy loss is a strong baseline for few-shot learning. When fine-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100 with the same hyper-parameters. The simplicit…
BERT fine-tuning is unstable due to optimization issues, not forgetting or dataset size.
problem Stability of fine-tuning BERT-based models across different random seeds.
method Analysis of BERT, RoBERTa, and ALBERT fine-tuned on GLUE datasets, identifying optimization difficulties as the cause of instability.
result Fine-tuning instability is due to optimization difficulties leading to vanishing gradients, not forgetting or dataset size.
Improved code translation by preserving structure with composed fine-tuning.
problem Improving code translation accuracy with unlabeled code outputs.
method Pre-trained denoiser to capture output structure, composed fine-tuning to fine-tune predictor.
result Composed fine-tuning significantly improves generalization over standard fine-tuning.
New findings on hyperbolicity of fine curve graphs and their subgraphs.
problem Investigating hyperbolicity of fine curve graphs and their subgraphs.
method Analyzing large subgraphs of fine curve graphs and computing distances in specific cases.
result Large subgraphs of fine curve graphs contain flats of every finite dimension, indicating they are not hyperbolic.
Improved DMs with DP-SGD for generating private images.
problem Low privacy of DMs and lack of good privacy-utility tradeoff.
method Adopt LDMs with DP-SGD on attention modules of LDMs.
result Generates high-quality DP images with reduced parameters.
The paper introduces a Hessian-based method to improve generalization in fine-tuned deep neural networks.
problem Improving generalization in fine-tuned deep neural networks, especially in noisy conditions.
method PAC-Bayesian analysis to identify a Hessian-based distance measure, proving generalization bounds, and developing an algorithm with a generalization error guarantee.
result Hessian-based distance measure correlates well with observed generalization gaps and can match the scale of these gaps in practice.
Homotopy types of curve and arc complexes are studied.
problem Understanding the homotopy types of curve and arc complexes.
method Proving homotopy equivalence and contractibility of complexes.
result Fine curve complex is homotopy equivalent to curve complex, fine arc complex is contractible.
Fine-tuning LLMs improves capability but harms safety, study finds.
problem Balancing capability and safety in LLM fine-tuning.
method Theoretical framework and numerical experiments for two safety-aware fine-tuning strategies.
result Characterization of fundamental limits of safety-capability trade-off in LLM fine-tuning.
Optimizes sparse fine-tuning for privacy in neural networks.
problem Performance gap between DP-SGD and non-private fine-tuning.
method Optimization-based approach using private gradient information for selecting trainable weights.
result Our selection method leads to better prediction accuracy compared to existing approaches.
Fine-tuning harms in-context learning, but restricting updates to the value matrix improves zero-shot performance.
problem Fine-tuning harms in-context learning, reducing zero-shot performance on unseen tasks.
method Theoretical analysis of linear attention models, identifying conditions for degraded few-shot performance.
result Restricting updates to the value matrix improves zero-shot performance while preserving in-context learning.
Measures consistency of tabular LLM predictions under fine-tuning multiplicity.
problem Conflicting predictions from fine-tuned tabular LLMs.
method Local stability measure in embedding space.
result Probabilistic guarantees on prediction consistency under multiplicity.
Self-play fine-tuning improves diffusion models for text-to-image generation.
problem Plateauing performance of diffusion models after data saturation.
method Self-play fine-tuning (SPIN-Diffusion) using competition among model versions.
result Significantly improved model performance and human preference alignment.
The paper develops a theory linking pretraining and fine-tuning in neural networks.
problem Understanding how initialization choices impact feature learning and generalization in neural networks.
method Analytical theory of diagonal linear networks, deriving generalization error as a function of initialization parameters and task statistics.
result Different initialization choices place networks into four fine-tuning regimes with varying abilities to support feature learning and generalization.
Automorphisms of fine 1-curve graph linked to surface homeomorphisms.
problem Understanding automorphisms of fine 1-curve graphs.
method Isomorphic mapping to surface homeomorphisms.
result Automorphism group is isomorphic to homeomorphism group of a surface.
New method reduces fine-tuning cost for reused models.
problem Repeating fine-tuning costs with outdated foundation models.
method Portable Reward Tuning (PRT) trains a reward model to maximize the same loss function as fine-tuning.
result PRT achieves comparable accuracy to inference-time tuning with less inference cost.
Feature representations from pre-trained deep neural networks have been known to exhibit excellent generalization and utility across a variety of related tasks. Fine-tuning is by far the simplest and most widely used approach that seeks to exploit and adapt these feature representations to novel tasks with limited data…
Compact models match or exceed GPT's performance in financial news sentiment analysis.
problem Improving financial sentiment analysis models without large computational costs.
method Fine-tuning non-generative, small-sized models (FinBERT, FinDRoBERTa) on a novel market score database.
result Fine-tuned models outperform GPT-3.5 and GPT-4 in zero-shot learning for financial news sentiment analysis.
ADAPT improves robustness of Vision Transformers without full model fine-tuning.
problem Vulnerability of Vision Transformers to adversarial attacks.
method Parameter-efficient prompt tuning with ADAPT framework for adaptive adversarial training.
result ADAPT achieves robust accuracy of ~40% w.r.t. SOTA methods using only ~1% of the parameters.
Automorphisms of fine graphs for surfaces and tori are studied.
problem Understanding automorphisms of fine graphs for surfaces and tori.
method Extending previous results to tori and discussing smooth versions.
result Automorphism groups of fine graphs for surfaces and tori are naturally isomorphic to homeomorphism groups.
Residual Prior Diffusion integrates coarse latent priors with diffusion models for better generative tasks.
problem Diffusion models struggle with representing both large-scale and fine-scale details in data distributions.
method Two-stage framework: first a coarse prior model captures large-scale structure, then a diffusion model represents the residual.
result RPD accurately captures fine-scale details while preserving large-scale structure, outperforming standard diffusion models.