RoBERTa outperforms other pre-trained models in NER tasks.
problem Improving Named Entity Recognition (NER) performance.
method Fine-tuning four pre-trained models (BERT, ERNIE, ERNIE2.0-tiny, RoBERTa) on NER task.
result RoBERTa achieved state-of-the-art results on MSRA-2006 dataset.
RoBERTa model detects counterfactual statements in text.
problem Detecting and extracting counterfactual statements from text.
method Used RoBERTa language representation model for both subtasks.
result RoBERTa achieved top performance in both subtasks at SemEval-2020.
Improved negation detection in Dutch clinical texts using machine learning.
problem Extracting negation from clinical text for better model development.
method Comparison of rule-based and machine learning methods (biLSTM, RoBERTa).
result BiLSTM and RoBERTa models outperform rule-based method in F1 score, precision, and recall.
Paper proposes a deep learning model for understanding e-commerce addresses.
problem Challenges in parsing shipping addresses with no fixed format.
method Combines NLP techniques with pre-processing steps for addresses, uses RoBERTa for vector representations.
result RoBERTa model achieves 90% accuracy in sub-region classification for North and South Indian cities.
New algorithms improve privacy and utility of large language models.
problem Privacy-preserving fine-tuning of large language models.
method Meta-framework for differentially private fine-tuning, inspired by recent success in fine-tuning.
result Private fine-tuned models achieve utility close to non-private models, with improved privacy and efficiency.
This study evaluates LLMs for sentiment analysis in stock price prediction.
problem Improving stock price prediction accuracy using LLMs for news sentiment analysis.
method Compared 3 LLMs (DeBERTa, RoBERTa, FinBERT) for sentiment-driven stock prediction.
result DeBERTa outperforms other models with 75% accuracy, and ensemble model increases accuracy to 80%.
The paper improves ESG taxonomy and classifies sentences as sustainable or unsustainable.
problem Improving ESG taxonomy and classifying sentences based on ESG factors.
method For ESG taxonomy, used Sentence-BERT models. For sentence classification, combined RoBERTa with a multi-layer perceptron.
result Significant performance improvement and high accuracy in classifying sentences.
SpanishTinyRoBERTa distills large Spanish models into efficient question-answering models.
problem Efficient Spanish question-answering models for resource-constrained environments.
method Knowledge distillation from large Spanish language models onto a smaller model.
result SpanishTinyRoBERTa achieves comparable performance to large models with faster inference.
BoC probe assesses neural network confidence coherence, revealing architecture-specific uncertainty.
problem Poor calibration and OOD detection in neural networks.
method Bag-of-Coins (BoC) probe compares softmax confidence to pairwise dominance probabilities.
result BoC reveals clear ID/OOD separation for some architectures but not others.
This paper improves neural network compression by using robust low-rank approximations.
problem Neural network compression sensitivity to outliers.
method Introduces robust low-rank approximations using ℓ p \ell_p ℓ p norms (for p ∈ [ 1 , 2 ] p\in [1,2] p ∈ [ 1 , 2 ] ) and provides efficient algorithms. result Achieves up to 28% compression with minimal accuracy loss compared to existing methods.
Averaging recent model checkpoints speeds up training time.
problem Training large vision or language models is time-consuming.
method Average the weights of the k latest checkpoints.
result Speeds up training by dozens of epochs, saving up to 68 GPU hours.
Study proposes memory-efficient backpropagation for linear layers in neural networks.
problem Significant memory usage in backpropagation through linear layers in neural networks.
method Randomized matrix multiplications to reduce memory usage with a moderate decrease in test accuracy.
result Demonstrated benefits of the proposed method on fine-tuning pre-trained models.
Large financial dataset tracks FOMC communications and their impact.
problem Understanding how FOMC communications influence financial markets.
method Constructed a large annotated dataset of FOMC speeches, minutes, and transcripts. Developed a hawk-dove classification task. Evaluated various models on the dataset and used RoBERTa-large for monetary policy stance measurement.
result Monetary policy stance measures derived from FOMC documents predict market performance.
This study evaluates zero-shot LLMs in finance, finding ChatGPT performs well but fine-tuned models are better.
problem Evaluating zero-shot LLMs in financial tasks.
method Comparison of ChatGPT and fine-tuned models on annotated data.
result Fine-tuned models generally outperform zero-shot LLMs.
DPZero fine-tunes large models privately without backpropagation.
problem Memory and privacy challenges in fine-tuning large language models.
method DPZero uses zeroth-order methods for private fine-tuning, avoiding backpropagation.
result DPZero achieves private fine-tuning of RoBERTa and OPT on various tasks.
Prodigy estimates learning rate without tuning, improving convergence.
problem Estimating optimal learning rate in adaptive methods.
method Prodigy modifies D-Adaptation to estimate distance to solution D D D . result Prodigy improves convergence rate by a factor of O ( log ( D / d 0 ) ) O(\sqrt{\log(D/d_0)}) O ( log ( D / d 0 ) ) . Paper presents a model for identifying informative COVID-19 tweets.
problem Identifying informative COVID-19 tweets on Twitter.
method Leveraged transformers (RoBERTa, XLNet, BERTweet) trained in Semi-Supervised Learning (SSL) setting.
result Achieved F1 score of 0.9011 on test set, ranking 7th on leaderboard.
Task-agnostic data augmentation shows little benefit for pretrained transformers.
problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.
method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.
result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.
Contrastive Code Representation Learning improves code summarization and type inference.
problem Code representations are sensitive to edits, hindering downstream semantic understanding tasks.
method ContraCode: a contrastive pre-training task that learns code functionality.
result Contrastive pre-training improves code summarization and type inference accuracy.
Gestalt combines two models to improve SQuAD2.0 performance.
problem Improving the accuracy of answering questions in context paragraphs.
method A stacking ensemble of ALBERT and RoBERTa models, combined with a CNN-based meta-model.
result Best ensemble achieved 87.117 EM and 90.306 F1 scores, improving baseline by 0.55% and 0.61% respectively.
SAM improves deep learning tasks by promoting balancedness, reducing outlier impact.
problem Improving generalization in deep learning tasks, especially with scale-invariant problems.
method Introduces balancedness as a new concept to depict global behaviors of SAM, focusing on the difference between squared norms of two variables.
result SAM promotes balancedness and is data-responsive, outperforming SGD in outlier scenarios.
Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of task…
We extend quantization-aware training to extreme model compression.
problem Maximizing model accuracy with minimal model size.
method Quantize a random subset of weights during training, allowing unbiased gradients through other weights.
result Established new state-of-the-art compromises between accuracy and model size.
LLMs improve financial sentiment analysis in finance.
problem Defining and measuring financial sentiment.
method Investigation of sentiment measurement methods and LLMs.
result LLMs enhance financial sentiment analysis.
DataInf efficiently approximates data influence in large models, improving transparency and identifying mislabeled data.
problem Efficiently estimating data influence in large-scale models like LoRA-tuned LLMs and diffusion models.
method DataInf uses a closed-form expression to approximate influence scores efficiently.
result DataInf outperforms existing methods in computational and memory efficiency, accurately identifying influential data points.
We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simpl…
Downscaled models outperform larger ones on GLUE tasks.
problem Difficulty in attributing performance changes to specific factors in large language models.
method Pre-trained down-scaled versions of Transformer-based architectures on a common corpus, benchmarked on GLUE tasks.
result MLM + NSP (BERT-style) consistently outperforms other objectives.
This thesis evaluates text-based vs audio-based classification of mental health interviews.
problem Classifying psychiatric illness using text-based methods.
method Design and evaluate a text classification network on mental health interviews, using belabBERT.
result Text-based classification is a strong alternative to audio-based methods.
BERT fine-tuning is unstable due to optimization issues, not forgetting or dataset size.
problem Stability of fine-tuning BERT-based models across different random seeds.
method Analysis of BERT, RoBERTa, and ALBERT fine-tuned on GLUE datasets, identifying optimization difficulties as the cause of instability.
result Fine-tuning instability is due to optimization difficulties leading to vanishing gradients, not forgetting or dataset size.
Allowing machines to choose whether to kill humans would be devastating for world peace and security. But how do we equip machines with the ability to learn ethical or even moral choices? Jentzsch et al.(2019) showed that applying machine learning to human texts can extract deontological ethical reasoning about "right"…
ALIEN improves uncertainty estimation of language models by refining entropy-based methods.
problem Overconfidence in uncertainty estimation for language models, especially for difficult inputs.
method ALIEN refines entropy-based uncertainty by aligning it with prediction reliability, using a lightweight uncertainty head.
result ALIEN consistently outperforms strong baselines in detecting incorrect predictions and achieving the lowest calibration error.
Qwen3-8B outperforms classical models in financial text classification.
problem Financial text classification for trading systems and sentiment analysis.
method Noisy Embedding Instruction Finetuning and Rank-stabilized Low-Rank Adaptation.
result Qwen3-8B achieves better classification accuracy and fewer training epochs.
A new method compresses NLP networks by using multiple subspaces instead of a single one.
problem Large errors in compressing NLP networks using a single subspace.
method Projective clustering to find multiple subspaces that minimize squared distances.
result Networks are more accurate and smaller compared to standard matrix factorization.