QUACKIE creates a new benchmark for NLP interpretability.
problem Evaluating NLP interpretability methods is challenging due to biased ground truths.
method Formulated a custom classification task from question-answering datasets, generating unbiased ground truths.
result Demonstrated the effectiveness of current interpretability methods on the new benchmark.
HUBERT combines BERT's structure with TPRs to improve NLP task transfer.
problem Improving transferability among NLP tasks.
method Combines BERT's bidirectional Transformer structure with TPRs to learn shared data structures.
result HUBERT outperforms BERT on GLUE and HANS datasets, showing better transferability.
New pruning method retains model expressiveness for NLP tasks.
problem Pruning large pretrained transformer models for real-world deployment.
method Mixture Gaussian Prior Pruning (MGPP) algorithm.
result MGPP outperforms existing pruning methods in high sparsity settings.
Improved NLP performance with fewer parameters and less data using conditional multi-task learning.
problem Challenges in transferring knowledge across different NLP tasks, including overfitting, forgetting, and negative transfer.
method Proposes a novel Transformer architecture with conditional attention and task-conditioned modules for efficient parameter sharing and mitigating forgetting.
result Achieves state-of-the-art performance on 26 NLP tasks with 66% less data and 50% fewer parameters compared to existing methods.
A2 Learning reduces redundant examples in AL for NLP tasks.
problem Redundant examples in AL strategies waste annotation effort.
method A2 Learning actively adapts to deep learning models to eliminate redundant examples.
result A2 Learning reduces data requirements by 3-25% on NLP tasks.
SusGen-GPT improves financial NLP and ESG report generation.
problem Lack of advanced NLP tools for finance and ESG domains.
method Developed SusGen-30K dataset and SusGen-GPT models.
result Achieved state-of-the-art performance in financial NLP tasks.
BERT outperforms traditional machine learning in text classification tasks.
problem Comparing BERT to traditional machine learning methods for text classification.
method Empirical testing of BERT against TF-IDF-based machine learning models in various scenarios.
result BERT demonstrates superior performance and independence from text features.
COCKATIEL explains neural net models on NLP tasks by identifying meaningful concepts.
problem Transformer models are complex and hard to interpret.
method COCKATIEL uses NMF and sensitivity analysis to identify and rank concepts used by the model.
result COCKATIEL provides accurate and meaningful explanations without affecting model performance.
Survey on using large models to train smaller datasets in NLP.
problem Lack of large datasets and computing resources for NLP tasks.
method Analysis of recent transfer learning approaches in NLP.
result Increased demand for transfer learning in NLP due to large models.
Model generates label-dependent paraphrases for NLP tasks.
problem Generating semantically different paraphrases for NLP tasks.
method Deep variational model with label-dependent generation.
result Model improves generative power of paraphrasing models.
Optimal Word2Vec hyper-parameters improve NLP tasks.
problem Finding the best Word2Vec hyper-parameters for NLP tasks.
method Empirical evaluation of various hyper-parameter combinations on NLP tasks.
result The best hyper-parameters vary by task, and high analogy scores don't always correlate with performance.
Unified framework improves NLP tasks by converting diverse problems into text-to-text format.
problem Improving natural language processing tasks through transfer learning.
method Unified text-to-text transformer framework, comparing various pre-training objectives and architectures.
result Achieved state-of-the-art results on multiple NLP benchmarks.
SCROLLS benchmarks long text NLP tasks, improving existing models.
problem Short NLP benchmarks ignore long texts; SCROLLS addresses this.
method Handpicked long-text datasets for summarization, QA, and inference tasks.
result Improvement potential on SCROLLS tasks, as indicated by initial baselines.
BERTino is a lightweight Italian DistilBERT model for NLP tasks.
problem High computational and memory demands of large language models.
method Developed a DistilBERT model tailored for Italian language.
result F1 scores comparable to BERTBASE with significant speed improvements.
Question semantic similarity (Q2Q) is a challenging task that is very useful in many NLP applications, such as detecting duplicate questions and question answering systems. In this paper, we present the results and findings of the shared task (Semantic Question Similarity in Arabic). The task was organized as part of t…
BigBird improves transformer performance on NLP tasks with longer sequences.
problem Quadratic dependency on sequence length in transformer models.
method Sparse attention mechanism reducing quadratic dependency to linear.
result Significant improvement in performance on NLP tasks like QA and summarization.
AFTER technique improves NLP models by preventing overfitting to task-specific domains.
problem Standard fine-tuning degrades pretraining domain representations.
method Complements task-specific loss with adversarial objective.
result AFTER leads to improved performance on various NLP tasks.
Study recommends PLM choices for minimizing calibration error in NLP tasks.
problem Minimizing calibration error in PLM-based NLP predictions.
method Compared various options for PLM encoding, size, uncertainty quantifier, and fine-tuning loss.
result Recommendations for a well-calibrated PLM-based prediction pipeline.
TX-Ray analyzes and quantifies model knowledge transfer in NLP.
problem Insufficient methods for explaining and quantifying model knowledge transfer in NLP.
method Modified computer vision explainability principle to NLP, visualizing feature preference distributions.
result TX-Ray reveals how self-supervised models learn linguistic abstractions and improves generalization.
This paper improves NLP interpretability by using sentence segments instead of words.
problem Limitations of word-based sampling in explaining complex BERT models.
method Using sentence segments as elementary building blocks for NLP interpretability.
result Improved fidelity of the explainer on a benchmark classification task.
Significant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate results, even when there is little labelled data, because these NLP models can benefit from training on both task-agnostic and task-specific unlabelled data. However, the…
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
LLM Pro Finance Suite enhances financial NLP with instruction-tuned models.
problem Limited NLP capabilities for financial tasks in generalist models.
method Instruction-tuned large language models fine-tuned on financial data.
result Consistent improvement over state-of-the-art baselines in finance tasks.
Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce tech…
Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learnin…
RoBERTa outperforms other pre-trained models in NER tasks.
problem Improving Named Entity Recognition (NER) performance.
method Fine-tuning four pre-trained models (BERT, ERNIE, ERNIE2.0-tiny, RoBERTa) on NER task.
result RoBERTa achieved state-of-the-art results on MSRA-2006 dataset.
Deep neural network models have recently achieved state-of-the-art performance gains in a variety of natural language processing (NLP) tasks (Young, Hazarika, Poria, & Cambria, 2017). However, these gains rely on the availability of large amounts of annotated examples, without which state-of-the-art performance is rare…
This paper predicts legal proceedings status using NLP and machine learning.
problem Classify Brazilian legal proceedings into archived, active, and suspended categories.
method Combined NLP techniques with machine learning to classify legal proceedings sequences.
result Achieved maximum accuracy of 93% and top average F1 Scores of 89% (macro) and 93% (weighted).
Improved language model for French clinical reports achieves state-of-the-art performance in medical NLP tasks.
problem Lack of specialized language models for French clinical reports.
method Adapted a general pre-trained language model (CamemBERT) to French clinical reports using a corpus of 21M reports.
result Pretrained and fine-tuned models improved F1-score by 3 percentage points on APMed task.
Study shows fairness metrics are unreliable for small datasets in NLP tasks.
problem Unreliable fairness metrics for small datasets in NLP tasks.
method Experiments on Bios dataset with varying model sizes.
result Common fairness indices provide unreliable results for small samples.
Universal Language Model for Fine-tuning [arXiv:1801.06146] (ULMFiT) is one of the first NLP methods for efficient inductive transfer learning. Unsupervised pretraining results in improvements on many NLP tasks for English. In this paper, we describe a new method that uses subword tokenization to adapt ULMFiT to langua…
Extracting information from electronic health records (EHR) is a challenging task since it requires prior knowledge of the reports and some natural language processing algorithm (NLP). With the growing number of EHR implementations, such knowledge is increasingly challenging to obtain in an efficient manner. We address…
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extens…
This work explains why GANs are less used for NLP tasks.
problem Why adversarial approaches like GANs are not widely used for NLP tasks.
method Theoretical analysis and reductions showing that maximizing likelihood is equivalent to minimizing distinguishability for certain models.
result Maximizing likelihood is as effective as minimizing distinguishability for NLP tasks.
Paper improves neural network models for MOOC student course prediction.
problem Improving neural network-based predictive methods for MOOC student course trajectory modeling.
method Enhanced LSTM networks and Transformer architecture applied to MOOC student data.
result Improved accuracy in predicting MOOC student course trajectories.
The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process (Frankle & Carbin, 2019). Intriguingly, this phenomenon suggests that initial…
HuSpaCy offers an industrial-grade Hungarian NLP toolkit.
problem Lack of suitable open-source Hungarian NLP pipelines.
method Built on spaCy, HuSpaCy includes lemmatization, morphosyntactic analysis, entity recognition, and word embeddings.
result HuSpaCy achieves high accuracy with resource-efficient prediction.
Following great success in the image processing field, the idea of adversarial training has been applied to tasks in the natural language processing (NLP) field. One promising approach directly applies adversarial training developed in the image processing field to the input word embedding space instead of the discrete…
SpanishTinyRoBERTa distills large Spanish models into efficient question-answering models.
problem Efficient Spanish question-answering models for resource-constrained environments.
method Knowledge distillation from large Spanish language models onto a smaller model.
result SpanishTinyRoBERTa achieves comparable performance to large models with faster inference.
Task-agnostic data augmentation shows little benefit for pretrained transformers.
problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.
method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.
result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.
First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various task…
Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find …
Cryptonite tests NLP models with cryptic crossword clues.
problem Ambiguity in language poses a challenge for NLP models.
method Cryptic crossword dataset based on cryptic clues.
result Fine-tuning T5-Large on Cryptonite achieves only 7.6% accuracy.
Supervised machine learning (ML) algorithms are aimed at maximizing classification performance under available energy and storage constraints. They try to map the training data to the corresponding labels while ensuring generalizability to unseen data. However, they do not integrate meaning-based relationships among la…
BERTopic enhances stock market prediction by analyzing sentiment in topic models.
problem Improving stock price prediction accuracy using sentiment analysis.
method Employed BERTopic for sentiment analysis of stock market comments integrated with deep learning models.
result Enhanced model performance through topic sentiment integration.
Combines foundation models with weak supervision to improve NLP and video tasks.
problem Leveraging weak supervision with foundation models without labeled data.
method Liger, a combination of foundation model embeddings and weak supervision techniques.
result Liger outperforms existing weak supervision methods by 14.1 points on benchmark NLP and video tasks.
New neural network layer handles OOV words in NLP tasks without pre-training.
problem Handling out-of-vocabulary words in natural language processing.
method Contextual-compositional neural network layer that attends to character sequence and context.
result Improves performance on 23 languages in joint tagging tasks.
Neural embeddings have been used with great success in Natural Language Processing (NLP). They provide compact representations that encapsulate word similarity and attain state-of-the-art performance in a range of linguistic tasks. The success of neural embeddings has prompted significant amounts of research into appli…