We develop a novel method, called PoWER-BERT, for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “BERT model”
BERT model improves cross-lingual document retrieval.
problem Improving cross-lingual information retrieval between English queries and foreign documents.
method Introduced a deep relevance matching model based on BERT, trained with weak supervision using CLIR data.
result Model outperforms competitive baseline approaches in retrieving Lithuanian documents against short English queries.
Combines BERT and GCN for better text classification.
problem Limited global information capture by BERT.
method Integrates BERT with VGCN for improved text classification.
result VGCN-BERT outperforms BERT and GCN alone.
Study compares machine learning models and BERT on SQuAD dataset.
problem Comparing performance of machine learning models and BERT on SQuAD dataset.
method Comparative analysis of machine learning models and BERT on SQuAD dataset.
result BERT model outperforms other models on SQuAD dataset but requires more time.
Sketch-BERT learns vector sketches using BERT-like self-supervised learning.
problem Lack of effective vector sketch representation for recognition and retrieval tasks.
method Generalized BERT to sketch domain with novel embedding networks and self-supervised sketch gestalt learning.
result Improved performance on sketch recognition, retrieval, and gestalt tasks.
BERT-XML automates ICD coding from EHR notes using BERT pretraining.
problem Manual ICD coding is time-consuming and expensive in hospitals.
method Trained a BERT model on EHR notes, adapting it for multi-label attention.
result First large-scale ICD-10 classification model using millions of EHR notes.
In graph instance representation learning, both the diverse graph instance sizes and the graph node orderless property have been the major obstacles that render existing representation learning models fail to work. In this paper, we will examine the effectiveness of GRAPH-BERT on graph instance representation learning,…
lamBERT learns language and actions using multimodal BERT.
problem Learning language and actions in complex environments.
method Extending BERT to multimodal representation and integrating with reinforcement learning.
result lamBERT model achieved higher rewards in multitask and transfer settings.
Study compares BERT with other sentiment analysis models.
problem Comparing sentiment analysis techniques.
method Used four models: Sent WordNet, logistic regression, LSTM, and BERT on IMDB movie reviews.
result BERT outperformed other models in sentiment classification.
BERT outperforms traditional machine learning in text classification tasks.
problem Comparing BERT to traditional machine learning methods for text classification.
method Empirical testing of BERT against TF-IDF-based machine learning models in various scenarios.
result BERT demonstrates superior performance and independence from text features.
BERT models outperform GPT in financial engineering sentiment analysis.
problem Using sentiment from news events for commodity trading.
method Benchmarked Transformer models (BERT, GPT) on financial engineering task.
result CopBERT models outperform GPT and vanilla BERT models.
Combines BERT and graph CNN for improved text classification.
problem Text classification problems
method Combining BERT embedding and graph convolutional neural network
result Graph CNN model performs better than classical models combined with BERT
Chinese word segmentation (CWS) is a fundamental task for Chinese language understanding. Recently, neural network-based models have attained superior performance in solving the in-domain CWS task. Last year, Bidirectional Encoder Representation from Transformers (BERT), a new language representation model, has been pr…
BERT learns claim descriptions to identify patent novelty.
problem Identifying novel patent claims among existing documents.
method Training BERT on concatenated claims and descriptions, scoring BERT's output.
result BERT identifies relevant X documents for patent novelty.
FinEAS models financial sentiment using BERT embeddings.
problem Financial sentiment analysis in markets.
method Supervised fine-tuning of BERT embeddings for financial texts.
result FinEAS outperforms vanilla BERT, LSTM, and FinBERT.
CERT improves language understanding by contrastively learning sentence-level semantics.
problem Lack of sentence-level semantics in existing pretraining tasks.
method Contrastive self-supervised learning at the sentence level using back-translation augmentations.
result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
The dominant graph neural networks (GNNs) over-rely on the graph links, several serious performance problems with which have been witnessed already, e.g., suspended animation problem and over-smoothing problem. What's more, the inherently inter-connected nature precludes parallelization within the graph, which becomes …
BERT fine-tuning is unstable due to optimization issues, not forgetting or dataset size.
problem Stability of fine-tuning BERT-based models across different random seeds.
method Analysis of BERT, RoBERTa, and ALBERT fine-tuned on GLUE datasets, identifying optimization difficulties as the cause of instability.
result Fine-tuning instability is due to optimization difficulties leading to vanishing gradients, not forgetting or dataset size.
Weight Squeezing transfers knowledge from large models to smaller ones, improving performance and speed.
problem Transfer learning and model compression for faster and more efficient training.
method Reparameterization of weights from a large model to a smaller one.
result Weight Squeezing outperforms other methods on GLUE benchmark with faster training.
BERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the current work, we focus on the interpretation of self-attention, which is one of the fundamental underlying components of BERT. Using a subset of G…
Paper proposes MA-BERT for efficient data-driven ATM models.
problem Long training time and need for large datasets in data-driven ATM models.
method Multi-Agent Bidirectional Encoder Representations from Transformers (MA-BERT) and transfer learning framework.
result MA-BERT saves training time and achieves high performance with little data.
Aspect-Based Sentiment Analysis (ABSA) deals with the extraction of sentiments and their targets. Collecting labeled data for this task in order to help neural networks generalize better can be laborious and time-consuming. As an alternative, similar data to the real-world examples can be produced artificially through …
FAT ALBERT: Finding Answers in Large Texts using Semantic Similarity Attention Layer based on BERTcs.CL
Model improves BERT for answering multiple-choice questions in large texts.
problem Improving machine comprehension of large text corpora for question answering.
method Developed a model using BERT with a semantic similarity attention layer to extract key sentences.
result Outperforms leading models in MovieQA challenge with 87.79% test accuracy.
Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a sep…
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-t…
Paper uses BERT to assess P2P borrowers' credit risk from loan descriptions.
problem Information asymmetry in P2P lending due to lack of borrower data.
method Fine-tunes BERT on Lending Club dataset to generate risk scores from loan descriptions.
result BERT-generated risk scores improve XGBoost classifier's performance in loan granting.
We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate …
This study improves sentence embeddings from BERT models.
problem Capturing the underlying meaning of sentences using BERT models.
method Comprehensive review and testing of various sentence embedding extraction and refinement methods.
result Representation-shaping techniques significantly improve sentence embeddings from BERT-based and simple baseline models.
Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysiscs.CL
Paper tackles counterfactual sentence detection and evaluation.
problem Detect and evaluate counterfactual sentences in natural language.
method Used a BERT base model for classification and a hybrid BERT Multi-Layer Perceptron for sequence identification. Introduced cascaded linear inputs to improve performance.
result Achieved an F1 score of 85.00% in Task 1 and 83.90% in Task 2.
BERT learns molecular substructures for chemistry problems.
problem Predicting chemical properties and synthesizing molecules.
method Transformer-based BERT model on molecule string representations, attention visualization.
result BERT learns to represent functional groups and atoms for various chemical properties.
Efficient algorithm finds fast Transformer models.
problem Slow inference time of Transformer models.
method Decompose Transformer architecture into components, use sampling-based one-shot search.
result Achieved 10% to 30% speedup on pre-trained BERT and 70% on top of a previous state-of-the-art model.
G5 universal GRAPH-BERT learns graph representations across different datasets.
problem Learning graph representations across diverse graph datasets with distinct input and output configurations.
method G5 introduces a pluggable model architecture with input and output components for each graph data source, connected via a unified layer and fusion layer.
result G5 removes obstacles for cross-graph representation learning and transfer, even for sparse data.
Efficiently trains BERT on academic GPUs in 12 days.
problem Training large-scale BERT models is expensive and time-consuming.
method Optimizes training on multiple GPUs and nodes, reducing costs.
result Trains BERT on academic GPUs in 12 days, not requiring expensive hardware.
Bayesian method improves reliability of BERT for hate speech detection.
problem Reliable detection of hate speech in user-generated content.
method Bayesian approach using Monte Carlo dropout in transformer models.
result Monte Carlo dropout provides well-calibrated reliability estimates.
HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detectioncs.CL
HMS-BERT detects cyberbullying in multiple languages and labels.
problem Multilingual and multi-label cyberbullying detection challenges.
method Hybrid multi-task self-training framework using BERT.
result Strong performance on multi-label and main classification tasks.
Predict stock movement with news headlines using BERT embeddings.
problem Predicting stock price movement after financial news.
method Fine-Tuned Contextualized-Embedding Recurrent Neural Network (FT-CE-RNN) using BERT.
result Obtains state-of-the-art results on stock movement prediction task.
The paper improves model robustness by regularizing posterior differences.
problem Improving model robustness in noisy input scenarios.
method Posterior differential regularization with -divergence.
result Regularizing with -divergence improves model robustness.
In this work we focus on fine-tuning a pre-trained BERT model and applying it to patent classification. When applied to large datasets of over two millions patents, our approach outperforms the state of the art by an approach using CNN with word embeddings. In addition, we focus on patent claims without other parts in …
BERT embeddings improve sequence quality metrics.
problem Measuring the quality of generated sequences against references.
method Employ contextual BERT embeddings for sequence-level reward.
result Contextual embeddings provide a more effective learning signal.
Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and, thus, are too resource-hungry and computation-intensive to suit low-capability devices or applications with strict latency re…
BERT improved for propaganda detection with imbalanced, dissimilar data.
problem BERT struggles with dissimilar imbalanced datasets in propaganda detection.
method Cost-sensitive BERT with dissimilarity measure for imbalanced, dissimilar datasets.
result Achieved second-highest score on sentence-level propaganda classification.
I-BERT extends Transformer's self-attention to arbitrary input lengths.
problem Transformer models struggle with inductive generalization to unseen input lengths.
method Replaces positional encodings with a recurrent layer.
result I-BERT achieves state-of-the-art results on algorithmic tasks.
FastSGT improves accuracy in BERT-based DST for SGD datasets.
problem Dialog State Tracking in goal-oriented dialogue systems.
method BERT-based model with two carry-over procedures and multi-head attention.
result Significantly improved accuracy compared to baseline model.
We develop a chatbot using Deep Bidirectional Transformer models (BERT) to handle client questions in financial investment customer service. The bot can recognize 381 intents, and decides when to say "I don't know" and escalates irrelevant/uncertain questions to human operators. Our main novel contribution is the discu…
Accelerates BERT pretraining from 3 days to 54 minutes.
problem Long training time of BERT due to large mini-batch sizes.
method LANS method and learning rate scheduler for large mini-batch training.
result Achieved fastest BERT training time of 54 minutes.
This paper fine-tunes BERT for stock market sentiment analysis and improves trading performance.
problem Improving trading performance in non-strongly efficient markets.
method Fine-tuning BERT on annotated data, combining with Alpha191 model for regression and prediction.
result Emotional factors significantly improve trading performance, increasing return rates by 73.8% compared to baseline.