Transformers improve Finnish language modeling, achieving lower perplexity scores.
problem Improving language modeling for Finnish using deep learning models.
method Used BERT and Transformer-XL models in a sub-word setting, compared to LSTM.
result Transformer-XL outperforms LSTM, achieving a 27% better perplexity score.
Study on stock portfolio concentration among Finnish households and investors.
problem Understanding the concentration of stock portfolios owned by Finnish households and investors.
method Analysis of stock portfolios using Herfindahl-Hirschman index over 20 years.
result High portfolio concentration observed in Finnish retail investors, similar to institutional investors.
The paper proposes to analyze a data set of Finnish ranks of academic publication channels with Extreme Learning Machine (ELM). The purpose is to introduce and test recently proposed ELM-based mislabel detection approach with a rich set of features characterizing a publication channel. We will compare the architecture,…
We investigate the trading behavior of Finnish individual investors trading the stocks selected to compute the OMXH25 index in 2003 by tracking the individual daily investment decisions. We verify that the set of investors is a highly heterogeneous system under many aspects. We introduce a correlation based method that…
Dissertation tackles geodesic ray transform on Riemannian manifolds.
problem Determining functions from line integrals along geodesics.
method Establishes conditions for unique and stable determination of functions.
result New numerical model for computed tomography imaging created.
Recently, a marked Poisson process (MPP) model for life catastrophe risk was proposed in [6]. We provide a justification and further support for the model by considering more general Poisson point processes in the context of extreme value theory (EVT), and basing the choice of model on statistical tests and model compa…
GP-NODE combines Gaussian processes and NeuralODEs for Bayesian system identification.
problem Bayesian systems identification from partial, noisy and irregular observations.
method Differentiable programming, Hamiltonian Monte Carlo, Gaussian Process priors, sparsity-promoting priors.
result Efficient inference of posterior distributions over plausible models with quantified uncertainty.
Multilayer networks are attracting growing attention in many fields, including finance. In this paper, we develop a new tractable procedure for multilayer aggregation based on statistical validation, which we apply to investor networks. Moreover, we propose two other improvements to their analysis: transaction bootstra…
A new causal graph framework identifies treatment effects without adjusting for confounders.
problem Invalid identification of causal effects due to unmeasured confounders.
method Developed the Napkin graph to identify causal effects through a ratio of g-formulas, using influence-function-based estimators.
result Demonstrated substantial efficiency gains in estimating causal effects using the Napkin graph.
Study quantifies gender bias in language models across 7 languages.
problem Measuring gender bias in language models across multiple languages.
method Curated dataset of politicians, multilingual language models, probing language models.
result Larger language models do not show significant gender bias compared to smaller ones.
New ASR system handles multiple languages without needing language-specific encoding.
problem Joint training of data-rich and data-scarce languages in a single model.
method Transforms all languages to a single writing system through transliteration, separating modeling and rendering.
result Language-agnostic multilingual ASR system reduces WER up to 10% over language-dependent models.
New technique reduces language biases in large language models.
problem Gap between language and thought in large language models.
method Proposes Language-of-Thoughts (LoT) technique to adjust order and tokens.
result Significantly reduces language biases and improves reasoning tasks.
In the wake of the ongoing global financial crisis, interdependencies among banks have come into focus in trying to assess systemic risk. To date, such analysis has largely been based on numerical data. By contrast, this study attempts to gain further insight into bank interconnections by tapping into financial discuss…
The paper applies math and physics to language models, introducing entropy and geometric concepts.
problem Understanding and improving language models to approximate intelligent language.
method Formal definitions, functional analysis, topology, thermodynamics, and set theory.
result Entropy function reveals key obstacles for LLMs and offers insights into language models.
Paper introduces TrufLL for language model training without labeled data.
problem Training language models from scratch with reinforcement learning.
method Dynamic vocabulary truncation in RL for language generation.
result TrufLL successfully learns language generation policies from scratch.
Multilingual end-to-end (E2E) models have shown great promise in expansion of automatic speech recognition (ASR) coverage of the world's languages. They have shown improvement over monolingual systems, and have simplified training and serving by eliminating language-specific acoustic, pronunciation, and language models…
Paper reviews neurolinguistics and language technologies, emphasizing mutual enrichment.
problem Understanding brain activity during language processing.
method Brain imaging studies and natural language representations.
result Development of brain-aware natural language representations.
New findings show language models can't simultaneously avoid hallucinations and capture all language richness.
problem Achieving both valid output and full language richness in language generation.
method Investigates language generation within a statistical setting, focusing on consistency and breadth.
result For most collections of candidate languages, a language model cannot simultaneously avoid hallucinations and capture all language richness.
Neural language modeling (LM) has led to significant improvements in several applications, including Automatic Speech Recognition. However, they typically require large amounts of training data, which is not available for many domains and languages. In this study, we propose a multilingual neural language model archite…
lamBERT learns language and actions using multimodal BERT.
problem Learning language and actions in complex environments.
method Extending BERT to multimodal representation and integrating with reinforcement learning.
result lamBERT model achieved higher rewards in multitask and transfer settings.
New model shows natural language exhibits phase transition similar to physics.
problem Understanding critical properties in natural language models.
method Created a context-sensitive random language model.
result Demonstrated a Berezinskii--Kosterlitz--Thouless phase transition.
Optimus pre-trains sentences in a latent space for various NLP tasks.
problem Training large-scale language models for diverse NLP tasks.
method Pre-trained Variational Autoencoder (VAE) on large text corpus, fine-tuned for various tasks.
result Optimus achieves state-of-the-art on VAE language modeling benchmarks.
Language models learn from training data and can leak private information.
problem Language models lack context understanding and can expose private data.
method Discussing the limitations of current privacy protection methods for language models.
result Existing privacy protection methods are insufficient for language models.
Large language models predict human sensory judgments across multiple modalities.
problem Determining the extent of perceptual information in language.
method State-of-the-art large language models were used to predict sensory judgments across six psychophysical datasets.
result Large language models can predict human sensory judgments across multiple modalities with significant correlation to human data.
Study finds phase transition in context-sensitive language model with short-range interactions.
problem Understanding phase transitions in language models with short-range interactions.
method Constructed a random language model with short-range interactions and investigated its statistical properties.
result Phase transition occurs in context-sensitive language models with constant context length.
Benchmark tests spoken language models for infant language learning.
problem Understanding how infants learn language from speech.
method Developed a language-acquisition-friendly benchmark.
result Benchmarking shows models' strengths and weaknesses.
Chronologically consistent models maintain accuracy with time-restricted data.
problem Training data introduces lookahead bias and training leakage in large language models.
method ChronoBERT and ChronoGPT trained with only available data at each time point.
result Models achieve strong performance and competitive with larger models, mitigating lookahead bias.
Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and gener…
LLMs translate natural language trading intents into correct option strategies using a domain-specific language.
problem Challenges in translating natural language trading intents into correct option strategies due to the complexity of option chain data.
method Introduce Option Query Language (OQL) as a domain-specific intermediate representation to abstract option markets into high-level primitives under grammatical rules. Use LLMs as semantic parsers and validate queries by an engine.
result Significantly improves execution accuracy and logical consistency over direct baselines.
When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is fluent in,even if the math lessons were only taught in one language. However, current representations in machine learning are language dependent. In this work, we pres…
Improved LID for multilingual speakers using context-aware models.
problem Low accuracy for languages spoken by multilingual speakers, especially with accented speech.
method Coarser-grained acoustic model and integration with interaction context signals.
result Average 97% accuracy across all language combinations, 60% improvement in worst-case accuracy.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
Introduces challenges and techniques for creating machine translation for indigenous languages.
problem Limited data for machine translation of indigenous languages.
method Introduction to challenges, concepts, and techniques for creating MT systems.
result Discussion of recent advances and open questions in NLP for these languages.
AudioPaLM combines text and speech models to improve speech processing and translation.
problem Improving speech understanding and generation models.
method Unified multimodal architecture combining text and speech models.
result AudioPaLM significantly outperforms existing speech translation systems.
We propose multi-way, multilingual neural machine translation. The proposed approach enables a single neural translation model to translate between multiple languages, with a number of parameters that grows only linearly with the number of languages. This is made possible by having a single attention mechanism that is …
Language models are essential for natural language processing (NLP) tasks, such as machine translation and text summarization. Remarkable performance has been demonstrated recently across many NLP domains via a Transformer-based language model with over a billion parameters, verifying the benefits of model size. Model …
The task of translating between programming languages differs from the challenge of translating natural languages in that programming languages are designed with a far more rigid set of structural and grammatical rules. Previous work has used a tree-to-tree encoder/decoder model to take advantage of the inherent tree s…
Neural network framework for language recognition considers sequence information and improves accuracy.
problem Challenging task of automatic language identification in noisy conditions.
method Proposes a neural network framework with bidirectional LSTM and attention modeling for relevance weighting.
result Significant improvements over conventional methods in noisy conditions and multi-speaker speech.
Language models can answer questions without external knowledge.
problem How much knowledge can be stored in language models?
method Fine-tuning pre-trained language models to answer questions without external context.
result Fine-tuned models perform competitively with open-domain systems.
Improved language generation with faster sampling speed.
problem Speed and coherence issues in autoregressive language models.
method Introduces Neural Flow Diffusion Models (NFDM) for discrete state spaces.
result Substantially reduces likelihood gap with autoregressive models.
Language models are at the heart of numerous works, notably in the text mining and information retrieval communities. These statistical models aim at extracting word distributions, from simple unigram models to recurrent approaches with latent variables that capture subtle dependencies in texts. However, those models a…
GPT-f uses language models to find new proofs in formal math.
problem Generating original mathematical terms for automated theorem proving.
method Transformer-based language model for automated theorem proving and proof assistant.
result GPT-f contributed new proofs to the Metamath library.
Generates biomedical abstracts from titles, years, and keywords.
problem Difficulties in understanding biomedical research papers due to specialized language.
method Conditional transformer-based language model with metadata conditioning.
result Generated abstracts are more relevant and non-trivial than GPT-2.
VALC provides concept-level interpretations of FLMs, overcoming word-level limitations.
problem Lack of higher-level structure interpretation in FLMs' attention weights.
method Formal definition of conceptual interpretation, variational Bayesian framework (VALC).
result VALC finds optimal language concepts for FLM predictions, providing concept-level interpretations.
We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language mo…
Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size an…
LLMs are compared to Markov chains for natural language processing.
problem Theoretical analysis of LLMs' generalization capabilities.
method Equivalence between LLMs and Markov chains, studying multi-step inference.
result Derives generalization bounds for LLMs, capturing their behavior in practice.
Language Rectified Flow improves diffusion language generation by simplifying complex steps.
problem Complexity in diffusion language models limits their implementation in NLP applications.
method Reformulates probabilistic flow models to learn neural ODE models for efficient domain transfer.
result Consistently outperforms baselines on fine-grained control tasks and text editing.