New findings show language models can't simultaneously avoid hallucinations and capture all language richness.
problem Achieving both valid output and full language richness in language generation.
method Investigates language generation within a statistical setting, focusing on consistency and breadth.
result For most collections of candidate languages, a language model cannot simultaneously avoid hallucinations and capture all language richness.
Paper introduces TrufLL for language model training without labeled data.
problem Training language models from scratch with reinforcement learning.
method Dynamic vocabulary truncation in RL for language generation.
result TrufLL successfully learns language generation policies from scratch.
This paper proposes a new method to connect language and physical actions in reinforcement learning.
problem Connecting linguistic representations to the physical world in embodied agents.
method Language-conditioned goal generators to decouple sensorimotor learning from language acquisition.
result Agents can demonstrate a diversity of behaviors for any given instruction.
Optimus pre-trains sentences in a latent space for various NLP tasks.
problem Training large-scale language models for diverse NLP tasks.
method Pre-trained Variational Autoencoder (VAE) on large text corpus, fine-tuned for various tasks.
result Optimus achieves state-of-the-art on VAE language modeling benchmarks.
New method shows cultural transmission improves language compositionality.
problem How cultural transmission affects language emergence and compositionality.
method Introduced cultural evolutionary dynamics into language emergence by periodically replacing agents in a population.
result Languages emerge with better compositional generalization due to implicit cultural transmission.
LLMs translate natural language trading intents into correct option strategies using a domain-specific language.
problem Challenges in translating natural language trading intents into correct option strategies due to the complexity of option chain data.
method Introduce Option Query Language (OQL) as a domain-specific intermediate representation to abstract option markets into high-level primitives under grammatical rules. Use LLMs as semantic parsers and validate queries by an engine.
result Significantly improves execution accuracy and logical consistency over direct baselines.
ScratchGAN trains language GANs without pre-training, achieving comparable results.
problem Training GANs for natural language is challenging due to gradient estimation, optimization instability, and mode collapse.
method Used large batch sizes, dense rewards, and discriminator regularization to stabilize and improve language GANs.
result ScratchGAN performs comparably to maximum likelihood training on quality and diversity metrics.
New approach decouples skill learning and language grounding for autonomous agents.
problem Autonomous acquisition of skills without external instructions and feedback.
method Language-Goal-Behavior (LGB) architecture with semantic representation.
result Decouples skill learning and language grounding, enabling diversity and strategy switching.
Paper uses language models to predict MBTI personality types with high accuracy.
problem Predicting Myers-Briggs personality types from text.
method Fine-tuned BERT model for predicting MBTI types and generating personality-specific language.
result BERT model achieves high accuracy in predicting MBTI types and personality-specific language generation.
Language Rectified Flow improves diffusion language generation by simplifying complex steps.
problem Complexity in diffusion language models limits their implementation in NLP applications.
method Reformulates probabilistic flow models to learn neural ODE models for efficient domain transfer.
result Consistently outperforms baselines on fine-grained control tasks and text editing.
Generates biomedical abstracts from titles, years, and keywords.
problem Difficulties in understanding biomedical research papers due to specialized language.
method Conditional transformer-based language model with metadata conditioning.
result Generated abstracts are more relevant and non-trivial than GPT-2.
We characterize language generation with stability and breadth, proving impossibility results.
problem Characterizing and proving impossibility results for language generation with stability and breadth.
method Analysis of existing notions of breadth and stability, proving lower bounds.
result Proven impossibility of generating with higher perplexity or lower hallucination rate for stable generators.
Stan models are compiled to generative languages and extended with new features.
problem Lack of direct support for variational inference and deep models in Stan.
method Comprehensive compilation scheme to convert Stan models to generative languages, and extension of Stan with new features.
result NumPyro backend yields a 2.3x speedup compared to Stan in geometric mean over 26 benchmarks.
A deep learning approach generates math word problems in multiple languages.
problem Template-based mechanisms for generating mathematical word problems lack customizability and creativity.
method Character Level Long Short Term Memory Network (LSTM) and POS tags are used to generate and resolve constraints in generated problems.
result The approach generates accurate math word problems in English and Sinhala with over 90% accuracy.
Project aims to improve coherence in language generation models.
problem Models often generate inconsistent text that diverges from the prompt.
method Trained a sentence pair coherence classifier and co-trained GPT-2 with this coherence objective.
result Fine-tuned model generates coherent paragraphs without diverging.
Improved language generation with faster sampling speed.
problem Speed and coherence issues in autoregressive language models.
method Introduces Neural Flow Diffusion Models (NFDM) for discrete state spaces.
result Substantially reduces likelihood gap with autoregressive models.
Deep learning models generate languages that lack abstract reasoning.
problem Lack of abstract reasoning in deep learning-generated languages.
method Analyzed emergent language from two multi-agent games with compositional measures.
result Deep learning solutions often fail to generalize to out-of-training examples.
Real-time spell checker adapts to new languages.
problem No real-time, language-adaptable spell checkers for non-English languages.
method Used Wikipedia and subtitles data to generate dictionaries, created noisy channel datasets, compared with industry tools.
result System performs well across 24 languages, outperforming existing tools.
Proposes a new RNN for language generation capturing long-range dependencies.
problem Capturing long-range word dependencies and sentence order in text corpora.
method Recurrent Hierarchical Topic-Guided RNN with dynamic deep topic model.
result Outperforms larger-context RNN-based language models and learns interpretable topics.
Improved robot navigation using multi-head attention for natural language instructions.
problem Improving robot navigation in unfamiliar environments.
method Proposes a multi-head attention mechanism blending layer in a neural network model.
result Significant performance gains in translating instructions for unseen environments.
Study quantifies gender bias in language models across 7 languages.
problem Measuring gender bias in language models across multiple languages.
method Curated dataset of politicians, multilingual language models, probing language models.
result Larger language models do not show significant gender bias compared to smaller ones.
Language helps RL agents learn complex relational and causal structures.
problem Learning relational and causal structure in complex environments.
method Training RL agents to predict language descriptions and explanations.
result Language aids agents in learning challenging relational and causal tasks.
Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and gener…
DLM-One speeds up language generation by 500x with continuous models.
problem Efficiently generating text sequences in natural language processing.
method Score-distillation of continuous diffusion language models.
result Achieves up to 500x speedup in inference time with competitive performance.
New model shows natural language exhibits phase transition similar to physics.
problem Understanding critical properties in natural language models.
method Created a context-sensitive random language model.
result Demonstrated a Berezinskii--Kosterlitz--Thouless phase transition.
GPT-f uses language models to find new proofs in formal math.
problem Generating original mathematical terms for automated theorem proving.
method Transformer-based language model for automated theorem proving and proof assistant.
result GPT-f contributed new proofs to the Metamath library.
The paper applies math and physics to language models, introducing entropy and geometric concepts.
problem Understanding and improving language models to approximate intelligent language.
method Formal definitions, functional analysis, topology, thermodynamics, and set theory.
result Entropy function reveals key obstacles for LLMs and offers insights into language models.
Generates detailed fashion feedback from outfit images.
problem Creating informative and diverse fashion feedback from outfit images.
method Trained deep generative models with visual attention, then improved with Maximum Mutual Information objective function.
result Generated sentences are more diverse and detailed.
LoRA-MCL improves language models by generating diverse sentence continuations.
problem Language models struggle with generating diverse, plausible sentence continuations.
method Low-Rank Adaptation combined with Multiple Choice Learning (MCL) to handle ambiguity.
result LoRA-MCL generates high-diversity and relevant outputs in various tasks.
Language models are miscalibrated, leading to overestimated entropy rates and memory usage.
problem Miscalibration of language models leading to overestimated entropy rates and memory usage.
method Calibration-based approach to measure discrepancies and improve models.
result State-of-the-art language models are miscalibrated, causing overestimated entropy rates and memory usage.
Self-taught optimizer improves code generation using language models.
problem Improving code generation using language models.
method Recursive self-improvement of a scaffolding program that generates code.
result Improved scaffolding program generates programs with significantly better performance.
AudioPaLM combines text and speech models to improve speech processing and translation.
problem Improving speech understanding and generation models.
method Unified multimodal architecture combining text and speech models.
result AudioPaLM significantly outperforms existing speech translation systems.
SLM models code syntax as trees to generate any programming language code.
problem Generating any piece of code in a given language without restrictions.
method Structural language modeling (SLM) decomposes code into ASTs and estimates probabilities over nodes.
result SLM model generates arbitrary code in any language, outperforming previous methods.
New bounds show large language models can generalize beyond training data.
problem Generalization of large language models beyond training data.
method Compression bound derivation using prediction smoothing and SubLoRA.
result Large language models can discover generalizable regularities.
LLMs are compared to Markov chains for natural language processing.
problem Theoretical analysis of LLMs' generalization capabilities.
method Equivalence between LLMs and Markov chains, studying multi-step inference.
result Derives generalization bounds for LLMs, capturing their behavior in practice.
KG-A2C agent learns natural language IF games by reasoning and constraining action spaces.
problem Challenges of natural language understanding, partial observability, and combinatorially large action spaces in IF games.
method Builds a dynamic knowledge graph while exploring, constraining actions using templates.
result Outperforms current IF agents across various games with larger action spaces.
Study explores various methods for detecting offensive language.
problem Detecting offensive language in text.
method Examined custom and off-the-shelf architectures, feature vector generation approaches.
result These methods perform well on small, noisy datasets.
iPrompt uses LLMs to generate natural-language explanations of data patterns.
problem Finding and explaining patterns in data using natural language.
method Interpretable autoprompting (iPrompt) that generates natural-language explanations based on LLMs.
result iPrompt can accurately find and explain data patterns, improving upon human-written prompts.
New models improve language generation by sharing intermediate states.
problem Language models struggle with past mistakes in sequence generation.
method Integrate second-order terms in hidden-state update, sharing intermediate states.
result Shared parametrization improves language modeling performance.
A new language for neural architecture search decouples search spaces and algorithms.
problem Current neural architecture search methods are limited to specific use-cases and lack general-purpose constructs.
method Proposes a formal language for encoding search spaces over general computational graphs, allowing modular, composable, and reusable encodings.
result The language enables easy experimentation with different search spaces and algorithms without reinventing the wheel.
Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks. These language models are almost universally autoregressive in nature, generating sentences one token at a time from left to right. This paper studies the influence o…
Study examines how decoding algorithms affect fairness in language generation models.
problem Impact of decoding algorithms on fairness in open-ended language generation.
method Systematic analysis of top-p, top-k, and temperature decoding algorithms. result Decoding algorithms significantly impact fairness across demographic groups.
Study on generating and identifying languages privately, showing privacy imposes costs and creates barriers.
problem Generating and identifying languages privately in the limit model.
method Introduced a continual release model under differential privacy constraints, proving both positive and negative results.
result Privacy imposes quantitative and qualitative costs, and creates fundamental barriers for identification.
Investigates how neural network capacity impacts the learning of compositional languages.
problem The importance of communicative bandwidth in emergent language learning.
method Exploration of how neural network capacity affects the learning of compositional languages.
result There is a specific range of model capacity and channel bandwidth that induces compositional structure.
Let G be a finitely generated group. We show that for any finite generating set A, the language consisting of all geodesics in Cay(G,A) with a contracting property is a regular language. As an application, we show that any finitely generated group containing an infinite contracting geodesic must be either virtual…
Language models exhibit low-rank structure, which can be used for generation.
problem Understanding the low-dimensional structure of large language models.
method Empirical demonstration and theoretical analysis of the approximate rank of language models' logits.
result Language models can generate responses using linear combinations of unrelated prompts.
This work investigates uncertainty quantification for black-box large language models in natural language generation.
problem Lack of trustworthiness in responses generated by black-box large language models.
method Differentiated uncertainty vs confidence, proposed and compared several confidence/uncertainty measures, applied to selective NLG.
result A simple measure for semantic dispersion can predict the quality of LLM responses.
This research explores discrete diffusion models for natural language generation.
problem Challenges in applying diffusion models to discrete data, especially natural language.
method Investigates Discrete Denoising Diffusion Probabilistic Model (D3PM) and compares it with autoregressive models.
result Discrete diffusion models achieve better processing speed than autoregressive models.