Generates biomedical abstracts from titles, years, and keywords.
problem Difficulties in understanding biomedical research papers due to specialized language.
method Conditional transformer-based language model with metadata conditioning.
result Generated abstracts are more relevant and non-trivial than GPT-2.
Large language models can't efficiently reason conditionally in a distribution-free setting.
problem Impossibility of conditional PAC-efficient reasoning in large language models.
method Proof of impossibility in a distribution-free setting for non-atomic input spaces.
result Any algorithm achieving conditional PAC efficiency must defer to the expert model with high probability.
Paper introduces TrufLL for language model training without labeled data.
problem Training language models from scratch with reinforcement learning.
method Dynamic vocabulary truncation in RL for language generation.
result TrufLL successfully learns language generation policies from scratch.
Project aims to improve coherence in language generation models.
problem Models often generate inconsistent text that diverges from the prompt.
method Trained a sentence pair coherence classifier and co-trained GPT-2 with this coherence objective.
result Fine-tuned model generates coherent paragraphs without diverging.
The paper distinguishes between conditional and marginal processes in language models and discusses conditions for usefulness.
problem The conditional nature of language models trained on observed sequences and the need for marginal text-only processes.
method Distinguishing between full conditional language process, marginal text-only process, and model-induced distribution; analyzing assumptions of stationarity and ergodicity.
result The marginal text-only law is useful only when the observed prefix is an approximately sufficient statistic for the latent circumstances relevant to continuation.
New method converts natural language commands into reward functions for robots.
problem Creating effective reward functions for autonomous machines.
method Language-conditioned reward learning (LC-RL) using inverse reinforcement learning.
result Model learns transferable rewards from natural language commands.
The semantic map calibrates uncertainty from language model probabilities.
problem Uncertainty in language model probabilities for professional decisions.
method Prespecified semantic map linking probabilities of verbal responses to probabilities of declared states.
result Language-derived probabilities outperform printed numerical probabilities and recover valid uncertainty coverage.
This paper proposes a new method to connect language and physical actions in reinforcement learning.
problem Connecting linguistic representations to the physical world in embodied agents.
method Language-conditioned goal generators to decouple sensorimotor learning from language acquisition.
result Agents can demonstrate a diversity of behaviors for any given instruction.
MGLM models all possible language channel factorizations for improved multilingual generation.
problem Generating multilingual text with flexibility and quality.
method Generative joint distribution model over language channels, marginalizing all possible factorizations.
result MGLM outperforms traditional models in multilingual generation tasks.
Paper studies model stealing for low-rank language models.
problem Model stealing threatens proprietary models' security and data privacy.
method Theoretical study of model stealing for Hidden Markov Models (HMMs) and low-rank language models using the conditional query model.
result Efficient algorithm for learning any low-rank distribution in the conditional query model.
SLM models code syntax as trees to generate any programming language code.
problem Generating any piece of code in a given language without restrictions.
method Structural language modeling (SLM) decomposes code into ASTs and estimates probabilities over nodes.
result SLM model generates arbitrary code in any language, outperforming previous methods.
Improved multilingual speech recognition with low latency for nine Indic languages.
problem Imbalance in training data across languages and low latency in interactive applications.
method Conditioning on a language vector and training language-specific adapter layers.
result Lower word error rate than monolingual E2E models and conventional systems.
Model captures author language diffusion over time.
problem Lack of author identity and temporal context in language models.
method Temporal language model conditioning on author and temporal vectors.
result Beat temporal and non-temporal baselines, learns time-varying author representations.
New approach decouples skill learning and language grounding for autonomous agents.
problem Autonomous acquisition of skills without external instructions and feedback.
method Language-Goal-Behavior (LGB) architecture with semantic representation.
result Decouples skill learning and language grounding, enabling diversity and strategy switching.
The paper explores how language models can provide reliable state measurements without being interpreted as beliefs.
problem How to use language models to reliably infer states without misinterpreting them as beliefs.
method Developed a semantic map and semiparametric inverse to link language probabilities to state probabilities, avoiding hidden models.
result Conditions for existence, identification, stable recovery, and uniform stability of posterior states from observable language probabilities.
Birch automates probabilistic modeling using a Turing-complete language.
problem Automating the matching of probabilistic models with inference methods.
method Formally describes models as programs, revealing structure and form dynamically.
result Probabilistic programming languages can tailor inference methods based on model structure and form.
Theory predicts neural scaling exponents from language statistics.
problem No existing theory could quantitatively predict neural scaling exponents.
method Isolated two key statistical properties of language.
result Derives a simple formula predicting neural scaling exponents.
Gaussian processes are conditioned on various types of data.
problem Exact inference in Gaussian processes is limited to linear-Gaussian settings.
method Established an equivalence between GPs and linear diffusion models, allowing for approximate inference in non-linear settings.
result A general-purpose GP inference scheme that handles various conditioning statements, including non-linear physics and natural language.
This paper investigates how large language models achieve neural collapse, a phenomenon linked to generalization.
problem Neural collapse in large language models under imbalanced and token-rich conditions.
method Empirical investigation of scaling and regularization effects on CLMs' progression towards neural collapse.
result Neural collapse properties develop with scale and regularization, linked to generalization in language modeling.
Paper tackles natural language generation using GANs, achieving state-of-the-art results.
problem Discrepancy in progress between image and natural language generation using GANs.
method Introduces a simple baseline for generating natural language from noise without gradient estimators.
result Achieves state-of-the-art results on a Chinese poem generation dataset.
Parallel decoding improves machine translation efficiency and accuracy.
problem Efficiently generating translations from left to right.
method Conditional masked language modeling for parallel decoding.
result Improves translation performance by over 4 BLEU points.
The study examines how language models learn to represent the world, identifying conditions for ecological veridicality.
problem Understanding when language models learn to represent the world accurately and how this learning process can fail.
method Analyzes the Bayes-optimal next-token cross-entropy decomposition and the role of training ecology in shaping model representations.
result The minimum-complexity zero-excess solution is the quotient partition by training equivalence, and this solution is not preserved in in-context learning or per-task adaptation.
Model detects depression from transcribed interviews using affective language.
problem Detecting depression from transcribed clinical interviews.
method Hierarchical Attention Network with affective conditioning.
result Model achieves state-of-the-art F1 scores in depression detection.
Pre-trained language model boosts RL efficiency.
problem Low sample efficiency in RL, especially in lifelong learning.
method Use a pre-trained task-independent language model for transfer learning.
result Goal-conditional RL agents become more sample efficient.
Current image captioning systems miss spatial location details.
problem Capturing spatial location information in image captions.
method Evaluation of image captioning systems from literature.
result Language models alone are insufficient for capturing spatial location in image captions.
Language models predict inorganic synthesis conditions and temperatures.
problem Limited data and heuristic approaches constrain inorganic synthesis planning.
method Language models without fine-tuning predict precursor conditions and temperatures.
result Language models achieve high accuracy in predicting synthesis conditions and temperatures.
A new method improves language model extrapolation without changing training.
problem Improving language model performance on sequences not seen during training.
method Perturbation-based training procedure for language models.
result The method consistently improves out-of-support prediction.
Generative Bridging Network improves sequence prediction models by penalizing confidence and smoothing language.
problem Data sparsity and overfitting in sequence prediction tasks.
method Introduces a bridge module to extend ground truth and minimize KL-divergence between bridge distribution and generator.
result Significant improvements over strong baselines in machine translation and text summarization tasks.
A new sampler for FLMs improves token-level decoding controls.
problem Sampling from FLMs using standard methods collapses marginals and produces invalid sequences.
method Samples clean one-hot endpoints from FLM token marginals and uses Ornstein-Uhlenbeck bridges conditioned on these endpoints.
result The method preserves token-wise posterior-predictive marginals and improves quality-diversity tradeoff.
Generative model creates detailed 3D shapes from text descriptions.
problem Creating high-resolution 3D models from natural language descriptions.
method Two-step process: first generating low-resolution shapes, then high-resolution shapes using Conditional Wasserstein GAN framework.
result Improved method generates 3D shapes more faithful to natural language.
New method uses low logit rank to simplify complex language models.
problem Understanding and learning from modern language models.
method Exploiting the low logit rank structure of language models for efficient learning.
result An efficient algorithm for learning low logit rank models from queries.
Learned feature representations and sub-phoneme posteriors from Deep Neural Networks (DNNs) have been used separately to produce significant performance gains for speaker and language recognition tasks. In this work we show how these gains are possible using a single DNN for both speaker and language recognition. The u…
Enhanced conformal methods improve validity of LLM outputs.
problem Lack of conditional validity and high false rejection rates in LLM validity guarantees.
method Adaptive conditional conformal procedure and improved scoring function differentiation.
result Demonstrated improved validity and utility on real datasets.
Tutorial on combining latent variable models with deep learning.
problem Combining latent variable models with deep learning to model natural language.
method Exploring variational inference to address intractable posterior inference and non-differentiability issues.
result Exploration of variational inference techniques to handle deep latent variable models.
Improves neural dialogue systems with snapshot learning and model architectures.
problem Improving interpretability and performance in neural dialogue systems.
method Various model architectures and snapshot learning for conditional generation.
result Snapshot learning leads to consistent performance improvements.
Paper estimates AI hallucinations in conditional generation tasks.
problem Estimating the frequency of AI-generated incorrect responses.
method Developed a method to estimate hallucination probability from generated responses and log probabilities.
result Method accurately estimates hallucination rate in natural language and synthetic tasks.
Combining watermark and non-watermark detectors improves LLM detection.
problem Challenges in detecting watermarked language models due to limited entropy.
method Investigates hybrid schemes combining watermark and non-watermark detectors.
result Hybrid schemes outperform either class of detector under various conditions.
Natural language guides reinforcement learning exploration in games.
problem Improving reinforcement learning in unseen environments.
method Used encoder-decoder networks to learn associations between natural language and state-action information, then applied this to guide agent exploration.
result Modified policy shaping algorithm outperformed Q-learning and baseline policy shaping in Frogger.
The study analyzes numerical stability in large language models using mixed-precision arithmetic.
problem Numerical stability of large language models using low-precision arithmetic.
method Developed a mixed-precision analysis of transformer inference, deriving bounds for condition numbers and forward error.
result Established that numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream.
Paper proposes a method to improve language model performance on unknown distributions.
problem Language models trained on diverse data can perform poorly on unseen distributions.
method Distributionally robust optimization (DRO) to minimize worst-case performance over a mixture of potential test distributions.
result Topic CVaR approach reduces perplexity by 5.5 points compared to standard maximum likelihood.
New method samples from LLM posterior for coherent, useful responses.
problem Hallucinations in large language models.
method Posterior sampling for conditional generation, with calibration.
result Achieves statistical guarantees with higher downstream utility.
Reduces gender bias in patient notes while maintaining medical classification accuracy.
problem Bias in natural language processing of patient notes.
method Identifying and removing gendered language using BERT-based classifiers, then augmenting data to maintain performance.
result Minimal degradation in health condition classification tasks with data augmentation.
It is time-consuming and error-prone to implement inference procedures for each new probabilistic model. Probabilistic programming addresses this problem by allowing a user to specify the model and having a compiler automatically generate an inference procedure for it. For this approach to be practical, it is important…
Language models allocate information storage, not collapsing into uniform representations.
problem Incomplete neural collapse in language model representations.
method Analyzing variance and information sharing across 14 models, proving an information floor.
result Within-class variance is allocated information storage, not collapsed into uniform representations.
UQE uses LLMs to analyze unstructured data efficiently.
problem Efficient analytics on unstructured data.
method Proposes UQE, a query engine that uses LLMs to interpret UQL queries.
result Demonstrates efficient analytics on various unstructured data types.
This work introduces RISE to explain LLMs more reliably by distinguishing essential context.
problem Identifying which context elements influence LLM outputs reliably.
method RISE (Redundancy-Insensitive Scoring of Explanation) method.
result RISE provides more robust explanations than traditional methods.
This research explores discrete diffusion models for natural language generation.
problem Challenges in applying diffusion models to discrete data, especially natural language.
method Investigates Discrete Denoising Diffusion Probabilistic Model (D3PM) and compares it with autoregressive models.
result Discrete diffusion models achieve better processing speed than autoregressive models.
ProGen models protein sequences for synthetic biology.
problem Generating proteins without structural annotations.
method Trained a 1.2B-parameter language model on 280M protein sequences.
result ProGen generates proteins with fine-grained control and accuracy.