arXiv research

Language-based methods improve human similarity approximations without requiring many human judgments.

Express improves causal attention guarantees for language models.

New method uses low logit rank to simplify complex language models.

Language models exhibit low-rank structure, which can be used for generation.

New RFs reduce kernel approximation variance and improve Transformer performance.

Paper proposes f-DPG for aligning language models with preferences.

Transformers struggle to approximate smooth functions, relying on piecewise constant approximations.

The paper applies math and physics to language models, introducing entropy and geometric concepts.

SLED improves factuality in LLMs without external knowledge.

Large language models can't efficiently reason conditionally in a distribution-free setting.

New method for efficient probabilistic inference using masked language modeling.

ABC improves uncertainty quantification in LLMs for clinical diagnostics.

This work investigates how multi-round reasoning improves LLM performance.

DOS improves language model generation by considering inter-token dependencies.

Improved ASR for English-isiZulu code-switched speech with semi-supervised training.

This work shows how approximate reward models can significantly improve inference-time scaling.

CALDERA compresses large language models by approximating weight matrices with low-rank, low-precision factors.

Transformer models align words through attention weights, closely approximating Optimal Transport.

We characterize language generation with stability and breadth, proving impossibility results.

A new method uses a frozen language model to improve sample efficiency in reinforcement learning.

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read}, compose and write operations. NSE c…

The paper analyzes how forgetting in LLMs is linked to simple task-upstream example associations.

Paper proposes efficient optimizers for large language models with fast convergence and low memory usage.

Study shows challenges in converting RNNs to FSMs due to computational complexity.

Prototype model improves model auditing and understanding.

Natural language processing often involves computations with semantic or syntactic graphs to facilitate sophisticated reasoning based on structural relationships. While convolution kernels provide a powerful tool for comparing graph structure based on node (word) level relationships, they are difficult to customize and…

Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a state-of-the-art neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing inp…

XPO enhances RLHF by encouraging diverse responses, offering improved sample efficiency.

Gaussian processes are a class of flexible nonparametric Bayesian tools that are widely used across the sciences, and in industry, to model complex data sources. Key to applying Gaussian process models is the availability of well-developed open source software, which is available in many programming languages. In this …

We present an approximated maximum likelihood method for the multifractal random walk processes of [E. Bacry et al., Phys. Rev. E 64, 026103 (2001)]. The likelihood is computed using a Laplace approximation and a truncation in the dependency structure for the latent volatility. The procedure is implemented as a package…

In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most pop…

This study examines how large language models learn in-context and provides insights into their performance and architecture.

VERA-V uses variational inference to discover vulnerabilities in multimodal vision-language models.

DatedGPT prevents lookahead bias in financial forecasting models.

We collect well known and less known facts about the bivariate normal distribution and translate them into copula language. In addition, we prove a very general formula for the bivariate normal copula, we compute Gini's gamma, and we provide improved bounds and approximations on the diagonal.

Paper addresses online alignment of large language models under uncertain preference feedback.

Paper develops a theory explaining contrastive pre-training for multimodal AI.

Study language generation with limited memory, showing different impacts on achievable densities and convergence.

The study analyzes the convergence rate of a large transformer model.

Proving that next-token prediction makes language models generate coherent long documents.

Influence functions help study large language model generalization, revealing surprising decay patterns.

wd1 improves reasoning in dLLMs by optimizing policies without policy ratios.

ChatGPT predicts stock market reactions from news headlines without financial training.

The study detects deceptive language in business communication using AI.

BLoB fine-tunes LLMs with Bayesian methods to improve uncertainty estimation.

FisherSFT selects informative examples to fine-tune LLMs efficiently.

We introduce a method for using deep neural networks to amortize the cost of inference in models from the family induced by universal probabilistic programming languages, establishing a framework that combines the strengths of probabilistic programming and deep learning methods. We call what we do "compilation of infer…

We analyze the Hessian spectra of large models up to 100B parameters.