MFMs enable efficient reward alignment for generative models.
problem Computational bottleneck in controlling generative models.
method Meta Flow Maps (MFMs) extend consistency models and flow maps to stochastic regime for efficient value function estimation.
result MFMs enable inference-time steering and unbiased, off-policy fine-tuning to general rewards efficiently.
ELS framework improves safety alignment by dynamically steering LLMs towards helpful responses.
problem Over-Refusal in Aligned Large Language Models
method Fine-tuning free framework using an Energy-Based Model (EBM) to dynamically steer LLMs during inference.
result Extensive experiments show a significant reduction in false refusals (from 57.3% to 82.6%) while maintaining safety performance.
Improves inference-time alignment for diffusion models without updating weights.
problem Aligning diffusion models without updating weights for high-reward outputs.
method Trust-Region Iterative Twisted Sequential Monte Carlo (TRI-TSMC) for variance reduction and efficiency.
result Improves primary alignment objectives on text generation tasks.
We show LLMs can be locally linear, enabling better control of activations.
problem Suboptimal control of LLM activations during generation.
method Model LLM inference as a linear dynamical system, compute feedback controllers using Jacobians, and adapt classical control theory.
result Robust, fine-grained control of LLM activations across models and tasks.
GLASS Flows improves flow and diffusion model performance by optimizing sampling efficiency.
problem Efficiency bottleneck in sampling Markov transitions for flow and diffusion models.
method Introduces GLASS Flows, a new sampling paradigm that simulates a 'flow matching model within a flow matching model' to sample Markov transitions efficiently.
result Eliminates the trade-off between stochastic evolution and efficiency in large-scale text-to-image models.
Unified Bayesian model explains in-context learning and activation steering in LLMs.
problem Understanding and controlling the behavior of large language models (LLMs) through prompts and activations.
method Developed a Bayesian model to explain and predict the effects of in-context learning and activation steering.
result Unified model predicts distinct phases and sudden shifts in LLM behavior, explaining prior empirical phenomena.
DTS improves inference-time alignment of diffusion models with less compute.
problem Inference-time alignment of diffusion models suffers from inaccurate value estimation and inefficient reuse of past computations.
method Diffusion Tree Sampling (DTS) uses a tree-based approach to propagate terminal rewards and iteratively refine value estimates.
result DTS produces asymptotically exact samples and matches the FID of best-performing baselines with up to 10x less compute.
DriftLite improves inference quality of diffusion models without retraining.
problem Adapting pre-trained diffusion models to new target distributions without retraining.
method Lightweight, training-free particle-based approach that steers inference dynamics with optimal stability control.
result Consistently reduces variance and improves sample quality over existing methods.
Inference-Time Scaling can be extended to domains prone to systematic failure using intrinsic statistics.
problem Scaling inference time in domains prone to systematic failure
method Intrinsic Selection (iS), Intrinsic Particle Filtering (iPF), and Particle Distillation (dPF)
result Intrinsic Selection improves engineering design selection by 20% and pass@1 by 6.1 points on average.
Itô maps provide a method for any-step SDE integration.
problem Stochastic dynamics
method Itô map formulation
result Empirical results on synthetic and image-generation benchmarks
New analysis shows interpretability doesn't guarantee steering utility in LLMs.
problem Does higher interpretability lead to better steering utility in large language models?
method Trained 90 SAEs across three LLMs, evaluated interpretability and steering utility, used Kendall's rank coefficients for analysis.
result Interpretability is only weakly associated with steering utility, and features selected by Delta Token Confidence improve steering performance.
The paper studies how to improve language model inference using particle filtering.
problem Understanding the accuracy-cost tradeoffs of inference-time methods for large language models.
method Introduces particle filtering algorithms like Sequential Monte Carlo (SMC) to study language model inference.
result Identifies criteria enabling non-asymptotic guarantees for SMC and fundamental limits faced by all particle filtering methods.
The ability of deep learning models to generalize well across different scenarios depends primarily on the quality and quantity of annotated data. Labeling large amounts of data for all possible scenarios that a model may encounter would not be feasible; if even possible. We propose a framework to deal with limited lab…
New method steers protein design towards desired properties.
problem Challenges in designing proteins with specific structures and properties.
method Feynman-Kac framework applied to RFdiffusion models with guiding potentials.
result Significant improvement in predicted interface energetics and binder designability.
Study designs incentives for adapting multi-agent systems without knowing their learning dynamics.
problem Designing incentives for an adapting population in multi-agent systems without prior knowledge of their learning dynamics.
method Introduces a model-based non-episodic Reinforcement Learning (RL) formulation for steering Markovian agents towards desired policies, focusing on history-dependent strategies to handle model uncertainty.
result Identifies conditions for the existence of steering strategies to guide agents to desired policies and provides empirical algorithms to approximately solve the objective.
Self-driving vehicles have expanded dramatically over the last few years. Udacity has release a dataset containing, among other data, a set of images with the steering angle captured during driving. The Udacity challenge aimed to predict steering angle based on only the provided images. We explore two different models …
Study designs steering rewards for MFGs with unknown dynamics and model uncertainty.
problem Designing incentives for large populations of agents in MFGs with uncertain model details.
method Developed optimistic exploration algorithms for agents with no-adaptive regret behaviors.
result Sub-linear regret guarantees for cumulative gaps between agent behaviors and desired outcomes.
New model shows most users on Q&A site Stack Overflow ignore badges.
problem Understanding user behavior in Q&A websites with badges.
method Probabilistic model applied to Stack Overflow data.
result Majority of users remain apathetic to badges but still contribute.
The paper explores how AI systems use information geometry to encode semantic structure.
problem How AI systems encode semantic structure into geometric representation spaces.
method Focuses on softmax distributions and develops dual steering method for robust concept manipulation.
result Dual steering optimally modifies target concepts while minimizing off-target changes.
New approach extracts AI model representations for steering and monitoring.
problem Understanding and steering AI models' internal knowledge representation.
method Feature learning advances for scalable linear concept representations.
result Concept representations enable steering and monitoring of AI models.
Painless Activation Steering automates post-training for LMs without manual intervention.
problem Manual post-training methods are time-consuming and labor-intensive.
method Painless Activation Steering (PAS) is a fully automated approach that requires no manual intervention.
result PAS reliably improves performance for behavior tasks but not for intelligence-oriented tasks.
Solves steering problem with continuous time, Hilbert-Schmidt cost, and matrix ODEs.
problem Fixed horizon linear quadratic covariance steering in continuous time with a specific terminal cost.
method Formulates necessary conditions as a coupled matrix ODE two-point boundary value problem, designs a matricial recursive algorithm, and proves convergence.
result Proposes and proves the convergence of a matricial recursive algorithm for solving the steering problem.
A new method steers Gaussian distributions with minimal effort.
problem Steering high-dimensional Gaussian distributions efficiently.
method Sliced feedback controller using one-dimensional projections and averaging.
result The method steers Gaussian distributions to targets efficiently.
Inference-aware meta-alignment of LLMs reduces computational cost.
problem Aligning LLMs to diverse human preferences is challenging due to conflicting criteria.
method IAMA trains a base model to be aligned to multiple tasks via different inference-time alignment algorithms, using non-linear GRPO for optimization.
result IAMA enables effective alignment of LLMs to multiple criteria with limited computational budget.
RNE provides a flexible framework for diffusion models, enabling inference-time control and energy-based training.
problem Insufficient knowledge of marginal densities in diffusion models.
method Introduces Radon-Nikodym Estimator (RNE) to reveal the connection between marginal densities and transition kernels.
result RNE delivers strong results in inference-time control and energy-based diffusion training.
Interactive steering improves hierarchical clustering for diverse user needs.
problem Existing hierarchical clustering methods fail to meet diverse user needs.
method Knowledge-driven and data-driven constraints, interactive steering through a visual interface.
result Facilitates the building of customized clustering trees efficiently and effectively.
Tutorial on optimizing diffusion model samples for specific metrics.
problem Optimizing diffusion model samples for specific downstream metrics.
method Review and exploration of inference-time guidance and alignment methods.
result Unified perspective on inference-time algorithms and novel methods.
This paper improves model training by using a reference model to guide target model training.
problem Improving generalization and data efficiency in model training.
method DRRho risk minimization framework based on Distributionally Robust Optimization (DRO).
result DRRho risk minimization improves generalization and data efficiency compared to training without a reference model.
A new method for steering large agent populations efficiently.
problem Controlling the configuration of a swarm of identical, interacting cooperative agents.
method Mean-Field Schrodinger Bridges with Gaussian Mixture Models.
result A highly efficient parameterization to approximate optimal solutions of the MFSB problem in closed form.
Model-free reinforcement learning has recently been shown to successfully learn navigation policies from raw sensor data. In this work, we address the problem of learning driving policies for an autonomous agent in a high-fidelity simulator. Building upon recent research that applies deep reinforcement learning to navi…
This work shows how approximate reward models can significantly improve inference-time scaling.
problem Improving the efficiency of inference for large language models.
method Identifying the Bellman error of approximate reward models and using Sequential Monte Carlo (SMC) for inference.
result Approximate reward models can reduce computational complexity from exponential to polynomial in T. New algorithm improves inference-time alignment without reward hacking.
problem Improving quality of responses from language models with limited compute.
method Inference-time alignment, focusing on extttInferenceTimePessimism algorithm. result Optimal performance and scaling-monotonicity of extttInferenceTimePessimism. New method prevents RLHF alignment collapse by accounting for policy's influence on reward model updates.
problem Iterative RLHF leads to alignment collapse where policies exploit RM's blind spots.
method Foresighted policy optimization (FPO) restores missing steering term via regularization.
result FPO prevents alignment collapse on LLM alignment pipelines using Llama-3.2-1B.
EHR-MPC optimizes sepsis treatment using digital twins and inference-time control.
problem Optimal sepsis treatment policies are contested and difficult to adapt during inference.
method EHR-MPC decouples learning patient dynamics from treatment optimization, enabling inference-time control over learned digital twins.
result EHR-MPC achieves comparable off-policy performance and improved simulation performance compared to RL baselines.
Bayesian Power Steering fine-tunes large diffusion models for domain adaptation.
problem Fine-tuning pre-trained diffusion models for tasks in a smaller probability space.
method Bayesian framework with a novel network structure (Bayesian Power Steering).
result Bayesian Power Steering achieves an FID score of 10.49 on the COCO17 dataset.
URGE improves diffusion model quality without gradients or Hessian.
problem Improving sample quality in diffusion models without gradient evaluations.
method Path-wise importance reweighting via Girsanov change of measure.
result URGE achieves better generation quality than existing methods.
New method aligns diffusion models for inference-time properties without retraining.
problem Aligning pre-trained diffusion models for desired inference-time properties.
method Variationally stable Doob's matching for provable guidance estimation.
result Consistent estimator of guidance with non-asymptotic convergence guarantees.
Improves diffusion model performance and efficiency through classical search.
problem Tackles inference-time control in diffusion models.
method Proposes a framework combining local and global search for efficient navigation.
result Significant gains in performance and efficiency across various domains.
We develop a novel method, called PoWER-BERT, for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate …
NegToMe uses images to guide text-based models away from unwanted visual elements.
problem Insufficient text-based adversarial guidance for complex visual concepts.
method Negative token merging (NegToMe) using visual features from reference images.
result Significantly enhances output diversity and reduces visual similarity to copyrighted content.
Regularization improves stability and consistency of sparse autoencoders.
problem Varying features across random seeds and training choices in SAEs.
method Added L1 or L2 penalties on encoder and decoder weights.
result L2 regularization increases cross-seed feature consistency.
Framework disentangles deep feature uncertainty for efficient inference.
problem Inference-time uncertainty estimation for reliable decision-making.
method Uncertainty-Guided Inference-Time Selection framework.
result Significantly tighter prediction intervals and 60% compute reduction.
End-to-end autonomous driving models get better uncertainty estimates.
problem Uncertainty quantification for end-to-end autonomous driving models.
method Approximate inference for implicit copula neural linear model.
result Densities for steering angle are marginally calibrated.
The paper explores linear representations in language models using counterfactuals.
problem Understanding linear representations and geometric concepts in large language models.
method Formalized linear representation in output and input spaces, identified causal inner product.
result Unified understanding of linear representations and their connection to interpretation and control.
Increased data gathering capacity, together with the spread of data analytics techniques, has prompted an unprecedented concentration of information related to the individuals' preferences in the hands of a few gatekeepers. In the present paper, we show how platforms' performances still appear astonishing in relation t…
As neural networks become widely deployed in different applications and on different hardware, it has become increasingly important to optimize inference time and model size along with model accuracy. Most current techniques optimize model size, model accuracy and inference time in different stages, resulting in subopt…
User engagement in online social networking depends critically on the level of social activity in the corresponding platform--the number of online actions, such as posts, shares or replies, taken by their users. Can we design data-driven algorithms to increase social activity? At a user level, such algorithms may incre…
While several approaches to face emotion recognition task are proposed in literature, none of them reports on power consumption nor inference time required to run the system in an embedded environment. Without adequate knowledge about these factors it is not clear whether we are actually able to provide accurate face e…