Paper proposes verifier engineering for improving foundation models.
problem Challenges in providing effective supervision signals for foundation models.
method Leverages automated verifiers to perform verification tasks and deliver feedback.
result Verifier engineering can enhance foundation models' capabilities.
Easy conditions found for simplifying complex systems.
problem Linearizing complex two-input systems.
method Endogenous dynamic feedback with a dimension of at most two.
result Necessary and sufficient conditions for linearizability.
New method uses correlated auxiliary feedback to reduce regret in parameterized bandits.
problem Reducing regret in parameterized bandits with correlated auxiliary feedback.
method Develops a reward estimator using auxiliary feedback with tight confidence bounds.
result Shows significant reduction in regret compared to standard methods.
New algorithm reduces regret in delayed feedback generalised linear bandits.
problem Regret in delayed feedback generalised linear bandits.
method Adaptation of optimistic algorithm to delayed feedback.
result Achieves a regret bound independent of the horizon's delay penalty.
Continuous-time distributed mirror descent with integral feedback converges to global optimum.
problem Distributed optimization of a global strongly convex function with local convex components.
method Continuous-time distributed mirror descent with integral feedback.
result Asymptotic convergence to global optimum with constant step-size.
Study improves understanding of why agentic theorem provers succeed.
problem Understanding which components of agentic theorem provers improve proof success.
method Statistical provability theory and finite-horizon reachability MDP model.
result Bounds provability gap and explains components' effectiveness.
Study on when RLVR can learn compositional problems.
problem Understanding when RLVR can learn compositional problems.
method Theoretical analysis of task-advantage ratio to characterize learnability.
result Identified conditions for learnability of compositional problems.
RLHF uses human feedback to train AI models, posing statistical challenges.
problem Aligning AI models with human preferences using noisy, subjective feedback.
method Supervised fine-tuning, reward modeling, policy optimization, statistical ideas.
result Statistical methods for reward function learning and policy optimization.
Policy gradient algorithms typically combine discounted future rewards with an estimated value function, to compute the direction and magnitude of parameter updates. However, for most Reinforcement Learning tasks, humans can provide additional insight to constrain the policy learning. We introduce a general method to i…
Paper addresses generalization error bounds for learning with censored feedback.
problem Impact of censored feedback on generalization error bounds.
method Derives an extension of DKW inequality for non-IID data due to censored feedback and uses it to bound generalization error.
result Existing generalization error bounds fail to account for censored feedback, necessitating new bounds.
Study online learning with off-policy feedback in adversarial bandit problems.
problem Learning with limited direct feedback in sequential decision making.
method Proposed algorithms that adapt pessimistic reward estimators to handle unknown behavior policy.
result Guaranteed regret bounds scaling with policy mismatch, improving performance against well-covered comparators.
New RLHF framework handles general preference oracles without reward functions.
problem Handling general preference oracles without assuming a reward function.
method Developed a minimax game between two LLMs for RLHF under a general preference oracle, focusing on KL-regularized preference.
result Proposed algorithms for efficient offline and online RLHF learning.
This paper presents a description of the mechanical operations of banking as used in modern banking systems regulated under the Basel Accords, in order to provide support for a verifiable and complete description of the banking system suitable for computer simulation. Feedback is requested on the contents of this docum…
The paper tackles exact linearization and control of flat discrete-time systems.
problem Exact linearization and control of flat nonlinear discrete-time systems.
method Investigates conditions for choosing new inputs and feedbacks that may depend on forward-shifts of the new input.
result Easily verifiable conditions for choosing a feasible input and a new input that minimizes forward-shifts of the flat output.
Proposes estimators for complex dose-response curves using kernel methods.
problem Estimating complex dose-response curves with continuous treatments, mediators, and covariates.
method Kernel ridge regression with sequential kernel embedding technique.
result Simple estimators for mediated and time-varying dose response curves with nonasymptotic uniform rates.
New method for identifying best designs in vector optimization with uncertain feedback.
problem Optimizing vector-valued outcomes with uncertain preferences.
method Stochastic bandit feedback, polyhedral ordering cone, (ε,δ)-PAC Pareto set identification. result Sample complexity characterized and matched by the naïve elimination algorithm.
Scheduling the transmission of time-sensitive data to multiple users over error-prone communication channels is studied with the goal of minimizing the long-term average age of information (AoI) at the users under a constraint on the average number of transmissions at the source node. After each transmission, the sourc…
An automatic machine learning (AutoML) task is to select the best algorithm and its hyper-parameters simultaneously. Previously, the hyper-parameters of all algorithms are joint as a single search space, which is not only huge but also redundant, because many dimensions of hyper-parameters are irrelevant with the selec…
Proposes RPG-RT for red-teaming T2I models without internal access.
problem Evaluating T2I models' security through red-teaming is challenging due to their closed-source nature and unknown defense mechanisms.
method Integrates LLM and rule-based preference modeling to dynamically adapt to unknown defense mechanisms.
result Demonstrates superior and practical approach for red-teaming T2I models.
A new method for identifying the best arm in multi-armed bandits with mediator feedback.
problem Sequential decision-making with unknown mediator policies.
method Best-arm identification under mediators' feedback (BAI-MF) with statistical lower bound and sequential decision-making strategy.
result The proposed algorithm matches the lower bound on sample complexity for identifying the best arm.
Model captures decision-making under bounded rationality with prior beliefs and market feedback.
problem Bounded rationality in decision-making with limited processing abilities.
method Maximum entropy principle applied to Quantal Response Statistical Equilibrium framework.
result Prior beliefs influence decision-making, altering the outcome of market feedback.
RLVR maintains safety while improving reasoning capabilities in LLMs.
problem Safety-capability tradeoff in fine-tuning LLMs.
method Reinforcement Learning with Verifiable Rewards (RLVR) and theoretical analysis.
result RLVR can enhance reasoning while maintaining safety guardrails.
Neural networks are commonly trained to make predictions through learning algorithms. Contrastive Hebbian learning, which is a powerful rule inspired by gradient backpropagation, is based on Hebb's rule and the contrastive divergence algorithm. It operates in two phases, the forward (or free) phase, where the data are …
New approach combines semi-supervised learning and bandits for better predictions.
problem Online semi-supervised learning with bandit feedback for applications like clinical trials and ad recommendations.
method Adjusted Graph Convolutional Network (GCN) for contextual bandits, with semi-supervised missing rewards imputation.
result Developed multi-GCN embedded contextual bandit algorithms verified on real-world datasets.
Extended flatness approach for discrete-time systems considers forward and backward shifts.
problem Defining flatness for discrete-time systems with forward-shifts.
method Introducing backward-shifts to extend flatness definition.
result Extended flat systems maintain key properties like reachability and controllability.
The procure to pay process (P2P) in large enterprises is a back-end business process which deals with the procurement of products and services for enterprise operations. Procurement is done by issuing purchase orders to impaneled vendors and invoices submitted by vendors are paid after they go through a rigorous valida…
The paper explores a Multi-Objective RL approach for trading that generalizes reward functions.
problem Improving performance in single-asset trading through adaptive reward functions.
method Developed a Multi-Objective Deep Reinforcement Learning algorithm to generalize reward functions and discount factors.
result The Multi-Objective algorithm demonstrates increased predictive stability and better performance in sparse reward scenarios.
Proposes SSL framework for improving item recommendations in large-scale systems.
problem Sparse feedback data for long-tail items in recommender systems.
method Multi-task self-supervised learning framework with data augmentation.
result Significant improvements in model performance, especially on slices lacking supervision.
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
Formalizes weak and strong verification for LLMs, controlling errors without assumptions.
problem Balancing cost and reliability in reasoning with LLMs.
method Formalizes weak-strong verification policies, introduces metrics, develops online algorithm.
result Optimal policies admit a two-threshold structure, and calibration and sharpness govern value of weak verifiers.
Paper presents neural network controllers for offset-free setpoint tracking.
problem Offset-free setpoint tracking using neural network controllers.
method Exploiting slope-restricted activation functions, linear matrix inequalities are used to verify stability.
result Global and local stability conditions for neural network controllers are derived.
The paper explores dual learning, a technique that improves machine translation and image transformation.
problem Understanding and improving dual learning's effectiveness and conditions.
method Theoretical analysis and algorithmic extension of dual learning.
result Multi-step dual learning boosts performance under mild conditions.
This research introduces an autonomous robot navigation method using reinforcement learning.
problem Improving robot navigation in complex environments.
method Deep Q Network (DQN) and Proximal Policy Optimization (PPO) models for path planning and decision-making.
result The models enhance robot navigation ability and adaptive learning in unknown environments.
New method calibrates probabilistic linear solver for online coverage guarantees.
problem Uncertainty in probabilistic linear solver solutions without coverage guarantees.
method Online conformal prediction-PLS (OCP-PLS) method to calibrate uncertainty thresholds.
result Validates online calibration of uncertainty thresholds via online conformal prediction.
Self-improvement refines language models by verifying their own outputs.
problem Improving language models without external feedback.
method Formalizing self-improvement as sharpening, using the model itself as a verifier.
result RLHF-based self-improvement can outperform SFT-based methods.
New insights into cascade feedback linearization of control systems.
problem Obtaining a cascade feedback linearization for invariant control systems.
method Introducing truncated versions of operators from the calculus of variations to prove new theorems.
result Established new geometry and foundational theorems for future work.
New method learns from either positive or negative feedback alone.
problem Limited applicability of existing preference optimization methods in scenarios with only unpaired feedback.
method Decouples learning from positive and negative feedback, using expectation-maximization (EM) to optimize probability of positive outcomes and explicitly incorporate negative examples.
result Stable learning from negative feedback alone demonstrated.
User preferences for items can be inferred from either explicit feedback, such as item ratings, or implicit feedback, such as rental histories. Research in collaborative filtering has concentrated on explicit feedback, resulting in the development of accurate and scalable models. However, since explicit feedback is oft…
This paper improves image retrieval accuracy through novel relevance feedback methods.
problem Improving image retrieval accuracy in Content-Based Image Retrieval (CBIR).
method Novel addition to feature re-weighting and classification techniques, focusing on 0-th iteration improvement.
result Significantly improved retrieval accuracy from relevance feedback.
Unified approach to RLHF tackles uncertainty in reward function.
problem Uncertainty in reward function learned from human feedback.
method Value-incentivized preference optimization (VPO) that regularizes the reward function with value function.
result Theoretical and practical guarantees for both online and offline RLHF settings.
Recommender systems recommend items more accurately by analyzing users' potential interest on different brands' items. In conjunction with users' rating similarity, the presence of users' implicit feedbacks like clicking items, viewing items specifications, watching videos etc. have been proved to be helpful for learni…
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
Develops a new model for RLHF accounting for partially observed states and intermediate feedback.
problem Lack of models for partially observed states and intermediate feedback in RLHF.
method PORRL model with cardinal and dueling feedback methods.
result Demonstrates improved learning and alignment with new model-based and model-free methods.
Classifier learns to ignore unreliable feedback from end users.
problem Improving classifier performance by filtering unreliable feedback.
method Modeling end users as autonomous agents, periodically retraining classifier with filtered feedback.
result Classifier can identify and filter out unreliable feedback, improving performance.
Paper characterizes minimax regret rates for online ranking with top-k feedback.
problem Analyzing online ranking with partial feedback.
method Developed techniques from partial monitoring to characterize minimax regret rates.
result Full characterization of minimax regret rates for Precision@n.
This paper studies the continuous time mean-variance portfolio selection problem with one kind of non-linear wealth dynamics. To deal the expectation constraint, an auxiliary stochastic control problem is firstly solved by two new generalized stochastic Riccati equations from which a candidate portfolio in feedback for…
Study on sample complexity for pure exploration in feedback graph settings.
problem Sample complexity of pure exploration in online learning with feedback graphs.
method Derive instance-specific lower bounds and present asymptotically optimal algorithm TaS-FG.
result TaS-FG is asymptotically optimal and efficient across different graph configurations.
Study agnostic feature-based dynamic pricing models with linear policies and noisy valuations.
problem Tackles dynamic pricing with unknown noise and no assumptions on data.
method Studies two agnostic models: linear policy and linear noisy valuation, presenting algorithms and regret bounds.
result Demonstrates no-regret learning is possible under weak assumptions, but noisy feedback is not significantly more useful than bandit feedback.